Trace number 4398371

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2.O parallel? (TO) 2520.9 2522.86

General information on the benchmark

NameOpd/Opd-sum-large/
Opd-04-350-100.xml
MD5SUM05a37308653f1418d8a9d31e27700b02
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark9
Best CPU time to get the best result obtained on this benchmark10.1781
Satisfiable
(Un)Satisfiability was proved
Number of variables1401
Number of constraints11
Number of domains2
Minimum domain size2
Maximum domain size342
Distribution of domain sizes[{"size":2,"count":1400},{"size":342,"count":1}]
Minimum variable degree5
Maximum variable degree7
Distribution of variable degrees[{"degree":5,"count":1400},{"degree":7,"count":1}]
Minimum constraint arity350
Maximum constraint arity1400
Distribution of constraint arities[{"arity":350,"count":4},{"arity":701,"count":6},{"arity":1400,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":10}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (14897 MiB free)
  memory of node 1: 16384 MiB (15296 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4398371-1568599498/watcher-4398371-1568599498 -o /tmp/evaluation-result-4398371-1568599498/solver-4398371-1568599498 -C 20160 -W 2520 -M 31000 --bin-var /tmp/runsolver-exch-node146-1568596936-27641 --watchdog 2580 cosoco -model -nbcores=8 HOME/instance-4398371-1568599498.xml 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 20160 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 20190 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=27951, runsolver pid=27948
Current StackSize limit: 8192 KiB


[startup+0.100156 s]*
/proc/loadavg: 8.42 8.38 7.85 2/193 27952
/proc/meminfo: memFree=30911056/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=34788 memory=11296 CPUtime=0.1 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 3008 0 0 0 10 0 0 0 20 0 1 0 435863795 35622912 2824 33554432000 4194304 6211092 140729483691120 140729483687000 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 8697 2824 846 493 0 2002 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 34788 KiB
Current children cumulated memory: 11296 KiB

[startup+0.209522 s]*
/proc/loadavg: 8.42 8.38 7.85 2/193 27952
/proc/meminfo: memFree=30910452/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=35580 memory=12088 CPUtime=0.2 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 3215 0 0 0 20 0 0 0 20 0 1 0 435863795 36433920 3022 33554432000 4194304 6211092 140729483691120 140729483687000 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 8895 3022 846 493 0 2200 0
Current children cumulated CPU time: 0.2 s
Current children cumulated vsize: 35580 KiB
Current children cumulated memory: 12088 KiB

[startup+0.308744 s]*
/proc/loadavg: 8.42 8.38 7.85 2/193 27952
/proc/meminfo: memFree=30909708/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=36372 memory=12880 CPUtime=0.3 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 3403 0 0 0 30 0 0 0 20 0 1 0 435863795 37244928 3220 33554432000 4194304 6211092 140729483691120 140729483687000 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 9093 3220 846 493 0 2398 0
Current children cumulated CPU time: 0.3 s
Current children cumulated vsize: 36372 KiB
Current children cumulated memory: 12880 KiB

[startup+0.700333 s]
/proc/loadavg: 8.42 8.38 7.85 2/193 27952
/proc/meminfo: memFree=30908344/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=39276 memory=15784 CPUtime=0.69 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4132 0 0 0 69 0 0 0 20 0 1 0 435863795 40218624 3946 33554432000 4194304 6211092 140729483691120 140729483687000 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 9819 3946 846 493 0 3124 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 39276 KiB
Current children cumulated memory: 15784 KiB

[startup+1.50033 s]
/proc/loadavg: 8.42 8.38 7.85 2/194 27954
/proc/meminfo: memFree=30904504/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=45348 memory=21856 CPUtime=1.49 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 5650 0 0 0 148 1 0 0 20 0 1 0 435863795 46436352 5464 33554432000 4194304 6211092 140729483691120 140729483550408 140268730829632 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 11337 5464 846 493 0 4642 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 45348 KiB
Current children cumulated memory: 21856 KiB

[startup+3.10873 s]
/proc/loadavg: 8.42 8.38 7.85 2/194 27954
/proc/meminfo: memFree=30889656/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=57556 memory=34004 CPUtime=3.1 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 9489 0 0 0 309 1 0 0 20 0 1 0 435863795 58937344 8501 33554432000 4194304 6211092 140729483691120 140729483550408 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 14389 8501 846 493 0 7694 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 57556 KiB
Current children cumulated memory: 34004 KiB

[startup+6.30032 s]
/proc/loadavg: 7.83 8.25 7.81 3/194 27954
/proc/meminfo: memFree=30867220/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=81704 memory=58124 CPUtime=6.29 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 16294 0 0 0 627 2 0 0 20 0 1 0 435863795 83664896 14531 33554432000 4194304 6211092 140729483691120 140729483550408 5446275 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 20426 14531 846 493 0 13731 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 81704 KiB
Current children cumulated memory: 58124 KiB

[startup+12.7003 s]
/proc/loadavg: 7.28 8.13 7.77 2/193 27954
/proc/meminfo: memFree=30822688/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=130200 memory=106424 CPUtime=12.69 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 29159 0 0 0 1265 4 0 0 20 0 1 0 435863795 133324800 26606 33554432000 4194304 6211092 140729483691120 140729483550408 140268721810839 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 32550 26606 846 493 0 25855 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 130200 KiB
Current children cumulated memory: 106424 KiB

[startup+25.5003 s]
/proc/loadavg: 5.89 7.78 7.66 2/193 27954
/proc/meminfo: memFree=30724736/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=226712 memory=203180 CPUtime=25.49 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 54886 0 0 0 2541 8 0 0 20 0 1 0 435863795 232153088 50795 33554432000 4194304 6211092 140729483691120 140729483550408 5454176 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 56678 50795 846 493 0 49983 0
Current children cumulated CPU time: 25.49 s
Current children cumulated vsize: 226712 KiB
Current children cumulated memory: 203180 KiB

[startup+51.1087 s]
/proc/loadavg: 4.22 7.24 7.49 2/192 27957
/proc/meminfo: memFree=30527780/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=420500 memory=396596 CPUtime=51.1 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 104012 0 0 0 5095 15 0 0 20 0 1 0 435863795 430592000 99149 33554432000 4194304 6211092 140729483691120 140729483550408 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 105125 99149 846 493 0 98430 0
Current children cumulated CPU time: 51.1 s
Current children cumulated vsize: 420500 KiB
Current children cumulated memory: 396596 KiB

[startup+102.3 s]
/proc/loadavg: 2.40 6.28 7.15 2/192 27959
/proc/meminfo: memFree=30138584/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=811112 memory=786416 CPUtime=102.29 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 201599 0 0 0 10200 29 0 0 20 0 1 0 435863795 830578688 196604 33554432000 4194304 6211092 140729483691120 140729483550408 5456306 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 202778 196604 846 493 0 196083 0
Current children cumulated CPU time: 102.29 s
Current children cumulated vsize: 811112 KiB
Current children cumulated memory: 786416 KiB

[startup+162.3 s]
/proc/loadavg: 1.51 5.32 6.76 2/191 27962
/proc/meminfo: memFree=29680044/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=1270388 memory=1245512 CPUtime=162.29 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 316370 0 0 0 16186 43 0 0 20 0 1 0 435863795 1300877312 311378 33554432000 4194304 6211092 140729483691120 140729483550408 140268721808214 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 317597 311378 846 493 0 310902 0
Current children cumulated CPU time: 162.29 s
Current children cumulated vsize: 1270388 KiB
Current children cumulated memory: 1245512 KiB

[startup+222.3 s]
/proc/loadavg: 1.19 4.53 6.40 2/191 27965
/proc/meminfo: memFree=29219700/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=1731008 memory=1704816 CPUtime=222.27 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 431194 0 0 0 22169 58 0 0 20 0 1 0 435863795 1772552192 426204 33554432000 4194304 6211092 140729483691120 140729483550408 5455650 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 432752 426204 846 493 0 426057 0
Current children cumulated CPU time: 222.27 s
Current children cumulated vsize: 1731008 KiB
Current children cumulated memory: 1704816 KiB

[startup+282.3 s]
/proc/loadavg: 1.10 3.90 6.07 2/191 27970
/proc/meminfo: memFree=28760028/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=2188520 memory=2164636 CPUtime=282.27 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 545596 0 0 0 28153 74 0 0 20 0 1 0 435863795 2241044480 541159 33554432000 4194304 6211092 140729483691120 140729483550408 140268721810862 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 547130 541159 846 493 0 540435 0
Current children cumulated CPU time: 282.27 s
Current children cumulated vsize: 2188520 KiB
Current children cumulated memory: 2164636 KiB

[startup+342.3 s]
/proc/loadavg: 1.04 3.37 5.75 2/191 27973
/proc/meminfo: memFree=28300188/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=2650628 memory=2623420 CPUtime=342.27 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 660357 0 0 0 34139 88 0 0 20 0 1 0 435863795 2714243072 655855 33554432000 4194304 6211092 140729483691120 140729483550408 5440624 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 662657 655855 846 493 0 655962 0
Current children cumulated CPU time: 342.27 s
Current children cumulated vsize: 2650628 KiB
Current children cumulated memory: 2623420 KiB

[startup+402.3 s]
/proc/loadavg: 1.08 2.97 5.46 2/191 27983
/proc/meminfo: memFree=27839984/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=3107876 memory=3083328 CPUtime=402.26 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 774871 0 0 0 40121 105 0 0 20 0 1 0 435863795 3182465024 770832 33554432000 4194304 6211092 140729483691120 140729483550408 5441074 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 776969 770832 846 493 0 770274 0
Current children cumulated CPU time: 402.26 s
Current children cumulated vsize: 3107876 KiB
Current children cumulated memory: 3083328 KiB

[startup+462.3 s]
/proc/loadavg: 1.03 2.61 5.18 2/191 27988
/proc/meminfo: memFree=27380252/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=3572416 memory=3543996 CPUtime=462.26 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 889767 0 0 0 46106 120 0 0 20 0 1 0 435863795 3658153984 885999 33554432000 4194304 6211092 140729483691120 140729483550408 5446275 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 893104 885999 846 493 0 886409 0
Current children cumulated CPU time: 462.26 s
Current children cumulated vsize: 3572416 KiB
Current children cumulated memory: 3543996 KiB


################
# More data... #
################

Current children cumulated vsize: 14585244 KiB
Current children cumulated memory: 14549856 KiB

[startup+1962.3 s]
/proc/loadavg: 1.00 1.02 1.84 2/191 28098
/proc/meminfo: memFree=15891704/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=15041568 memory=15008268 CPUtime=1962.14 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 3758267 0 0 0 195569 645 0 0 20 0 1 0 435863795 15402565632 3752067 33554432000 4194304 6211092 140729483691120 140729483550408 5446606 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 3760392 3752067 846 493 0 3753697 0
Current children cumulated CPU time: 1962.14 s
Current children cumulated vsize: 15041568 KiB
Current children cumulated memory: 15008268 KiB

[startup+2022.3 s]
/proc/loadavg: 1.40 1.11 1.82 2/191 28101
/proc/meminfo: memFree=15432524/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=15497232 memory=15465976 CPUtime=2022.13 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 3872844 0 0 0 201537 676 0 0 20 0 1 0 435863795 15869165568 3866494 33554432000 4194304 6211092 140729483691120 140729483550408 140268721810835 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 3874308 3866494 846 493 0 3867613 0
Current children cumulated CPU time: 2022.13 s
Current children cumulated vsize: 15497232 KiB
Current children cumulated memory: 15465976 KiB

[startup+2082.3 s]
/proc/loadavg: 1.25 1.13 1.79 2/191 28106
/proc/meminfo: memFree=14973544/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=15952896 memory=15923596 CPUtime=2082.12 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 3987308 0 0 0 207502 710 0 0 20 0 1 0 435863795 16335765504 3980899 33554432000 4194304 6211092 140729483691120 140729483550408 5452215 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 3988224 3980899 846 493 0 3981529 0
Current children cumulated CPU time: 2082.12 s
Current children cumulated vsize: 15952896 KiB
Current children cumulated memory: 15923596 KiB

[startup+2142.3 s]
/proc/loadavg: 1.09 1.11 1.74 2/191 28109
/proc/meminfo: memFree=14514644/32770624 swapFree=67001240/67108860
[pid=27951] ppid=27948 vsize=16408296 memory=16381040 CPUtime=2142.12 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4101825 0 0 0 213468 744 0 0 20 0 1 0 435863795 16802095104 4095260 33554432000 4194304 6211092 140729483691120 140729483550408 140268721805200 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4102074 4095260 846 493 0 4095379 0
Current children cumulated CPU time: 2142.12 s
Current children cumulated vsize: 16408296 KiB
Current children cumulated memory: 16381040 KiB

[startup+2202.3 s]
/proc/loadavg: 1.03 1.09 1.69 2/192 28120
/proc/meminfo: memFree=14056160/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=16863696 memory=16838660 CPUtime=2202.11 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4216216 0 0 0 219432 779 0 0 20 0 1 0 435863795 17268424704 4209665 33554432000 4194304 6211092 140729483691120 140729483550408 5441131 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4215924 4209665 846 493 0 4209229 0
Current children cumulated CPU time: 2202.11 s
Current children cumulated vsize: 16863696 KiB
Current children cumulated memory: 16838660 KiB

[startup+2262.3 s]
/proc/loadavg: 1.01 1.07 1.65 2/192 28125
/proc/meminfo: memFree=13600104/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=17319096 memory=17294324 CPUtime=2262.11 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4330847 0 0 0 225398 813 0 0 20 0 1 0 435863795 17734754304 4323581 33554432000 4194304 6211092 140729483691120 140729483550408 5446649 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4329774 4323581 846 493 0 4323079 0
Current children cumulated CPU time: 2262.11 s
Current children cumulated vsize: 17319096 KiB
Current children cumulated memory: 17294324 KiB

[startup+2322.3 s]
/proc/loadavg: 1.00 1.06 1.61 2/192 28128
/proc/meminfo: memFree=13139968/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=17811408 memory=17753588 CPUtime=2322.1 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4445802 0 0 0 231361 849 0 0 20 0 1 0 435863795 18238881792 4438397 33554432000 4194304 6211092 140729483691120 140729483550408 5013126 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4452852 4438397 846 493 0 4446157 0
Current children cumulated CPU time: 2322.1 s
Current children cumulated vsize: 17811408 KiB
Current children cumulated memory: 17753588 KiB

[startup+2382.3 s]
/proc/loadavg: 1.00 1.05 1.57 2/192 28131
/proc/meminfo: memFree=12683608/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=18266808 memory=18208900 CPUtime=2382.09 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4560191 0 0 0 237324 885 0 0 20 0 1 0 435863795 18705211392 4552225 33554432000 4194304 6211092 140729483691120 140729483550408 5455634 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4566702 4552225 846 493 0 4560007 0
Current children cumulated CPU time: 2382.09 s
Current children cumulated vsize: 18266808 KiB
Current children cumulated memory: 18208900 KiB

[startup+2442.3 s]
/proc/loadavg: 1.00 1.04 1.53 2/192 28136
/proc/meminfo: memFree=12225484/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=18721812 memory=18666076 CPUtime=2442.08 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4674603 0 0 0 243284 924 0 0 20 0 1 0 435863795 19171135488 4666519 33554432000 4194304 6211092 140729483691120 140729483550408 5451510 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4680453 4666519 846 493 0 4673758 0
Current children cumulated CPU time: 2442.08 s
Current children cumulated vsize: 18721812 KiB
Current children cumulated memory: 18666076 KiB

[startup+2502.3 s]
/proc/loadavg: 1.00 1.04 1.50 2/191 28139
/proc/meminfo: memFree=11767376/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=19177080 memory=19123524 CPUtime=2502.07 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4789094 0 0 0 249248 959 0 0 20 0 1 0 435863795 19637329920 4780881 33554432000 4194304 6211092 140729483691120 140729483550408 5013005 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4794270 4780881 846 493 0 4787575 0
Current children cumulated CPU time: 2502.07 s
Current children cumulated vsize: 19177080 KiB
Current children cumulated memory: 19123524 KiB



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+2520 s]*
/proc/loadavg: 1.00 1.03 1.49 2/191 28140
/proc/meminfo: memFree=11628216/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=19311456 memory=19259856 CPUtime=2519.77 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 4202496 4822897 0 0 0 251008 969 0 0 20 0 1 0 435863795 19774930944 4814964 33554432000 4194304 6211092 140729483691120 140729483550408 140268720884709 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4827864 4814964 846 493 0 4821169 0
Current children cumulated CPU time: 2519.77 s
Current children cumulated vsize: 19311456 KiB
Current children cumulated memory: 19259856 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+2522.86 s]
# the end of solver process 27951 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=2510.34 s, system=10.5544 s

Solver just ended.
Dumping a history of the last processes samples

[startup+2520.8 s]
/proc/loadavg: 1.00 1.03 1.49 2/191 28140
/proc/meminfo: memFree=11628216/32770624 swapFree=67001244/67108860
[pid=27951] ppid=27948 vsize=19311456 memory=19259856 CPUtime=2519.88 cores=0-7
/proc/27951/stat : 27951 (cosoco) S 27948 27951 27614 0 -1 512 4822902 0 0 0 251018 970 0 0 20 0 1 0 435863795 19774930944 4814964 33554432000 4194304 6211092 140729483691120 140729483550408 5189550 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4827864 4814964 846 493 0 4821169 0
Current children cumulated CPU time: 2519.88 s
Current children cumulated vsize: 19311456 KiB
Current children cumulated memory: 19259856 KiB

[startup+2521.6 s]
/proc/loadavg: 1.00 1.03 1.49 2/193 28143
/proc/meminfo: memFree=11183984/32770624 swapFree=67001248/67108860
[pid=27951] ppid=27948 vsize=19311456 memory=19259856 CPUtime=2519.98 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 512 4822902 0 0 0 251028 970 0 0 20 0 1 0 435863795 19774930944 4814964 33554432000 4194304 6211092 140729483691120 140729483550408 5189550 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4827864 4814964 846 493 0 4821169 0
Current children cumulated CPU time: 2519.98 s
Current children cumulated vsize: 19311456 KiB
Current children cumulated memory: 19259856 KiB

[startup+2522 s]*
/proc/loadavg: 1.00 1.03 1.49 2/193 28143
/proc/meminfo: memFree=10690820/32770624 swapFree=67001248/67108860
[pid=27951] ppid=27948 vsize=19311456 memory=19259856 CPUtime=2520.04 cores=0-7
/proc/27951/stat : 27951 (cosoco) R 27948 27951 27614 0 -1 512 4822902 0 0 0 251034 970 0 0 20 0 1 0 435863795 19774930944 4814964 33554432000 4194304 6211092 140729483691120 140729483550408 5189550 0 16384 4096 8405026 0 0 0 17 6 0 0 0 0 0 8312316 8317760 39055360 140729483695121 140729483695213 140729483695213 140729483698126 0
/proc/27951/statm: 4827864 4814964 846 493 0 4821169 0
Current children cumulated CPU time: 2520.04 s
Current children cumulated vsize: 19311456 KiB
Current children cumulated memory: 19259856 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 2522.86
CPU time (s): 2520.9
CPU user time (s): 2510.34
CPU system time (s): 10.5544
CPU usage (%): 99.9224
Max. virtual memory (cumulated for all children) (KiB): 19311456
Max. memory (cumulated for all children) (KiB): 19259856

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2510.34
system time used= 10.5544
maximum resident set size= 19259940
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4822902
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 117341
involuntary context switches= 3175


# summary of solver processes directly reported to runsolver:
#   pid: 27951
#   total CPU time (s): 2520.9
#   total CPU user time (s): 2510.34
#   total CPU system time (s): 10.5544

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 12.637 second user time and 12.6674 second system time

The end

Launcher Data 

Begin job on node146 at 2019-09-16 04:04:17
IDJOB=4398371
IDBENCH=122932
IDSOLVER=2851
FILE ID=node146/4398371-1568599498
RUNJOBID= node146-1568596936-27641
PBS_JOBID= 21703402
Free space on /tmp= 59028 MiB

SOLVER NAME= cosoco 2.O parallel
BENCH NAME= XCSP17/Opd/Opd-sum-large/Opd-04-350-100.xml
COMMAND LINE= cosoco -model -nbcores=NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4398371-1568599498/watcher-4398371-1568599498 -o /tmp/evaluation-result-4398371-1568599498/solver-4398371-1568599498 -C 20160 -W 2520 -M 31000 --bin-var /tmp/runsolver-exch-node146-1568596936-27641 --watchdog 2580  cosoco -model -nbcores=8 HOME/instance-4398371-1568599498.xml

TIME LIMIT= 20160 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 05a37308653f1418d8a9d31e27700b02
RANDOM SEED=1424078961

node146.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        30919288 kB
MemAvailable:   31871812 kB
Buffers:          213744 kB
Cached:          1010200 kB
SwapCached:         4372 kB
Active:           580960 kB
Inactive:         648516 kB
Active(anon):      23024 kB
Inactive(anon):    29884 kB
Active(file):     557936 kB
Inactive(file):   618632 kB
Unevictable:      110700 kB
Mlocked:          110700 kB
SwapTotal:      67108860 kB
SwapFree:       67001240 kB
Dirty:              2804 kB
Writeback:             0 kB
AnonPages:        145936 kB
Mapped:            45452 kB
Shmem:             38760 kB
Slab:             213680 kB
SReclaimable:     183580 kB
SUnreclaim:        30100 kB
KernelStack:        3856 kB
PageTables:         5396 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     507056 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    100352 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      109952 kB
DirectMap2M:    33435648 kB

Free space on /tmp at the end= 58576 MiB
End job on node146 at 2019-09-16 04:46:20