Trace number 4390366

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2? (MO) 1271.28 1273.15

General information on the benchmark

NameOpd/Opd-sum-small/
Opd-10-030-009.xml
MD5SUM902db1136df2968cd1e3d1d445559775
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark2
Best CPU time to get the best result obtained on this benchmark7.29626
Satisfiable
(Un)Satisfiability was proved
Number of variables301
Number of constraints56
Number of domains2
Minimum domain size2
Maximum domain size29
Distribution of domain sizes[{"size":2,"count":300},{"size":29,"count":1}]
Minimum variable degree11
Maximum variable degree46
Distribution of variable degrees[{"degree":11,"count":300},{"degree":46,"count":1}]
Minimum constraint arity30
Maximum constraint arity300
Distribution of constraint arities[{"arity":30,"count":10},{"arity":61,"count":45},{"arity":300,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":55}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (6665 MiB free)
  memory of node 1: 16384 MiB (13311 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4390366-1568337061/watcher-4390366-1568337061 -o /tmp/evaluation-result-4390366-1568337061/solver-4390366-1568337061 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node103-1568329709-13076 --watchdog 2560 cosoco -lc -model -restarts -stick HOME/instance-4390366-1568337061.xml 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2500 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=14478, runsolver pid=14475
Current StackSize limit: 8192 KiB


[startup+0.100154 s]*
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20453872/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=28724 memory=5012 CPUtime=0.1 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 1473 0 0 0 10 0 0 0 20 0 1 0 409959173 29413376 1253 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 7181 1253 844 493 0 486 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 28724 KiB
Current children cumulated memory: 5012 KiB

[startup+0.209485 s]*
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20451268/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=30140 memory=6332 CPUtime=0.21 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 1810 0 0 0 21 0 0 0 20 0 1 0 409959173 30863360 1583 33554432000 4194304 6210612 140734567623984 140734567606152 5454016 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 7535 1583 845 493 0 840 0
Current children cumulated CPU time: 0.21 s
Current children cumulated vsize: 30140 KiB
Current children cumulated memory: 6332 KiB

[startup+0.308978 s]*
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20448664/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=31412 memory=7712 CPUtime=0.31 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 2118 0 0 0 31 0 0 0 20 0 1 0 409959173 32165888 1928 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 7853 1928 845 493 0 1158 0
Current children cumulated CPU time: 0.31 s
Current children cumulated vsize: 31412 KiB
Current children cumulated memory: 7712 KiB

[startup+0.700283 s]
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20444200/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=36216 memory=12360 CPUtime=0.69 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 3303 0 0 0 69 0 0 0 20 0 1 0 409959173 37085184 3090 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 9054 3090 845 493 0 2359 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 36216 KiB
Current children cumulated memory: 12360 KiB

[startup+1.50029 s]
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20430000/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=46004 memory=22388 CPUtime=1.49 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 5777 0 0 0 149 0 0 0 20 0 1 0 409959173 47108096 5597 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 11501 5597 845 493 0 4806 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 46004 KiB
Current children cumulated memory: 22388 KiB

[startup+3.10963 s]
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20379388/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=66080 memory=42288 CPUtime=3.1 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 10796 0 0 0 309 1 0 0 20 0 1 0 409959173 67665920 10572 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 16520 10572 845 493 0 9825 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 66080 KiB
Current children cumulated memory: 42288 KiB

[startup+6.30027 s]
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20302376/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=106100 memory=81748 CPUtime=6.28 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 20714 0 0 0 626 2 0 0 20 0 1 0 409959173 108646400 20437 33554432000 4194304 6210612 140734567623984 140734567606152 139840152194009 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 26525 20437 845 493 0 19830 0
Current children cumulated CPU time: 6.28 s
Current children cumulated vsize: 106100 KiB
Current children cumulated memory: 81748 KiB

[startup+12.7003 s]
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=20150136/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=187040 memory=160752 CPUtime=12.69 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 40542 0 0 0 1264 5 0 0 20 0 1 0 409959173 191528960 40188 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 46760 40188 845 493 0 40065 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 187040 KiB
Current children cumulated memory: 160752 KiB

[startup+25.5003 s]
/proc/loadavg: 2.01 2.10 2.10 3/201 14479
/proc/meminfo: memFree=19815920/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=343036 memory=319416 CPUtime=25.48 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 79243 0 0 0 2537 11 0 0 20 0 1 0 409959173 351268864 79854 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 85759 79854 845 493 0 79064 0
Current children cumulated CPU time: 25.48 s
Current children cumulated vsize: 343036 KiB
Current children cumulated memory: 319416 KiB

[startup+51.1096 s]
/proc/loadavg: 2.01 2.09 2.09 3/201 14482
/proc/meminfo: memFree=19149632/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=663944 memory=637660 CPUtime=51.1 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 155781 0 0 0 5092 18 0 0 20 0 1 0 409959173 679878656 159415 33554432000 4194304 6210612 140734567623984 140734567606152 5456150 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 165986 159415 845 493 0 159291 0
Current children cumulated CPU time: 51.1 s
Current children cumulated vsize: 663944 KiB
Current children cumulated memory: 637660 KiB

[startup+102.3 s]
/proc/loadavg: 2.00 2.07 2.09 3/201 14485
/proc/meminfo: memFree=17846592/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=1309960 memory=1268872 CPUtime=102.28 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 308021 0 0 0 10194 34 0 0 20 0 1 0 409959173 1341399040 317218 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 327490 317218 845 493 0 320795 0
Current children cumulated CPU time: 102.28 s
Current children cumulated vsize: 1309960 KiB
Current children cumulated memory: 1268872 KiB

[startup+162.3 s]
/proc/loadavg: 2.00 2.06 2.08 3/202 14490
/proc/meminfo: memFree=16310200/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=2056220 memory=2011184 CPUtime=162.28 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 485919 0 0 0 16173 55 0 0 20 0 1 0 409959173 2105569280 502796 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 514055 502796 845 493 0 507360 0
Current children cumulated CPU time: 162.28 s
Current children cumulated vsize: 2056220 KiB
Current children cumulated memory: 2011184 KiB

[startup+222.3 s]
/proc/loadavg: 2.05 2.07 2.08 3/203 14496
/proc/meminfo: memFree=14774380/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=2820988 memory=2753672 CPUtime=222.27 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 663903 0 0 0 22149 78 0 0 20 0 1 0 409959173 2888691712 688418 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 705247 688418 845 493 0 698552 0
Current children cumulated CPU time: 222.27 s
Current children cumulated vsize: 2820988 KiB
Current children cumulated memory: 2753672 KiB

[startup+282.3 s]
/proc/loadavg: 2.02 2.05 2.08 3/202 14500
/proc/meminfo: memFree=13239188/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=3531136 memory=3497656 CPUtime=282.27 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 841312 0 0 0 28127 100 0 0 20 0 1 0 409959173 3615883264 874414 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 882784 874414 845 493 0 876089 0
Current children cumulated CPU time: 282.27 s
Current children cumulated vsize: 3531136 KiB
Current children cumulated memory: 3497656 KiB

[startup+342.3 s]
/proc/loadavg: 2.01 2.04 2.07 3/201 14504
/proc/meminfo: memFree=11705232/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=4323020 memory=4239128 CPUtime=342.26 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 1019119 0 0 0 34101 125 0 0 20 0 1 0 409959173 4426772480 1059782 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 1080755 1059782 845 493 0 1074060 0
Current children cumulated CPU time: 342.26 s
Current children cumulated vsize: 4323020 KiB
Current children cumulated memory: 4239128 KiB

[startup+402.3 s]
/proc/loadavg: 2.00 2.04 2.07 3/202 14510
/proc/meminfo: memFree=10171212/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=5033888 memory=4981448 CPUtime=402.26 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 1196665 0 0 0 40075 151 0 0 20 0 1 0 409959173 5154701312 1245362 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 1258472 1245362 845 493 0 1251777 0
Current children cumulated CPU time: 402.26 s
Current children cumulated vsize: 5033888 KiB
Current children cumulated memory: 4981448 KiB

[startup+462.3 s]
/proc/loadavg: 2.00 2.03 2.06 3/200 14514
/proc/meminfo: memFree=8636284/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=5866900 memory=5723572 CPUtime=462.25 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 1374674 0 0 0 46046 179 0 0 20 0 1 0 409959173 6007705600 1430893 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 1466725 1430893 845 493 0 1460030 0
Current children cumulated CPU time: 462.25 s
Current children cumulated vsize: 5866900 KiB
Current children cumulated memory: 5723572 KiB


################
# More data... #
################

Current children cumulated vsize: 9601832 KiB
Current children cumulated memory: 9433548 KiB

[startup+822.3 s]
/proc/loadavg: 2.01 2.02 2.05 3/200 14547
/proc/meminfo: memFree=192268/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=10309220 memory=10173792 CPUtime=822.23 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 2439808 0 0 0 81861 362 0 0 20 0 1 0 409959173 10556641280 2543448 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 2577305 2543448 845 493 0 2570610 0
Current children cumulated CPU time: 822.23 s
Current children cumulated vsize: 10309220 KiB
Current children cumulated memory: 10173792 KiB

[startup+882.3 s]
/proc/loadavg: 1.95 2.00 2.04 3/200 14551
/proc/meminfo: memFree=203036/32770624 swapFree=66997368/67108860
[pid=14478] ppid=14475 vsize=11013572 memory=10911832 CPUtime=882.23 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 2617820 0 0 0 87804 419 0 0 20 0 1 0 409959173 11277897728 2727958 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 2753393 2727958 845 493 0 2746698 0
Current children cumulated CPU time: 882.23 s
Current children cumulated vsize: 11013572 KiB
Current children cumulated memory: 10911832 KiB

[startup+942.3 s]
/proc/loadavg: 2.35 2.09 2.07 3/203 14625
/proc/meminfo: memFree=15900644/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=11719244 memory=11650992 CPUtime=942.22 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 2795588 0 0 0 93760 462 0 0 20 0 1 0 409959173 12000505856 2912748 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 2929811 2912748 845 493 0 2923116 0
Current children cumulated CPU time: 942.22 s
Current children cumulated vsize: 11719244 KiB
Current children cumulated memory: 11650992 KiB

[startup+1002.3 s]
/proc/loadavg: 2.13 2.08 2.06 3/202 14756
/proc/meminfo: memFree=15152696/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=12426764 memory=12391872 CPUtime=1002.21 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 2972871 0 0 0 99722 499 0 0 20 0 1 0 409959173 12725006336 3097968 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3106691 3097968 845 493 0 3099996 0
Current children cumulated CPU time: 1002.21 s
Current children cumulated vsize: 12426764 KiB
Current children cumulated memory: 12391872 KiB

[startup+1062.3 s]
/proc/loadavg: 2.05 2.06 2.06 3/202 14760
/proc/meminfo: memFree=14410096/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=13421724 memory=13133184 CPUtime=1062.2 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 3150733 0 0 0 105681 539 0 0 20 0 1 0 409959173 13743845376 3283296 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3355431 3283296 845 493 0 3348736 0
Current children cumulated CPU time: 1062.2 s
Current children cumulated vsize: 13421724 KiB
Current children cumulated memory: 13133184 KiB

[startup+1122.3 s]
/proc/loadavg: 2.02 2.05 2.05 3/203 14767
/proc/meminfo: memFree=13665404/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=14128848 memory=13875372 CPUtime=1122.17 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 3327808 0 0 0 111638 579 0 0 20 0 1 0 409959173 14467940352 3468843 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3532212 3468843 845 493 0 3525517 0
Current children cumulated CPU time: 1122.17 s
Current children cumulated vsize: 14128848 KiB
Current children cumulated memory: 13875372 KiB

[startup+1182.3 s]
/proc/loadavg: 2.01 2.04 2.05 3/202 14771
/proc/meminfo: memFree=12924348/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=14836104 memory=14615368 CPUtime=1182.15 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 3504814 0 0 0 117595 620 0 0 20 0 1 0 409959173 15192170496 3653842 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3709026 3653842 845 493 0 3702331 0
Current children cumulated CPU time: 1182.15 s
Current children cumulated vsize: 14836104 KiB
Current children cumulated memory: 14615368 KiB

[startup+1242.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/200 14780
/proc/meminfo: memFree=12184140/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15542964 memory=15355028 CPUtime=1242.12 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 3681774 0 0 0 123554 658 0 0 20 0 1 0 409959173 15915995136 3838757 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3885741 3838757 845 493 0 3879046 0
Current children cumulated CPU time: 1242.12 s
Current children cumulated vsize: 15542964 KiB
Current children cumulated memory: 15355028 KiB



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1270.3 s]
/proc/loadavg: 2.00 2.03 2.05 3/200 14783
/proc/meminfo: memFree=11834240/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15872436 memory=15702808 CPUtime=1270.1 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 4202496 3764240 0 0 0 126332 678 0 0 20 0 1 0 409959173 16253374464 3925702 33554432000 4194304 6210612 140734567623984 140734567606152 139840152124389 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3968109 3925702 845 493 0 3961414 0
Current children cumulated CPU time: 1270.1 s
Current children cumulated vsize: 15872436 KiB
Current children cumulated memory: 15702808 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+1273.15 s]
# the end of solver process 14478 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=1263.58 s, system=7.69486 s

Solver just ended.
Dumping a history of the last processes samples

[startup+1270.4 s]
/proc/loadavg: 2.00 2.03 2.05 3/200 14783
/proc/meminfo: memFree=11834240/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15872436 memory=15702808 CPUtime=1270.11 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) S 14475 14478 13044 0 -1 512 3764245 0 0 0 126333 678 0 0 20 0 1 0 409959173 16253374464 3925702 33554432000 4194304 6210612 140734567623984 140734567606152 5188952 0 16384 4096 8405026 0 0 0 17 1 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3968109 3925702 845 493 0 3961414 0
Current children cumulated CPU time: 1270.11 s
Current children cumulated vsize: 15872436 KiB
Current children cumulated memory: 15702808 KiB

[startup+1271.2 s]
/proc/loadavg: 2.00 2.03 2.05 3/201 14784
/proc/meminfo: memFree=11458752/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15872436 memory=15702808 CPUtime=1270.24 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) S 14475 14478 13044 0 -1 512 3764245 0 0 0 126346 678 0 0 20 0 1 0 409959173 16253374464 3925702 33554432000 4194304 6210612 140734567623984 140734567606152 5188952 0 16384 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3968109 3925702 845 493 0 3961414 0
Current children cumulated CPU time: 1270.24 s
Current children cumulated vsize: 15872436 KiB
Current children cumulated memory: 15702808 KiB

[startup+1272 s]*
/proc/loadavg: 2.00 2.03 2.05 3/201 14784
/proc/meminfo: memFree=10915252/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15872436 memory=15702808 CPUtime=1270.37 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) S 14475 14478 13044 0 -1 512 3764245 0 0 0 126353 684 0 0 20 0 1 0 409959173 16253374464 3925702 33554432000 4194304 6210612 140734567623984 140734567606152 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3968109 3925702 845 493 0 3961414 0
Current children cumulated CPU time: 1270.37 s
Current children cumulated vsize: 15872436 KiB
Current children cumulated memory: 15702808 KiB

[startup+1272.2 s]
/proc/loadavg: 2.00 2.03 2.05 3/201 14784
/proc/meminfo: memFree=10915252/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15872436 memory=15702808 CPUtime=1270.4 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 512 3764245 0 0 0 126356 684 0 0 20 0 1 0 409959173 16253374464 3925702 33554432000 4194304 6210612 140734567623984 140734567606152 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3968109 3925702 845 493 0 3961414 0
Current children cumulated CPU time: 1270.4 s
Current children cumulated vsize: 15872436 KiB
Current children cumulated memory: 15702808 KiB

[startup+1272.3 s]
/proc/loadavg: 2.00 2.03 2.05 3/201 14784
/proc/meminfo: memFree=10915252/32770624 swapFree=66997372/67108860
[pid=14478] ppid=14475 vsize=15872436 memory=15702808 CPUtime=1270.42 cores=1,3,5,7
/proc/14478/stat : 14478 (cosoco) R 14475 14478 13044 0 -1 512 3764245 0 0 0 126358 684 0 0 20 0 1 0 409959173 16253374464 3925702 33554432000 4194304 6210612 140734567623984 140734567606152 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8308220 8313664 26038272 140734567629835 140734567629937 140734567629937 140734567632846 0
/proc/14478/statm: 3968109 3925702 845 493 0 3961414 0
Current children cumulated CPU time: 1270.42 s
Current children cumulated vsize: 15872436 KiB
Current children cumulated memory: 15702808 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1273.15
CPU time (s): 1271.28
CPU user time (s): 1263.58
CPU system time (s): 7.69486
CPU usage (%): 99.8525
Max. virtual memory (cumulated for all children) (KiB): 15872436
Max. memory (cumulated for all children) (KiB): 15702808

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1263.58
system time used= 7.69486
maximum resident set size= 15703008
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3764245
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 132854
involuntary context switches= 1596


# summary of solver processes directly reported to runsolver:
#   pid: 14478
#   total CPU time (s): 1271.28
#   total CPU user time (s): 1263.58
#   total CPU system time (s): 7.69486

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 4.64488 second user time and 8.65595 second system time

The end

Launcher Data 

Begin job on node103 at 2019-09-13 03:10:02
IDJOB=4390366
IDBENCH=122986
IDSOLVER=2831
FILE ID=node103/4390366-1568337061
RUNJOBID= node103-1568329709-13076
PBS_JOBID= 21701158
Free space on /tmp= 57264 MiB

SOLVER NAME= cosoco 2
BENCH NAME= XCSP17/Opd/Opd-sum-small/Opd-10-030-009.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4390366-1568337061/watcher-4390366-1568337061 -o /tmp/evaluation-result-4390366-1568337061/solver-4390366-1568337061 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node103-1568329709-13076 --watchdog 2560  cosoco -lc -model -restarts -stick HOME/instance-4390366-1568337061.xml

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 902db1136df2968cd1e3d1d445559775
RANDOM SEED=1205917562

node103.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        20457208 kB
MemAvailable:   28522332 kB
Buffers:          153868 kB
Cached:          8130868 kB
SwapCached:         8800 kB
Active:          5026252 kB
Inactive:        6594964 kB
Active(anon):    3324976 kB
Inactive(anon):    53120 kB
Active(file):    1701276 kB
Inactive(file):  6541844 kB
Unevictable:      107456 kB
Mlocked:          107456 kB
SwapTotal:      67108860 kB
SwapFree:       66997368 kB
Dirty:              2748 kB
Writeback:             0 kB
AnonPages:       3436692 kB
Mapped:            45400 kB
Shmem:             32848 kB
Slab:             280672 kB
SReclaimable:     250464 kB
SUnreclaim:        30208 kB
KernelStack:        3520 kB
PageTables:        12196 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:    3922144 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    368640 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      105856 kB
DirectMap2M:    33439744 kB

Free space on /tmp at the end= 56744 MiB
End job on node103 at 2019-09-13 03:31:15