Trace number 4397033

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2.0? (MO) 1528.03 1530.6

General information on the benchmark

NameOpd/Opd-sum-large/
Opd-15-035-010.xml
MD5SUMb4ffb396ba331e6a6f6e6b1bc4ae482f
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark3
Best CPU time to get the best result obtained on this benchmark6.18633
Satisfiable
(Un)Satisfiability was proved
Number of variables526
Number of constraints121
Number of domains2
Minimum domain size2
Maximum domain size33
Distribution of domain sizes[{"size":2,"count":525},{"size":33,"count":1}]
Minimum variable degree16
Maximum variable degree106
Distribution of variable degrees[{"degree":16,"count":525},{"degree":106,"count":1}]
Minimum constraint arity35
Maximum constraint arity525
Distribution of constraint arities[{"arity":35,"count":15},{"arity":71,"count":105},{"arity":525,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":120}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (9929 MiB free)
  memory of node 1: 16384 MiB (15153 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4397033-1568575547/watcher-4397033-1568575547 -o /tmp/evaluation-result-4397033-1568575547/solver-4397033-1568575547 -C 2520 -W 2520 -M 15500 --bin-var /tmp/runsolver-exch-node149-1568572746-32297 --watchdog 2580 cosoco -lc -model -restarts -stick HOME/instance-4397033-1568575547.xml 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2550 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=384, runsolver pid=381
Current StackSize limit: 8192 KiB


[startup+0.100152 s]*
/proc/loadavg: 2.00 1.99 1.89 3/199 385
/proc/meminfo: memFree=25683616/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=28680 memory=5008 CPUtime=0.1 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 1454 0 0 0 10 0 0 0 20 0 1 0 433793379 29368320 1252 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 7170 1252 845 493 0 475 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 28680 KiB
Current children cumulated memory: 5008 KiB

[startup+0.209419 s]*
/proc/loadavg: 2.00 1.99 1.89 3/199 385
/proc/meminfo: memFree=25682236/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=29768 memory=6064 CPUtime=0.2 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 1734 0 0 0 20 0 0 0 20 0 1 0 433793379 30482432 1516 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 7442 1516 846 493 0 747 0
Current children cumulated CPU time: 0.2 s
Current children cumulated vsize: 29768 KiB
Current children cumulated memory: 6064 KiB

[startup+0.308906 s]*
/proc/loadavg: 2.00 1.99 1.89 3/199 385
/proc/meminfo: memFree=25681120/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=30864 memory=7120 CPUtime=0.3 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 1990 0 0 0 30 0 0 0 20 0 1 0 433793379 31604736 1780 33554432000 4194304 6211092 140723234704496 140723234684744 5446275 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 7716 1780 846 493 0 1021 0
Current children cumulated CPU time: 0.3 s
Current children cumulated vsize: 30864 KiB
Current children cumulated memory: 7120 KiB

[startup+0.700272 s]
/proc/loadavg: 2.00 1.99 1.89 3/199 385
/proc/meminfo: memFree=25678888/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=34768 memory=11072 CPUtime=0.69 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 2986 0 0 0 69 0 0 0 20 0 1 0 433793379 35602432 2768 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 8692 2768 846 493 0 1997 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 34768 KiB
Current children cumulated memory: 11072 KiB

[startup+1.50028 s]
/proc/loadavg: 2.00 1.99 1.89 3/199 385
/proc/meminfo: memFree=25673388/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=43164 memory=19400 CPUtime=1.49 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 5045 0 0 0 149 0 0 0 20 0 1 0 433793379 44199936 4850 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 10791 4850 846 493 0 4096 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 43164 KiB
Current children cumulated memory: 19400 KiB

[startup+3.10939 s]
/proc/loadavg: 2.00 1.99 1.89 3/199 385
/proc/meminfo: memFree=25652816/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=60072 memory=35860 CPUtime=3.09 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 9235 0 0 0 309 0 0 0 20 0 1 0 433793379 61513728 8965 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 15018 8965 846 493 0 8323 0
Current children cumulated CPU time: 3.09 s
Current children cumulated vsize: 60072 KiB
Current children cumulated memory: 35860 KiB

[startup+6.30027 s]
/proc/loadavg: 2.00 1.99 1.90 3/199 385
/proc/meminfo: memFree=25622296/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=93756 memory=68684 CPUtime=6.29 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 17501 0 0 0 627 2 0 0 20 0 1 0 433793379 96006144 17171 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 23439 17171 846 493 0 16744 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 93756 KiB
Current children cumulated memory: 68684 KiB

[startup+12.7003 s]
/proc/loadavg: 2.00 1.99 1.90 3/199 385
/proc/meminfo: memFree=25561008/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=158748 memory=134712 CPUtime=12.68 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 34022 0 0 0 1265 3 0 0 20 0 1 0 433793379 162557952 33678 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 39687 33678 846 493 0 32992 0
Current children cumulated CPU time: 12.68 s
Current children cumulated vsize: 158748 KiB
Current children cumulated memory: 134712 KiB

[startup+25.5003 s]
/proc/loadavg: 2.00 1.99 1.90 3/200 387
/proc/meminfo: memFree=25425372/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=292652 memory=267776 CPUtime=25.48 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 66399 0 0 0 2541 7 0 0 20 0 1 0 433793379 299675648 66944 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 73163 66944 846 493 0 66468 0
Current children cumulated CPU time: 25.48 s
Current children cumulated vsize: 292652 KiB
Current children cumulated memory: 267776 KiB

[startup+51.1094 s]
/proc/loadavg: 2.00 1.99 1.90 3/199 387
/proc/meminfo: memFree=25156792/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=562604 memory=531072 CPUtime=51.06 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 130255 0 0 0 5092 14 0 0 20 0 1 0 433793379 576106496 132768 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 140651 132768 846 493 0 133956 0
Current children cumulated CPU time: 51.06 s
Current children cumulated vsize: 562604 KiB
Current children cumulated memory: 531072 KiB

[startup+102.3 s]
/proc/loadavg: 2.00 1.99 1.90 3/199 391
/proc/meminfo: memFree=24631780/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=1082348 memory=1057144 CPUtime=102.23 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 256741 0 0 0 10194 29 0 0 20 0 1 0 433793379 1108324352 264286 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 270587 264286 846 493 0 263892 0
Current children cumulated CPU time: 102.23 s
Current children cumulated vsize: 1082348 KiB
Current children cumulated memory: 1057144 KiB

[startup+162.3 s]
/proc/loadavg: 2.00 1.99 1.91 3/199 397
/proc/meminfo: memFree=24010692/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=1734028 memory=1676080 CPUtime=162.18 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 405311 0 0 0 16170 48 0 0 20 0 1 0 433793379 1775644672 419020 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 433507 419020 846 493 0 426812 0
Current children cumulated CPU time: 162.18 s
Current children cumulated vsize: 1734028 KiB
Current children cumulated memory: 1676080 KiB

[startup+222.3 s]
/proc/loadavg: 2.00 1.99 1.92 3/199 402
/proc/meminfo: memFree=23390728/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=2324128 memory=2294808 CPUtime=222.15 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 552851 0 0 0 22147 68 0 0 20 0 1 0 433793379 2379907072 573702 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 581032 573702 846 493 0 574337 0
Current children cumulated CPU time: 222.15 s
Current children cumulated vsize: 2324128 KiB
Current children cumulated memory: 2294808 KiB

[startup+282.3 s]
/proc/loadavg: 2.00 1.99 1.92 3/199 406
/proc/meminfo: memFree=22772572/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=2969948 memory=2911752 CPUtime=282.12 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 700984 0 0 0 28125 87 0 0 20 0 1 0 433793379 3041226752 727938 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 742487 727938 846 493 0 735792 0
Current children cumulated CPU time: 282.12 s
Current children cumulated vsize: 2969948 KiB
Current children cumulated memory: 2911752 KiB

[startup+342.3 s]
/proc/loadavg: 2.03 2.00 1.93 3/199 420
/proc/meminfo: memFree=22152420/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=3560284 memory=3530668 CPUtime=342.08 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 848545 0 0 0 34101 107 0 0 20 0 1 0 433793379 3645730816 882667 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 890071 882667 846 493 0 883376 0
Current children cumulated CPU time: 342.08 s
Current children cumulated vsize: 3560284 KiB
Current children cumulated memory: 3530668 KiB

[startup+402.3 s]
/proc/loadavg: 2.01 2.00 1.94 3/199 424
/proc/meminfo: memFree=21533636/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=4233236 memory=4148676 CPUtime=402.04 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 997006 0 0 0 40078 126 0 0 20 0 1 0 433793379 4334833664 1037169 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 1058309 1037169 846 493 0 1051614 0
Current children cumulated CPU time: 402.04 s
Current children cumulated vsize: 4233236 KiB
Current children cumulated memory: 4148676 KiB

[startup+462.3 s]
/proc/loadavg: 2.00 2.00 1.95 3/199 428
/proc/meminfo: memFree=20913796/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=4822880 memory=4767148 CPUtime=462 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 1144595 0 0 0 46054 146 0 0 20 0 1 0 433793379 4938629120 1191787 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 1205720 1191787 846 493 0 1199025 0
Current children cumulated CPU time: 462 s
Current children cumulated vsize: 4822880 KiB
Current children cumulated memory: 4767148 KiB


################
# More data... #
################

Current children cumulated vsize: 10436748 KiB
Current children cumulated memory: 10318152 KiB

[startup+1062.3 s]
/proc/loadavg: 2.00 2.00 1.95 3/199 486
/proc/meminfo: memFree=14735456/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=11025204 memory=10933340 CPUtime=1061.62 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 2619853 0 0 0 105798 364 0 0 20 0 1 0 433793379 11289808896 2733335 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 2756301 2733335 846 493 0 2749606 0
Current children cumulated CPU time: 1061.62 s
Current children cumulated vsize: 11025204 KiB
Current children cumulated memory: 10933340 KiB

[startup+1122.3 s]
/proc/loadavg: 2.44 2.11 1.99 3/199 490
/proc/meminfo: memFree=14115592/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=11615952 memory=11551624 CPUtime=1121.59 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 2767447 0 0 0 111771 388 0 0 20 0 1 0 433793379 11894734848 2887906 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 2903988 2887906 846 493 0 2897293 0
Current children cumulated CPU time: 1121.59 s
Current children cumulated vsize: 11615952 KiB
Current children cumulated memory: 11551624 KiB

[startup+1182.3 s]
/proc/loadavg: 2.23 2.12 2.00 3/199 494
/proc/meminfo: memFree=13498212/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=12204540 memory=12167336 CPUtime=1181.56 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 2914976 0 0 0 117743 413 0 0 20 0 1 0 433793379 12497448960 3041834 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3051135 3041834 846 493 0 3044440 0
Current children cumulated CPU time: 1181.56 s
Current children cumulated vsize: 12204540 KiB
Current children cumulated memory: 12167336 KiB

[startup+1242.3 s]
/proc/loadavg: 2.08 2.10 2.00 3/199 501
/proc/meminfo: memFree=12880516/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=13072880 memory=12784368 CPUtime=1241.53 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 3062800 0 0 0 123713 440 0 0 20 0 1 0 433793379 13386629120 3196092 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3268220 3196092 846 493 0 3261525 0
Current children cumulated CPU time: 1241.53 s
Current children cumulated vsize: 13072880 KiB
Current children cumulated memory: 12784368 KiB

[startup+1302.3 s]
/proc/loadavg: 2.03 2.08 2.00 3/199 506
/proc/meminfo: memFree=12265092/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=13661072 memory=13399292 CPUtime=1301.5 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 3210113 0 0 0 129684 466 0 0 20 0 1 0 433793379 13988937728 3349823 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3415268 3349823 846 493 0 3408573 0
Current children cumulated CPU time: 1301.5 s
Current children cumulated vsize: 13661072 KiB
Current children cumulated memory: 13399292 KiB

[startup+1362.3 s]
/proc/loadavg: 2.01 2.07 2.00 3/199 510
/proc/meminfo: memFree=11646412/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=14249264 memory=14016260 CPUtime=1361.47 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 3357408 0 0 0 135654 493 0 0 20 0 1 0 433793379 14591246336 3504065 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3562316 3504065 846 493 0 3555621 0
Current children cumulated CPU time: 1361.47 s
Current children cumulated vsize: 14249264 KiB
Current children cumulated memory: 14016260 KiB

[startup+1422.3 s]
/proc/loadavg: 2.00 2.05 2.00 3/199 516
/proc/meminfo: memFree=11028280/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=14837852 memory=14633544 CPUtime=1421.45 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 3504902 0 0 0 141622 523 0 0 20 0 1 0 433793379 15193960448 3658386 33554432000 4194304 6211092 140723234704496 140723234684744 140236544391998 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3709463 3658386 846 493 0 3702768 0
Current children cumulated CPU time: 1421.45 s
Current children cumulated vsize: 14837852 KiB
Current children cumulated memory: 14633544 KiB

[startup+1482.3 s]
/proc/loadavg: 2.05 2.06 2.01 3/199 520
/proc/meminfo: memFree=10411576/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15425780 memory=15248452 CPUtime=1481.41 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 3652216 0 0 0 147589 552 0 0 20 0 1 0 433793379 15795998720 3812113 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3856445 3812113 846 493 0 3849750 0
Current children cumulated CPU time: 1481.41 s
Current children cumulated vsize: 15425780 KiB
Current children cumulated memory: 15248452 KiB



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1527.9 s]
/proc/loadavg: 2.03 2.05 2.01 3/199 535
/proc/meminfo: memFree=9946364/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15872336 memory=15715752 CPUtime=1527 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 4202496 3764135 0 0 0 152122 578 0 0 20 0 1 0 433793379 16253272064 3928938 33554432000 4194304 6211092 140723234704496 140723234684744 140236544396261 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3968084 3928938 846 493 0 3961389 0
Current children cumulated CPU time: 1527 s
Current children cumulated vsize: 15872336 KiB
Current children cumulated memory: 15715752 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+1530.6 s]
# the end of solver process 384 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=1521.52 s, system=6.5042 s

Solver just ended.
Dumping a history of the last processes samples

[startup+1528 s]*
/proc/loadavg: 2.03 2.05 2.01 3/200 536
/proc/meminfo: memFree=9886068/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15872336 memory=15715752 CPUtime=1527.01 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) S 381 384 32265 0 -1 512 3764139 0 0 0 152123 578 0 0 20 0 1 0 433793379 16253272064 3928938 33554432000 4194304 6211092 140723234704496 140723234684744 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3968084 3928938 846 493 0 3961389 0
Current children cumulated CPU time: 1527.01 s
Current children cumulated vsize: 15872336 KiB
Current children cumulated memory: 15715752 KiB

[startup+1528.8 s]
/proc/loadavg: 2.03 2.05 2.01 3/200 536
/proc/meminfo: memFree=9886068/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15872336 memory=15715752 CPUtime=1527.14 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) S 381 384 32265 0 -1 512 3764139 0 0 0 152135 579 0 0 20 0 1 0 433793379 16253272064 3928938 33554432000 4194304 6211092 140723234704496 140723234684744 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3968084 3928938 846 493 0 3961389 0
Current children cumulated CPU time: 1527.14 s
Current children cumulated vsize: 15872336 KiB
Current children cumulated memory: 15715752 KiB

[startup+1529.6 s]
/proc/loadavg: 2.02 2.05 2.01 4/200 536
/proc/meminfo: memFree=9331160/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15872336 memory=15715752 CPUtime=1527.26 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) S 381 384 32265 0 -1 512 3764139 0 0 0 152147 579 0 0 20 0 1 0 433793379 16253272064 3928938 33554432000 4194304 6211092 140723234704496 140723234684744 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3968084 3928938 846 493 0 3961389 0
Current children cumulated CPU time: 1527.26 s
Current children cumulated vsize: 15872336 KiB
Current children cumulated memory: 15715752 KiB

[startup+1529.8 s]
/proc/loadavg: 2.02 2.05 2.01 4/200 536
/proc/meminfo: memFree=9331160/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15872336 memory=15715752 CPUtime=1527.3 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 512 3764139 0 0 0 152150 580 0 0 20 0 1 0 433793379 16253272064 3928938 33554432000 4194304 6211092 140723234704496 140723234684744 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3968084 3928938 846 493 0 3961389 0
Current children cumulated CPU time: 1527.3 s
Current children cumulated vsize: 15872336 KiB
Current children cumulated memory: 15715752 KiB

[startup+1529.9 s]
/proc/loadavg: 2.02 2.05 2.01 4/200 536
/proc/meminfo: memFree=9331160/32770624 swapFree=66941236/67108860
[pid=384] ppid=381 vsize=15872336 memory=15715752 CPUtime=1527.32 cores=1,3,5,7
/proc/384/stat : 384 (cosoco) R 381 384 32265 0 -1 512 3764139 0 0 0 152152 580 0 0 20 0 1 0 433793379 16253272064 3928938 33554432000 4194304 6211092 140723234704496 140723234684744 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8312316 8317760 9830400 140723234706443 140723234706545 140723234706545 140723234709454 0
/proc/384/statm: 3968084 3928938 846 493 0 3961389 0
Current children cumulated CPU time: 1527.32 s
Current children cumulated vsize: 15872336 KiB
Current children cumulated memory: 15715752 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1530.6
CPU time (s): 1528.03
CPU user time (s): 1521.52
CPU system time (s): 6.5042
CPU usage (%): 99.8316
Max. virtual memory (cumulated for all children) (KiB): 15872336
Max. memory (cumulated for all children) (KiB): 15715752

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1521.52
system time used= 6.5042
maximum resident set size= 15715904
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3764139
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 138113
involuntary context switches= 7153


# summary of solver processes directly reported to runsolver:
#   pid: 384
#   total CPU time (s): 1528.03
#   total CPU user time (s): 1521.52
#   total CPU system time (s): 6.5042

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 5.05974 second user time and 10.588 second system time

The end

Launcher Data 

Begin job on node149 at 2019-09-15 21:25:09
IDJOB=4397033
IDBENCH=122936
IDSOLVER=2848
FILE ID=node149/4397033-1568575547
RUNJOBID= node149-1568572746-32297
PBS_JOBID= 21703308
Free space on /tmp= 61420 MiB

SOLVER NAME= cosoco 2.0
BENCH NAME= XCSP17/Opd/Opd-sum-large/Opd-15-035-010.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4397033-1568575547/watcher-4397033-1568575547 -o /tmp/evaluation-result-4397033-1568575547/solver-4397033-1568575547 -C 2520 -W 2520 -M 15500 --bin-var /tmp/runsolver-exch-node149-1568572746-32297 --watchdog 2580  cosoco -lc -model -restarts -stick HOME/instance-4397033-1568575547.xml

TIME LIMIT= 2520 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b4ffb396ba331e6a6f6e6b1bc4ae482f
RANDOM SEED=1007110592

node149.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        25684904 kB
MemAvailable:   31759036 kB
Buffers:          207112 kB
Cached:          6168560 kB
SwapCached:        61260 kB
Active:          2588168 kB
Inactive:        3935308 kB
Active(anon):     115196 kB
Inactive(anon):    80936 kB
Active(file):    2472972 kB
Inactive(file):  3854372 kB
Unevictable:      111216 kB
Mlocked:          111216 kB
SwapTotal:      67108860 kB
SwapFree:       66941236 kB
Dirty:              2820 kB
Writeback:             0 kB
AnonPages:        199500 kB
Mapped:            48672 kB
Shmem:             39836 kB
Slab:             154516 kB
SReclaimable:     125248 kB
SUnreclaim:        29268 kB
KernelStack:        3536 kB
PageTables:         5412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     622136 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    112640 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      144768 kB
DirectMap2M:    33400832 kB

Free space on /tmp at the end= 60880 MiB
End job on node149 at 2019-09-15 21:50:40