Trace number 4389753

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2? (MO) 1521.29 1523.14

General information on the benchmark

NameOpd/Opd-sum-large/
Opd-15-035-010.xml
MD5SUMb4ffb396ba331e6a6f6e6b1bc4ae482f
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark3
Best CPU time to get the best result obtained on this benchmark6.18633
Satisfiable
(Un)Satisfiability was proved
Number of variables526
Number of constraints121
Number of domains2
Minimum domain size2
Maximum domain size33
Distribution of domain sizes[{"size":2,"count":525},{"size":33,"count":1}]
Minimum variable degree16
Maximum variable degree106
Distribution of variable degrees[{"degree":16,"count":525},{"degree":106,"count":1}]
Minimum constraint arity35
Maximum constraint arity525
Distribution of constraint arities[{"arity":35,"count":15},{"arity":71,"count":105},{"arity":525,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":120}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (15280 MiB free)
  memory of node 1: 16384 MiB (15562 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4389753-1568321648/watcher-4389753-1568321648 -o /tmp/evaluation-result-4389753-1568321648/solver-4389753-1568321648 -C 2520 -W 2520 -M 15500 --bin-var /tmp/runsolver-exch-node141-1568313599-15141 --watchdog 2580 cosoco -lc -model -restarts -stick HOME/instance-4389753-1568321648.xml 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2550 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=16826, runsolver pid=16823
Current StackSize limit: 8192 KiB


[startup+0.100155 s]*
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31581932/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=28680 memory=5008 CPUtime=0.1 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 1465 0 0 0 10 0 0 0 20 0 1 0 408102730 29368320 1252 33554432000 4194304 6210612 140730838700416 140730838696952 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 7170 1252 844 493 0 475 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 28680 KiB
Current children cumulated memory: 5008 KiB

[startup+0.2098 s]*
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31580816/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=29900 memory=6328 CPUtime=0.21 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 1748 0 0 0 21 0 0 0 20 0 1 0 408102730 30617600 1582 33554432000 4194304 6210612 140730838700416 140730838696952 5452090 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 7475 1582 845 493 0 780 0
Current children cumulated CPU time: 0.21 s
Current children cumulated vsize: 29900 KiB
Current children cumulated memory: 6328 KiB

[startup+0.309204 s]*
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31579700/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=30864 memory=7120 CPUtime=0.31 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2006 0 0 0 31 0 0 0 20 0 1 0 408102730 31604736 1780 33554432000 4194304 6210612 140730838700416 140730838696952 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 7716 1780 845 493 0 1021 0
Current children cumulated CPU time: 0.31 s
Current children cumulated vsize: 30864 KiB
Current children cumulated memory: 7120 KiB

[startup+0.700291 s]
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31577840/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=34900 memory=11068 CPUtime=0.69 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2995 0 0 0 69 0 0 0 20 0 1 0 408102730 35737600 2767 33554432000 4194304 6210612 140730838700416 140730838696952 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 8725 2767 845 493 0 2030 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 34900 KiB
Current children cumulated memory: 11068 KiB

[startup+1.50029 s]
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31572632/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=43296 memory=19396 CPUtime=1.49 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 5063 0 0 0 149 0 0 0 20 0 1 0 408102730 44335104 4849 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 10824 4849 845 493 0 4129 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 43296 KiB
Current children cumulated memory: 19396 KiB

[startup+3.10971 s]
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31551508/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=60204 memory=36064 CPUtime=3.1 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 9278 0 0 0 309 1 0 0 20 0 1 0 408102730 61648896 9016 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 15051 9016 845 493 0 8356 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 60204 KiB
Current children cumulated memory: 36064 KiB

[startup+6.30028 s]
/proc/loadavg: 2.06 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31520348/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=94152 memory=69208 CPUtime=6.29 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 17580 0 0 0 627 2 0 0 20 0 1 0 408102730 96411648 17302 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 23538 17302 845 493 0 16843 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 94152 KiB
Current children cumulated memory: 69208 KiB

[startup+12.7003 s]
/proc/loadavg: 2.05 2.07 2.06 3/200 16827
/proc/meminfo: memFree=31458128/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=159408 memory=135236 CPUtime=12.69 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 34175 0 0 0 1265 4 0 0 20 0 1 0 408102730 163233792 33809 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 39852 33809 845 493 0 33157 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 159408 KiB
Current children cumulated memory: 135236 KiB

[startup+25.5003 s]
/proc/loadavg: 2.04 2.07 2.06 3/200 16828
/proc/meminfo: memFree=31323220/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=293972 memory=267812 CPUtime=25.49 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 66929 0 0 0 2541 8 0 0 20 0 1 0 408102730 301027328 66953 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 73493 66953 845 493 0 66798 0
Current children cumulated CPU time: 25.49 s
Current children cumulated vsize: 293972 KiB
Current children cumulated memory: 267812 KiB

[startup+51.1098 s]
/proc/loadavg: 2.03 2.06 2.06 3/200 16830
/proc/meminfo: memFree=31052200/32770624 swapFree=66917236/67108860
[pid=16826] ppid=16823 vsize=564980 memory=534480 CPUtime=51.11 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 131065 0 0 0 5094 17 0 0 20 0 1 0 408102730 578539520 133620 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 141245 133620 845 493 0 134550 0
Current children cumulated CPU time: 51.11 s
Current children cumulated vsize: 564980 KiB
Current children cumulated memory: 534480 KiB

[startup+102.3 s]
/proc/loadavg: 2.01 2.05 2.06 3/199 16835
/proc/meminfo: memFree=30524908/32770624 swapFree=66917244/67108860
[pid=16826] ppid=16823 vsize=1087100 memory=1062448 CPUtime=102.29 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 258077 0 0 0 10198 31 0 0 20 0 1 0 408102730 1113190400 265612 33554432000 4194304 6210612 140730838700416 140730838680664 140039456208432 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 271775 265612 845 493 0 265080 0
Current children cumulated CPU time: 102.29 s
Current children cumulated vsize: 1087100 KiB
Current children cumulated memory: 1062448 KiB

[startup+162.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/199 16839
/proc/meminfo: memFree=29902648/32770624 swapFree=66917244/67108860
[pid=16826] ppid=16823 vsize=1740232 memory=1682940 CPUtime=162.29 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 407088 0 0 0 16178 51 0 0 20 0 1 0 408102730 1781997568 420735 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 435058 420735 845 493 0 428363 0
Current children cumulated CPU time: 162.29 s
Current children cumulated vsize: 1740232 KiB
Current children cumulated memory: 1682940 KiB

[startup+222.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/200 16845
/proc/meminfo: memFree=29281396/32770624 swapFree=66917244/67108860
[pid=16826] ppid=16823 vsize=2332180 memory=2303560 CPUtime=222.28 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 555057 0 0 0 22158 70 0 0 20 0 1 0 408102730 2388152320 575890 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 583045 575890 845 493 0 576350 0
Current children cumulated CPU time: 222.28 s
Current children cumulated vsize: 2332180 KiB
Current children cumulated memory: 2303560 KiB

[startup+282.3 s]
/proc/loadavg: 2.00 2.03 2.05 2/194 16858
/proc/meminfo: memFree=28737068/32770624 swapFree=66917248/67108860
[pid=16826] ppid=16823 vsize=2979848 memory=2922616 CPUtime=282.27 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 703692 0 0 0 28140 87 0 0 20 0 1 0 408102730 3051364352 730654 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 744962 730654 845 493 0 738267 0
Current children cumulated CPU time: 282.27 s
Current children cumulated vsize: 2979848 KiB
Current children cumulated memory: 2922616 KiB

[startup+342.3 s]
/proc/loadavg: 2.19 2.07 2.06 3/199 16914
/proc/meminfo: memFree=27543460/32770624 swapFree=66917248/67108860
[pid=16826] ppid=16823 vsize=3572428 memory=3543656 CPUtime=342.27 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 851808 0 0 0 34121 106 0 0 20 0 1 0 408102730 3658166272 885914 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 893107 885914 845 493 0 886412 0
Current children cumulated CPU time: 342.27 s
Current children cumulated vsize: 3572428 KiB
Current children cumulated memory: 3543656 KiB

[startup+402.3 s]
/proc/loadavg: 2.07 2.06 2.05 3/202 16929
/proc/meminfo: memFree=26345912/32770624 swapFree=66917264/67108860
[pid=16826] ppid=16823 vsize=4247228 memory=4163368 CPUtime=402.26 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 1000233 0 0 0 40101 125 0 0 20 0 1 0 408102730 4349161472 1040842 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 1061807 1040842 845 493 0 1055112 0
Current children cumulated CPU time: 402.26 s
Current children cumulated vsize: 4247228 KiB
Current children cumulated memory: 4163368 KiB

[startup+462.3 s]
/proc/loadavg: 2.03 2.05 2.05 3/202 16956
/proc/meminfo: memFree=25146324/32770624 swapFree=66917268/67108860
[pid=16826] ppid=16823 vsize=4839116 memory=4784164 CPUtime=462.26 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 1148374 0 0 0 46081 145 0 0 20 0 1 0 408102730 4955254784 1196041 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 1209779 1196041 845 493 0 1203084 0
Current children cumulated CPU time: 462.26 s
Current children cumulated vsize: 4839116 KiB
Current children cumulated memory: 4784164 KiB


################
# More data... #
################

Current children cumulated vsize: 9296808 KiB
Current children cumulated memory: 9124316 KiB

[startup+942.3 s]
/proc/loadavg: 2.01 2.02 2.05 3/200 16994
/proc/meminfo: memFree=15573528/32770624 swapFree=66917268/67108860
[pid=16826] ppid=16823 vsize=9890268 memory=9745508 CPUtime=942.2 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2335443 0 0 0 93899 321 0 0 20 0 1 0 408102730 10127634432 2436377 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 2472567 2436377 845 493 0 2465872 0
Current children cumulated CPU time: 942.2 s
Current children cumulated vsize: 9890268 KiB
Current children cumulated memory: 9745508 KiB

[startup+1002.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/200 17006
/proc/meminfo: memFree=14375804/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=10482288 memory=10366620 CPUtime=1002.19 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2483780 0 0 0 99875 344 0 0 20 0 1 0 408102730 10733862912 2591655 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 2620572 2591655 845 493 0 2613877 0
Current children cumulated CPU time: 1002.19 s
Current children cumulated vsize: 10482288 KiB
Current children cumulated memory: 10366620 KiB

[startup+1062.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/200 17010
/proc/meminfo: memFree=13182352/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=11074176 memory=10985356 CPUtime=1062.18 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2632065 0 0 0 105849 369 0 0 20 0 1 0 408102730 11339956224 2746339 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 2768544 2746339 845 493 0 2761849 0
Current children cumulated CPU time: 1062.18 s
Current children cumulated vsize: 11074176 KiB
Current children cumulated memory: 10985356 KiB

[startup+1122.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/201 17016
/proc/meminfo: memFree=11984448/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=11667960 memory=11606280 CPUtime=1122.18 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2780436 0 0 0 111825 393 0 0 20 0 1 0 408102730 11947991040 2901570 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 2916990 2901570 845 493 0 2910295 0
Current children cumulated CPU time: 1122.18 s
Current children cumulated vsize: 11667960 KiB
Current children cumulated memory: 11606280 KiB

[startup+1182.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/200 17020
/proc/meminfo: memFree=10788808/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=12259452 memory=12225336 CPUtime=1182.18 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 2928761 0 0 0 117797 421 0 0 20 0 1 0 408102730 12553678848 3056334 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3064863 3056334 845 493 0 3058168 0
Current children cumulated CPU time: 1182.18 s
Current children cumulated vsize: 12259452 KiB
Current children cumulated memory: 12225336 KiB

[startup+1242.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/200 17024
/proc/meminfo: memFree=9591588/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=13130960 memory=12845188 CPUtime=1242.17 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 3077365 0 0 0 123770 447 0 0 20 0 1 0 408102730 13446103040 3211297 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3282740 3211297 845 493 0 3276045 0
Current children cumulated CPU time: 1242.17 s
Current children cumulated vsize: 13130960 KiB
Current children cumulated memory: 12845188 KiB

[startup+1302.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/200 17030
/proc/meminfo: memFree=8398900/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=13722452 memory=13463420 CPUtime=1302.16 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 3225537 0 0 0 129742 474 0 0 20 0 1 0 408102730 14051790848 3365855 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3430613 3365855 845 493 0 3423918 0
Current children cumulated CPU time: 1302.16 s
Current children cumulated vsize: 13722452 KiB
Current children cumulated memory: 13463420 KiB

[startup+1362.3 s]
/proc/loadavg: 2.61 2.17 2.10 3/199 17034
/proc/meminfo: memFree=7200852/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=14313944 memory=14083736 CPUtime=1362.16 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 3373682 0 0 0 135717 499 0 0 20 0 1 0 408102730 14657478656 3520934 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3578486 3520934 845 493 0 3571791 0
Current children cumulated CPU time: 1362.16 s
Current children cumulated vsize: 14313944 KiB
Current children cumulated memory: 14083736 KiB

[startup+1422.3 s]
/proc/loadavg: 2.29 2.17 2.10 3/199 17038
/proc/meminfo: memFree=6003976/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=14905172 memory=14704128 CPUtime=1422.15 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 3521892 0 0 0 141690 525 0 0 20 0 1 0 408102730 15262896128 3676032 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3726293 3676032 845 493 0 3719598 0
Current children cumulated CPU time: 1422.15 s
Current children cumulated vsize: 14905172 KiB
Current children cumulated memory: 14704128 KiB

[startup+1482.3 s]
/proc/loadavg: 2.11 2.14 2.09 3/200 17044
/proc/meminfo: memFree=4809228/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=15496664 memory=15322404 CPUtime=1482.14 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 3670077 0 0 0 147660 554 0 0 20 0 1 0 408102730 15868583936 3830601 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3874166 3830601 845 493 0 3867471 0
Current children cumulated CPU time: 1482.14 s
Current children cumulated vsize: 15496664 KiB
Current children cumulated memory: 15322404 KiB



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1520.4 s]
/proc/loadavg: 2.05 2.12 2.09 3/199 17045
/proc/meminfo: memFree=4051660/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=15872204 memory=15716268 CPUtime=1520.24 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) R 16823 16826 15109 0 -1 4202496 3764170 0 0 0 151453 571 0 0 20 0 1 0 408102730 16253136896 3929067 33554432000 4194304 6210612 140730838700416 140730838680664 140039455284197 0 0 4096 8405026 0 0 0 17 5 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3968051 3929067 845 493 0 3961356 0
Current children cumulated CPU time: 1520.24 s
Current children cumulated vsize: 15872204 KiB
Current children cumulated memory: 15716268 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+1523.14 s]
# the end of solver process 16826 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=1514.74 s, system=6.54249 s

Solver just ended.
Dumping a history of the last processes samples

[startup+1521.6 s]
/proc/loadavg: 2.05 2.12 2.09 4/200 17046
/proc/meminfo: memFree=3707040/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=15872204 memory=15716268 CPUtime=1520.43 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) S 16823 16826 15109 0 -1 512 3764175 0 0 0 151468 575 0 0 20 0 1 0 408102730 16253136896 3929067 33554432000 4194304 6210612 140730838700416 140730838680664 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3968051 3929067 845 493 0 3961356 0
Current children cumulated CPU time: 1520.43 s
Current children cumulated vsize: 15872204 KiB
Current children cumulated memory: 15716268 KiB

[startup+1522 s]*
/proc/loadavg: 2.05 2.12 2.09 4/200 17046
/proc/meminfo: memFree=3220828/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=15872204 memory=15716268 CPUtime=1520.49 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) S 16823 16826 15109 0 -1 512 3764175 0 0 0 151471 578 0 0 20 0 1 0 408102730 16253136896 3929067 33554432000 4194304 6210612 140730838700416 140730838680664 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3968051 3929067 845 493 0 3961356 0
Current children cumulated CPU time: 1520.49 s
Current children cumulated vsize: 15872204 KiB
Current children cumulated memory: 15716268 KiB

[startup+1522.4 s]
/proc/loadavg: 2.05 2.12 2.09 4/200 17046
/proc/meminfo: memFree=3220828/32770624 swapFree=66917276/67108860
[pid=16826] ppid=16823 vsize=15872204 memory=15716268 CPUtime=1520.54 cores=1,3,5,7
/proc/16826/stat : 16826 (cosoco) S 16823 16826 15109 0 -1 512 3764175 0 0 0 151474 580 0 0 20 0 1 0 408102730 16253136896 3929067 33554432000 4194304 6210612 140730838700416 140730838680664 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 7 0 0 0 0 0 8308220 8313664 36429824 140730838705163 140730838705265 140730838705265 140730838708174 0
/proc/16826/statm: 3968051 3929067 845 493 0 3961356 0
Current children cumulated CPU time: 1520.54 s
Current children cumulated vsize: 15872204 KiB
Current children cumulated memory: 15716268 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1523.14
CPU time (s): 1521.29
CPU user time (s): 1514.74
CPU system time (s): 6.54249
CPU usage (%): 99.8783
Max. virtual memory (cumulated for all children) (KiB): 15872204
Max. memory (cumulated for all children) (KiB): 15716268

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1514.74
system time used= 6.54249
maximum resident set size= 15716516
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3764175
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 121500
involuntary context switches= 1223


# summary of solver processes directly reported to runsolver:
#   pid: 16826
#   total CPU time (s): 1521.29
#   total CPU user time (s): 1514.74
#   total CPU system time (s): 6.54249

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 5.84906 second user time and 10.2983 second system time

The end

Launcher Data 

Begin job on node141 at 2019-09-12 22:53:27
IDJOB=4389753
IDBENCH=122936
IDSOLVER=2829
FILE ID=node141/4389753-1568321648
RUNJOBID= node141-1568313599-15141
PBS_JOBID= 21701135
Free space on /tmp= 56108 MiB

SOLVER NAME= cosoco 2
BENCH NAME= XCSP17/Opd/Opd-sum-large/Opd-15-035-010.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4389753-1568321648/watcher-4389753-1568321648 -o /tmp/evaluation-result-4389753-1568321648/solver-4389753-1568321648 -C 2520 -W 2520 -M 15500 --bin-var /tmp/runsolver-exch-node141-1568313599-15141 --watchdog 2580  cosoco -lc -model -restarts -stick HOME/instance-4389753-1568321648.xml

TIME LIMIT= 2520 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b4ffb396ba331e6a6f6e6b1bc4ae482f
RANDOM SEED=1326346931

node141.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        31583296 kB
MemAvailable:   31771680 kB
Buffers:          120928 kB
Cached:           356232 kB
SwapCached:        81948 kB
Active:           298548 kB
Inactive:         338508 kB
Active(anon):      84024 kB
Inactive(anon):   106104 kB
Active(file):     214524 kB
Inactive(file):   232404 kB
Unevictable:      109212 kB
Mlocked:          109212 kB
SwapTotal:      67108860 kB
SwapFree:       66917236 kB
Dirty:              2724 kB
Writeback:             0 kB
AnonPages:        249016 kB
Mapped:            34964 kB
Shmem:             21288 kB
Slab:             144332 kB
SReclaimable:     114580 kB
SUnreclaim:        29752 kB
KernelStack:        3568 kB
PageTables:         5968 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     602020 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    196608 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      128384 kB
DirectMap2M:    33417216 kB

Free space on /tmp at the end= 55632 MiB
End job on node141 at 2019-09-12 23:18:50