Trace number 4390424

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2? (MO) 1117.74 1119.82

General information on the benchmark

NameOpd/Opd-sum-small/
Opd-19-020-009.xml
MD5SUM62ee24bcba75d1d33fbbd4bb45a854a0
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark4
Best CPU time to get the best result obtained on this benchmark1381.62
Satisfiable
(Un)Satisfiability was proved
Number of variables381
Number of constraints191
Number of domains2
Minimum domain size2
Maximum domain size17
Distribution of domain sizes[{"size":2,"count":380},{"size":17,"count":1}]
Minimum variable degree20
Maximum variable degree172
Distribution of variable degrees[{"degree":20,"count":380},{"degree":172,"count":1}]
Minimum constraint arity20
Maximum constraint arity380
Distribution of constraint arities[{"arity":20,"count":19},{"arity":41,"count":171},{"arity":380,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":190}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (14459 MiB free)
  memory of node 1: 16384 MiB (2631 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4390424-1568336997/watcher-4390424-1568336997 -o /tmp/evaluation-result-4390424-1568336997/solver-4390424-1568336997 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node134-1568335411-5299 --watchdog 2560 cosoco -lc -model -restarts -stick HOME/instance-4390424-1568336997.xml 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2500 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=5553, runsolver pid=5550
Current StackSize limit: 8192 KiB


[startup+0.100209 s]*
/proc/loadavg: 2.00 1.98 1.79 4/205 5554
/proc/meminfo: memFree=17498780/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=28988 memory=5276 CPUtime=0.1 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 1529 0 0 0 10 0 0 0 20 0 1 0 409950258 29683712 1319 33554432000 4194304 6210612 140729851382976 140729851379512 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 7247 1319 845 493 0 552 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 28988 KiB
Current children cumulated memory: 5276 KiB

[startup+0.210398 s]*
/proc/loadavg: 2.00 1.98 1.79 3/205 5554
/proc/meminfo: memFree=17496216/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=30564 memory=6856 CPUtime=0.21 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 1912 0 0 0 21 0 0 0 20 0 1 0 409950258 31297536 1714 33554432000 4194304 6210612 140729851382976 140729851379512 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 7641 1714 845 493 0 946 0
Current children cumulated CPU time: 0.21 s
Current children cumulated vsize: 30564 KiB
Current children cumulated memory: 6856 KiB

[startup+0.310024 s]*
/proc/loadavg: 2.00 1.98 1.79 3/205 5554
/proc/meminfo: memFree=17493984/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=32008 memory=8176 CPUtime=0.31 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 2261 0 0 0 31 0 0 0 20 0 1 0 409950258 32776192 2044 33554432000 4194304 6210612 140729851382976 140729851379512 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 8002 2044 845 493 0 1307 0
Current children cumulated CPU time: 0.31 s
Current children cumulated vsize: 32008 KiB
Current children cumulated memory: 8176 KiB

[startup+0.700387 s]
/proc/loadavg: 2.00 1.98 1.79 3/205 5554
/proc/meminfo: memFree=17489768/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=37368 memory=13712 CPUtime=0.69 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 3595 0 0 0 69 0 0 0 20 0 1 0 409950258 38264832 3428 33554432000 4194304 6210612 140729851382976 140729851379512 5012488 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 9342 3428 845 493 0 2647 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 37368 KiB
Current children cumulated memory: 13712 KiB

[startup+1.50041 s]
/proc/loadavg: 2.00 1.98 1.79 3/205 5554
/proc/meminfo: memFree=17478980/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=48996 memory=24856 CPUtime=1.49 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 6396 0 0 0 148 1 0 0 20 0 1 0 409950258 50171904 6214 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 12249 6214 845 493 0 5554 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 48996 KiB
Current children cumulated memory: 24856 KiB

[startup+3.10992 s]
/proc/loadavg: 2.24 2.03 1.81 3/205 5554
/proc/meminfo: memFree=17430600/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=72384 memory=47284 CPUtime=3.1 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 12067 0 0 0 308 2 0 0 20 0 1 0 409950258 74121216 11821 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 18096 11821 845 493 0 11401 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 72384 KiB
Current children cumulated memory: 47284 KiB

[startup+6.30037 s]
/proc/loadavg: 2.24 2.03 1.81 3/206 5555
/proc/meminfo: memFree=17343388/32770624 swapFree=66915836/67108860
[pid=5553] ppid=5550 vsize=116256 memory=92008 CPUtime=6.29 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 23282 0 0 0 626 3 0 0 20 0 1 0 409950258 119046144 23002 33554432000 4194304 6210612 140729851382976 140729851368984 139971951896126 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 29064 23002 845 493 0 22369 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 116256 KiB
Current children cumulated memory: 92008 KiB

[startup+12.7003 s]*
/proc/loadavg: 2.43 2.08 1.83 3/205 5558
/proc/meminfo: memFree=18116548/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=208044 memory=183148 CPUtime=12.69 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 45078 0 0 0 1263 6 0 0 20 0 1 0 409950258 213037056 45787 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 52011 45787 845 493 0 45316 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 208044 KiB
Current children cumulated memory: 183148 KiB

[startup+25.5003 s]
/proc/loadavg: 2.36 2.08 1.83 3/205 5561
/proc/meminfo: memFree=17857032/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=393688 memory=362024 CPUtime=25.49 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 87845 0 0 0 2539 10 0 0 20 0 1 0 409950258 403136512 90506 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 98422 90506 845 493 0 91727 0
Current children cumulated CPU time: 25.49 s
Current children cumulated vsize: 393688 KiB
Current children cumulated memory: 362024 KiB

[startup+51.1003 s]
/proc/loadavg: 2.24 2.07 1.84 3/205 5566
/proc/meminfo: memFree=17292780/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=745008 memory=719928 CPUtime=51.08 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 172214 0 0 0 5087 21 0 0 20 0 1 0 409950258 762888192 179982 33554432000 4194304 6210612 140729851382976 140729851368984 5012624 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 186252 179982 845 493 0 179557 0
Current children cumulated CPU time: 51.08 s
Current children cumulated vsize: 745008 KiB
Current children cumulated memory: 719928 KiB

[startup+102.3 s]
/proc/loadavg: 2.10 2.06 1.85 3/205 5571
/proc/meminfo: memFree=16183188/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=1474828 memory=1437232 CPUtime=102.27 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 340310 0 0 0 10185 42 0 0 20 0 1 0 409950258 1510223872 359308 33554432000 4194304 6210612 140729851382976 140729851368984 5440460 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 368707 359308 845 493 0 362012 0
Current children cumulated CPU time: 102.27 s
Current children cumulated vsize: 1474828 KiB
Current children cumulated memory: 1437232 KiB

[startup+162.3 s]
/proc/loadavg: 2.04 2.05 1.86 3/205 5577
/proc/meminfo: memFree=14874716/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=2314200 memory=2276508 CPUtime=162.26 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 536840 0 0 0 16159 67 0 0 20 0 1 0 409950258 2369740800 569127 33554432000 4194304 6210612 140729851382976 140729851368984 5446100 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 578550 569127 845 493 0 571855 0
Current children cumulated CPU time: 162.26 s
Current children cumulated vsize: 2314200 KiB
Current children cumulated memory: 2276508 KiB

[startup+222.3 s]
/proc/loadavg: 1.95 2.01 1.86 3/205 5581
/proc/meminfo: memFree=13570660/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=3179232 memory=3113032 CPUtime=222.25 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 733196 0 0 0 22134 91 0 0 20 0 1 0 409950258 3255533568 778258 33554432000 4194304 6210612 140729851382976 140729851368984 5442952 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 794808 778258 845 493 0 788113 0
Current children cumulated CPU time: 222.25 s
Current children cumulated vsize: 3179232 KiB
Current children cumulated memory: 3113032 KiB

[startup+282.3 s]
/proc/loadavg: 1.93 2.00 1.86 3/205 5585
/proc/meminfo: memFree=12266688/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=4086196 memory=3950824 CPUtime=282.25 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 929403 0 0 0 28106 119 0 0 20 0 1 0 409950258 4184264704 987706 33554432000 4194304 6210612 140729851382976 140729851368984 5012705 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 1021549 987706 845 493 0 1014854 0
Current children cumulated CPU time: 282.25 s
Current children cumulated vsize: 4086196 KiB
Current children cumulated memory: 3950824 KiB

[startup+342.3 s]
/proc/loadavg: 1.97 2.00 1.87 3/204 5591
/proc/meminfo: memFree=10961108/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=4868276 memory=4787952 CPUtime=342.24 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 1125009 0 0 0 34078 146 0 0 20 0 1 0 409950258 4985114624 1196988 33554432000 4194304 6210612 140729851382976 140729851368984 139971951898048 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 1217069 1196988 845 493 0 1210374 0
Current children cumulated CPU time: 342.24 s
Current children cumulated vsize: 4868276 KiB
Current children cumulated memory: 4787952 KiB

[startup+402.3 s]
/proc/loadavg: 1.99 2.00 1.88 3/203 5595
/proc/meminfo: memFree=9656820/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=5650412 memory=5624968 CPUtime=402.23 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 1320538 0 0 0 40051 172 0 0 20 0 1 0 409950258 5786021888 1406242 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 1412603 1406242 845 493 0 1405908 0
Current children cumulated CPU time: 402.23 s
Current children cumulated vsize: 5650412 KiB
Current children cumulated memory: 5624968 KiB

[startup+462.3 s]
/proc/loadavg: 1.92 1.98 1.88 3/204 5600
/proc/meminfo: memFree=8349416/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=6621084 memory=6465572 CPUtime=462.22 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 1516965 0 0 0 46021 201 0 0 20 0 1 0 409950258 6779990016 1616393 33554432000 4194304 6210612 140729851382976 140729851368984 5446470 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 1655271 1616393 845 493 0 1648576 0
Current children cumulated CPU time: 462.22 s
Current children cumulated vsize: 6621084 KiB
Current children cumulated memory: 6465572 KiB


################
# More data... #
################

Current children cumulated vsize: 8186988 KiB
Current children cumulated memory: 8141392 KiB

[startup+642.3 s]
/proc/loadavg: 2.07 2.07 1.93 3/205 5785
/proc/meminfo: memFree=21410600/32770624 swapFree=66916308/67108860
[pid=5553] ppid=5550 vsize=9251348 memory=8981280 CPUtime=642.15 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 2105345 0 0 0 63930 285 0 0 20 0 1 0 409950258 9473380352 2245320 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 2312837 2245320 845 493 0 2306142 0
Current children cumulated CPU time: 642.15 s
Current children cumulated vsize: 9251348 KiB
Current children cumulated memory: 8981280 KiB

[startup+702.3 s]
/proc/loadavg: 2.03 2.06 1.94 3/205 5798
/proc/meminfo: memFree=20570720/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=10034768 memory=9822300 CPUtime=702.11 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 2301432 0 0 0 69898 313 0 0 20 0 1 0 409950258 10275602432 2455575 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 2508692 2455575 845 493 0 2501997 0
Current children cumulated CPU time: 702.11 s
Current children cumulated vsize: 10034768 KiB
Current children cumulated memory: 9822300 KiB

[startup+762.3 s]
/proc/loadavg: 2.01 2.04 1.94 3/205 5803
/proc/meminfo: memFree=19726612/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=10818980 memory=10661760 CPUtime=762.08 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 2497710 0 0 0 75867 341 0 0 20 0 1 0 409950258 11078635520 2665440 33554432000 4194304 6210612 140729851382976 140729851368984 139971950971877 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 2704745 2665440 845 493 0 2698050 0
Current children cumulated CPU time: 762.08 s
Current children cumulated vsize: 10818980 KiB
Current children cumulated memory: 10661760 KiB

[startup+822.3 s]
/proc/loadavg: 2.00 2.04 1.95 3/203 5807
/proc/meminfo: memFree=18885288/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=11603192 memory=11501472 CPUtime=822.05 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 2693995 0 0 0 81835 370 0 0 20 0 1 0 409950258 11881668608 2875368 33554432000 4194304 6210612 140729851382976 140729851368984 5451458 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 2900798 2875368 845 493 0 2894103 0
Current children cumulated CPU time: 822.05 s
Current children cumulated vsize: 11603192 KiB
Current children cumulated memory: 11501472 KiB

[startup+882.3 s]
/proc/loadavg: 2.08 2.05 1.96 3/203 5877
/proc/meminfo: memFree=17980244/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=12388772 memory=12340916 CPUtime=882.02 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 2890157 0 0 0 87797 405 0 0 20 0 1 0 409950258 12686102528 3085229 33554432000 4194304 6210612 140729851382976 140729851368984 139971951896132 0 0 4096 8405026 0 0 0 17 0 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3097193 3085229 845 493 0 3090498 0
Current children cumulated CPU time: 882.02 s
Current children cumulated vsize: 12388772 KiB
Current children cumulated memory: 12340916 KiB

[startup+942.3 s]
/proc/loadavg: 2.03 2.04 1.96 3/203 5881
/proc/meminfo: memFree=17141356/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=13590900 memory=13178212 CPUtime=941.99 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 3086919 0 0 0 93759 440 0 0 20 0 1 0 409950258 13917081600 3294553 33554432000 4194304 6210612 140729851382976 140729851368984 5012715 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3397725 3294553 845 493 0 3391030 0
Current children cumulated CPU time: 941.99 s
Current children cumulated vsize: 13590900 KiB
Current children cumulated memory: 13178212 KiB

[startup+1002.3 s]
/proc/loadavg: 2.09 2.05 1.97 3/203 5886
/proc/meminfo: memFree=16300644/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=14374188 memory=14017664 CPUtime=1001.96 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 3283092 0 0 0 99721 475 0 0 20 0 1 0 409950258 14719168512 3504416 33554432000 4194304 6210612 140729851382976 140729851368984 5451361 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3593547 3504416 845 493 0 3586852 0
Current children cumulated CPU time: 1001.96 s
Current children cumulated vsize: 14374188 KiB
Current children cumulated memory: 14017664 KiB

[startup+1062.3 s]
/proc/loadavg: 2.03 2.04 1.97 3/203 5893
/proc/meminfo: memFree=15458464/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15159264 memory=14855728 CPUtime=1061.93 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 3478944 0 0 0 105679 514 0 0 20 0 1 0 409950258 15523086336 3713932 33554432000 4194304 6210612 140729851382976 140729851368984 139971960916824 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3789816 3713932 845 493 0 3783121 0
Current children cumulated CPU time: 1061.93 s
Current children cumulated vsize: 15159264 KiB
Current children cumulated memory: 14855728 KiB



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1117 s]
/proc/loadavg: 2.01 2.04 1.97 3/203 5897
/proc/meminfo: memFree=14688988/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15872196 memory=15622128 CPUtime=1116.6 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 4202496 3657409 0 0 0 111109 551 0 0 20 0 1 0 409950258 16253128704 3905532 33554432000 4194304 6210612 140729851382976 140729851368984 5012555 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3968049 3905532 845 493 0 3961354 0
Current children cumulated CPU time: 1116.6 s
Current children cumulated vsize: 15872196 KiB
Current children cumulated memory: 15622128 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+1119.82 s]
# the end of solver process 5553 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=1111.37 s, system=6.36357 s

Solver just ended.
Dumping a history of the last processes samples

[startup+1117.5 s]
/proc/loadavg: 2.01 2.04 1.97 3/203 5897
/proc/meminfo: memFree=14688988/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15872196 memory=15622128 CPUtime=1116.69 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) R 5550 5553 5268 0 -1 512 3657414 0 0 0 111117 552 0 0 20 0 1 0 409950258 16253128704 3905532 33554432000 4194304 6210612 140729851382976 140729851368984 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3968049 3905532 845 493 0 3961354 0
Current children cumulated CPU time: 1116.69 s
Current children cumulated vsize: 15872196 KiB
Current children cumulated memory: 15622128 KiB

[startup+1118.3 s]
/proc/loadavg: 1.93 2.02 1.96 2/204 5898
/proc/meminfo: memFree=14282560/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15872196 memory=15622128 CPUtime=1116.81 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) S 5550 5553 5268 0 -1 512 3657414 0 0 0 111128 553 0 0 20 0 1 0 409950258 16253128704 3905532 33554432000 4194304 6210612 140729851382976 140729851368984 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3968049 3905532 845 493 0 3961354 0
Current children cumulated CPU time: 1116.81 s
Current children cumulated vsize: 15872196 KiB
Current children cumulated memory: 15622128 KiB

[startup+1118.7 s]*
/proc/loadavg: 1.93 2.02 1.96 4/204 5898
/proc/meminfo: memFree=13777112/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15872196 memory=15622128 CPUtime=1116.86 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) S 5550 5553 5268 0 -1 512 3657414 0 0 0 111133 553 0 0 20 0 1 0 409950258 16253128704 3905532 33554432000 4194304 6210612 140729851382976 140729851368984 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3968049 3905532 845 493 0 3961354 0
Current children cumulated CPU time: 1116.86 s
Current children cumulated vsize: 15872196 KiB
Current children cumulated memory: 15622128 KiB

[startup+1118.9 s]
/proc/loadavg: 1.93 2.02 1.96 4/204 5898
/proc/meminfo: memFree=13777112/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15872196 memory=15622128 CPUtime=1116.89 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) S 5550 5553 5268 0 -1 512 3657414 0 0 0 111136 553 0 0 20 0 1 0 409950258 16253128704 3905532 33554432000 4194304 6210612 140729851382976 140729851368984 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3968049 3905532 845 493 0 3961354 0
Current children cumulated CPU time: 1116.89 s
Current children cumulated vsize: 15872196 KiB
Current children cumulated memory: 15622128 KiB

[startup+1119 s]
/proc/loadavg: 1.93 2.02 1.96 4/204 5898
/proc/meminfo: memFree=13777112/32770624 swapFree=66916316/67108860
[pid=5553] ppid=5550 vsize=15872196 memory=15622128 CPUtime=1116.91 cores=0,2,4,6
/proc/5553/stat : 5553 (cosoco) S 5550 5553 5268 0 -1 512 3657414 0 0 0 111137 554 0 0 20 0 1 0 409950258 16253128704 3905532 33554432000 4194304 6210612 140729851382976 140729851368984 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 19050496 140729851388939 140729851389041 140729851389041 140729851391950 0
/proc/5553/statm: 3968049 3905532 845 493 0 3961354 0
Current children cumulated CPU time: 1116.91 s
Current children cumulated vsize: 15872196 KiB
Current children cumulated memory: 15622128 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1119.82
CPU time (s): 1117.74
CPU user time (s): 1111.37
CPU system time (s): 6.36357
CPU usage (%): 99.814
Max. virtual memory (cumulated for all children) (KiB): 15872196
Max. memory (cumulated for all children) (KiB): 15622128

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1111.37
system time used= 6.36357
maximum resident set size= 15622144
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3657414
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 129997
involuntary context switches= 2261


# summary of solver processes directly reported to runsolver:
#   pid: 5553
#   total CPU time (s): 1117.74
#   total CPU user time (s): 1111.37
#   total CPU system time (s): 6.36357

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 4.13146 second user time and 7.80059 second system time

The end

Launcher Data 

Begin job on node134 at 2019-09-13 03:09:16
IDJOB=4390424
IDBENCH=122984
IDSOLVER=2831
FILE ID=node134/4390424-1568336997
RUNJOBID= node134-1568335411-5299
PBS_JOBID= 21701162
Free space on /tmp= 55684 MiB

SOLVER NAME= cosoco 2
BENCH NAME= XCSP17/Opd/Opd-sum-small/Opd-19-020-009.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4390424-1568336997/watcher-4390424-1568336997 -o /tmp/evaluation-result-4390424-1568336997/solver-4390424-1568336997 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node134-1568335411-5299 --watchdog 2560  cosoco -lc -model -restarts -stick HOME/instance-4390424-1568336997.xml

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 62ee24bcba75d1d33fbbd4bb45a854a0
RANDOM SEED=968648875

node134.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        17501264 kB
MemAvailable:   19778596 kB
Buffers:          151628 kB
Cached:          2381152 kB
SwapCached:        90592 kB
Active:         12359744 kB
Inactive:        2248260 kB
Active(anon):   12008752 kB
Inactive(anon):   105884 kB
Active(file):     350992 kB
Inactive(file):  2142376 kB
Unevictable:      110920 kB
Mlocked:          110920 kB
SwapTotal:      67108860 kB
SwapFree:       66915836 kB
Dirty:            618664 kB
Writeback:         81588 kB
AnonPages:      12097016 kB
Mapped:            45456 kB
Shmem:             30796 kB
Slab:             229332 kB
SReclaimable:     199596 kB
SUnreclaim:        29736 kB
KernelStack:        3648 kB
PageTables:        29320 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:   12534288 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2623488 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      118144 kB
DirectMap2M:    33427456 kB

Free space on /tmp at the end= 55608 MiB
End job on node134 at 2019-09-13 03:27:56