Trace number 4390253

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2? (MO) 1531.27 1533.45

General information on the benchmark

NameOpd/Opd-sum-large/
Opd-15-035-010.xml
MD5SUMb4ffb396ba331e6a6f6e6b1bc4ae482f
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark3
Best CPU time to get the best result obtained on this benchmark6.16104
Satisfiable
(Un)Satisfiability was proved
Number of variables526
Number of constraints121
Number of domains2
Minimum domain size2
Maximum domain size33
Distribution of domain sizes[{"size":2,"count":525},{"size":33,"count":1}]
Minimum variable degree16
Maximum variable degree106
Distribution of variable degrees[{"degree":16,"count":525},{"degree":106,"count":1}]
Minimum constraint arity35
Maximum constraint arity525
Distribution of constraint arities[{"arity":35,"count":15},{"arity":71,"count":105},{"arity":525,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":120}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (9915 MiB free)
  memory of node 1: 16384 MiB (13303 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4390253-1568335273/watcher-4390253-1568335273 -o /tmp/evaluation-result-4390253-1568335273/solver-4390253-1568335273 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node103-1568329709-13075 --watchdog 2560 cosoco -lc -model -restarts -stick HOME/instance-4390253-1568335273.xml 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2500 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
Current StackSize limit: 8192 KiB

solver pid=14166, runsolver pid=14163

[startup+0.100198 s]*
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23774008/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=28680 memory=5008 CPUtime=0.1 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 1463 0 0 0 10 0 0 0 20 0 1 0 409780405 29368320 1252 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 7170 1252 844 493 0 475 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 28680 KiB
Current children cumulated memory: 5008 KiB

[startup+0.209479 s]*
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23772892/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=29768 memory=6064 CPUtime=0.21 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 1743 0 0 0 21 0 0 0 20 0 1 0 409780405 30482432 1516 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 7442 1516 845 493 0 747 0
Current children cumulated CPU time: 0.21 s
Current children cumulated vsize: 29768 KiB
Current children cumulated memory: 6064 KiB

[startup+0.309126 s]*
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23772440/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=30864 memory=7120 CPUtime=0.31 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 1998 0 0 0 31 0 0 0 20 0 1 0 409780405 31604736 1780 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 7716 1780 845 493 0 1021 0
Current children cumulated CPU time: 0.31 s
Current children cumulated vsize: 30864 KiB
Current children cumulated memory: 7120 KiB

[startup+0.700338 s]
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23770208/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=34768 memory=11068 CPUtime=0.69 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 2979 0 0 0 69 0 0 0 20 0 1 0 409780405 35602432 2767 33554432000 4194304 6210612 140727574464608 140727574444856 5013016 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 8692 2767 845 493 0 1997 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 34768 KiB
Current children cumulated memory: 11068 KiB

[startup+1.50034 s]
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23765256/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=43164 memory=19396 CPUtime=1.49 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 5032 0 0 0 148 1 0 0 20 0 1 0 409780405 44199936 4849 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 10791 4849 845 493 0 4096 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 43164 KiB
Current children cumulated memory: 19396 KiB

[startup+3.10944 s]
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23745216/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=59940 memory=35800 CPUtime=3.1 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 9212 0 0 0 309 1 0 0 20 0 1 0 409780405 61378560 8950 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 14985 8950 845 493 0 8290 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 59940 KiB
Current children cumulated memory: 35800 KiB

[startup+6.30033 s]
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23714340/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=93624 memory=68680 CPUtime=6.29 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 17452 0 0 0 626 3 0 0 20 0 1 0 409780405 95870976 17170 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 23406 17170 845 493 0 16711 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 93624 KiB
Current children cumulated memory: 68680 KiB

[startup+12.7003 s]
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23652588/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=158484 memory=134180 CPUtime=12.69 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 33931 0 0 0 1264 5 0 0 20 0 1 0 409780405 162287616 33545 33554432000 4194304 6210612 140727574464608 140727574444856 140581443386816 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 39621 33545 845 493 0 32926 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 158484 KiB
Current children cumulated memory: 134180 KiB

[startup+25.5003 s]
/proc/loadavg: 2.01 2.07 2.05 3/200 14167
/proc/meminfo: memFree=23516988/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=291992 memory=266684 CPUtime=25.48 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 66121 0 0 0 2537 11 0 0 20 0 1 0 409780405 298999808 66671 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 72998 66671 845 493 0 66303 0
Current children cumulated CPU time: 25.48 s
Current children cumulated vsize: 291992 KiB
Current children cumulated memory: 266684 KiB

[startup+51.1095 s]
/proc/loadavg: 2.00 2.06 2.05 3/200 14170
/proc/meminfo: memFree=23250872/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=561152 memory=529140 CPUtime=51.09 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 130264 0 0 0 5090 19 0 0 20 0 1 0 409780405 574619648 132285 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 140288 132285 845 493 0 133593 0
Current children cumulated CPU time: 51.09 s
Current children cumulated vsize: 561152 KiB
Current children cumulated memory: 529140 KiB

[startup+102.3 s]
/proc/loadavg: 2.00 2.05 2.05 3/200 14174
/proc/meminfo: memFree=22724864/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=1079576 memory=1055688 CPUtime=102.26 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 256328 0 0 0 10188 38 0 0 20 0 1 0 409780405 1105485824 263922 33554432000 4194304 6210612 140727574464608 140727574444856 5455493 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 269894 263922 845 493 0 263199 0
Current children cumulated CPU time: 102.26 s
Current children cumulated vsize: 1079576 KiB
Current children cumulated memory: 1055688 KiB

[startup+162.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/200 14178
/proc/meminfo: memFree=22107672/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=1729540 memory=1671060 CPUtime=162.25 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 404012 0 0 0 16165 60 0 0 20 0 1 0 409780405 1771048960 417765 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 432385 417765 845 493 0 425690 0
Current children cumulated CPU time: 162.25 s
Current children cumulated vsize: 1729540 KiB
Current children cumulated memory: 1671060 KiB

[startup+222.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/200 14184
/proc/meminfo: memFree=21489472/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=2317792 memory=2288032 CPUtime=222.22 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 551077 0 0 0 22137 85 0 0 20 0 1 0 409780405 2373419008 572008 33554432000 4194304 6210612 140727574464608 140727574444856 5451368 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 579448 572008 845 493 0 572753 0
Current children cumulated CPU time: 222.22 s
Current children cumulated vsize: 2317792 KiB
Current children cumulated memory: 2288032 KiB

[startup+282.3 s]
/proc/loadavg: 2.00 2.03 2.05 3/200 14188
/proc/meminfo: memFree=20873400/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=2961632 memory=2903128 CPUtime=282.22 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 698724 0 0 0 28113 109 0 0 20 0 1 0 409780405 3032711168 725782 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 740408 725782 845 493 0 733713 0
Current children cumulated CPU time: 282.22 s
Current children cumulated vsize: 2961632 KiB
Current children cumulated memory: 2903128 KiB

[startup+342.3 s]
/proc/loadavg: 2.00 2.02 2.05 3/199 14192
/proc/meminfo: memFree=20253852/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=3550648 memory=3520668 CPUtime=342.19 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 845979 0 0 0 34091 128 0 0 20 0 1 0 409780405 3635863552 880167 33554432000 4194304 6210612 140727574464608 140727574444856 5440930 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 887662 880167 845 493 0 880967 0
Current children cumulated CPU time: 342.19 s
Current children cumulated vsize: 3550648 KiB
Current children cumulated memory: 3520668 KiB

[startup+402.3 s]
/proc/loadavg: 2.00 2.02 2.05 3/199 14198
/proc/meminfo: memFree=19636572/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=4221884 memory=4136860 CPUtime=402.18 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 993456 0 0 0 40068 150 0 0 20 0 1 0 409780405 4323209216 1034215 33554432000 4194304 6210612 140727574464608 140727574444856 140581443386788 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 1055471 1034215 845 493 0 1048776 0
Current children cumulated CPU time: 402.18 s
Current children cumulated vsize: 4221884 KiB
Current children cumulated memory: 4136860 KiB

[startup+462.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/199 14202
/proc/meminfo: memFree=19021068/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=4810076 memory=4753744 CPUtime=462.16 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 1140578 0 0 0 46044 172 0 0 20 0 1 0 409780405 4925517824 1188436 33554432000 4194304 6210612 140727574464608 140727574444856 5454016 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 1202519 1188436 845 493 0 1195824 0
Current children cumulated CPU time: 462.16 s
Current children cumulated vsize: 4810076 KiB
Current children cumulated memory: 4753744 KiB


################
# More data... #
################

Current children cumulated vsize: 10420380 KiB
Current children cumulated memory: 10301204 KiB

[startup+1062.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/199 14257
/proc/meminfo: memFree=12845576/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=11007384 memory=10915240 CPUtime=1061.98 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 2614374 0 0 0 105778 420 0 0 20 0 1 0 409780405 11271561216 2728810 33554432000 4194304 6210612 140727574464608 140727574444856 140581443384154 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 2751846 2728810 845 493 0 2745151 0
Current children cumulated CPU time: 1061.98 s
Current children cumulated vsize: 11007384 KiB
Current children cumulated memory: 10915240 KiB

[startup+1122.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/199 14264
/proc/meminfo: memFree=12228280/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=11596416 memory=11531580 CPUtime=1121.97 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 2761402 0 0 0 111745 452 0 0 20 0 1 0 409780405 11874729984 2882895 33554432000 4194304 6210612 140727574464608 140727574444856 5451458 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 2899104 2882895 845 493 0 2892409 0
Current children cumulated CPU time: 1121.97 s
Current children cumulated vsize: 11596416 KiB
Current children cumulated memory: 11531580 KiB

[startup+1182.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/199 14268
/proc/meminfo: memFree=11613108/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=12183024 memory=12145352 CPUtime=1181.95 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 2908356 0 0 0 117710 485 0 0 20 0 1 0 409780405 12475416576 3036338 33554432000 4194304 6210612 140727574464608 140727574444856 5453752 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3045756 3036338 845 493 0 3039061 0
Current children cumulated CPU time: 1181.95 s
Current children cumulated vsize: 12183024 KiB
Current children cumulated memory: 12145352 KiB

[startup+1242.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/201 14281
/proc/meminfo: memFree=10997604/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=13050836 memory=12759716 CPUtime=1241.93 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 3055904 0 0 0 123678 515 0 0 20 0 1 0 409780405 13364056064 3189929 33554432000 4194304 6210612 140727574464608 140727574444856 5452063 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3262709 3189929 845 493 0 3256014 0
Current children cumulated CPU time: 1241.93 s
Current children cumulated vsize: 13050836 KiB
Current children cumulated memory: 12759716 KiB

[startup+1302.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/202 14309
/proc/meminfo: memFree=10380284/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=13637840 memory=13375532 CPUtime=1301.91 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 3202823 0 0 0 129638 553 0 0 20 0 1 0 409780405 13965148160 3343883 33554432000 4194304 6210612 140727574464608 140727574444856 5451322 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3409460 3343883 845 493 0 3402765 0
Current children cumulated CPU time: 1301.91 s
Current children cumulated vsize: 13637840 KiB
Current children cumulated memory: 13375532 KiB

[startup+1362.3 s]
/proc/loadavg: 2.29 2.08 2.07 3/202 14313
/proc/meminfo: memFree=9763592/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=14224844 memory=13991348 CPUtime=1361.9 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 3349735 0 0 0 135602 588 0 0 20 0 1 0 409780405 14566240256 3497837 33554432000 4194304 6210612 140727574464608 140727574444856 5451993 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3556211 3497837 845 493 0 3549516 0
Current children cumulated CPU time: 1361.9 s
Current children cumulated vsize: 14224844 KiB
Current children cumulated memory: 13991348 KiB

[startup+1422.3 s]
/proc/loadavg: 2.35 2.15 2.09 3/202 14317
/proc/meminfo: memFree=9148284/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=14812244 memory=14605640 CPUtime=1421.88 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 3496775 0 0 0 141565 623 0 0 20 0 1 0 409780405 15167737856 3651410 33554432000 4194304 6210612 140727574464608 140727574444856 5442920 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3703061 3651410 845 493 0 3696366 0
Current children cumulated CPU time: 1421.88 s
Current children cumulated vsize: 14812244 KiB
Current children cumulated memory: 14605640 KiB

[startup+1482.3 s]
/proc/loadavg: 2.13 2.12 2.09 3/202 14323
/proc/meminfo: memFree=8531196/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=15399248 memory=15221612 CPUtime=1481.86 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 3643756 0 0 0 147530 656 0 0 20 0 1 0 409780405 15768829952 3805403 33554432000 4194304 6210612 140727574464608 140727574444856 140581443386829 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3849812 3805403 845 493 0 3843117 0
Current children cumulated CPU time: 1481.86 s
Current children cumulated vsize: 15399248 KiB
Current children cumulated memory: 15221612 KiB



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1530.7 s]
/proc/loadavg: 2.06 2.11 2.08 3/200 14326
/proc/meminfo: memFree=8040020/32770624 swapFree=66997112/67108860
[pid=14166] ppid=14163 vsize=15872600 memory=15717524 CPUtime=1530.24 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 4202496 3762252 0 0 0 152334 690 0 0 20 0 1 0 409780405 16253542400 3929381 33554432000 4194304 6210612 140727574464608 140727574444856 140581442460645 0 0 4096 8405026 0 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3968150 3929381 845 493 0 3961455 0
Current children cumulated CPU time: 1530.24 s
Current children cumulated vsize: 15872600 KiB
Current children cumulated memory: 15717524 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+1533.45 s]
# the end of solver process 14166 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=1523.59 s, system=7.67448 s

Solver just ended.
Dumping a history of the last processes samples

[startup+1531.2 s]
/proc/loadavg: 2.05 2.10 2.08 4/201 14327
/proc/meminfo: memFree=7914948/32770624 swapFree=66997116/67108860
[pid=14166] ppid=14163 vsize=15872600 memory=15717524 CPUtime=1530.3 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) S 14163 14166 13044 0 -1 512 3762256 0 0 0 152339 691 0 0 20 0 1 0 409780405 16253542400 3929381 33554432000 4194304 6210612 140727574464608 140727574444856 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3968150 3929381 845 493 0 3961455 0
Current children cumulated CPU time: 1530.3 s
Current children cumulated vsize: 15872600 KiB
Current children cumulated memory: 15717524 KiB

[startup+1532 s]*
/proc/loadavg: 2.05 2.10 2.08 3/201 14328
/proc/meminfo: memFree=7406904/32770624 swapFree=66997116/67108860
[pid=14166] ppid=14163 vsize=15872600 memory=15717524 CPUtime=1530.42 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) S 14163 14166 13044 0 -1 512 3762256 0 0 0 152350 692 0 0 20 0 1 0 409780405 16253542400 3929381 33554432000 4194304 6210612 140727574464608 140727574444856 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3968150 3929381 845 493 0 3961455 0
Current children cumulated CPU time: 1530.42 s
Current children cumulated vsize: 15872600 KiB
Current children cumulated memory: 15717524 KiB

[startup+1532.4 s]
/proc/loadavg: 2.05 2.10 2.08 3/201 14328
/proc/meminfo: memFree=7406904/32770624 swapFree=66997116/67108860
[pid=14166] ppid=14163 vsize=15872600 memory=15717524 CPUtime=1530.47 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) S 14163 14166 13044 0 -1 512 3762256 0 0 0 152355 692 0 0 20 0 1 0 409780405 16253542400 3929381 33554432000 4194304 6210612 140727574464608 140727574444856 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3968150 3929381 845 493 0 3961455 0
Current children cumulated CPU time: 1530.47 s
Current children cumulated vsize: 15872600 KiB
Current children cumulated memory: 15717524 KiB

[startup+1532.6 s]
/proc/loadavg: 2.05 2.10 2.08 3/201 14328
/proc/meminfo: memFree=7406904/32770624 swapFree=66997116/67108860
[pid=14166] ppid=14163 vsize=15872600 memory=15717524 CPUtime=1530.5 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) R 14163 14166 13044 0 -1 512 3762256 0 0 0 152358 692 0 0 20 0 1 0 409780405 16253542400 3929381 33554432000 4194304 6210612 140727574464608 140727574444856 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3968150 3929381 845 493 0 3961455 0
Current children cumulated CPU time: 1530.5 s
Current children cumulated vsize: 15872600 KiB
Current children cumulated memory: 15717524 KiB

[startup+1532.7 s]
/proc/loadavg: 2.05 2.10 2.08 3/201 14328
/proc/meminfo: memFree=7406904/32770624 swapFree=66997116/67108860
[pid=14166] ppid=14163 vsize=15872600 memory=15717524 CPUtime=1530.51 cores=0,2,4,6
/proc/14166/stat : 14166 (cosoco) S 14163 14166 13044 0 -1 512 3762256 0 0 0 152359 692 0 0 20 0 1 0 409780405 16253542400 3929381 33554432000 4194304 6210612 140727574464608 140727574444856 5188952 0 16384 4096 8405026 18446744071580971376 0 0 17 4 0 0 0 0 0 8308220 8313664 37593088 140727574471691 140727574471793 140727574471793 140727574474702 0
/proc/14166/statm: 3968150 3929381 845 493 0 3961455 0
Current children cumulated CPU time: 1530.51 s
Current children cumulated vsize: 15872600 KiB
Current children cumulated memory: 15717524 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1533.45
CPU time (s): 1531.27
CPU user time (s): 1523.59
CPU system time (s): 7.67448
CPU usage (%): 99.8578
Max. virtual memory (cumulated for all children) (KiB): 15872600
Max. memory (cumulated for all children) (KiB): 15717524

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1523.59
system time used= 7.67448
maximum resident set size= 15717704
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3762256
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 119405
involuntary context switches= 1175


# summary of solver processes directly reported to runsolver:
#   pid: 14166
#   total CPU time (s): 1531.27
#   total CPU user time (s): 1523.59
#   total CPU system time (s): 7.67448

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 5.69207 second user time and 10.3705 second system time

The end

Launcher Data 

Begin job on node103 at 2019-09-13 02:40:14
IDJOB=4390253
IDBENCH=122936
IDSOLVER=2831
FILE ID=node103/4390253-1568335273
RUNJOBID= node103-1568329709-13075
PBS_JOBID= 21701158
Free space on /tmp= 57264 MiB

SOLVER NAME= cosoco 2
BENCH NAME= XCSP17/Opd/Opd-sum-large/Opd-15-035-010.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4390253-1568335273/watcher-4390253-1568335273 -o /tmp/evaluation-result-4390253-1568335273/solver-4390253-1568335273 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node103-1568329709-13075 --watchdog 2560  cosoco -lc -model -restarts -stick HOME/instance-4390253-1568335273.xml

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b4ffb396ba331e6a6f6e6b1bc4ae482f
RANDOM SEED=176197299

node103.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        23776296 kB
MemAvailable:   31828868 kB
Buffers:          151276 kB
Cached:          8121160 kB
SwapCached:         8920 kB
Active:          1724492 kB
Inactive:        6584560 kB
Active(anon):      24948 kB
Inactive(anon):    53036 kB
Active(file):    1699544 kB
Inactive(file):  6531524 kB
Unevictable:      107456 kB
Mlocked:          107456 kB
SwapTotal:      67108860 kB
SwapFree:       66997112 kB
Dirty:              2692 kB
Writeback:             0 kB
AnonPages:        263280 kB
Mapped:            45236 kB
Shmem:             32812 kB
Slab:             280300 kB
SReclaimable:     249964 kB
SUnreclaim:        30336 kB
KernelStack:        3776 kB
PageTables:         5564 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     547984 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    217088 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      105856 kB
DirectMap2M:    33439744 kB

Free space on /tmp at the end= 56800 MiB
End job on node103 at 2019-09-13 03:05:48