Trace number 4398213

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
cosoco 2.O parallel? (TO) 2520.6 2522.86

General information on the benchmark

NameBibd/Bibd-sum-open/
Bibd-sum-85-085-36-36-15.xml
MD5SUMe7b94b237cf4deebd8c7bf96e74cfc29
Bench CategoryCSP (decision problem)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables7225
Number of constraints3741
Number of domains1
Minimum domain size2
Maximum domain size2
Distribution of domain sizes[{"size":2,"count":7225}]
Minimum variable degree87
Maximum variable degree87
Distribution of variable degrees[{"degree":87,"count":7225}]
Minimum constraint arity85
Maximum constraint arity7225
Distribution of constraint arities[{"arity":85,"count":170},{"arity":170,"count":3570},{"arity":7225,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":3740}]
Optimization problemNO
Type of objective

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (15281 MiB free)
  memory of node 1: 16384 MiB (15819 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4398213-1568601268/watcher-4398213-1568601268 -o /tmp/evaluation-result-4398213-1568601268/solver-4398213-1568601268 -C 20160 -W 2520 -M 31000 --bin-var /tmp/runsolver-exch-node121-1568598281-5777 --watchdog 2580 cosoco -model -nbcores=8 HOME/instance-4398213-1568601268.xml 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 20160 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 20190 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=6264, runsolver pid=6261
Current StackSize limit: 8192 KiB


[startup+0.100159 s]*
/proc/loadavg: 8.45 8.58 8.30 2/192 6265
/proc/meminfo: memFree=31805100/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=74392 memory=50556 CPUtime=0.09 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 12854 0 0 0 8 1 0 0 20 0 1 0 436389493 76177408 12639 33554432000 4194304 6211092 140736800559552 140736800556088 139808383416224 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 18598 12639 836 493 0 11903 0
Current children cumulated CPU time: 0.09 s
Current children cumulated vsize: 74392 KiB
Current children cumulated memory: 50556 KiB

[startup+0.208879 s]*
/proc/loadavg: 8.45 8.58 8.30 2/192 6265
/proc/meminfo: memFree=31773232/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=101892 memory=78188 CPUtime=0.2 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 19730 0 0 0 18 2 0 0 20 0 1 0 436389493 104337408 19547 33554432000 4194304 6211092 140736800559552 140736800556088 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 25473 19547 842 493 0 18778 0
Current children cumulated CPU time: 0.2 s
Current children cumulated vsize: 101892 KiB
Current children cumulated memory: 78188 KiB

[startup+0.308694 s]*
/proc/loadavg: 8.45 8.58 8.30 2/192 6265
/proc/meminfo: memFree=31772240/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=102684 memory=78980 CPUtime=0.3 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 19921 0 0 0 28 2 0 0 20 0 1 0 436389493 105148416 19745 33554432000 4194304 6211092 140736800559552 140736800556088 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 25671 19745 842 493 0 18976 0
Current children cumulated CPU time: 0.3 s
Current children cumulated vsize: 102684 KiB
Current children cumulated memory: 78980 KiB

[startup+0.700328 s]
/proc/loadavg: 8.45 8.58 8.30 2/192 6265
/proc/meminfo: memFree=31770752/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=105612 memory=81884 CPUtime=0.69 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 20653 0 0 0 67 2 0 0 20 0 1 0 436389493 108146688 20471 33554432000 4194304 6211092 140736800559552 140736800556088 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 26403 20471 842 493 0 19708 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 105612 KiB
Current children cumulated memory: 81884 KiB

[startup+1.50033 s]
/proc/loadavg: 8.45 8.58 8.30 2/192 6265
/proc/meminfo: memFree=31767476/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=111676 memory=87916 CPUtime=1.49 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 22175 0 0 0 146 3 0 0 20 0 1 0 436389493 114356224 21979 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 27919 21979 842 493 0 21224 0
Current children cumulated CPU time: 1.49 s
Current children cumulated vsize: 111676 KiB
Current children cumulated memory: 87916 KiB

[startup+3.10859 s]
/proc/loadavg: 7.85 8.45 8.26 2/192 6265
/proc/meminfo: memFree=31752676/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=124004 memory=100140 CPUtime=3.1 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 25494 0 0 0 307 3 0 0 20 0 1 0 436389493 126980096 25035 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 31001 25035 842 493 0 24306 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 124004 KiB
Current children cumulated memory: 100140 KiB

[startup+6.30032 s]
/proc/loadavg: 7.85 8.45 8.26 2/192 6265
/proc/meminfo: memFree=31729340/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=148476 memory=124484 CPUtime=6.29 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 31825 0 0 0 625 4 0 0 20 0 1 0 436389493 152039424 31121 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 37119 31121 842 493 0 30424 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 148476 KiB
Current children cumulated memory: 124484 KiB

[startup+12.7003 s]
/proc/loadavg: 7.31 8.33 8.22 2/192 6265
/proc/meminfo: memFree=31683996/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=197728 memory=173216 CPUtime=12.69 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 44263 0 0 0 1263 6 0 0 20 0 1 0 436389493 202473472 43304 33554432000 4194304 6211092 140736800559552 140736800520792 139808384292414 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 49432 43304 842 493 0 42737 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 197728 KiB
Current children cumulated memory: 173216 KiB

[startup+25.5003 s]
/proc/loadavg: 5.91 7.97 8.10 2/192 6266
/proc/meminfo: memFree=31585732/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=294604 memory=270636 CPUtime=25.48 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 69139 0 0 0 2539 9 0 0 20 0 1 0 436389493 301674496 67659 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 73651 67659 842 493 0 66956 0
Current children cumulated CPU time: 25.48 s
Current children cumulated vsize: 294604 KiB
Current children cumulated memory: 270636 KiB

[startup+51.1086 s]
/proc/loadavg: 4.23 7.41 7.91 2/192 6268
/proc/meminfo: memFree=31385200/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=490876 memory=466856 CPUtime=51.09 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 117854 0 0 0 5093 16 0 0 20 0 1 0 436389493 502657024 116714 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 122719 116714 842 493 0 116024 0
Current children cumulated CPU time: 51.09 s
Current children cumulated vsize: 490876 KiB
Current children cumulated memory: 466856 KiB

[startup+102.3 s]
/proc/loadavg: 2.40 6.42 7.55 2/191 6271
/proc/meminfo: memFree=30996832/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=884772 memory=856348 CPUtime=102.26 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 214297 0 0 0 10198 28 0 0 20 0 1 0 436389493 906006528 214087 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 221193 214087 842 493 0 214498 0
Current children cumulated CPU time: 102.26 s
Current children cumulated vsize: 884772 KiB
Current children cumulated memory: 856348 KiB

[startup+162.3 s]
/proc/loadavg: 1.52 5.44 7.14 2/191 6274
/proc/meminfo: memFree=30538248/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=1344376 memory=1315172 CPUtime=162.24 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 326963 0 0 0 16181 43 0 0 20 0 1 0 436389493 1376641024 328793 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 336094 328793 842 493 0 329399 0
Current children cumulated CPU time: 162.24 s
Current children cumulated vsize: 1344376 KiB
Current children cumulated memory: 1315172 KiB

[startup+222.3 s]
/proc/loadavg: 1.19 4.63 6.76 2/191 6279
/proc/meminfo: memFree=30081684/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=1809052 memory=1771632 CPUtime=222.23 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 439580 0 0 0 22165 58 0 0 20 0 1 0 436389493 1852469248 442908 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 452263 442908 842 493 0 445568 0
Current children cumulated CPU time: 222.23 s
Current children cumulated vsize: 1809052 KiB
Current children cumulated memory: 1771632 KiB

[startup+282.3 s]
/proc/loadavg: 1.07 3.97 6.40 2/191 6282
/proc/meminfo: memFree=29622616/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=2257324 memory=2230124 CPUtime=282.22 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 551633 0 0 0 28150 72 0 0 20 0 1 0 436389493 2311499776 557531 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 564331 557531 842 493 0 557636 0
Current children cumulated CPU time: 282.22 s
Current children cumulated vsize: 2257324 KiB
Current children cumulated memory: 2230124 KiB

[startup+342.3 s]
/proc/loadavg: 1.03 3.43 6.06 2/191 6285
/proc/meminfo: memFree=29162748/32770624 swapFree=66985140/67108860
[pid=6264] ppid=6261 vsize=2729804 memory=2688416 CPUtime=342.21 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 664138 0 0 0 34134 87 0 0 20 0 1 0 436389493 2795319296 672104 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 682451 672104 842 493 0 675756 0
Current children cumulated CPU time: 342.21 s
Current children cumulated vsize: 2729804 KiB
Current children cumulated memory: 2688416 KiB

[startup+402.3 s]
/proc/loadavg: 1.01 2.99 5.74 2/191 6297
/proc/meminfo: memFree=28705096/32770624 swapFree=66985152/67108860
[pid=6264] ppid=6261 vsize=3178472 memory=3145176 CPUtime=402.19 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 776432 0 0 0 40117 102 0 0 20 0 1 0 436389493 3254755328 786294 33554432000 4194304 6211092 140736800559552 140736800520792 5451482 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 794618 786294 842 493 0 787923 0
Current children cumulated CPU time: 402.19 s
Current children cumulated vsize: 3178472 KiB
Current children cumulated memory: 3145176 KiB

[startup+462.3 s]
/proc/loadavg: 1.00 2.63 5.45 2/191 6300
/proc/meminfo: memFree=28247724/32770624 swapFree=66985152/67108860
[pid=6264] ppid=6261 vsize=3626216 memory=3602148 CPUtime=462.18 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 888785 0 0 0 46101 117 0 0 20 0 1 0 436389493 3713245184 900537 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 906554 900537 842 493 0 899859 0
Current children cumulated CPU time: 462.18 s
Current children cumulated vsize: 3626216 KiB
Current children cumulated memory: 3602148 KiB


################
# More data... #
################

Current children cumulated vsize: 14665888 KiB
Current children cumulated memory: 14571576 KiB

[startup+1962.3 s]
/proc/loadavg: 1.01 1.08 1.93 2/191 6431
/proc/meminfo: memFree=16797348/32770624 swapFree=66985176/67108860
[pid=6264] ppid=6261 vsize=15112576 memory=15028656 CPUtime=1961.88 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 3701068 0 0 0 195550 638 0 0 20 0 1 0 436389493 15475277824 3757164 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 6 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 3778144 3757164 842 493 0 3771449 0
Current children cumulated CPU time: 1961.88 s
Current children cumulated vsize: 15112576 KiB
Current children cumulated memory: 15028656 KiB

[startup+2022.3 s]
/proc/loadavg: 1.00 1.07 1.87 2/191 6436
/proc/meminfo: memFree=16342968/32770624 swapFree=66985176/67108860
[pid=6264] ppid=6261 vsize=15559396 memory=15483832 CPUtime=2021.86 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 3813518 0 0 0 201524 662 0 0 20 0 1 0 436389493 15932821504 3870958 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 3889849 3870958 842 493 0 3883154 0
Current children cumulated CPU time: 2021.86 s
Current children cumulated vsize: 15559396 KiB
Current children cumulated memory: 15483832 KiB

[startup+2082.3 s]
/proc/loadavg: 1.05 1.07 1.82 2/191 6439
/proc/meminfo: memFree=15884388/32770624 swapFree=66985176/67108860
[pid=6264] ppid=6261 vsize=16005688 memory=15940168 CPUtime=2081.84 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 3925678 0 0 0 207491 693 0 0 20 0 1 0 436389493 16389824512 3985042 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4001422 3985042 842 493 0 3994727 0
Current children cumulated CPU time: 2081.84 s
Current children cumulated vsize: 16005688 KiB
Current children cumulated memory: 15940168 KiB

[startup+2142.3 s]
/proc/loadavg: 1.02 1.06 1.77 2/191 6442
/proc/meminfo: memFree=15429948/32770624 swapFree=66985176/67108860
[pid=6264] ppid=6261 vsize=16451716 memory=16394460 CPUtime=2141.83 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4037801 0 0 0 213455 728 0 0 20 0 1 0 436389493 16846557184 4098615 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4112929 4098615 842 493 0 4106234 0
Current children cumulated CPU time: 2141.83 s
Current children cumulated vsize: 16451716 KiB
Current children cumulated memory: 16394460 KiB

[startup+2202.3 s]
/proc/loadavg: 1.01 1.05 1.72 2/192 6455
/proc/meminfo: memFree=14971200/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=16898140 memory=16851012 CPUtime=2201.82 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4150014 0 0 0 219422 760 0 0 20 0 1 0 436389493 17303695360 4212753 33554432000 4194304 6211092 140736800559552 140736800520792 5451552 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4224535 4212753 842 493 0 4217840 0
Current children cumulated CPU time: 2201.82 s
Current children cumulated vsize: 16898140 KiB
Current children cumulated memory: 16851012 KiB

[startup+2262.3 s]
/proc/loadavg: 1.00 1.04 1.68 2/192 6458
/proc/meminfo: memFree=14513480/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=17344696 memory=17307968 CPUtime=2261.8 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4262403 0 0 0 225386 794 0 0 20 0 1 0 436389493 17760968704 4326992 33554432000 4194304 6211092 140736800559552 140736800520792 5478709 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4336174 4326992 842 493 0 4329479 0
Current children cumulated CPU time: 2261.8 s
Current children cumulated vsize: 17344696 KiB
Current children cumulated memory: 17307968 KiB

[startup+2322.3 s]
/proc/loadavg: 1.00 1.03 1.64 2/192 6461
/proc/meminfo: memFree=14057676/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=17790724 memory=17762040 CPUtime=2321.78 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4374617 0 0 0 231348 830 0 0 20 0 1 0 436389493 18217701376 4440510 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4447681 4440510 842 493 0 4440986 0
Current children cumulated CPU time: 2321.78 s
Current children cumulated vsize: 17790724 KiB
Current children cumulated memory: 17762040 KiB

[startup+2382.3 s]
/proc/loadavg: 1.00 1.03 1.60 2/192 6466
/proc/meminfo: memFree=13601344/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=18423476 memory=18217680 CPUtime=2381.77 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4487086 0 0 0 237312 865 0 0 20 0 1 0 436389493 18865639424 4554420 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4605869 4554420 842 493 0 4599174 0
Current children cumulated CPU time: 2381.77 s
Current children cumulated vsize: 18423476 KiB
Current children cumulated memory: 18217680 KiB

[startup+2442.3 s]
/proc/loadavg: 1.00 1.02 1.56 2/192 6469
/proc/meminfo: memFree=13144352/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=18869768 memory=18674108 CPUtime=2441.76 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4599385 0 0 0 243274 902 0 0 20 0 1 0 436389493 19322642432 4668527 33554432000 4194304 6211092 140736800559552 140736800520792 139808384294331 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4717442 4668527 842 493 0 4710747 0
Current children cumulated CPU time: 2441.76 s
Current children cumulated vsize: 18869768 KiB
Current children cumulated memory: 18674108 KiB

[startup+2502.3 s]
/proc/loadavg: 1.08 1.03 1.53 2/191 6472
/proc/meminfo: memFree=12687168/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=19315796 memory=19130436 CPUtime=2501.75 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4711486 0 0 0 249234 941 0 0 20 0 1 0 436389493 19779375104 4782609 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4828949 4782609 842 493 0 4822254 0
Current children cumulated CPU time: 2501.75 s
Current children cumulated vsize: 19315796 KiB
Current children cumulated memory: 19130436 KiB



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+2520 s]*
/proc/loadavg: 1.05 1.03 1.52 2/191 6474
/proc/meminfo: memFree=12551344/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=19447532 memory=19264312 CPUtime=2519.45 cores=0-7
/proc/6264/stat : 6264 (cosoco) R 6261 6264 5749 0 -1 4202496 4744673 0 0 0 250995 950 0 0 20 0 1 0 436389493 19914272768 4816078 33554432000 4194304 6211092 140736800559552 140736800520792 139808383368165 0 0 4096 8405026 0 0 0 17 2 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4861883 4816078 842 493 0 4855188 0
Current children cumulated CPU time: 2519.45 s
Current children cumulated vsize: 19447532 KiB
Current children cumulated memory: 19264312 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+2522.86 s]
# the end of solver process 6264 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=2510.24 s, system=10.3668 s

Solver just ended.
Dumping a history of the last processes samples

[startup+2520.8 s]
/proc/loadavg: 1.05 1.03 1.52 2/191 6474
/proc/meminfo: memFree=12551344/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=19447532 memory=19264312 CPUtime=2519.58 cores=0-7
/proc/6264/stat : 6264 (cosoco) S 6261 6264 5749 0 -1 512 4744679 0 0 0 251007 951 0 0 20 0 1 0 436389493 19914272768 4816078 33554432000 4194304 6211092 140736800559552 140736800520792 5189550 0 16384 4096 8405026 18446744071580971376 0 0 17 6 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4861883 4816078 842 493 0 4855188 0
Current children cumulated CPU time: 2519.58 s
Current children cumulated vsize: 19447532 KiB
Current children cumulated memory: 19264312 KiB

[startup+2521.6 s]
/proc/loadavg: 1.05 1.03 1.52 2/192 6475
/proc/meminfo: memFree=12034000/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=19447532 memory=19264312 CPUtime=2519.68 cores=0-7
/proc/6264/stat : 6264 (cosoco) S 6261 6264 5749 0 -1 512 4744679 0 0 0 251017 951 0 0 20 0 1 0 436389493 19914272768 4816078 33554432000 4194304 6211092 140736800559552 140736800520792 5189550 0 16384 4096 8405026 18446744071580971376 0 0 17 6 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4861883 4816078 842 493 0 4855188 0
Current children cumulated CPU time: 2519.68 s
Current children cumulated vsize: 19447532 KiB
Current children cumulated memory: 19264312 KiB

[startup+2522 s]*
/proc/loadavg: 1.05 1.03 1.52 2/192 6475
/proc/meminfo: memFree=11524484/32770624 swapFree=66985180/67108860
[pid=6264] ppid=6261 vsize=19447532 memory=19264312 CPUtime=2519.74 cores=0-7
/proc/6264/stat : 6264 (cosoco) S 6261 6264 5749 0 -1 512 4744679 0 0 0 251023 951 0 0 20 0 1 0 436389493 19914272768 4816078 33554432000 4194304 6211092 140736800559552 140736800520792 5189550 0 16384 4096 8405026 18446744071580971376 0 0 17 6 0 0 0 0 0 8312316 8317760 17002496 140736800564241 140736800564333 140736800564333 140736800567246 0
/proc/6264/statm: 4861883 4816078 842 493 0 4855188 0
Current children cumulated CPU time: 2519.74 s
Current children cumulated vsize: 19447532 KiB
Current children cumulated memory: 19264312 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 2522.86
CPU time (s): 2520.6
CPU user time (s): 2510.24
CPU system time (s): 10.3668
CPU usage (%): 99.9105
Max. virtual memory (cumulated for all children) (KiB): 19447532
Max. memory (cumulated for all children) (KiB): 19264312

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2510.24
system time used= 10.3668
maximum resident set size= 19264372
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4744679
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 128244
involuntary context switches= 2502


# summary of solver processes directly reported to runsolver:
#   pid: 6264
#   total CPU time (s): 2520.6
#   total CPU user time (s): 2510.24
#   total CPU system time (s): 10.3668

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 9.43388 second user time and 15.6828 second system time

The end

Launcher Data 

Begin job on node121 at 2019-09-16 04:33:46
IDJOB=4398213
IDBENCH=124071
IDSOLVER=2851
FILE ID=node121/4398213-1568601268
RUNJOBID= node121-1568598281-5777
PBS_JOBID= 21703423
Free space on /tmp= 61408 MiB

SOLVER NAME= cosoco 2.O parallel
BENCH NAME= XCSP17/Bibd/Bibd-sum-open/Bibd-sum-85-085-36-36-15.xml
COMMAND LINE= cosoco -model -nbcores=NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4398213-1568601268/watcher-4398213-1568601268 -o /tmp/evaluation-result-4398213-1568601268/solver-4398213-1568601268 -C 20160 -W 2520 -M 31000 --bin-var /tmp/runsolver-exch-node121-1568598281-5777 --watchdog 2580  cosoco -model -nbcores=8 HOME/instance-4398213-1568601268.xml

TIME LIMIT= 20160 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e7b94b237cf4deebd8c7bf96e74cfc29
RANDOM SEED=1357735838

node121.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        31847684 kB
MemAvailable:   31905856 kB
Buffers:           50384 kB
Cached:           313536 kB
SwapCached:        16008 kB
Active:           214384 kB
Inactive:         169100 kB
Active(anon):       4132 kB
Inactive(anon):    57100 kB
Active(file):     210252 kB
Inactive(file):   112000 kB
Unevictable:      109352 kB
Mlocked:          109352 kB
SwapTotal:      67108860 kB
SwapFree:       66985140 kB
Dirty:              2792 kB
Writeback:           292 kB
AnonPages:        113524 kB
Mapped:            40316 kB
Shmem:             33404 kB
Slab:             132452 kB
SReclaimable:     103508 kB
SUnreclaim:        28944 kB
KernelStack:        3376 kB
PageTables:         5476 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     506276 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:     67584 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      122240 kB
DirectMap2M:    33423360 kB

Free space on /tmp at the end= 60912 MiB
End job on node121 at 2019-09-16 05:15:49