Trace number 4409631

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2.0 parallel? (TO) 2520.57 2522.9

General information on the benchmark

NameOpd/Opd-sum-large/
Opd-04-350-100.xml
MD5SUM05a37308653f1418d8a9d31e27700b02
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark9
Best CPU time to get the best result obtained on this benchmark10.1781
Satisfiable
(Un)Satisfiability was proved
Number of variables1401
Number of constraints11
Number of domains2
Minimum domain size2
Maximum domain size342
Distribution of domain sizes[{"size":2,"count":1400},{"size":342,"count":1}]
Minimum variable degree5
Maximum variable degree7
Distribution of variable degrees[{"degree":5,"count":1400},{"degree":7,"count":1}]
Minimum constraint arity350
Maximum constraint arity1400
Distribution of constraint arities[{"arity":350,"count":4},{"arity":701,"count":6},{"arity":1400,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":10}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (14680 MiB free)
  memory of node 1: 16384 MiB (13065 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4409631-1569819904/watcher-4409631-1569819904 -o /tmp/evaluation-result-4409631-1569819904/solver-4409631-1569819904 -C 20160 -W 2520 -M 31000 --bin-var /tmp/runsolver-exch-node151-1569817277-6491 --watchdog 2580 cosoco -model -nbcores=8 HOME/instance-4409631-1569819904.xml 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 20160 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 20190 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2520 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=6972, runsolver pid=6969
Current StackSize limit: 8192 KiB


[startup+0.100126 s]*
/proc/loadavg: 7.91 8.23 7.86 2/194 6973
/proc/meminfo: memFree=28403368/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=34784 memory=11292 CPUtime=0.09 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 3005 0 0 0 9 0 0 0 20 0 1 0 558261675 35618816 2823 33554432000 4194304 6211132 140736187096480 140736187093016 140428583920536 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 8696 2823 846 493 0 2001 0
Current children cumulated CPU time: 0.09 s
Current children cumulated vsize: 34784 KiB
Current children cumulated memory: 11292 KiB

[startup+0.209469 s]*
/proc/loadavg: 7.91 8.23 7.86 2/194 6973
/proc/meminfo: memFree=28402624/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=35576 memory=12084 CPUtime=0.2 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 3211 0 0 0 20 0 0 0 20 0 1 0 558261675 36429824 3021 33554432000 4194304 6211132 140736187096480 140736187093016 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 8894 3021 846 493 0 2199 0
Current children cumulated CPU time: 0.2 s
Current children cumulated vsize: 35576 KiB
Current children cumulated memory: 12084 KiB

[startup+0.308718 s]*
/proc/loadavg: 7.91 8.23 7.86 2/194 6973
/proc/meminfo: memFree=28401880/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=36368 memory=12876 CPUtime=0.3 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 3399 0 0 0 30 0 0 0 20 0 1 0 558261675 37240832 3219 33554432000 4194304 6211132 140736187096480 140736187093016 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 9092 3219 846 493 0 2397 0
Current children cumulated CPU time: 0.3 s
Current children cumulated vsize: 36368 KiB
Current children cumulated memory: 12876 KiB

[startup+0.700256 s]
/proc/loadavg: 7.91 8.23 7.86 2/194 6973
/proc/meminfo: memFree=28400392/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=39272 memory=15780 CPUtime=0.69 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4123 0 0 0 69 0 0 0 20 0 1 0 558261675 40214528 3945 33554432000 4194304 6211132 140736187096480 140736187093016 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 9818 3945 846 493 0 3123 0
Current children cumulated CPU time: 0.69 s
Current children cumulated vsize: 39272 KiB
Current children cumulated memory: 15780 KiB

[startup+1.50026 s]
/proc/loadavg: 7.91 8.23 7.86 2/194 6973
/proc/meminfo: memFree=28396548/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=45212 memory=21852 CPUtime=1.48 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 5632 0 0 0 148 0 0 0 20 0 1 0 558261675 46297088 5463 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 11303 5463 846 493 0 4608 0
Current children cumulated CPU time: 1.48 s
Current children cumulated vsize: 45212 KiB
Current children cumulated memory: 21852 KiB

[startup+3.10931 s]
/proc/loadavg: 7.91 8.23 7.86 2/194 6973
/proc/meminfo: memFree=28381668/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=57420 memory=33740 CPUtime=3.1 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 9452 0 0 0 309 1 0 0 20 0 1 0 558261675 58798080 8435 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 14355 8435 846 493 0 7660 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 57420 KiB
Current children cumulated memory: 33740 KiB

[startup+6.30025 s]
/proc/loadavg: 7.36 8.11 7.82 2/193 6973
/proc/meminfo: memFree=28358560/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=81436 memory=57856 CPUtime=6.29 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 16239 0 0 0 627 2 0 0 20 0 1 0 558261675 83390464 14464 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 20359 14464 846 493 0 13664 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 81436 KiB
Current children cumulated memory: 57856 KiB

[startup+12.7003 s]
/proc/loadavg: 6.85 7.99 7.79 2/193 6973
/proc/meminfo: memFree=28313796/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=129800 memory=106156 CPUtime=12.69 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 29071 0 0 0 1265 4 0 0 20 0 1 0 558261675 132915200 26539 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 32450 26539 846 493 0 25755 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 129800 KiB
Current children cumulated memory: 106156 KiB

[startup+25.5003 s]
/proc/loadavg: 5.55 7.65 7.68 2/193 6974
/proc/meminfo: memFree=28215720/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=226180 memory=202652 CPUtime=25.49 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 54735 0 0 0 2540 9 0 0 20 0 1 0 558261675 231608320 50663 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 56545 50663 846 493 0 49850 0
Current children cumulated CPU time: 25.49 s
Current children cumulated vsize: 226180 KiB
Current children cumulated memory: 202652 KiB

[startup+51.1092 s]
/proc/loadavg: 4.00 7.11 7.50 2/192 6976
/proc/meminfo: memFree=28019008/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=419440 memory=395536 CPUtime=51.1 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 103763 0 0 0 5092 18 0 0 20 0 1 0 558261675 429506560 98884 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 104860 98884 846 493 0 98165 0
Current children cumulated CPU time: 51.1 s
Current children cumulated vsize: 419440 KiB
Current children cumulated memory: 395536 KiB

[startup+102.3 s]
/proc/loadavg: 2.30 6.17 7.16 2/191 6979
/proc/meminfo: memFree=27630156/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=810316 memory=785904 CPUtime=102.27 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 201325 0 0 0 10196 31 0 0 20 0 1 0 558261675 829763584 196476 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 202579 196476 846 493 0 195884 0
Current children cumulated CPU time: 102.27 s
Current children cumulated vsize: 810316 KiB
Current children cumulated memory: 785904 KiB

[startup+162.3 s]
/proc/loadavg: 1.48 5.23 6.77 2/191 6982
/proc/meminfo: memFree=27169732/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=1269988 memory=1245264 CPUtime=162.25 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 316184 0 0 0 16174 51 0 0 20 0 1 0 558261675 1300467712 311316 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 317497 311316 846 493 0 310802 0
Current children cumulated CPU time: 162.25 s
Current children cumulated vsize: 1269988 KiB
Current children cumulated memory: 1245264 KiB

[startup+222.3 s]
/proc/loadavg: 1.18 4.46 6.41 2/191 6985
/proc/meminfo: memFree=26709756/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=1730872 memory=1704832 CPUtime=222.23 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 431054 0 0 0 22154 69 0 0 20 0 1 0 558261675 1772412928 426208 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 432718 426208 846 493 0 426023 0
Current children cumulated CPU time: 222.23 s
Current children cumulated vsize: 1730872 KiB
Current children cumulated memory: 1704832 KiB

[startup+282.3 s]
/proc/loadavg: 1.06 3.83 6.07 2/191 6990
/proc/meminfo: memFree=26249024/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=2188252 memory=2164192 CPUtime=282.21 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 545891 0 0 0 28136 85 0 0 20 0 1 0 558261675 2240770048 541048 33554432000 4194304 6211132 140736187096480 140736186955768 5443124 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 547063 541048 846 493 0 540368 0
Current children cumulated CPU time: 282.21 s
Current children cumulated vsize: 2188252 KiB
Current children cumulated memory: 2164192 KiB

[startup+342.3 s]
/proc/loadavg: 1.02 3.31 5.76 2/191 7000
/proc/meminfo: memFree=25787740/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=2650624 memory=2624756 CPUtime=342.18 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 660536 0 0 0 34116 102 0 0 20 0 1 0 558261675 2714238976 656189 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 662656 656189 846 493 0 655961 0
Current children cumulated CPU time: 342.18 s
Current children cumulated vsize: 2650624 KiB
Current children cumulated memory: 2624756 KiB

[startup+402.3 s]
/proc/loadavg: 1.01 2.89 5.46 2/191 7003
/proc/meminfo: memFree=25327320/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=3108136 memory=3084236 CPUtime=402.17 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 774943 0 0 0 40096 121 0 0 20 0 1 0 558261675 3182731264 771059 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 777034 771059 846 493 0 770339 0
Current children cumulated CPU time: 402.17 s
Current children cumulated vsize: 3108136 KiB
Current children cumulated memory: 3084236 KiB

[startup+462.3 s]
/proc/loadavg: 1.00 2.55 5.18 2/191 7008
/proc/meminfo: memFree=24868516/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=3572808 memory=3542236 CPUtime=462.15 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 889494 0 0 0 46075 140 0 0 20 0 1 0 558261675 3658555392 885559 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 893202 885559 846 493 0 886507 0
Current children cumulated CPU time: 462.15 s
Current children cumulated vsize: 3572808 KiB
Current children cumulated memory: 3542236 KiB


################
# More data... #
################

Current children cumulated vsize: 14589332 KiB
Current children cumulated memory: 14554776 KiB

[startup+1962.3 s]
/proc/loadavg: 1.00 1.01 1.83 2/191 7117
/proc/meminfo: memFree=13375468/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=15045260 memory=15012608 CPUtime=1961.84 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 3748647 0 0 0 195384 800 0 0 20 0 1 0 558261675 15406346240 3753152 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 3761315 3753152 846 493 0 3754620 0
Current children cumulated CPU time: 1961.84 s
Current children cumulated vsize: 15045260 KiB
Current children cumulated memory: 15012608 KiB

[startup+2022.3 s]
/proc/loadavg: 1.05 1.03 1.78 2/191 7120
/proc/meminfo: memFree=12915928/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=15501320 memory=15470896 CPUtime=2021.83 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 3862787 0 0 0 201348 835 0 0 20 0 1 0 558261675 15873351680 3867724 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 3875330 3867724 846 493 0 3868635 0
Current children cumulated CPU time: 2021.83 s
Current children cumulated vsize: 15501320 KiB
Current children cumulated memory: 15470896 KiB

[startup+2082.3 s]
/proc/loadavg: 1.02 1.02 1.74 2/191 7125
/proc/meminfo: memFree=12457480/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=15957116 memory=15928728 CPUtime=2081.82 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 3976824 0 0 0 207311 871 0 0 20 0 1 0 558261675 16340086784 3982182 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 3989279 3982182 846 493 0 3982584 0
Current children cumulated CPU time: 2081.82 s
Current children cumulated vsize: 15957116 KiB
Current children cumulated memory: 15928728 KiB

[startup+2142.3 s]
/proc/loadavg: 1.01 1.02 1.69 2/191 7135
/proc/meminfo: memFree=12000744/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=16412780 memory=16384440 CPUtime=2141.81 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4090870 0 0 0 213273 908 0 0 20 0 1 0 558261675 16806686720 4096110 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4103195 4096110 846 493 0 4096500 0
Current children cumulated CPU time: 2141.81 s
Current children cumulated vsize: 16412780 KiB
Current children cumulated memory: 16384440 KiB

[startup+2202.3 s]
/proc/loadavg: 1.00 1.01 1.65 2/191 7138
/proc/meminfo: memFree=11542672/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=16868312 memory=16842016 CPUtime=2201.8 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4204866 0 0 0 219234 946 0 0 20 0 1 0 558261675 17273151488 4210504 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4217078 4210504 846 493 0 4210383 0
Current children cumulated CPU time: 2201.8 s
Current children cumulated vsize: 16868312 KiB
Current children cumulated memory: 16842016 KiB

[startup+2262.3 s]
/proc/loadavg: 1.00 1.01 1.61 2/191 7143
/proc/meminfo: memFree=11084076/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=17323844 memory=17299508 CPUtime=2261.78 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4318850 0 0 0 225193 985 0 0 20 0 1 0 558261675 17739616256 4324877 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4330961 4324877 846 493 0 4324266 0
Current children cumulated CPU time: 2261.78 s
Current children cumulated vsize: 17323844 KiB
Current children cumulated memory: 17299508 KiB

[startup+2322.3 s]
/proc/loadavg: 1.47 1.12 1.60 2/191 7146
/proc/meminfo: memFree=10626592/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=17816288 memory=17756500 CPUtime=2321.77 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4433226 0 0 0 231154 1023 0 0 20 0 1 0 558261675 18243878912 4439125 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4454072 4439125 846 493 0 4447377 0
Current children cumulated CPU time: 2321.77 s
Current children cumulated vsize: 17816288 KiB
Current children cumulated memory: 17756500 KiB

[startup+2382.3 s]
/proc/loadavg: 1.57 1.25 1.62 2/191 7149
/proc/meminfo: memFree=10167800/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=18271556 memory=18214600 CPUtime=2381.77 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4547295 0 0 0 237113 1064 0 0 20 0 1 0 558261675 18710073344 4553650 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4567889 4553650 846 493 0 4561194 0
Current children cumulated CPU time: 2381.77 s
Current children cumulated vsize: 18271556 KiB
Current children cumulated memory: 18214600 KiB

[startup+2442.3 s]
/proc/loadavg: 1.21 1.20 1.58 2/191 7154
/proc/meminfo: memFree=9709376/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=18726560 memory=18671564 CPUtime=2441.76 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4661192 0 0 0 243068 1108 0 0 20 0 1 0 558261675 19175997440 4667891 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4681640 4667891 846 493 0 4674945 0
Current children cumulated CPU time: 2441.76 s
Current children cumulated vsize: 18726560 KiB
Current children cumulated memory: 18671564 KiB

[startup+2502.3 s]
/proc/loadavg: 1.08 1.16 1.54 2/191 7157
/proc/meminfo: memFree=9251380/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=19181960 memory=19129060 CPUtime=2501.75 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4775146 0 0 0 249029 1146 0 0 20 0 1 0 558261675 19642327040 4782265 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4795490 4782265 846 493 0 4788795 0
Current children cumulated CPU time: 2501.75 s
Current children cumulated vsize: 19181960 KiB
Current children cumulated memory: 19129060 KiB



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+2520 s]*
/proc/loadavg: 1.06 1.16 1.53 2/192 7159
/proc/meminfo: memFree=9115092/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=19316204 memory=19263352 CPUtime=2519.45 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 4202496 4808738 0 0 0 250788 1157 0 0 20 0 1 0 558261675 19779792896 4815838 33554432000 4194304 6211132 140736187096480 140736186955768 140428583914469 0 0 4096 8405026 0 0 0 17 3 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4829051 4815838 846 493 0 4822356 0
Current children cumulated CPU time: 2519.45 s
Current children cumulated vsize: 19316204 KiB
Current children cumulated memory: 19263352 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+2522.9 s]
# the end of solver process 6972 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=2508.1 s, system=12.4757 s

Solver just ended.
Dumping a history of the last processes samples

[startup+2520.8 s]
/proc/loadavg: 1.06 1.16 1.53 2/192 7159
/proc/meminfo: memFree=9115092/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=19316204 memory=19263352 CPUtime=2519.5 cores=0-7
/proc/6972/stat : 6972 (cosoco) R 6969 6972 6464 0 -1 512 4808744 0 0 0 250792 1158 0 0 20 0 1 0 558261675 19779792896 4815838 33554432000 4194304 6211132 140736187096480 140736186955768 5189582 0 16384 4096 8405026 0 0 0 17 5 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4829051 4815838 846 493 0 4822356 0
Current children cumulated CPU time: 2519.5 s
Current children cumulated vsize: 19316204 KiB
Current children cumulated memory: 19263352 KiB

[startup+2521.6 s]
/proc/loadavg: 1.05 1.15 1.53 2/193 7160
/proc/meminfo: memFree=8865556/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=19316204 memory=19263352 CPUtime=2519.61 cores=0-7
/proc/6972/stat : 6972 (cosoco) S 6969 6972 6464 0 -1 512 4808744 0 0 0 250803 1158 0 0 20 0 1 0 558261675 19779792896 4815838 33554432000 4194304 6211132 140736187096480 140736186955768 5189582 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4829051 4815838 846 493 0 4822356 0
Current children cumulated CPU time: 2519.61 s
Current children cumulated vsize: 19316204 KiB
Current children cumulated memory: 19263352 KiB

[startup+2522 s]*
/proc/loadavg: 1.05 1.15 1.53 2/193 7160
/proc/meminfo: memFree=8326032/32770624 swapFree=66956692/67108860
[pid=6972] ppid=6969 vsize=19316204 memory=19263352 CPUtime=2519.67 cores=0-7
/proc/6972/stat : 6972 (cosoco) S 6969 6972 6464 0 -1 512 4808744 0 0 0 250809 1158 0 0 20 0 1 0 558261675 19779792896 4815838 33554432000 4194304 6211132 140736187096480 140736186955768 5189582 0 16384 4096 8405026 18446744071580971376 0 0 17 1 0 0 0 0 0 8312316 8317760 25108480 140736187098129 140736187098221 140736187098221 140736187101134 0
/proc/6972/statm: 4829051 4815838 846 493 0 4822356 0
Current children cumulated CPU time: 2519.67 s
Current children cumulated vsize: 19316204 KiB
Current children cumulated memory: 19263352 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 2522.9
CPU time (s): 2520.57
CPU user time (s): 2508.1
CPU system time (s): 12.4757
CPU usage (%): 99.9078
Max. virtual memory (cumulated for all children) (KiB): 19316204
Max. memory (cumulated for all children) (KiB): 19263352

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2508.1
system time used= 12.4757
maximum resident set size= 19263464
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4808744
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 98793
involuntary context switches= 2018


# summary of solver processes directly reported to runsolver:
#   pid: 6972
#   total CPU time (s): 2520.57
#   total CPU user time (s): 2508.1
#   total CPU system time (s): 12.4757

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 8.57347 second user time and 16.7482 second system time

The end

Launcher Data 

Begin job on node151 at 2019-09-30 07:04:23
IDJOB=4409631
IDBENCH=122932
IDSOLVER=2877
FILE ID=node151/4409631-1569819904
RUNJOBID= node151-1569817277-6491
PBS_JOBID= 21705917
Free space on /tmp= 62032 MiB

SOLVER NAME= cosoco 2.0 parallel
BENCH NAME= XCSP17/Opd/Opd-sum-large/Opd-04-350-100.xml
COMMAND LINE= cosoco -model -nbcores=NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4409631-1569819904/watcher-4409631-1569819904 -o /tmp/evaluation-result-4409631-1569819904/solver-4409631-1569819904 -C 20160 -W 2520 -M 31000 --bin-var /tmp/runsolver-exch-node151-1569817277-6491 --watchdog 2580  cosoco -model -nbcores=8 HOME/instance-4409631-1569819904.xml

TIME LIMIT= 20160 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 05a37308653f1418d8a9d31e27700b02
RANDOM SEED=1444516667

node151.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5335.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5335.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5335.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5335.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        28411556 kB
MemAvailable:   31815108 kB
Buffers:          177820 kB
Cached:          3551164 kB
SwapCached:        50572 kB
Active:          1875840 kB
Inactive:        1907572 kB
Active(anon):      22732 kB
Inactive(anon):   111420 kB
Active(file):    1853108 kB
Inactive(file):  1796152 kB
Unevictable:      108524 kB
Mlocked:          108524 kB
SwapTotal:      67108860 kB
SwapFree:       66956692 kB
Dirty:              2768 kB
Writeback:            32 kB
AnonPages:        145804 kB
Mapped:            26276 kB
Shmem:             70732 kB
Slab:             169412 kB
SReclaimable:     140252 kB
SUnreclaim:        29160 kB
KernelStack:        3600 kB
PageTables:         5076 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:     539068 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:    102400 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      138624 kB
DirectMap2M:    33406976 kB

Free space on /tmp at the end= 61648 MiB
End job on node151 at 2019-09-30 07:46:26