Trace number 4397708

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
cosoco 2.0? (MO) 1157.92 1159.78

General information on the benchmark

NameOpd/Opd-sum-small/
Opd-15-015-004.xml
MD5SUM1a7c138b4bdf03d361f31c5b4036df8a
Bench CategoryCOP (optimization problem)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark2.19765
Satisfiable
(Un)Satisfiability was proved
Number of variables226
Number of constraints121
Number of domains2
Minimum domain size2
Maximum domain size15
Distribution of domain sizes[{"size":2,"count":225},{"size":15,"count":1}]
Minimum variable degree16
Maximum variable degree106
Distribution of variable degrees[{"degree":16,"count":225},{"degree":106,"count":1}]
Minimum constraint arity15
Maximum constraint arity225
Distribution of constraint arities[{"arity":15,"count":15},{"arity":31,"count":105},{"arity":225,"count":1}]
Number of extensional constraints0
Number of intensional constraints0
Distribution of constraint types[{"type":"lex","count":1},{"type":"sum","count":120}]
Optimization problemYES
Type of objectivemin VAR

Solver Data


Verifier Data 

One s line expected

Watcher Data 

runsolver version 3.4.0 (svn: 3012) Copyright (C) 2010-2013 Olivier ROUSSEL

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

NUMA information:
  number of nodes: 2
  memory of node 0: 16374 MiB (2605 MiB free)
  memory of node 1: 16384 MiB (15443 MiB free)
  node distances:
     10  21
     21  10

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-4397708-1568577098/watcher-4397708-1568577098 -o /tmp/evaluation-result-4397708-1568577098/solver-4397708-1568577098 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node102-1568572739-2567 --watchdog 2560 cosoco -lc -model -restarts -stick HOME/instance-4397708-1568577098.xml 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2500 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Linux 3.10.0-514.16.1.el7.x86_64
Starting watchdog thread
solver pid=3346, runsolver pid=3343
Current StackSize limit: 8192 KiB


[startup+0.100123 s]*
/proc/loadavg: 2.05 2.16 2.05 3/199 3347
/proc/meminfo: memFree=18479932/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=28796 memory=5008 CPUtime=0.1 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1461 0 0 0 10 0 0 0 20 0 1 0 433961235 29487104 1252 33554432000 4194304 6211092 140730102004528 140730101992456 140686898894769 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 7199 1252 846 493 0 504 0
Current children cumulated CPU time: 0.1 s
Current children cumulated vsize: 28796 KiB
Current children cumulated memory: 5008 KiB

[startup+0.209433 s]*
/proc/loadavg: 2.05 2.16 2.05 3/199 3347
/proc/meminfo: memFree=18477328/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=30192 memory=6440 CPUtime=0.2 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1816 0 0 0 20 0 0 0 20 0 1 0 433961235 30916608 1610 33554432000 4194304 6211092 140730102004528 140730101992456 5012856 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 7548 1610 846 493 0 853 0
Current children cumulated CPU time: 0.2 s
Current children cumulated vsize: 30192 KiB
Current children cumulated memory: 6440 KiB

[startup+0.308824 s]*
/proc/loadavg: 2.05 2.16 2.05 3/199 3348
/proc/meminfo: memFree=18475096/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=31572 memory=7832 CPUtime=0.3 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2144 0 0 0 30 0 0 0 20 0 1 0 433961235 32329728 1958 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 7893 1958 846 493 0 1198 0
Current children cumulated CPU time: 0.3 s
Current children cumulated vsize: 31572 KiB
Current children cumulated memory: 7832 KiB

[startup+0.700277 s]
/proc/loadavg: 1.97 2.14 2.04 3/199 3348
/proc/meminfo: memFree=18470500/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=36932 memory=12808 CPUtime=0.68 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 3410 0 0 0 68 0 0 0 20 0 1 0 433961235 37818368 3202 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 9233 3202 846 493 0 2538 0
Current children cumulated CPU time: 0.68 s
Current children cumulated vsize: 36932 KiB
Current children cumulated memory: 12808 KiB

[startup+1.50028 s]
/proc/loadavg: 1.97 2.14 2.04 3/199 3348
/proc/meminfo: memFree=18459340/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=47148 memory=23344 CPUtime=1.48 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 6042 0 0 0 148 0 0 0 20 0 1 0 433961235 48279552 5836 33554432000 4194304 6211092 140730102004528 140730101992456 140686897969982 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 11787 5836 846 493 0 5092 0
Current children cumulated CPU time: 1.48 s
Current children cumulated vsize: 47148 KiB
Current children cumulated memory: 23344 KiB

[startup+3.10911 s]
/proc/loadavg: 1.97 2.14 2.04 3/199 3348
/proc/meminfo: memFree=18412404/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=68680 memory=44576 CPUtime=3.1 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 11394 0 0 0 309 1 0 0 20 0 1 0 433961235 70328320 11144 33554432000 4194304 6211092 140730102004528 140730101992456 5013228 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 17170 11144 846 493 0 10475 0
Current children cumulated CPU time: 3.1 s
Current children cumulated vsize: 68680 KiB
Current children cumulated memory: 44576 KiB

[startup+6.30028 s]
/proc/loadavg: 1.97 2.14 2.04 3/200 3350
/proc/meminfo: memFree=18343872/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=112280 memory=87332 CPUtime=6.29 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 21633 0 0 0 627 2 0 0 20 0 1 0 433961235 114974720 21833 33554432000 4194304 6211092 140730102004528 140730101992456 5451572 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 28070 21833 846 493 0 21375 0
Current children cumulated CPU time: 6.29 s
Current children cumulated vsize: 112280 KiB
Current children cumulated memory: 87332 KiB

[startup+12.7003 s]
/proc/loadavg: 1.97 2.13 2.04 3/199 3350
/proc/meminfo: memFree=18207032/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=200912 memory=170880 CPUtime=12.69 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 42016 0 0 0 1265 4 0 0 20 0 1 0 433961235 205733888 42720 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 50228 42720 846 493 0 43533 0
Current children cumulated CPU time: 12.69 s
Current children cumulated vsize: 200912 KiB
Current children cumulated memory: 170880 KiB

[startup+25.5003 s]
/proc/loadavg: 1.98 2.13 2.04 3/199 3350
/proc/meminfo: memFree=17910348/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=365116 memory=340728 CPUtime=25.49 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 80920 0 0 0 2540 9 0 0 20 0 1 0 433961235 373878784 85182 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 91279 85182 846 493 0 84584 0
Current children cumulated CPU time: 25.49 s
Current children cumulated vsize: 365116 KiB
Current children cumulated memory: 340728 KiB

[startup+51.1092 s]
/proc/loadavg: 1.99 2.12 2.04 3/200 3352
/proc/meminfo: memFree=17315016/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=713060 memory=678028 CPUtime=51.09 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 158676 0 0 0 5091 18 0 0 20 0 1 0 433961235 730173440 169507 33554432000 4194304 6211092 140730102004528 140730101992456 5441106 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 178265 169507 846 493 0 171570 0
Current children cumulated CPU time: 51.09 s
Current children cumulated vsize: 713060 KiB
Current children cumulated memory: 678028 KiB

[startup+102.3 s]
/proc/loadavg: 1.99 2.10 2.04 3/200 3357
/proc/meminfo: memFree=16150080/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=1421164 memory=1354980 CPUtime=102.29 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 312568 0 0 0 10190 39 0 0 20 0 1 0 433961235 1455271936 338745 33554432000 4194304 6211092 140730102004528 140730101992456 140686898900383 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 355291 338745 846 493 0 348596 0
Current children cumulated CPU time: 102.29 s
Current children cumulated vsize: 1421164 KiB
Current children cumulated memory: 1354980 KiB

[startup+162.3 s]
/proc/loadavg: 2.00 2.08 2.04 3/200 3361
/proc/meminfo: memFree=14778460/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=2223904 memory=2145320 CPUtime=162.29 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 491746 0 0 0 16168 61 0 0 20 0 1 0 433961235 2277277696 536330 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 555976 536330 846 493 0 549281 0
Current children cumulated CPU time: 162.29 s
Current children cumulated vsize: 2223904 KiB
Current children cumulated memory: 2145320 KiB

[startup+222.3 s]
/proc/loadavg: 2.00 2.07 2.04 3/200 3365
/proc/meminfo: memFree=13410232/32770624 swapFree=66998468/67108860
[pid=3346] ppid=3343 vsize=3071812 memory=2936520 CPUtime=222.3 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 671164 0 0 0 22145 85 0 0 20 0 1 0 433961235 3145535488 734130 33554432000 4194304 6211092 140730102004528 140730101992456 140686898900383 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 767953 734130 846 493 0 761258 0
Current children cumulated CPU time: 222.3 s
Current children cumulated vsize: 3071812 KiB
Current children cumulated memory: 2936520 KiB

[startup+282.3 s]
/proc/loadavg: 2.00 2.06 2.04 3/200 3372
/proc/meminfo: memFree=12037660/32770624 swapFree=66998472/67108860
[pid=3346] ppid=3343 vsize=3800356 memory=3728356 CPUtime=282.3 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 849812 0 0 0 28120 110 0 0 20 0 1 0 433961235 3891564544 932089 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 950089 932089 846 493 0 943394 0
Current children cumulated CPU time: 282.3 s
Current children cumulated vsize: 3800356 KiB
Current children cumulated memory: 3728356 KiB

[startup+342.3 s]
/proc/loadavg: 2.40 2.18 2.08 3/200 3440
/proc/meminfo: memFree=26342172/32770624 swapFree=66998492/67108860
[pid=3346] ppid=3343 vsize=4698296 memory=4519340 CPUtime=342.29 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1028204 0 0 0 34096 133 0 0 20 0 1 0 433961235 4811055104 1129835 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 1174574 1129835 846 493 0 1167879 0
Current children cumulated CPU time: 342.29 s
Current children cumulated vsize: 4698296 KiB
Current children cumulated memory: 4519340 KiB

[startup+402.3 s]
/proc/loadavg: 2.15 2.15 2.08 3/199 3444
/proc/meminfo: memFree=24818608/32770624 swapFree=66998492/67108860
[pid=3346] ppid=3343 vsize=5433720 memory=5307956 CPUtime=402.27 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1206528 0 0 0 40069 158 0 0 20 0 1 0 433961235 5564129280 1326989 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 1358430 1326989 846 493 0 1351735 0
Current children cumulated CPU time: 402.27 s
Current children cumulated vsize: 5433720 KiB
Current children cumulated memory: 5307956 KiB

[startup+462.3 s]
/proc/loadavg: 2.05 2.12 2.07 3/199 3451
/proc/meminfo: memFree=23294276/32770624 swapFree=66998492/67108860
[pid=3346] ppid=3343 vsize=6144276 memory=6098552 CPUtime=462.25 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1384302 0 0 0 46041 184 0 0 20 0 1 0 433961235 6291738624 1524638 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 1536069 1524638 846 493 0 1529374 0
Current children cumulated CPU time: 462.25 s
Current children cumulated vsize: 6144276 KiB
Current children cumulated memory: 6098552 KiB


################
# More data... #
################

Current children cumulated vsize: 7134980 KiB
Current children cumulated memory: 6885844 KiB

[startup+582.3 s]
/proc/loadavg: 2.01 2.08 2.06 3/200 3469
/proc/meminfo: memFree=20249776/32770624 swapFree=66998492/67108860
[pid=3346] ppid=3343 vsize=7882316 memory=7676236 CPUtime=582.21 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1740623 0 0 0 57980 241 0 0 20 0 1 0 433961235 8071491584 1919059 33554432000 4194304 6211092 140730102004528 140730101992456 5013015 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 1970579 1919059 846 493 0 1963884 0
Current children cumulated CPU time: 582.21 s
Current children cumulated vsize: 7882316 KiB
Current children cumulated memory: 7676236 KiB

[startup+642.3 s]
/proc/loadavg: 2.00 2.07 2.06 3/202 3499
/proc/meminfo: memFree=18722332/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=8592740 memory=8466568 CPUtime=642.19 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 1918351 0 0 0 63953 266 0 0 20 0 1 0 433961235 8798965760 2116642 33554432000 4194304 6211092 140730102004528 140730101992456 5454183 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 2148185 2116642 846 493 0 2141490 0
Current children cumulated CPU time: 642.19 s
Current children cumulated vsize: 8592740 KiB
Current children cumulated memory: 8466568 KiB

[startup+702.3 s]
/proc/loadavg: 2.00 2.05 2.05 3/202 3503
/proc/meminfo: memFree=17200756/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=9303164 memory=9254736 CPUtime=702.17 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2096081 0 0 0 69922 295 0 0 20 0 1 0 433961235 9526439936 2313684 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 2325791 2313684 846 493 0 2319096 0
Current children cumulated CPU time: 702.17 s
Current children cumulated vsize: 9303164 KiB
Current children cumulated memory: 9254736 KiB

[startup+762.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/202 3507
/proc/meminfo: memFree=15679744/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=10489244 memory=10044816 CPUtime=762.16 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2274832 0 0 0 75894 322 0 0 20 0 1 0 433961235 10740985856 2511204 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 2622311 2511204 846 493 0 2615616 0
Current children cumulated CPU time: 762.16 s
Current children cumulated vsize: 10489244 KiB
Current children cumulated memory: 10044816 KiB

[startup+822.3 s]
/proc/loadavg: 2.00 2.04 2.05 3/202 3513
/proc/meminfo: memFree=14157952/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=11199536 memory=10833152 CPUtime=822.15 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2452578 0 0 0 81860 355 0 0 20 0 1 0 433961235 11468324864 2708288 33554432000 4194304 6211092 140730102004528 140730101992456 140686897969982 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 2799884 2708288 846 493 0 2793189 0
Current children cumulated CPU time: 822.15 s
Current children cumulated vsize: 11199536 KiB
Current children cumulated memory: 10833152 KiB

[startup+882.3 s]
/proc/loadavg: 2.00 2.03 2.05 3/201 3517
/proc/meminfo: memFree=12633668/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=11909696 memory=11623108 CPUtime=882.13 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2630280 0 0 0 87828 385 0 0 20 0 1 0 433961235 12195528704 2905777 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 2977424 2905777 846 493 0 2970729 0
Current children cumulated CPU time: 882.13 s
Current children cumulated vsize: 11909696 KiB
Current children cumulated memory: 11623108 KiB

[startup+942.3 s]
/proc/loadavg: 2.00 2.03 2.05 3/199 3521
/proc/meminfo: memFree=11110984/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=12619592 memory=12411020 CPUtime=942.12 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2807917 0 0 0 93795 417 0 0 20 0 1 0 433961235 12922462208 3102755 33554432000 4194304 6211092 140730102004528 140730101992456 5453896 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3154898 3102755 846 493 0 3148203 0
Current children cumulated CPU time: 942.12 s
Current children cumulated vsize: 12619592 KiB
Current children cumulated memory: 12411020 KiB

[startup+1002.3 s]
/proc/loadavg: 2.00 2.02 2.05 3/199 3527
/proc/meminfo: memFree=9590924/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=13329356 memory=13198556 CPUtime=1002.11 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 2985538 0 0 0 99759 452 0 0 20 0 1 0 433961235 13649260544 3299639 33554432000 4194304 6211092 140730102004528 140730101992456 4982624 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3332339 3299639 846 493 0 3325644 0
Current children cumulated CPU time: 1002.11 s
Current children cumulated vsize: 13329356 KiB
Current children cumulated memory: 13198556 KiB

[startup+1062.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/199 3531
/proc/meminfo: memFree=8069992/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=14039120 memory=13990180 CPUtime=1062.09 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 3163154 0 0 0 105723 486 0 0 20 0 1 0 433961235 14376058880 3497545 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3509780 3497545 846 493 0 3503085 0
Current children cumulated CPU time: 1062.09 s
Current children cumulated vsize: 14039120 KiB
Current children cumulated memory: 13990180 KiB

[startup+1122.3 s]
/proc/loadavg: 2.00 2.01 2.05 3/199 3535
/proc/meminfo: memFree=6546208/32770624 swapFree=66998504/67108860
[pid=3346] ppid=3343 vsize=15462164 memory=14775964 CPUtime=1122.08 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 3341897 0 0 0 111689 519 0 0 20 0 1 0 433961235 15833255936 3693991 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3865541 3693991 846 493 0 3858846 0
Current children cumulated CPU time: 1122.08 s
Current children cumulated vsize: 15462164 KiB
Current children cumulated memory: 14775964 KiB



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1157 s]*
/proc/loadavg: 2.00 2.01 2.05 3/200 3547
/proc/meminfo: memFree=5659544/32770624 swapFree=66998512/67108860
[pid=3346] ppid=3343 vsize=15872420 memory=15231380 CPUtime=1156.78 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 4202496 3444592 0 0 0 115138 540 0 0 20 0 1 0 433961235 16253358080 3807845 33554432000 4194304 6211092 140730102004528 140730101992456 140686897974245 0 0 4096 8405026 0 0 0 17 1 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3968105 3807845 846 493 0 3961410 0
Current children cumulated CPU time: 1156.78 s
Current children cumulated vsize: 15872420 KiB
Current children cumulated memory: 15231380 KiB

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)
??? end of watcher thread

[startup+1159.78 s]
# the end of solver process 3346 was just reported to runsolver
# cumulated CPU time of all completed processes:  user=1151.67 s, system=6.2417 s

Solver just ended.
Dumping a history of the last processes samples

[startup+1158.4 s]
/proc/loadavg: 2.00 2.01 2.05 3/201 3548
/proc/meminfo: memFree=5065012/32770624 swapFree=66998512/67108860
[pid=3346] ppid=3343 vsize=15872420 memory=15231644 CPUtime=1157.04 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) S 3343 3346 2535 0 -1 512 3444602 0 0 0 115158 546 0 0 20 0 1 0 433961235 16253358080 3807911 33554432000 4194304 6211092 140730102004528 140730101992456 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3968105 3807911 846 493 0 3961410 0
Current children cumulated CPU time: 1157.04 s
Current children cumulated vsize: 15872420 KiB
Current children cumulated memory: 15231644 KiB

[startup+1158.8 s]
/proc/loadavg: 2.00 2.01 2.05 3/201 3548
/proc/meminfo: memFree=5065012/32770624 swapFree=66998512/67108860
[pid=3346] ppid=3343 vsize=15872420 memory=15231644 CPUtime=1157.11 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) R 3343 3346 2535 0 -1 512 3444602 0 0 0 115164 547 0 0 20 0 1 0 433961235 16253358080 3807911 33554432000 4194304 6211092 140730102004528 140730101992456 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3968105 3807911 846 493 0 3961410 0
Current children cumulated CPU time: 1157.11 s
Current children cumulated vsize: 15872420 KiB
Current children cumulated memory: 15231644 KiB

[startup+1159 s]*
/proc/loadavg: 2.00 2.01 2.05 4/201 3548
/proc/meminfo: memFree=4536656/32770624 swapFree=66998512/67108860
[pid=3346] ppid=3343 vsize=15872420 memory=15231644 CPUtime=1157.14 cores=1,3,5,7
/proc/3346/stat : 3346 (cosoco) S 3343 3346 2535 0 -1 512 3444602 0 0 0 115167 547 0 0 20 0 1 0 433961235 16253358080 3807911 33554432000 4194304 6211092 140730102004528 140730101992456 5189768 0 16384 4096 8405026 18446744071580971376 0 0 17 5 0 0 0 0 0 8312316 8317760 9084928 140730102006795 140730102006897 140730102006897 140730102009806 0
/proc/3346/statm: 3968105 3807911 846 493 0 3961410 0
Current children cumulated CPU time: 1157.14 s
Current children cumulated vsize: 15872420 KiB
Current children cumulated memory: 15231644 KiB

Child ended because it received signal 9 (SIGKILL)
Real time (s): 1159.78
CPU time (s): 1157.92
CPU user time (s): 1151.67
CPU system time (s): 6.2417
CPU usage (%): 99.8395
Max. virtual memory (cumulated for all children) (KiB): 15872420
Max. memory (cumulated for all children) (KiB): 15231644

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1151.67
system time used= 6.2417
maximum resident set size= 15231668
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3444602
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 134254
involuntary context switches= 1451


# summary of solver processes directly reported to runsolver:
#   pid: 3346
#   total CPU time (s): 1157.92
#   total CPU user time (s): 1151.67
#   total CPU system time (s): 6.2417

??? join timestamper begins
??? join timestamper ends
??? end of timestamper thread
runsolver used 4.18801 second user time and 7.74662 second system time

The end

Launcher Data 

Begin job on node102 at 2019-09-15 21:50:53
IDJOB=4397708
IDBENCH=122978
IDSOLVER=2850
FILE ID=node102/4397708-1568577098
RUNJOBID= node102-1568572739-2567
PBS_JOBID= 21703338
Free space on /tmp= 59324 MiB

SOLVER NAME= cosoco 2.0
BENCH NAME= XCSP17/Opd/Opd-sum-small/Opd-15-015-004.xml
COMMAND LINE= cosoco -lc -model -restarts -stick BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-4397708-1568577098/watcher-4397708-1568577098 -o /tmp/evaluation-result-4397708-1568577098/solver-4397708-1568577098 -C 2400 -W 2500 -M 15500 --bin-var /tmp/runsolver-exch-node102-1568572739-2567 --watchdog 2560  cosoco -lc -model -restarts -stick HOME/instance-4397708-1568577098.xml

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1a7c138b4bdf03d361f31c5b4036df8a
RANDOM SEED=443601585

node102.alineos.net Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017

/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
performance

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.13
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.13
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.13
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.13
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2668.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32770624 kB
MemFree:        18482484 kB
MemAvailable:   18993408 kB
Buffers:          196336 kB
Cached:           614964 kB
SwapCached:         8044 kB
Active:         13252348 kB
Inactive:         461208 kB
Active(anon):   12900736 kB
Inactive(anon):    43476 kB
Active(file):     351612 kB
Inactive(file):   417732 kB
Unevictable:      110236 kB
Mlocked:          110236 kB
SwapTotal:      67108860 kB
SwapFree:       66998468 kB
Dirty:              2804 kB
Writeback:             0 kB
AnonPages:      13007432 kB
Mapped:            41760 kB
Shmem:             33080 kB
Slab:             143308 kB
SReclaimable:     114832 kB
SUnreclaim:        28476 kB
KernelStack:        3552 kB
PageTables:        31032 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83494172 kB
Committed_AS:   13683328 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      350232 kB
VmallocChunk:   34341910524 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2377728 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      118144 kB
DirectMap2M:    33427456 kB

Free space on /tmp at the end= 58792 MiB
End job on node102 at 2019-09-15 22:10:12