Trace number 3386531

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (TO) 5000.06 4979.4

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v20000-c84000-S357264423-067-UNKNOWN.cnf
MD5SUM425b4c5e5535f7d6b9d17ca8feb0ac3e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark186.403
Satisfiable
(Un)Satisfiability was proved
Number of variables20000
Number of clauses84000
Sum of the clauses size252000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 384000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/0.67	c INFO: solving HOME/instance-3386531-1305586255.cnf
0.39/0.67	c INFO: computing task features
0.66/0.97	c INFO: computed variable-clause graph statistics
0.66/0.98	c INFO: computed clause balance statistics
2.16/2.42	c INFO: computed variable balance statistics
3.26/3.57	c INFO: computed small-clause counts
3.26/3.57	c INFO: computed Horn-clause counts
3.86/4.12	c INFO: computed variable graph statistics
3.86/4.12	c INFO: collected features for HOME/instance-3386531-1305586255.cnf in 3.44 s
3.86/4.15	c INFO: running hybridGM3@0 for 97 with 4996 remaining (b = 0.43)
107.16/107.04	c INFO: running hybridGM3@0 for 97 with 4893 remaining (b = 0.26)
205.33/204.89	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.19)
306.44/305.71	c INFO: running hybridGM3@102 for 97 with 4695 remaining (b = 0.15)
398.14/397.11	c INFO: running hybridGM3@97 for 97 with 4603 remaining (b = 0.15)
499.06/497.59	c INFO: running TNM@0 for 97 with 4503 remaining (b = 0.16)
596.97/595.07	c INFO: running hybridGM3@100 for 97 with 4405 remaining (b = 0.14)
694.80/692.50	c INFO: running hybridGM3@0 for 194 with 4308 remaining (b = 0.26)
890.66/887.71	c INFO: running hybridGM3@0 for 194 with 4113 remaining (b = 0.20)
1087.77/1084.00	c INFO: running hybridGM3@0 for 194 with 3917 remaining (b = 0.15)
1283.91/1279.27	c INFO: running hybridGM3@0 for 194 with 3721 remaining (b = 0.12)
1479.01/1473.57	c INFO: running TNM@97 for 291 with 3527 remaining (b = 0.14)
1772.59/1765.89	c INFO: running TNM@0 for 97 with 3235 remaining (b = 0.08)
1871.50/1864.36	c INFO: running hybridGM3@0 for 194 with 3136 remaining (b = 0.08)
2069.70/2061.64	c INFO: running TNM@389 for 194 with 2939 remaining (b = 0.06)
2263.29/2254.48	c INFO: running TNM@582 for 873 with 2746 remaining (b = 0.24)
3145.87/3133.23	c INFO: running gnovelty+2@0 for 97 with 1867 remaining (b = 0.04)
3245.77/3232.76	c INFO: running hybridGM3@0 for 194 with 1768 remaining (b = 0.07)
3442.94/3429.03	c INFO: running hybridGM3@0 for 194 with 1572 remaining (b = 0.05)
3639.07/3624.31	c INFO: running TNM@98 for 1376 with 1376 remaining (b = 0.23)
5000.06/4979.38	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386531-1305586255/watcher-3386531-1305586255 -o /tmp/evaluation-result-3386531-1305586255/solver-3386531-1305586255 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386531-1305586255.cnf -seed 1302764447 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.64 0.58 0.69 2/530 13345
/proc/meminfo: memFree=12358672/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 935 0 0 0 0 0 0 0 17 0 1 0 670224277 16154624 814 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272687104 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 0
/proc/13345/statm: 3944 814 356 364 0 431 0

[startup+0.0831411 s]
/proc/loadavg: 1.64 0.58 0.69 2/530 13345
/proc/meminfo: memFree=12358672/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=50336 CPUtime=0.05 cores=1,3,5,7
/proc/13345/stat : 13345 (python) R 13343 13345 13258 0 -1 4202496 2010 0 0 0 4 1 0 0 17 0 1 0 670224277 51544064 1770 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272684901 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/13345/statm: 12584 1770 480 364 0 1284 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50336

[startup+0.100139 s]
/proc/loadavg: 1.64 0.58 0.69 2/530 13345
/proc/meminfo: memFree=12358672/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=87704 CPUtime=0.07 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 2638 0 0 0 5 2 0 0 17 0 1 0 670224277 89808896 2354 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272687104 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 0
/proc/13345/statm: 21926 2354 717 364 0 1622 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 87704

[startup+0.300106 s]
/proc/loadavg: 1.64 0.58 0.69 2/530 13345
/proc/meminfo: memFree=12358672/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=135748 CPUtime=0.19 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 5976 933 1 0 13 6 0 0 17 0 1 0 670224277 139005952 4149 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272685061 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/13345/statm: 33937 4149 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.701032 s]
/proc/loadavg: 1.64 0.58 0.69 2/530 13345
/proc/meminfo: memFree=12358672/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=281236 CPUtime=0.46 cores=1,3,5,7
/proc/13345/stat : 13345 (python) R 13343 13345 13258 0 -1 4202496 9958 933 1 0 29 17 0 0 15 0 1 0 670224277 287985664 7626 33554432000 4194304 5683644 140733394096704 18446744073709551615 47770996854722 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/13345/statm: 70309 7627 1761 364 0 6233 0
Current children cumulated CPU time (s) 0.46
Current children cumulated vsize (KiB) 281236

[startup+1.50089 s]
/proc/loadavg: 1.64 0.58 0.69 3/535 13354
/proc/meminfo: memFree=12282248/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=297692 CPUtime=1.26 cores=1,3,5,7
/proc/13345/stat : 13345 (python) R 13343 13345 13258 0 -1 4202496 16370 933 1 0 109 17 0 0 16 0 1 0 670224277 304836608 11808 33554432000 4194304 5683644 140733394096704 18446744073709551615 47770826242125 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/13345/statm: 74423 11808 1826 364 0 10347 0
Current children cumulated CPU time (s) 1.26
Current children cumulated vsize (KiB) 297692

[startup+3.1006 s]
/proc/loadavg: 1.74 0.62 0.70 3/535 13354
/proc/meminfo: memFree=12280748/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=297692 CPUtime=2.86 cores=1,3,5,7
/proc/13345/stat : 13345 (python) R 13343 13345 13258 0 -1 4202496 16371 933 1 0 269 17 0 0 18 0 1 0 670224277 304836608 11809 33554432000 4194304 5683644 140733394096704 18446744073709551615 4314500 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/13345/statm: 74423 11809 1827 364 0 10347 0
Current children cumulated CPU time (s) 2.86
Current children cumulated vsize (KiB) 297692

[startup+6.30103 s]
/proc/loadavg: 1.74 0.62 0.70 3/538 13357
/proc/meminfo: memFree=12264232/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=296980 CPUtime=3.93 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 16721 933 1 0 375 18 0 0 22 0 2 0 670224277 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/13345/statm: 74245 9133 1875 364 0 10163 0
[pid=13355] ppid=13345 vsize=287828 CPUtime=0 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 745 0 0 0 0 0 0 0 18 0 1 0 670224693 294735872 7610 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709135 0 0 16781312 514 11951358246196838662 0 0 17 7 0 0 0
/proc/13355/statm: 71957 7610 355 364 0 7875 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=2.12 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) R 13355 13357 13357 0 -1 4202496 6660 0 0 0 212 0 0 0 25 0 1 0 670224693 274866176 6397 33554432000 134512640 135204385 4293574352 18446744073709551615 134545851 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13357/statm: 67106 6397 64 169 0 66936 0
[pid=13345/tid=13356] ppid=13343 vsize=296980 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 31 933 0 0 0 0 0 0 22 0 2 0 670224693 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.05
Current children cumulated vsize (KiB) 853232

[startup+12.701 s]
/proc/loadavg: 1.78 0.66 0.71 3/538 13357
/proc/meminfo: memFree=12263504/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=296980 CPUtime=3.93 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 16721 933 1 0 375 18 0 0 22 0 2 0 670224277 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/13345/statm: 74245 9133 1875 364 0 10163 0
[pid=13355] ppid=13345 vsize=287828 CPUtime=0.02 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 899 0 0 0 1 1 0 0 18 0 1 0 670224693 294735872 7632 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709135 0 0 16781312 514 11951358246196838662 0 0 17 7 0 0 0
/proc/13355/statm: 71957 7632 377 364 0 7875 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=8.52 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) R 13355 13357 13357 0 -1 4202496 6663 0 0 0 852 0 0 0 25 0 1 0 670224693 274866176 6400 33554432000 134512640 135204385 4293574352 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13357/statm: 67106 6400 64 169 0 66936 0
[pid=13345/tid=13356] ppid=13343 vsize=296980 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 31 933 0 0 0 0 0 0 22 0 2 0 670224693 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 12.47
Current children cumulated vsize (KiB) 853232

[startup+25.5007 s]
/proc/loadavg: 1.82 0.71 0.72 3/538 13358
/proc/meminfo: memFree=12264160/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=296980 CPUtime=3.93 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 16721 933 1 0 375 18 0 0 22 0 2 0 670224277 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/13345/statm: 74245 9133 1875 364 0 10163 0
[pid=13355] ppid=13345 vsize=287828 CPUtime=0.07 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 904 0 0 0 4 3 0 0 18 0 1 0 670224693 294735872 7633 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709135 0 0 16781312 514 9228790430156423430 0 0 17 7 0 0 0
/proc/13355/statm: 71957 7633 378 364 0 7875 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=21.32 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) R 13355 13357 13357 0 -1 4202496 6663 0 0 0 2132 0 0 0 25 0 1 0 670224693 274866176 6400 33554432000 134512640 135204385 4293574352 18446744073709551615 134546157 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13357/statm: 67106 6400 64 169 0 66936 0
[pid=13345/tid=13356] ppid=13343 vsize=296980 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 31 933 0 0 0 0 0 0 22 0 2 0 670224693 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 25.32
Current children cumulated vsize (KiB) 853232

[startup+51.1011 s]
/proc/loadavg: 1.88 0.81 0.76 3/538 13358
/proc/meminfo: memFree=12263600/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=296980 CPUtime=3.93 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 16721 933 1 0 375 18 0 0 22 0 2 0 670224277 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/13345/statm: 74245 9133 1875 364 0 10163 0
[pid=13355] ppid=13345 vsize=287828 CPUtime=0.18 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 916 0 0 0 10 8 0 0 18 0 1 0 670224693 294735872 7633 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709135 0 0 16781312 514 9228790430156423430 0 0 17 7 0 0 0
/proc/13355/statm: 71957 7633 378 364 0 7875 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=46.92 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) R 13355 13357 13357 0 -1 4202496 6663 0 0 0 4692 0 0 0 25 0 1 0 670224693 274866176 6400 33554432000 134512640 135204385 4293574352 18446744073709551615 134537042 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13357/statm: 67106 6400 64 169 0 66936 0
[pid=13345/tid=13356] ppid=13343 vsize=296980 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 31 933 0 0 0 0 0 0 22 0 2 0 670224693 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 51.03
Current children cumulated vsize (KiB) 853232

[startup+102.32 s]
/proc/loadavg: 1.95 1.01 0.83 3/542 13364
/proc/meminfo: memFree=12229712/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=296980 CPUtime=3.93 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 16721 933 1 0 375 18 0 0 22 0 2 0 670224277 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/13345/statm: 74245 9133 1875 364 0 10163 0
[pid=13355] ppid=13345 vsize=287828 CPUtime=0.4 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 934 0 0 0 21 19 0 0 18 0 1 0 670224693 294735872 7633 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709135 0 0 16781312 514 9228790430156423430 0 0 17 7 0 0 0
/proc/13355/statm: 71957 7633 378 364 0 7875 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=98.14 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) R 13355 13357 13357 0 -1 4202496 6667 0 0 0 9814 0 0 0 25 0 1 0 670224693 274866176 6404 33554432000 134512640 135204385 4293574352 18446744073709551615 134536929 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13357/statm: 67106 6404 64 169 0 66936 0
[pid=13345/tid=13356] ppid=13343 vsize=296980 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 31 933 0 0 0 0 0 0 22 0 2 0 670224693 304107520 9133 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 102.47
Current children cumulated vsize (KiB) 853232

[startup+162.301 s]
/proc/loadavg: 1.98 1.19 0.90 5/546 13369
/proc/meminfo: memFree=12197672/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=307844 CPUtime=3.95 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 17612 933 1 0 377 18 0 0 20 0 3 0 670224277 315232256 9275 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13345/statm: 76961 9275 1878 364 0 12876 0
[pid=13355] ppid=13345 vsize=298072 CPUtime=0.43 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 1091 0 0 0 23 20 0 0 19 0 2 0 670224693 305225728 7670 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13355/statm: 74518 7670 410 364 0 10436 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=102.83 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) T 13355 13357 13357 0 -1 4202496 6667 0 0 0 10283 0 0 0 25 0 1 0 670224693 274866176 6404 33554432000 134512640 135204385 4293574352 18446744073709551615 134536952 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13357/statm: 67106 6404 64 169 0 66936 0
[pid=13355/tid=13365] ppid=13345 vsize=298072 CPUtime=0 cores=1,3,5,7
/proc/13355/task/13365/stat : 13365 (python) S 13345 13345 13258 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670234978 305225728 7670 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604459990302976 0 0 -1 7 0 0 0
[pid=13366] ppid=13345 vsize=297828 CPUtime=0.21 cores=1,3,5,7
/proc/13366/stat : 13366 (python) S 13345 13345 13258 0 -1 4202560 916 0 0 0 11 10 0 0 15 0 1 0 670234981 304975872 7766 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709190 0 0 16781312 514 9228790430156423430 0 0 17 7 0 0 0
/proc/13366/statm: 74457 7766 372 364 0 10372 0
[pid=13368] ppid=13366 vsize=268420 CPUtime=55.24 cores=1,3,5,7
/proc/13368/stat : 13368 (hybridGM3) R 13366 13368 13368 0 -1 4202496 6660 0 0 0 5524 0 0 0 25 0 1 0 670234982 274862080 6397 33554432000 134512640 135204385 4288408400 18446744073709551615 134537471 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13368/statm: 67105 6397 64 169 0 66935 0
[pid=13345/tid=13356] ppid=13343 vsize=307844 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 31 933 0 0 0 0 0 0 22 0 3 0 670224693 315232256 9275 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13367] ppid=13343 vsize=307844 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13367/stat : 13367 (python) S 13343 13345 13258 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 3 0 670234982 315232256 9275 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 162.66
Current children cumulated vsize (KiB) 1440588

[startup+222.301 s]
/proc/loadavg: 2.07 1.35 0.97 3/551 13376
/proc/meminfo: memFree=12165628/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=318100 CPUtime=3.99 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 18478 933 1 0 381 18 0 0 19 0 4 0 670224277 325734400 9281 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13345/statm: 79525 9281 1880 364 0 15437 0

################
# More data... #
################

[pid=13345/tid=13561] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13561/stat : 13561 (python) S 13343 13345 13258 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 14 0 670400867 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13572] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13572/stat : 13572 (python) S 13343 13345 13258 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 670410714 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13685] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13685/stat : 13685 (python) S 13343 13345 13258 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 670537601 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 3 0 0 0
[pid=13345/tid=13695] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13695/stat : 13695 (python) S 13343 13345 13258 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 14 0 670547554 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13713] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13713/stat : 13713 (python) S 13343 13345 13258 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 14 0 670567181 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5000.06
Current children cumulated vsize (KiB) 9599780

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+4979.3 s]
/proc/loadavg: 2.11 2.07 2.01 3/639 13800
/proc/meminfo: memFree=11839688/32950928 swapFree=60199076/67111528
[pid=13345] ppid=13343 vsize=421028 CPUtime=4.45 cores=1,3,5,7
/proc/13345/stat : 13345 (python) S 13343 13345 13258 0 -1 4202496 28814 933 1 0 424 21 0 0 18 0 14 0 670224277 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13345/statm: 105257 9455 1900 364 0 41139 0
[pid=13355] ppid=13345 vsize=298072 CPUtime=0.81 cores=1,3,5,7
/proc/13355/stat : 13355 (python) S 13345 13345 13258 0 -1 4202560 1125 0 0 0 44 37 0 0 18 0 2 0 670224693 305225728 7671 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13355/statm: 74518 7671 411 364 0 10436 0
[pid=13357] ppid=13355 vsize=268424 CPUtime=194.12 cores=1,3,5,7
/proc/13357/stat : 13357 (hybridGM3) T 13355 13357 13357 0 -1 4202496 6667 0 0 0 19412 0 0 0 25 0 1 0 670224693 274866176 6404 33554432000 134512640 135204385 4293574352 18446744073709551615 134549085 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13357/statm: 67106 6404 64 169 0 66936 0
[pid=13355/tid=13365] ppid=13345 vsize=298072 CPUtime=0 cores=1,3,5,7
/proc/13355/task/13365/stat : 13365 (python) S 13345 13345 13258 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 670234978 305225728 7671 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604459990302976 0 0 -1 7 0 0 0
[pid=13366] ppid=13345 vsize=297828 CPUtime=0.84 cores=1,3,5,7
/proc/13366/stat : 13366 (python) S 13345 13345 13258 0 -1 4202560 1119 0 0 0 42 42 0 0 16 0 2 0 670234981 304975872 7795 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13366/statm: 74457 7795 400 364 0 10372 0
[pid=13368] ppid=13366 vsize=268420 CPUtime=198.24 cores=1,3,5,7
/proc/13368/stat : 13368 (hybridGM3) T 13366 13368 13368 0 -1 4202496 6661 0 0 0 19824 0 0 0 25 0 1 0 670234982 274862080 6398 33554432000 134512640 135204385 4288408400 18446744073709551615 134536986 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13368/statm: 67105 6398 64 169 0 66935 0
[pid=13366/tid=13373] ppid=13345 vsize=297828 CPUtime=0 cores=1,3,5,7
/proc/13366/task/13373/stat : 13373 (python) S 13345 13345 13258 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 670244763 304975872 7795 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604466232762688 0 0 -1 7 0 0 0
[pid=13374] ppid=13345 vsize=308168 CPUtime=0.81 cores=1,3,5,7
/proc/13374/stat : 13374 (python) S 13345 13345 13258 0 -1 4202560 1131 0 0 0 41 40 0 0 15 0 2 0 670244766 315564032 7799 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13374/statm: 77042 7799 400 364 0 12954 0
[pid=13376] ppid=13374 vsize=268424 CPUtime=198 cores=1,3,5,7
/proc/13376/stat : 13376 (hybridGM3) T 13374 13376 13376 0 -1 4202496 6667 0 0 0 19799 1 0 0 25 0 1 0 670244767 274866176 6402 33554432000 134512640 135204385 4289626000 18446744073709551615 134537583 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13376/statm: 67106 6402 64 169 0 66936 0
[pid=13374/tid=13413] ppid=13345 vsize=308168 CPUtime=0 cores=1,3,5,7
/proc/13374/task/13413/stat : 13413 (python) S 13345 13345 13258 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 670254846 315564032 7799 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604459609776320 0 0 -1 5 0 0 0
[pid=13427] ppid=13345 vsize=318964 CPUtime=6.26 cores=1,3,5,7
/proc/13427/stat : 13427 (python) S 13345 13345 13258 0 -1 4202560 1150 0 0 0 322 304 0 0 18 0 2 0 670274037 326619136 7805 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13427/statm: 79741 7805 400 364 0 15650 0
[pid=13429] ppid=13427 vsize=458972 CPUtime=1461.28 cores=1,3,5,7
/proc/13429/stat : 13429 (TNM) T 13427 13429 13429 0 -1 4202496 3589 0 0 0 146128 0 0 0 25 0 1 0 670274037 469987328 3328 33554432000 134512640 135008748 4293184192 18446744073709551615 134524890 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/13429/statm: 114743 3328 46 122 0 113643 0
[pid=13427/tid=13437] ppid=13345 vsize=318964 CPUtime=0 cores=1,3,5,7
/proc/13427/task/13437/stat : 13437 (python) S 13345 13345 13258 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 670283782 326619136 7805 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604459990304768 0 0 -1 7 0 0 0
[pid=13454] ppid=13345 vsize=328736 CPUtime=0.78 cores=1,3,5,7
/proc/13454/stat : 13454 (python) S 13345 13345 13258 0 -1 4202560 1146 0 0 0 40 38 0 0 16 0 2 0 670293527 336625664 7898 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/13454/statm: 82184 7898 399 364 0 18090 0
[pid=13456] ppid=13454 vsize=268424 CPUtime=195.17 cores=1,3,5,7
/proc/13456/stat : 13456 (hybridGM3) T 13454 13456 13456 0 -1 4202496 6667 0 0 0 19517 0 0 0 25 0 1 0 670293528 274866176 6401 33554432000 134512640 135204385 4294071712 18446744073709551615 134537548 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/13456/statm: 67106 6401 64 169 0 66936 0
[pid=13454/tid=13466] ppid=13345 vsize=328736 CPUtime=0 cores=1,3,5,7
/proc/13454/task/13466/stat : 13466 (python) S 13345 13345 13258 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 670313045 336625664 7898 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604459990303872 0 0 -1 7 0 0 0
[pid=13467] ppid=13345 vsize=339152 CPUtime=0.84 cores=1,3,5,7
/proc/13467/stat : 13467 (python) S 13345 13345 13258 0 -1 4202560 1150 0 0 0 43 41 0 0 17 0 2 0 670313048 347291648 7903 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/13467/statm: 84788 7903 400 364 0 20691 0
[pid=13469] ppid=13467 vsize=268424 CPUtime=196.25 cores=1,3,5,7
/proc/13469/stat : 13469 (hybridGM3) T 13467 13469 13469 0 -1 4202496 6665 0 0 0 19625 0 0 0 25 0 1 0 670313049 274866176 6400 33554432000 134512640 135204385 4287333520 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/13469/statm: 67106 6400 64 169 0 66936 0
[pid=13467/tid=13479] ppid=13345 vsize=339152 CPUtime=0 cores=1,3,5,7
/proc/13467/task/13479/stat : 13479 (python) S 13345 13345 13258 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 670332674 347291648 7903 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604457396959296 0 0 -1 3 0 0 0
[pid=13480] ppid=13345 vsize=349520 CPUtime=0.85 cores=1,3,5,7
/proc/13480/stat : 13480 (python) S 13345 13345 13258 0 -1 4202560 1210 0 0 0 42 43 0 0 17 0 2 0 670332677 357908480 7913 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13480/statm: 87380 7913 406 364 0 23280 0
[pid=13482] ppid=13480 vsize=268420 CPUtime=195.23 cores=1,3,5,7
/proc/13482/stat : 13482 (hybridGM3) T 13480 13482 13482 0 -1 4202496 6667 0 0 0 19523 0 0 0 25 0 1 0 670332678 274862080 6399 33554432000 134512640 135204385 4292250896 18446744073709551615 134547471 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/13482/statm: 67105 6399 64 169 0 66935 0
[pid=13480/tid=13491] ppid=13345 vsize=349520 CPUtime=0 cores=1,3,5,7
/proc/13480/task/13491/stat : 13491 (python) S 13345 13345 13258 0 -1 4202560 91 0 0 0 0 0 0 0 17 0 2 0 670352202 357908480 7913 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604460112930624 0 0 -1 7 0 0 0
[pid=13492] ppid=13345 vsize=359640 CPUtime=0.83 cores=1,3,5,7
/proc/13492/stat : 13492 (python) S 13345 13345 13258 0 -1 4202560 1201 0 0 0 43 40 0 0 17 0 2 0 670352205 368271360 7917 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13492/statm: 89910 7917 406 364 0 25807 0
[pid=13494] ppid=13492 vsize=268420 CPUtime=194.25 cores=1,3,5,7
/proc/13494/stat : 13494 (hybridGM3) T 13492 13494 13494 0 -1 4202496 6667 0 0 0 19425 0 0 0 25 0 1 0 670352205 274862080 6397 33554432000 134512640 135204385 4288601664 18446744073709551615 134537249 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13494/statm: 67105 6397 64 169 0 66935 0
[pid=13492/tid=13539] ppid=13345 vsize=359640 CPUtime=0 cores=1,3,5,7
/proc/13492/task/13539/stat : 13539 (python) S 13345 13345 13258 0 -1 4202560 49 0 0 0 0 0 0 0 17 0 2 0 670371632 368271360 7917 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604458923005056 0 0 -1 7 0 0 0
[pid=13560] ppid=13345 vsize=370152 CPUtime=6.35 cores=1,3,5,7
/proc/13560/stat : 13560 (python) S 13345 13345 13258 0 -1 4202560 1174 0 0 0 323 312 0 0 18 0 2 0 670400867 379035648 7915 33554432000 4194304 5683644 140733394096704 18446744073709551615 268272709190 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13560/statm: 92538 7915 400 364 0 28432 0
[pid=13562] ppid=13560 vsize=458972 CPUtime=1453.48 cores=1,3,5,7
/proc/13562/stat : 13562 (TNM) R 13560 13562 13562 0 -1 4202496 3605 0 0 0 145348 0 0 0 25 0 1 0 670400867 469987328 3328 33554432000 134512640 135008748 4293406752 18446744073709551615 134527477 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/13562/statm: 114743 3328 46 122 0 113643 0
[pid=13560/tid=13570] ppid=13345 vsize=370152 CPUtime=0 cores=1,3,5,7
/proc/13560/task/13570/stat : 13570 (python) S 13345 13345 13258 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 670410711 379035648 7915 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604460700480448 0 0 -1 7 0 0 0
[pid=13571] ppid=13345 vsize=380152 CPUtime=0.85 cores=1,3,5,7
/proc/13571/stat : 13571 (python) S 13345 13345 13258 0 -1 4202560 1168 0 0 0 43 42 0 0 16 0 2 0 670410714 389275648 7919 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13571/statm: 95038 7919 400 364 0 30929 0
[pid=13573] ppid=13571 vsize=268428 CPUtime=197.25 cores=1,3,5,7
/proc/13573/stat : 13573 (hybridGM3) T 13571 13573 13573 0 -1 4202496 6671 0 0 0 19724 1 0 0 25 0 1 0 670410714 274870272 6401 33554432000 134512640 135204385 4293158928 18446744073709551615 134536767 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/13573/statm: 67107 6401 64 169 0 66937 0
[pid=13571/tid=13587] ppid=13345 vsize=380152 CPUtime=0 cores=1,3,5,7
/proc/13571/task/13587/stat : 13587 (python) S 13345 13345 13258 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 670430439 389275648 7919 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604460700481344 0 0 -1 3 0 0 0
[pid=13684] ppid=13345 vsize=390692 CPUtime=0.42 cores=1,3,5,7
/proc/13684/stat : 13684 (python) S 13345 13345 13258 0 -1 4202560 1132 0 0 0 21 21 0 0 19 0 2 0 670537601 400068608 7923 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13684/statm: 97673 7923 400 364 0 33561 0
[pid=13686] ppid=13684 vsize=1023176 CPUtime=99.5 cores=1,3,5,7
/proc/13686/stat : 13686 (gnovelty+2) T 13684 13686 13686 0 -1 4202496 2183 0 0 0 9950 0 0 0 25 0 1 0 670537601 1047732224 1881 33554432000 134512640 135663704 4286957088 18446744073709551615 134546755 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/13686/statm: 255794 1881 113 282 0 255510 0
[pid=13684/tid=13693] ppid=13345 vsize=390692 CPUtime=0 cores=1,3,5,7
/proc/13684/task/13693/stat : 13693 (python) S 13345 13345 13258 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670547552 400068608 7923 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604461961294144 0 0 -1 7 0 0 0
[pid=13694] ppid=13345 vsize=401100 CPUtime=0.85 cores=1,3,5,7
/proc/13694/stat : 13694 (python) S 13345 13345 13258 0 -1 4202560 1154 0 0 0 45 40 0 0 16 0 2 0 670547554 410726400 7927 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/13694/statm: 100275 7927 400 364 0 36160 0
[pid=13696] ppid=13694 vsize=268420 CPUtime=196.24 cores=1,3,5,7
/proc/13696/stat : 13696 (hybridGM3) T 13694 13696 13696 0 -1 4202496 6669 0 0 0 19623 1 0 0 25 0 1 0 670547554 274862080 6402 33554432000 134512640 135204385 4288111088 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13696/statm: 67105 6402 64 169 0 66935 0
[pid=13694/tid=13711] ppid=13345 vsize=401100 CPUtime=0 cores=1,3,5,7
/proc/13694/task/13711/stat : 13711 (python) S 13345 13345 13258 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 670567179 410726400 7927 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604456104128640 0 0 -1 3 0 0 0
[pid=13712] ppid=13345 vsize=411228 CPUtime=0.86 cores=1,3,5,7
/proc/13712/stat : 13712 (python) S 13345 13345 13258 0 -1 4202560 1163 0 0 0 46 40 0 0 16 0 2 0 670567181 421097472 7953 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284515787 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13712/statm: 102807 7953 400 364 0 38689 0
[pid=13714] ppid=13712 vsize=268424 CPUtime=195.25 cores=1,3,5,7
/proc/13714/stat : 13714 (hybridGM3) T 13712 13714 13714 0 -1 4202496 6669 0 0 0 19525 0 0 0 25 0 1 0 670567181 274866176 6400 33554432000 134512640 135204385 4293365648 18446744073709551615 134545844 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13714/statm: 67106 6400 64 169 0 66936 0
[pid=13712/tid=13728] ppid=13345 vsize=411228 CPUtime=0 cores=1,3,5,7
/proc/13712/task/13728/stat : 13728 (python) S 13345 13345 13258 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 670586707 421097472 7953 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 514 18446604456104130432 0 0 -1 7 0 0 0
[pid=13345/tid=13356] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13356/stat : 13356 (python) S 13343 13345 13258 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 14 0 670224693 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
[pid=13345/tid=13367] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13367/stat : 13367 (python) S 13343 13345 13258 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 14 0 670234982 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
[pid=13345/tid=13375] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13375/stat : 13375 (python) S 13343 13345 13258 0 -1 4202560 28 933 0 0 0 0 0 0 18 0 14 0 670244766 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
[pid=13345/tid=13428] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13428/stat : 13428 (python) S 13343 13345 13258 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 14 0 670274037 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
[pid=13345/tid=13455] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13455/stat : 13455 (python) S 13343 13345 13258 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 14 0 670293528 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13468] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13468/stat : 13468 (python) S 13343 13345 13258 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 14 0 670313048 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 1 0 0 0
[pid=13345/tid=13481] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13481/stat : 13481 (python) S 13343 13345 13258 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 14 0 670332677 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 1 0 0 0
[pid=13345/tid=13493] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13493/stat : 13493 (python) S 13343 13345 13258 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 14 0 670352205 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 3 0 0 0
[pid=13345/tid=13561] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13561/stat : 13561 (python) S 13343 13345 13258 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 14 0 670400867 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13572] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13572/stat : 13572 (python) S 13343 13345 13258 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 670410714 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13685] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13685/stat : 13685 (python) S 13343 13345 13258 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 670537601 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 3 0 0 0
[pid=13345/tid=13695] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13695/stat : 13695 (python) S 13343 13345 13258 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 14 0 670547554 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 7 0 0 0
[pid=13345/tid=13713] ppid=13343 vsize=421028 CPUtime=0 cores=1,3,5,7
/proc/13345/task/13713/stat : 13713 (python) S 13343 13345 13258 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 14 0 670567181 431132672 9455 33554432000 4194304 5683644 140733394096704 18446744073709551615 268284512689 0 0 16781312 2 18446604464025366848 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5000.06
Current children cumulated vsize (KiB) 9599780

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 13345 and gives
#  childrusage.ru_utime.tv_sec=1959
#  childrusage.ru_utime.tv_usec=810063
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=705372
# CPU time returned by wait4() is 1970.52
# while last known CPU time is 5000.06
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 4979.4
CPU time (s): 5000.06
CPU user time (s): 4989.42
CPU system time (s): 10.64
CPU usage (%): 100.415
Max. virtual memory (cumulated for all children) (KiB): 9599784

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1959.81
system time used= 10.7054
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 68608
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8698
involuntary context switches= 3367

runsolver used 37.8762 second user time and 88.6635 second system time

The end

Launcher Data 

Begin job on node122 at 2011-05-17 00:50:55
IDJOB=3386531
IDBENCH=82596
IDSOLVER=1872
FILE ID=node122/3386531-1305586255
RUNJOBID= node122-1305586147-13277
PBS_JOBID= 13350356
Free space on /tmp= 73344 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v20000-c84000-S357264423-067-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386531-1305586255/watcher-3386531-1305586255 -o /tmp/evaluation-result-3386531-1305586255/solver-3386531-1305586255 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386531-1305586255.cnf -seed 1302764447 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 425b4c5e5535f7d6b9d17ca8feb0ac3e
RANDOM SEED=1302764447

node122.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      12358952 kB
Buffers:        334600 kB
Cached:        4475576 kB
SwapCached:      50104 kB
Active:       15806140 kB
Inactive:      4489104 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      12358952 kB
SwapTotal:    67111528 kB
SwapFree:     60199076 kB
Dirty:           33448 kB
Writeback:          20 kB
AnonPages:    15446036 kB
Mapped:          36976 kB
Slab:           139016 kB
PageTables:      87952 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 160962100 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73312 MiB
End job on node122 at 2011-05-17 02:13:56