Trace number 3386581

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (TO) 5000.1 5000.48

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v20000-c84000-S1298436766-097-UNKNOWN.cnf
MD5SUM28b704f4b0f31dccf5e0c8fedfb14daa
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark85.363
Satisfiable
(Un)Satisfiability was proved
Number of variables20000
Number of clauses84000
Sum of the clauses size252000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 384000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.44/7.38	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.44/7.38	c INFO: solving HOME/instance-3386581-1307741588.cnf
0.44/7.38	c INFO: computing task features
0.66/7.67	c INFO: computed variable-clause graph statistics
0.66/7.68	c INFO: computed clause balance statistics
2.06/9.06	c INFO: computed variable balance statistics
3.16/10.16	c INFO: computed small-clause counts
3.16/10.17	c INFO: computed Horn-clause counts
3.76/10.72	c INFO: computed variable graph statistics
3.76/10.72	c INFO: collected features for HOME/instance-3386581-1307741588.cnf in 3.33 s
3.76/10.79	c INFO: running hybridGM3@0 for 97 with 4996 remaining (b = 0.43)
104.40/111.33	c INFO: running hybridGM3@0 for 97 with 4896 remaining (b = 0.26)
204.99/211.87	c INFO: running hybridGM3@0 for 97 with 4795 remaining (b = 0.19)
305.70/312.41	c INFO: running hybridGM3@100 for 97 with 4695 remaining (b = 0.15)
402.91/409.55	c INFO: running hybridGM3@100 for 97 with 4597 remaining (b = 0.15)
500.14/506.68	c INFO: running TNM@0 for 97 with 4500 remaining (b = 0.16)
600.44/606.83	c INFO: running hybridGM3@100 for 97 with 4400 remaining (b = 0.14)
697.66/703.97	c INFO: running hybridGM3@0 for 194 with 4303 remaining (b = 0.26)
894.60/900.63	c INFO: running hybridGM3@0 for 194 with 4106 remaining (b = 0.20)
1091.58/1097.32	c INFO: running hybridGM3@0 for 194 with 3910 remaining (b = 0.15)
1288.51/1294.03	c INFO: running hybridGM3@0 for 194 with 3713 remaining (b = 0.12)
1485.35/1490.69	c INFO: running TNM@100 for 291 with 3516 remaining (b = 0.14)
1775.15/1780.06	c INFO: running TNM@0 for 97 with 3227 remaining (b = 0.08)
1875.47/1880.23	c INFO: running hybridGM3@0 for 194 with 3127 remaining (b = 0.08)
2071.43/2075.94	c INFO: running TNM@389 for 194 with 2931 remaining (b = 0.06)
2264.89/2269.19	c INFO: running TNM@582 for 873 with 2738 remaining (b = 0.24)
3142.18/3145.26	c INFO: running gnovelty+2@0 for 97 with 1862 remaining (b = 0.04)
3241.59/3244.50	c INFO: running hybridGM3@0 for 194 with 1763 remaining (b = 0.07)
3438.46/3441.19	c INFO: running hybridGM3@0 for 194 with 1566 remaining (b = 0.05)
3635.56/3637.92	c INFO: running TNM@100 for 1369 with 1369 remaining (b = 0.23)
5000.10/5000.47	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 988)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386581-1307741588/watcher-3386581-1307741588 -o /tmp/evaluation-result-3386581-1307741588/solver-3386581-1307741588 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386581-1307741588.cnf -seed 794573573 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.16 0.03 0.01 2/176 7635
/proc/meminfo: memFree=31586680/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=11804 CPUtime=0 cores=0,2,4,6
/proc/7635/stat : 7635 (runsolver) S 7631 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 1 0 885757618 12087296 56 33554432000 4194304 5436572 140733572984576 18446744073709551615 5057063 0 2147483391 4096 24578 18446744071699207358 0 0 17 0 0 0 0
/proc/7635/statm: 2951 56 36 304 0 2643 0

[startup+0.0976429 s]
/proc/loadavg: 0.16 0.03 0.01 2/176 7635
/proc/meminfo: memFree=31586680/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 713 0 0 0 0 0 0 0 17 0 1 0 885757618 15548416 603 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/7635/statm: 3796 603 319 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.100617 s]
/proc/loadavg: 0.16 0.03 0.01 2/176 7635
/proc/meminfo: memFree=31586680/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 713 0 0 0 0 0 0 0 17 0 1 0 885757618 15548416 603 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/7635/statm: 3796 603 319 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.300618 s]
/proc/loadavg: 0.16 0.03 0.01 2/176 7635
/proc/meminfo: memFree=31586680/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=15772 CPUtime=0 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 928 0 0 0 0 0 0 0 15 0 1 0 885757618 16150528 806 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/7635/statm: 3943 806 356 364 0 430 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15772

[startup+0.700502 s]
/proc/loadavg: 0.16 0.03 0.01 2/176 7635
/proc/meminfo: memFree=31586680/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=16532 CPUtime=0.01 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 1186 0 0 0 1 0 0 0 15 0 1 0 885757618 16928768 1045 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/7635/statm: 4133 1045 417 364 0 620 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16532

[startup+1.50038 s]
/proc/loadavg: 0.16 0.03 0.01 2/178 7637
/proc/meminfo: memFree=31583548/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=38904 CPUtime=0.04 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 1631 0 0 0 3 1 0 0 16 0 1 0 885757618 39837696 1458 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/7635/statm: 9726 1458 455 364 0 1026 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 38904

[startup+3.10119 s]
/proc/loadavg: 0.16 0.03 0.01 2/178 7637
/proc/meminfo: memFree=31577844/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=104216 CPUtime=0.06 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 3041 0 0 0 4 2 0 0 15 0 1 0 885757618 106717184 2727 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204645376 0 2147483391 16781312 2 18446744071699207358 0 0 17 6 0 0 0
/proc/7635/statm: 26054 2727 847 364 0 1802 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104216
heavy processes:

[startup+6.30458 s]
/proc/loadavg: 0.15 0.03 0.01 4/178 7645
/proc/meminfo: memFree=31559220/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=246044 CPUtime=0.37 cores=0,2,4,6
/proc/7635/stat : 7635 (python) R 7631 7635 7590 0 -1 4202496 8074 933 1 0 26 11 0 0 15 0 1 0 885757618 251949056 5962 33554432000 4194304 5683644 140735096224432 18446744073709551615 4572327 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/7635/statm: 61511 5962 1621 364 0 4513 0
Current children cumulated CPU time (s) 0.37
Current children cumulated vsize (KiB) 246044
heavy processes:

[startup+12.7005 s]
/proc/loadavg: 0.29 0.06 0.02 4/184 7652
/proc/meminfo: memFree=31505736/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=296976 CPUtime=3.81 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 16708 933 1 0 367 14 0 0 22 0 2 0 885757618 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 74244 9132 1875 364 0 10162 0
[pid=7650] ppid=7635 vsize=287824 CPUtime=0 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 740 0 0 0 0 0 0 0 15 0 1 0 885758697 294731776 7609 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204667407 0 0 16781312 514 4358539138467594500 0 0 17 2 0 0 0
/proc/7650/statm: 71956 7609 355 364 0 7874 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=1.88 cores=0,2,4,6
/proc/7652/stat : 7652 (hybridGM3) R 7650 7652 7652 0 -1 4202496 6658 0 0 0 188 0 0 0 25 0 1 0 885758698 274866176 6395 33554432000 134512640 135204385 4287466320 18446744073709551615 134537363 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/7652/statm: 67106 6395 64 169 0 66936 0
[pid=7635/tid=7651] ppid=7631 vsize=296976 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7651/stat : 7651 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 22 0 2 0 885758698 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.69
Current children cumulated vsize (KiB) 853224
heavy processes:

[startup+25.5004 s]
/proc/loadavg: 0.67 0.16 0.05 4/184 7652
/proc/meminfo: memFree=31506824/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=296976 CPUtime=3.81 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 16708 933 1 0 367 14 0 0 22 0 2 0 885757618 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 74244 9132 1875 364 0 10162 0
[pid=7650] ppid=7635 vsize=287824 CPUtime=0 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 886 0 0 0 0 0 0 0 15 0 1 0 885758697 294731776 7632 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204667407 0 0 16781312 514 15975886359399203076 0 0 17 2 0 0 0
/proc/7650/statm: 71956 7632 378 364 0 7874 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=14.68 cores=0,2,4,6
/proc/7652/stat : 7652 (hybridGM3) R 7650 7652 7652 0 -1 4202496 6659 0 0 0 1468 0 0 0 25 0 1 0 885758698 274866176 6396 33554432000 134512640 135204385 4287466320 18446744073709551615 134547366 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/7652/statm: 67106 6396 64 169 0 66936 0
[pid=7635/tid=7651] ppid=7631 vsize=296976 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7651/stat : 7651 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 22 0 2 0 885758698 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 18.49
Current children cumulated vsize (KiB) 853224
heavy processes:

[startup+51.1011 s]
/proc/loadavg: 1.12 0.30 0.10 4/184 7653
/proc/meminfo: memFree=31506380/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=296976 CPUtime=3.81 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 16708 933 1 0 367 14 0 0 22 0 2 0 885757618 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 74244 9132 1875 364 0 10162 0
[pid=7650] ppid=7635 vsize=287824 CPUtime=0.03 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 895 0 0 0 2 1 0 0 15 0 1 0 885758697 294731776 7632 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204667407 0 0 16781312 514 15975886359399203076 0 0 17 2 0 0 0
/proc/7650/statm: 71956 7632 378 364 0 7874 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=40.28 cores=0,2,4,6
/proc/7652/stat : 7652 (hybridGM3) R 7650 7652 7652 0 -1 4202496 6660 0 0 0 4028 0 0 0 25 0 1 0 885758698 274866176 6397 33554432000 134512640 135204385 4287466320 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/7652/statm: 67106 6397 64 169 0 66936 0
[pid=7635/tid=7651] ppid=7631 vsize=296976 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7651/stat : 7651 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 22 0 2 0 885758698 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 44.12
Current children cumulated vsize (KiB) 853224

[startup+102.307 s]
/proc/loadavg: 1.62 0.56 0.20 4/184 7654
/proc/meminfo: memFree=31506972/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=296976 CPUtime=3.81 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 16708 933 1 0 367 14 0 0 22 0 2 0 885757618 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 74244 9132 1875 364 0 10162 0
[pid=7650] ppid=7635 vsize=287824 CPUtime=0.1 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 914 0 0 0 6 4 0 0 15 0 1 0 885758697 294731776 7632 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204667407 0 0 16781312 514 15975886359399203076 0 0 17 2 0 0 0
/proc/7650/statm: 71956 7632 378 364 0 7874 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=91.49 cores=0,2,4,6
/proc/7652/stat : 7652 (hybridGM3) R 7650 7652 7652 0 -1 4202496 6660 0 0 0 9149 0 0 0 25 0 1 0 885758698 274866176 6397 33554432000 134512640 135204385 4287466320 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/7652/statm: 67106 6397 64 169 0 66936 0
[pid=7635/tid=7651] ppid=7631 vsize=296976 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7651/stat : 7651 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 22 0 2 0 885758698 304103424 9132 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 95.4
Current children cumulated vsize (KiB) 853224

[startup+162.301 s]
/proc/loadavg: 1.86 0.82 0.31 4/192 7664
/proc/meminfo: memFree=31464460/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=307840 CPUtime=3.85 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 17602 933 1 0 371 14 0 0 20 0 3 0 885757618 315228160 9274 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 76960 9274 1878 364 0 12875 0
[pid=7650] ppid=7635 vsize=298068 CPUtime=0.11 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 1074 0 0 0 7 4 0 0 16 0 2 0 885758697 305221632 7669 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7650/statm: 74517 7669 410 364 0 10435 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=100.49 cores=0,2,4,6
/proc/7652/stat : 7652 (hybridGM3) T 7650 7652 7652 0 -1 4202496 6660 0 0 0 10049 0 0 0 25 0 1 0 885758698 274866176 6397 33554432000 134512640 135204385 4287466320 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7652/statm: 67106 6397 64 169 0 66936 0
[pid=7650/tid=7659] ppid=7635 vsize=298068 CPUtime=0 cores=0,2,4,6
/proc/7650/task/7659/stat : 7659 (python) S 7635 7635 7590 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 885768748 305221632 7669 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453927494720 0 0 -1 2 0 0 0
[pid=7660] ppid=7635 vsize=297824 CPUtime=0.04 cores=0,2,4,6
/proc/7660/stat : 7660 (python) S 7635 7635 7590 0 -1 4202560 898 0 0 0 3 1 0 0 15 0 1 0 885768751 304971776 7765 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204667462 0 0 16781312 514 15975886359399203076 0 0 17 2 0 0 0
/proc/7660/statm: 74456 7765 372 364 0 10371 0
[pid=7662] ppid=7660 vsize=268428 CPUtime=50.94 cores=0,2,4,6
/proc/7662/stat : 7662 (hybridGM3) R 7660 7662 7662 0 -1 4202496 6663 0 0 0 5094 0 0 0 25 0 1 0 885768752 274870272 6397 33554432000 134512640 135204385 4293122320 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/7662/statm: 67107 6397 64 169 0 66937 0
[pid=7635/tid=7651] ppid=7631 vsize=307840 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7651/stat : 7651 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 22 0 3 0 885758698 315228160 9274 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7661] ppid=7631 vsize=307840 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7661/stat : 7661 (python) S 7631 7635 7590 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 3 0 885768752 315228160 9274 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 155.43
Current children cumulated vsize (KiB) 1440584

[startup+222.301 s]
/proc/loadavg: 1.95 1.03 0.41 4/196 7669
/proc/meminfo: memFree=31426184/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=318096 CPUtime=3.88 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 18465 933 1 0 374 14 0 0 20 0 4 0 885757618 325730304 9280 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 79524 9280 1880 364 0 15436 0
[pid=7650] ppid=7635 vsize=298068 CPUtime=0.11 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 1074 0 0 0 7 4 0 0 16 0 2 0 885758697 305221632 7669 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7650/statm: 74517 7669 410 364 0 10435 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=100.49 cores=0,2,4,6

################
# More data... #
################

[pid=7635/tid=7815] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7815/stat : 7815 (python) S 7631 7635 7590 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 14 0 886072145 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7822] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7822/stat : 7822 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 886082069 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7831] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7831/stat : 7831 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 886101737 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4961.95
Current children cumulated vsize (KiB) 9603460



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5000.4 s]
/proc/loadavg: 1.00 1.01 1.23 2/224 7899
/proc/meminfo: memFree=31232428/32951124 swapFree=67054524/67111528
[pid=7635] ppid=7631 vsize=421044 CPUtime=4.36 cores=0,2,4,6
/proc/7635/stat : 7635 (python) S 7631 7635 7590 0 -1 4202496 28516 933 1 0 419 17 0 0 18 0 14 0 885757618 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/7635/statm: 105261 9437 1900 364 0 41143 0
[pid=7650] ppid=7635 vsize=298068 CPUtime=0.23 cores=0,2,4,6
/proc/7650/stat : 7650 (python) S 7635 7635 7590 0 -1 4202560 1110 0 0 0 14 9 0 0 15 0 2 0 885758697 305221632 7670 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7650/statm: 74517 7670 411 364 0 10435 0
[pid=7652] ppid=7650 vsize=268424 CPUtime=197.59 cores=0,2,4,6
/proc/7652/stat : 7652 (hybridGM3) T 7650 7652 7652 0 -1 4202496 6662 0 0 0 19759 0 0 0 25 0 1 0 885758698 274866176 6399 33554432000 134512640 135204385 4287466320 18446744073709551615 134546064 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7652/statm: 67106 6399 64 169 0 66936 0
[pid=7650/tid=7659] ppid=7635 vsize=298068 CPUtime=0 cores=0,2,4,6
/proc/7650/task/7659/stat : 7659 (python) S 7635 7635 7590 0 -1 4202560 51 0 0 0 0 0 0 0 15 0 2 0 885768748 305221632 7670 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453927494720 0 0 -1 2 0 0 0
[pid=7660] ppid=7635 vsize=297824 CPUtime=0.23 cores=0,2,4,6
/proc/7660/stat : 7660 (python) S 7635 7635 7590 0 -1 4202560 1101 0 0 0 15 8 0 0 15 0 2 0 885768751 304971776 7794 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7660/statm: 74456 7794 400 364 0 10371 0
[pid=7662] ppid=7660 vsize=268428 CPUtime=197.59 cores=0,2,4,6
/proc/7662/stat : 7662 (hybridGM3) T 7660 7662 7662 0 -1 4202496 6663 0 0 0 19758 1 0 0 25 0 1 0 885768752 274870272 6397 33554432000 134512640 135204385 4293122320 18446744073709551615 134536737 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7662/statm: 67107 6397 64 169 0 66937 0
[pid=7660/tid=7666] ppid=7635 vsize=297824 CPUtime=0 cores=0,2,4,6
/proc/7660/task/7666/stat : 7666 (python) S 7635 7635 7590 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 885778802 304971776 7794 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604440278878272 0 0 -1 2 0 0 0
[pid=7667] ppid=7635 vsize=308164 CPUtime=0.24 cores=0,2,4,6
/proc/7667/stat : 7667 (python) S 7635 7635 7590 0 -1 4202560 1116 0 0 0 13 11 0 0 15 0 2 0 885778805 315559936 7798 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7667/statm: 77041 7798 400 364 0 12953 0
[pid=7669] ppid=7667 vsize=268420 CPUtime=197.6 cores=0,2,4,6
/proc/7669/stat : 7669 (hybridGM3) T 7667 7669 7669 0 -1 4202496 6664 0 0 0 19760 0 0 0 25 0 1 0 885778806 274862080 6399 33554432000 134512640 135204385 4293922096 18446744073709551615 134537272 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/7669/statm: 67105 6399 64 169 0 66935 0
[pid=7667/tid=7673] ppid=7635 vsize=308164 CPUtime=0 cores=0,2,4,6
/proc/7667/task/7673/stat : 7673 (python) S 7635 7635 7590 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 885788856 315559936 7798 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453894721856 0 0 -1 2 0 0 0
[pid=7683] ppid=7635 vsize=318508 CPUtime=1.95 cores=0,2,4,6
/proc/7683/stat : 7683 (python) S 7635 7635 7590 0 -1 4202560 1126 0 0 0 105 90 0 0 18 0 2 0 885808286 326152192 7802 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7683/statm: 79627 7802 400 364 0 15536 0
[pid=7685] ppid=7683 vsize=462808 CPUtime=1458.75 cores=0,2,4,6
/proc/7685/stat : 7685 (TNM) T 7683 7685 7685 0 -1 4202496 3591 0 0 0 145875 0 0 0 25 0 1 0 885808286 473915392 3329 33554432000 134512640 135008748 4287998704 18446744073709551615 134529367 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7685/statm: 115702 3329 46 122 0 114602 0
[pid=7683/tid=7689] ppid=7635 vsize=318508 CPUtime=0 cores=0,2,4,6
/proc/7683/task/7689/stat : 7689 (python) S 7635 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 885818298 326152192 7802 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453912630208 0 0 -1 0 0 0 0
[pid=7693] ppid=7635 vsize=329236 CPUtime=0.25 cores=0,2,4,6
/proc/7693/stat : 7693 (python) S 7635 7635 7590 0 -1 4202560 1125 0 0 0 15 10 0 0 16 0 2 0 885828015 337137664 7806 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/7693/statm: 82309 7806 400 364 0 18215 0
[pid=7695] ppid=7693 vsize=268420 CPUtime=196.62 cores=0,2,4,6
/proc/7695/stat : 7695 (hybridGM3) T 7693 7695 7695 0 -1 4202496 6663 0 0 0 19661 1 0 0 25 0 1 0 885828016 274862080 6398 33554432000 134512640 135204385 4293016320 18446744073709551615 134545866 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/7695/statm: 67105 6398 64 169 0 66935 0
[pid=7693/tid=7701] ppid=7635 vsize=329236 CPUtime=0 cores=0,2,4,6
/proc/7693/task/7701/stat : 7701 (python) S 7635 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 885847678 337137664 7806 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453887605568 0 0 -1 4 0 0 0
[pid=7702] ppid=7635 vsize=339008 CPUtime=0.28 cores=0,2,4,6
/proc/7702/stat : 7702 (python) S 7635 7635 7590 0 -1 4202560 1133 0 0 0 16 12 0 0 16 0 2 0 885847682 347144192 7906 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/7702/statm: 84752 7906 399 364 0 20655 0
[pid=7704] ppid=7702 vsize=268424 CPUtime=196.64 cores=0,2,4,6
/proc/7704/stat : 7704 (hybridGM3) T 7702 7704 7704 0 -1 4202496 6669 0 0 0 19664 0 0 0 25 0 1 0 885847682 274866176 6401 33554432000 134512640 135204385 4286589232 18446744073709551615 134537123 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/7704/statm: 67106 6401 64 169 0 66936 0
[pid=7702/tid=7711] ppid=7635 vsize=339008 CPUtime=0 cores=0,2,4,6
/proc/7702/task/7711/stat : 7711 (python) S 7635 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 885867347 347144192 7906 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453917014016 0 0 -1 0 0 0 0
[pid=7712] ppid=7635 vsize=349424 CPUtime=0.25 cores=0,2,4,6
/proc/7712/stat : 7712 (python) S 7635 7635 7590 0 -1 4202560 1181 0 0 0 16 9 0 0 16 0 2 0 885867350 357810176 7917 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7712/statm: 87356 7917 406 364 0 23256 0
[pid=7714] ppid=7712 vsize=268424 CPUtime=196.66 cores=0,2,4,6
/proc/7714/stat : 7714 (hybridGM3) T 7712 7714 7714 0 -1 4202496 6668 0 0 0 19666 0 0 0 25 0 1 0 885867350 274866176 6399 33554432000 134512640 135204385 4287606928 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7714/statm: 67106 6399 64 169 0 66936 0
[pid=7712/tid=7724] ppid=7635 vsize=349424 CPUtime=0 cores=0,2,4,6
/proc/7712/task/7724/stat : 7724 (python) S 7635 7635 7590 0 -1 4202560 89 0 0 0 0 0 0 0 16 0 2 0 885887017 357810176 7917 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453931591808 0 0 -1 2 0 0 0
[pid=7725] ppid=7635 vsize=359920 CPUtime=0.26 cores=0,2,4,6
/proc/7725/stat : 7725 (python) S 7635 7635 7590 0 -1 4202560 1174 0 0 0 15 11 0 0 16 0 2 0 885887021 368558080 7921 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7725/statm: 89980 7921 406 364 0 25877 0
[pid=7727] ppid=7725 vsize=268424 CPUtime=196.62 cores=0,2,4,6
/proc/7727/stat : 7727 (hybridGM3) T 7725 7727 7727 0 -1 4202496 6667 0 0 0 19661 1 0 0 25 0 1 0 885887021 274866176 6397 33554432000 134512640 135204385 4290915648 18446744073709551615 134545851 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/7727/statm: 67106 6397 64 169 0 66936 0
[pid=7725/tid=7733] ppid=7635 vsize=359920 CPUtime=0 cores=0,2,4,6
/proc/7725/task/7733/stat : 7733 (python) S 7635 7635 7590 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 885906684 368558080 7921 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453919593472 0 0 -1 2 0 0 0
[pid=7755] ppid=7635 vsize=369948 CPUtime=2.18 cores=0,2,4,6
/proc/7755/stat : 7755 (python) S 7635 7635 7590 0 -1 4202560 1157 0 0 0 120 98 0 0 18 0 2 0 885935624 378826752 7919 33554432000 4194304 5683644 140735096224432 18446744073709551615 210204667462 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/7755/statm: 92487 7919 400 364 0 28381 0
[pid=7757] ppid=7755 vsize=458972 CPUtime=1462.62 cores=0,2,4,6
/proc/7757/stat : 7757 (TNM) R 7755 7757 7757 0 -1 4202496 3595 0 0 0 146260 2 0 0 25 0 1 0 885935625 469987328 3328 33554432000 134512640 135008748 4290890592 18446744073709551615 134527920 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/7757/statm: 114743 3328 46 122 0 113643 0
[pid=7755/tid=7763] ppid=7635 vsize=369948 CPUtime=0 cores=0,2,4,6
/proc/7755/task/7763/stat : 7763 (python) S 7635 7635 7590 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 885945639 378826752 7919 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453919595264 0 0 -1 0 0 0 0
[pid=7764] ppid=7635 vsize=380476 CPUtime=0.26 cores=0,2,4,6
/proc/7764/stat : 7764 (python) S 7635 7635 7590 0 -1 4202560 1148 0 0 0 13 13 0 0 16 0 2 0 885945642 389607424 7923 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7764/statm: 95119 7923 400 364 0 31010 0
[pid=7766] ppid=7764 vsize=268420 CPUtime=195.66 cores=0,2,4,6
/proc/7766/stat : 7766 (hybridGM3) T 7764 7766 7766 0 -1 4202496 6671 0 0 0 19565 1 0 0 25 0 1 0 885945642 274862080 6400 33554432000 134512640 135204385 4294156832 18446744073709551615 134536646 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7766/statm: 67105 6400 64 169 0 66935 0
[pid=7764/tid=7777] ppid=7635 vsize=380476 CPUtime=0 cores=0,2,4,6
/proc/7764/task/7777/stat : 7777 (python) S 7635 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 885965209 389607424 7923 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453919592576 0 0 -1 2 0 0 0
[pid=7814] ppid=7635 vsize=390620 CPUtime=0.13 cores=0,2,4,6
/proc/7814/stat : 7814 (python) S 7635 7635 7590 0 -1 4202560 1110 0 0 0 7 6 0 0 19 0 2 0 886072145 399994880 7927 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/7814/statm: 97655 7927 400 364 0 33543 0
[pid=7816] ppid=7814 vsize=1023176 CPUtime=99.2 cores=0,2,4,6
/proc/7816/stat : 7816 (gnovelty+2) T 7814 7816 7816 0 -1 4202496 2187 0 0 0 9920 0 0 0 25 0 1 0 886072145 1047732224 1886 33554432000 134512640 135663704 4286762864 18446744073709551615 134545693 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7816/statm: 255794 1886 113 282 0 255510 0
[pid=7814/tid=7820] ppid=7635 vsize=390620 CPUtime=0 cores=0,2,4,6
/proc/7814/task/7820/stat : 7820 (python) S 7635 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 886082066 399994880 7927 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604450421949440 0 0 -1 6 0 0 0
[pid=7821] ppid=7635 vsize=401024 CPUtime=0.29 cores=0,2,4,6
/proc/7821/stat : 7821 (python) S 7635 7635 7590 0 -1 4202560 1152 0 0 0 15 14 0 0 16 0 2 0 886082068 410648576 7931 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/7821/statm: 100256 7931 400 364 0 36141 0
[pid=7823] ppid=7821 vsize=268428 CPUtime=196.65 cores=0,2,4,6
/proc/7823/stat : 7823 (hybridGM3) T 7821 7823 7823 0 -1 4202496 6670 0 0 0 19665 0 0 0 25 0 1 0 886082069 274870272 6401 33554432000 134512640 135204385 4290143824 18446744073709551615 134546102 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/7823/statm: 67107 6401 64 169 0 66937 0
[pid=7821/tid=7829] ppid=7635 vsize=401024 CPUtime=0 cores=0,2,4,6
/proc/7821/task/7829/stat : 7829 (python) S 7635 7635 7590 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 886101735 410648576 7931 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453901059136 0 0 -1 2 0 0 0
[pid=7830] ppid=7635 vsize=411004 CPUtime=0.29 cores=0,2,4,6
/proc/7830/stat : 7830 (python) S 7635 7635 7590 0 -1 4202560 1152 0 0 0 14 15 0 0 16 0 2 0 886101737 420868096 7935 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216474059 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/7830/statm: 102751 7935 400 364 0 38633 0
[pid=7832] ppid=7830 vsize=268424 CPUtime=196.7 cores=0,2,4,6
/proc/7832/stat : 7832 (hybridGM3) T 7830 7832 7832 0 -1 4202496 6670 0 0 0 19669 1 0 0 25 0 1 0 886101737 274866176 6400 33554432000 134512640 135204385 4292211024 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/7832/statm: 67106 6400 64 169 0 66936 0
[pid=7830/tid=7838] ppid=7635 vsize=411004 CPUtime=0 cores=0,2,4,6
/proc/7830/task/7838/stat : 7838 (python) S 7635 7635 7590 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 886121408 420868096 7935 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 514 18446604453887604672 0 0 -1 0 0 0 0
[pid=7635/tid=7651] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7651/stat : 7651 (python) S 7631 7635 7590 0 -1 4202560 31 933 0 0 0 0 0 0 18 0 14 0 885758698 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7661] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7661/stat : 7661 (python) S 7631 7635 7590 0 -1 4202560 31 933 0 0 0 0 0 0 18 0 14 0 885768752 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 6 0 0 0
[pid=7635/tid=7668] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7668/stat : 7668 (python) S 7631 7635 7590 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 14 0 885778805 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7684] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7684/stat : 7684 (python) S 7631 7635 7590 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 14 0 885808286 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 6 0 0 0
[pid=7635/tid=7694] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7694/stat : 7694 (python) S 7631 7635 7590 0 -1 4202560 32 933 0 0 0 0 0 0 20 0 14 0 885828015 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 6 0 0 0
[pid=7635/tid=7703] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7703/stat : 7703 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 885847682 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 6 0 0 0
[pid=7635/tid=7713] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7713/stat : 7713 (python) S 7631 7635 7590 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 14 0 885867350 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 2 0 0 0
[pid=7635/tid=7726] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7726/stat : 7726 (python) S 7631 7635 7590 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 14 0 885887021 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 2 0 0 0
[pid=7635/tid=7756] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7756/stat : 7756 (python) S 7631 7635 7590 0 -1 4202560 34 933 0 0 0 0 0 0 18 0 14 0 885935625 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 2 0 0 0
[pid=7635/tid=7765] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7765/stat : 7765 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 885945642 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 2 0 0 0
[pid=7635/tid=7815] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7815/stat : 7815 (python) S 7631 7635 7590 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 14 0 886072145 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7822] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7822/stat : 7822 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 886082069 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
[pid=7635/tid=7831] ppid=7631 vsize=421044 CPUtime=0 cores=0,2,4,6
/proc/7635/task/7831/stat : 7831 (python) S 7631 7635 7590 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 14 0 886101737 431149056 9437 33554432000 4194304 5683644 140735096224432 18446744073709551615 210216470961 0 0 16781312 2 18446604453917643968 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5000.1
Current children cumulated vsize (KiB) 9603460

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7635 and gives
#  childrusage.ru_utime.tv_sec=1963
#  childrusage.ru_utime.tv_usec=287534
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=364488
# CPU time returned by wait4() is 1966.65
# while last known CPU time is 5000.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.48
CPU time (s): 5000.1
CPU user time (s): 4996.8
CPU system time (s): 3.3
CPU usage (%): 99.9923
Max. virtual memory (cumulated for all children) (KiB): 9603464

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1963.29
system time used= 3.36449
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 66480
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 14644
involuntary context switches= 3495

runsolver used 21.8817 second user time and 45.0991 second system time

The end

Launcher Data 

Begin job on node130 at 2011-06-10 23:33:08
IDJOB=3386581
IDBENCH=82646
IDSOLVER=1872
FILE ID=node130/3386581-1307741588
RUNJOBID= node130-1307741586-7608
PBS_JOBID= 13550773
Free space on /tmp= 73952 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v20000-c84000-S1298436766-097-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386581-1307741588/watcher-3386581-1307741588 -o /tmp/evaluation-result-3386581-1307741588/solver-3386581-1307741588 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386581-1307741588.cnf -seed 794573573 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 28b704f4b0f31dccf5e0c8fedfb14daa
RANDOM SEED=794573573

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31590016 kB
Buffers:        220692 kB
Cached:         925328 kB
SwapCached:      10548 kB
Active:         496492 kB
Inactive:       668604 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31590016 kB
SwapTotal:    67111528 kB
SwapFree:     67054524 kB
Dirty:           65160 kB
Writeback:           0 kB
AnonPages:       16540 kB
Mapped:          12220 kB
Slab:           131264 kB
PageTables:       4624 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   192060 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73916 MiB
End job on node130 at 2011-06-11 00:56:34