Trace number 3386571

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 3548.43 3536.54

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S971556279-001-UNKNOWN.cnf
MD5SUM9224277e61359574518ca553717e02ee
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark640.831
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.38/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.38/0.67	c INFO: solving HOME/instance-3386571-1305587432.cnf
0.38/0.67	c INFO: computing task features
0.56/0.87	c INFO: computed variable-clause graph statistics
0.56/0.88	c INFO: computed clause balance statistics
1.66/1.95	c INFO: computed variable balance statistics
2.26/2.55	c INFO: computed small-clause counts
2.26/2.55	c INFO: computed Horn-clause counts
2.26/2.59	c INFO: computed variable graph statistics
2.26/2.59	c INFO: collected features for HOME/instance-3386571-1305587432.cnf in 1.92 s
2.36/2.62	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.38)
103.05/103.01	c INFO: running gnovelty+2@0 for 97 with 4897 remaining (b = 0.13)
202.77/202.41	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
303.35/302.78	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.07)
403.07/402.14	c INFO: running adaptg2wsat2009++@0 for 97 with 4598 remaining (b = 0.05)
503.69/502.50	c INFO: running gnovelty+2@0 for 97 with 4498 remaining (b = 0.05)
603.40/601.92	c INFO: running FH@0 for 97 with 4399 remaining (b = 0.03)
702.12/700.31	c INFO: running adaptg2wsat2009++@0 for 97 with 4300 remaining (b = 0.03)
802.74/800.67	c INFO: running adaptg2wsat2009++@0 for 97 with 4200 remaining (b = 0.02)
903.46/901.05	c INFO: running FH@0 for 97 with 4100 remaining (b = 0.02)
1003.18/1000.43	c INFO: running hybridGM3@0 for 194 with 4000 remaining (b = 0.04)
1198.64/1195.23	c INFO: running gnovelty+2@0 for 97 with 3805 remaining (b = 0.02)
1298.35/1294.68	c INFO: running hybridGM3@0 for 194 with 3706 remaining (b = 0.03)
1493.95/1489.54	c INFO: running adaptg2wsat2009++@0 for 97 with 3511 remaining (b = 0.01)
1593.69/1588.94	c INFO: running hybridGM3@0 for 194 with 3412 remaining (b = 0.03)
1790.17/1784.80	c INFO: running hybridGM3@0 for 194 with 3216 remaining (b = 0.02)
1986.90/1980.76	c INFO: running TNM@0 for 1552 with 3020 remaining (b = 0.19)
3547.90/3535.98	c INFO: running FH@0 for 97 with 1465 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386571-1305587432/watcher-3386571-1305587432 -o /tmp/evaluation-result-3386571-1305587432/solver-3386571-1305587432 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386571-1305587432.cnf -seed 1675891500 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 2.00 1.67 2/452 6368
/proc/meminfo: memFree=19661444/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=15384 CPUtime=0 cores=0,2,4,6
/proc/6368/stat : 6368 (python) R 6366 6368 6138 0 -1 4202496 739 0 0 0 0 0 0 0 24 0 1 0 748973905 15753216 628 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905087981 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/6368/statm: 3846 648 322 364 0 333 0

[startup+0.0166101 s]
/proc/loadavg: 2.00 2.00 1.67 2/452 6368
/proc/meminfo: memFree=19661444/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=16240 CPUtime=0 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 1000 0 0 0 0 0 0 0 23 0 1 0 748973905 16629760 873 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 0
/proc/6368/statm: 4060 873 371 364 0 547 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16240

[startup+0.100566 s]
/proc/loadavg: 2.00 2.00 1.67 2/452 6368
/proc/meminfo: memFree=19661444/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=89768 CPUtime=0.06 cores=0,2,4,6
/proc/6368/stat : 6368 (python) R 6366 6368 6138 0 -1 4202496 2672 0 0 0 4 2 0 0 24 0 1 0 748973905 91922432 2388 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905392128 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/6368/statm: 22442 2388 750 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89768

[startup+0.30054 s]
/proc/loadavg: 2.00 2.00 1.67 2/452 6368
/proc/meminfo: memFree=19661444/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=135748 CPUtime=0.19 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 5971 936 1 0 14 5 0 0 17 0 1 0 748973905 139005952 4149 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905390085 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 0
/proc/6368/statm: 33937 4149 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700506 s]
/proc/loadavg: 2.00 2.00 1.67 2/452 6368
/proc/meminfo: memFree=19661444/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=280856 CPUtime=0.45 cores=0,2,4,6
/proc/6368/stat : 6368 (python) R 6366 6368 6138 0 -1 4202496 9883 936 1 0 33 12 0 0 16 0 1 0 748973905 287596544 7557 33554432000 4194304 5683644 140733971424320 18446744073709551615 47227147719635 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/6368/statm: 70214 7557 1761 364 0 6138 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 280856

[startup+1.50037 s]
/proc/loadavg: 2.00 2.00 1.67 3/453 6374
/proc/meminfo: memFree=19626716/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=290800 CPUtime=1.26 cores=0,2,4,6
/proc/6368/stat : 6368 (python) R 6366 6368 6138 0 -1 4202496 14018 936 1 0 113 13 0 0 17 0 1 0 748973905 297779200 10085 33554432000 4194304 5683644 140733971424320 18446744073709551615 47226976948362 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/6368/statm: 72700 10085 1826 364 0 8624 0
Current children cumulated CPU time (s) 1.26
Current children cumulated vsize (KiB) 290800

[startup+3.10102 s]
/proc/loadavg: 2.00 2.00 1.67 3/453 6374
/proc/meminfo: memFree=19626092/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=294136 CPUtime=2.38 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 14389 936 1 0 225 13 0 0 21 0 2 0 748973905 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6368/statm: 73534 8424 1877 364 0 9452 0
Current children cumulated CPU time (s) 2.38
Current children cumulated vsize (KiB) 294136
heavy processes:

[startup+6.30043 s]
/proc/loadavg: 2.00 2.00 1.67 3/456 6377
/proc/meminfo: memFree=19626196/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=294136 CPUtime=2.38 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 14389 936 1 0 225 13 0 0 21 0 2 0 748973905 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6368/statm: 73534 8424 1877 364 0 9452 0
[pid=6375] ppid=6368 vsize=284984 CPUtime=0 cores=0,2,4,6
/proc/6375/stat : 6375 (python) S 6368 6368 6138 0 -1 4202560 731 0 0 0 0 0 0 0 18 0 1 0 748974168 291823616 6899 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414159 0 0 16781312 514 9321484487353139460 0 0 17 4 0 0 0
/proc/6375/statm: 71246 6899 355 364 0 7164 0
[pid=6377] ppid=6375 vsize=1020308 CPUtime=3.66 cores=0,2,4,6
/proc/6377/stat : 6377 (gnovelty+2) R 6375 6377 6377 0 -1 4202496 1157 0 0 0 366 0 0 0 25 0 1 0 748974168 1044795392 892 33554432000 134512640 135663704 4293971952 18446744073709551615 134546036 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6377/statm: 255077 892 113 282 0 254793 0
[pid=6368/tid=6376] ppid=6366 vsize=294136 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 2 0 748974168 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 6.04
Current children cumulated vsize (KiB) 1599428

[startup+12.7003 s]
/proc/loadavg: 2.00 2.00 1.67 3/456 6377
/proc/meminfo: memFree=19625008/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=294136 CPUtime=2.38 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 14389 936 1 0 225 13 0 0 21 0 2 0 748973905 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6368/statm: 73534 8424 1877 364 0 9452 0
[pid=6375] ppid=6368 vsize=284984 CPUtime=0.01 cores=0,2,4,6
/proc/6375/stat : 6375 (python) S 6368 6368 6138 0 -1 4202560 875 0 0 0 0 1 0 0 18 0 1 0 748974168 291823616 6922 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414159 0 0 16781312 514 9321484487353139460 0 0 17 4 0 0 0
/proc/6375/statm: 71246 6922 378 364 0 7164 0
[pid=6377] ppid=6375 vsize=1020308 CPUtime=10.06 cores=0,2,4,6
/proc/6377/stat : 6377 (gnovelty+2) R 6375 6377 6377 0 -1 4202496 1157 0 0 0 1006 0 0 0 25 0 1 0 748974168 1044795392 892 33554432000 134512640 135663704 4293971952 18446744073709551615 134525684 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6377/statm: 255077 892 113 282 0 254793 0
[pid=6368/tid=6376] ppid=6366 vsize=294136 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 2 0 748974168 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12.45
Current children cumulated vsize (KiB) 1599428

[startup+25.501 s]
/proc/loadavg: 2.00 2.00 1.67 3/456 6377
/proc/meminfo: memFree=19626100/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=294136 CPUtime=2.38 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 14389 936 1 0 225 13 0 0 21 0 2 0 748973905 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6368/statm: 73534 8424 1877 364 0 9452 0
[pid=6375] ppid=6368 vsize=284984 CPUtime=0.06 cores=0,2,4,6
/proc/6375/stat : 6375 (python) S 6368 6368 6138 0 -1 4202560 882 0 0 0 3 3 0 0 18 0 1 0 748974168 291823616 6922 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414159 0 0 16781312 514 16151210085517852932 0 0 17 4 0 0 0
/proc/6375/statm: 71246 6922 378 364 0 7164 0
[pid=6377] ppid=6375 vsize=1020308 CPUtime=22.86 cores=0,2,4,6
/proc/6377/stat : 6377 (gnovelty+2) R 6375 6377 6377 0 -1 4202496 1157 0 0 0 2286 0 0 0 25 0 1 0 748974168 1044795392 892 33554432000 134512640 135663704 4293971952 18446744073709551615 134524916 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6377/statm: 255077 892 113 282 0 254793 0
[pid=6368/tid=6376] ppid=6366 vsize=294136 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 2 0 748974168 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 25.3
Current children cumulated vsize (KiB) 1599428

[startup+51.1003 s]
/proc/loadavg: 2.00 2.00 1.68 3/456 6378
/proc/meminfo: memFree=19627112/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=294136 CPUtime=2.38 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 14389 936 1 0 225 13 0 0 21 0 2 0 748973905 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6368/statm: 73534 8424 1877 364 0 9452 0
[pid=6375] ppid=6368 vsize=284984 CPUtime=0.14 cores=0,2,4,6
/proc/6375/stat : 6375 (python) S 6368 6368 6138 0 -1 4202560 893 0 0 0 7 7 0 0 18 0 1 0 748974168 291823616 6922 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414159 0 0 16781312 514 16151210085517852932 0 0 17 4 0 0 0
/proc/6375/statm: 71246 6922 378 364 0 7164 0
[pid=6377] ppid=6375 vsize=1020308 CPUtime=48.46 cores=0,2,4,6
/proc/6377/stat : 6377 (gnovelty+2) R 6375 6377 6377 0 -1 4202496 1157 0 0 0 4846 0 0 0 25 0 1 0 748974168 1044795392 892 33554432000 134512640 135663704 4293971952 18446744073709551615 134524888 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6377/statm: 255077 892 113 282 0 254793 0
[pid=6368/tid=6376] ppid=6366 vsize=294136 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 2 0 748974168 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.98
Current children cumulated vsize (KiB) 1599428

[startup+102.316 s]
/proc/loadavg: 2.00 2.00 1.70 3/456 6379
/proc/meminfo: memFree=19626080/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=294136 CPUtime=2.38 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 14389 936 1 0 225 13 0 0 21 0 2 0 748973905 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/6368/statm: 73534 8424 1877 364 0 9452 0
[pid=6375] ppid=6368 vsize=284984 CPUtime=0.29 cores=0,2,4,6
/proc/6375/stat : 6375 (python) S 6368 6368 6138 0 -1 4202560 910 0 0 0 15 14 0 0 18 0 1 0 748974168 291823616 6922 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414159 0 0 16781312 514 16151210085517852932 0 0 17 6 0 0 0
/proc/6375/statm: 71246 6922 378 364 0 7164 0
[pid=6377] ppid=6375 vsize=1020308 CPUtime=99.68 cores=0,2,4,6
/proc/6377/stat : 6377 (gnovelty+2) R 6375 6377 6377 0 -1 4202496 1157 0 0 0 9968 0 0 0 25 0 1 0 748974168 1044795392 892 33554432000 134512640 135663704 4293971952 18446744073709551615 134524572 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6377/statm: 255077 892 113 282 0 254793 0
[pid=6368/tid=6376] ppid=6366 vsize=294136 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 2 0 748974168 301195264 8424 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 102.35
Current children cumulated vsize (KiB) 1599428

[startup+162.301 s]
/proc/loadavg: 2.00 2.00 1.71 3/460 6385
/proc/meminfo: memFree=19617716/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=305000 CPUtime=2.41 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 15287 936 1 0 228 13 0 0 20 0 3 0 748973905 312320000 8566 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6368/statm: 76250 8566 1880 364 0 12165 0
[pid=6375] ppid=6368 vsize=295228 CPUtime=0.31 cores=0,2,4,6
/proc/6375/stat : 6375 (python) S 6368 6368 6138 0 -1 4202560 1072 0 0 0 16 15 0 0 19 0 2 0 748974168 302313472 6959 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6375/statm: 73807 6959 410 364 0 9725 0
[pid=6377] ppid=6375 vsize=1020308 CPUtime=100.35 cores=0,2,4,6
/proc/6377/stat : 6377 (gnovelty+2) T 6375 6377 6377 0 -1 4202496 1157 0 0 0 10035 0 0 0 25 0 1 0 748974168 1044795392 892 33554432000 134512640 135663704 4293971952 18446744073709551615 134525715 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6377/statm: 255077 892 113 282 0 254793 0
[pid=6375/tid=6380] ppid=6368 vsize=295228 CPUtime=0 cores=0,2,4,6
/proc/6375/task/6380/stat : 6380 (python) S 6368 6368 6138 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 748984204 302313472 6959 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604446493085888 0 0 -1 2 0 0 0
[pid=6381] ppid=6368 vsize=294984 CPUtime=0.18 cores=0,2,4,6
/proc/6381/stat : 6381 (python) S 6368 6368 6138 0 -1 4202560 918 0 0 0 10 8 0 0 18 0 1 0 748984207 302063616 7055 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414214 0 0 16781312 514 16151210085517852930 0 0 17 4 0 0 0
/proc/6381/statm: 73746 7055 372 364 0 9661 0
[pid=6383] ppid=6381 vsize=1020308 CPUtime=59.27 cores=0,2,4,6
/proc/6383/stat : 6383 (gnovelty+2) R 6381 6383 6383 0 -1 4202496 1156 0 0 0 5927 0 0 0 25 0 1 0 748984207 1044795392 892 33554432000 134512640 135663704 4293853856 18446744073709551615 134544186 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/6383/statm: 255077 892 113 282 0 254793 0
[pid=6368/tid=6376] ppid=6366 vsize=305000 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 3 0 748974168 312320000 8566 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
[pid=6368/tid=6382] ppid=6366 vsize=305000 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6382/stat : 6382 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 3 0 748984207 312320000 8566 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 162.52
Current children cumulated vsize (KiB) 2935828

[startup+222.301 s]
/proc/loadavg: 2.09 2.03 1.74 3/468 6394
/proc/meminfo: memFree=19566248/32950928 swapFree=65730892/67111528
[pid=6368] ppid=6366 vsize=315256 CPUtime=2.44 cores=0,2,4,6
/proc/6368/stat : 6368 (python) S 6366 6368 6138 0 -1 4202496 16136 936 1 0 231 13 0 0 20 0 4 0 748973905 322822144 8572 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0

################
# More data... #
################

/proc/6405/statm: 105850 2356 59 146 0 104726 0
[pid=6403/tid=6408] ppid=6368 vsize=315672 CPUtime=0 cores=0,2,4,6
/proc/6403/task/6408/stat : 6408 (python) S 6368 6368 6138 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749014116 323248128 7092 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604453984475840 0 0 -1 6 0 0 0
[pid=6409] ppid=6368 vsize=326120 CPUtime=0.33 cores=0,2,4,6
/proc/6409/stat : 6409 (python) S 6368 6368 6138 0 -1 4202560 1112 0 0 0 18 15 0 0 19 0 2 0 749014119 333946880 7096 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/6409/statm: 81530 7096 400 364 0 17436 0
[pid=6411] ppid=6409 vsize=423400 CPUtime=100.2 cores=0,2,4,6
/proc/6411/stat : 6411 (adaptg2wsat2009) T 6409 6411 6411 0 -1 4202496 2624 0 0 0 10020 0 0 0 25 0 1 0 749014120 433561600 2358 33554432000 134512640 135107411 4289081664 18446744073709551615 134532291 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/6411/statm: 105850 2358 59 146 0 104726 0
[pid=6409/tid=6419] ppid=6368 vsize=326120 CPUtime=0 cores=0,2,4,6
/proc/6409/task/6419/stat : 6419 (python) S 6368 6368 6138 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749024152 333946880 7096 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604446866492096 0 0 -1 6 0 0 0
[pid=6420] ppid=6368 vsize=336196 CPUtime=0.33 cores=0,2,4,6
/proc/6420/stat : 6420 (python) S 6368 6368 6138 0 -1 4202560 1120 0 0 0 17 16 0 0 19 0 2 0 749024156 344264704 7128 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6420/statm: 84049 7128 400 364 0 19952 0
[pid=6422] ppid=6420 vsize=1020308 CPUtime=99.37 cores=0,2,4,6
/proc/6422/stat : 6422 (gnovelty+2) T 6420 6422 6422 0 -1 4202496 1161 0 0 0 9937 0 0 0 25 0 1 0 749024156 1044795392 891 33554432000 134512640 135663704 4288611552 18446744073709551615 134524854 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6422/statm: 255077 891 113 282 0 254793 0
[pid=6420/tid=6426] ppid=6368 vsize=336196 CPUtime=0 cores=0,2,4,6
/proc/6420/task/6426/stat : 6426 (python) S 6368 6368 6138 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749034094 344264704 7128 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604451218808896 0 0 -1 6 0 0 0
[pid=6427] ppid=6368 vsize=346564 CPUtime=0.33 cores=0,2,4,6
/proc/6427/stat : 6427 (python) S 6368 6368 6138 0 -1 4202560 1180 0 0 0 18 15 0 0 19 0 2 0 749034098 354881536 7138 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6427/statm: 86641 7138 406 364 0 22541 0
[pid=6429] ppid=6427 vsize=253036 CPUtime=98.34 cores=0,2,4,6
/proc/6429/stat : 6429 (FH) T 6427 6429 6429 0 -1 4202496 2501 0 0 0 9834 0 0 0 25 0 1 0 749034098 259108864 2234 33554432000 134512640 134936988 4293953088 18446744073709551615 134547044 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6429/statm: 63259 2234 41 104 0 62763 0
[pid=6427/tid=6436] ppid=6368 vsize=346564 CPUtime=0 cores=0,2,4,6
/proc/6427/task/6436/stat : 6436 (python) S 6368 6368 6138 0 -1 4202560 102 0 0 0 0 0 0 0 19 0 2 0 749043932 354881536 7138 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604445752718400 0 0 -1 6 0 0 0
[pid=6437] ppid=6368 vsize=356724 CPUtime=0.33 cores=0,2,4,6
/proc/6437/stat : 6437 (python) S 6368 6368 6138 0 -1 4202560 1161 0 0 0 18 15 0 0 19 0 2 0 749043936 365285376 7142 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/6437/statm: 89181 7142 406 364 0 25078 0
[pid=6439] ppid=6437 vsize=423400 CPUtime=100.32 cores=0,2,4,6
/proc/6439/stat : 6439 (adaptg2wsat2009) T 6437 6439 6439 0 -1 4202496 2630 0 0 0 10032 0 0 0 25 0 1 0 749043936 433561600 2357 33554432000 134512640 135107411 4287860784 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/6439/statm: 105850 2357 59 146 0 104726 0
[pid=6437/tid=6447] ppid=6368 vsize=356724 CPUtime=0 cores=0,2,4,6
/proc/6437/task/6447/stat : 6447 (python) S 6368 6368 6138 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749053969 365285376 7142 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604446866490304 0 0 -1 6 0 0 0
[pid=6448] ppid=6368 vsize=367236 CPUtime=0.33 cores=0,2,4,6
/proc/6448/stat : 6448 (python) S 6368 6368 6138 0 -1 4202560 1128 0 0 0 18 15 0 0 19 0 2 0 749053973 376049664 7140 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6448/statm: 91809 7140 400 364 0 27703 0
[pid=6450] ppid=6448 vsize=423400 CPUtime=100.31 cores=0,2,4,6
/proc/6450/stat : 6450 (adaptg2wsat2009) T 6448 6450 6450 0 -1 4202496 2631 0 0 0 10031 0 0 0 25 0 1 0 749053973 433561600 2357 33554432000 134512640 135107411 4292338560 18446744073709551615 134532347 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6450/statm: 105850 2357 59 146 0 104726 0
[pid=6448/tid=6458] ppid=6368 vsize=367236 CPUtime=0 cores=0,2,4,6
/proc/6448/task/6458/stat : 6458 (python) S 6368 6368 6138 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749064006 376049664 7140 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604443468371136 0 0 -1 6 0 0 0
[pid=6459] ppid=6368 vsize=377328 CPUtime=0.34 cores=0,2,4,6
/proc/6459/stat : 6459 (python) S 6368 6368 6138 0 -1 4202560 1134 0 0 0 18 16 0 0 19 0 2 0 749064010 386383872 7167 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6459/statm: 94332 7167 400 364 0 30223 0
[pid=6461] ppid=6459 vsize=253032 CPUtime=99.33 cores=0,2,4,6
/proc/6461/stat : 6461 (FH) T 6459 6461 6461 0 -1 4202496 2506 0 0 0 9933 0 0 0 25 0 1 0 749064010 259104768 2234 33554432000 134512640 134936988 4289195648 18446744073709551615 134553566 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6461/statm: 63258 2234 41 104 0 62762 0
[pid=6459/tid=6468] ppid=6368 vsize=377328 CPUtime=0 cores=0,2,4,6
/proc/6459/task/6468/stat : 6468 (python) S 6368 6368 6138 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749073944 386383872 7167 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604452179344256 0 0 -1 6 0 0 0
[pid=6469] ppid=6368 vsize=387724 CPUtime=0.67 cores=0,2,4,6
/proc/6469/stat : 6469 (python) S 6368 6368 6138 0 -1 4202560 1150 0 0 0 35 32 0 0 19 0 2 0 749073948 397029376 7171 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6469/statm: 96931 7171 400 364 0 32819 0
[pid=6471] ppid=6469 vsize=268420 CPUtime=194.75 cores=0,2,4,6
/proc/6471/stat : 6471 (hybridGM3) T 6469 6471 6471 0 -1 4202496 6188 0 0 0 19475 0 0 0 25 0 1 0 749073949 274862080 5918 33554432000 134512640 135204385 4293611440 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6471/statm: 67105 5918 63 169 0 66935 0
[pid=6469/tid=6485] ppid=6368 vsize=387724 CPUtime=0 cores=0,2,4,6
/proc/6469/task/6485/stat : 6485 (python) S 6368 6368 6138 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749093425 397029376 7171 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604449015814272 0 0 -1 6 0 0 0
[pid=6486] ppid=6368 vsize=463668 CPUtime=0.34 cores=0,2,4,6
/proc/6486/stat : 6486 (python) S 6368 6368 6138 0 -1 4202560 1136 0 0 0 19 15 0 0 19 0 2 0 749093428 474796032 7177 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6486/statm: 115917 7177 400 364 0 51802 0
[pid=6488] ppid=6486 vsize=1020308 CPUtime=99.39 cores=0,2,4,6
/proc/6488/stat : 6488 (gnovelty+2) T 6486 6488 6488 0 -1 4202496 1163 0 0 0 9939 0 0 0 25 0 1 0 749093429 1044795392 891 33554432000 134512640 135663704 4292565920 18446744073709551615 134525309 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6488/statm: 255077 891 113 282 0 254793 0
[pid=6486/tid=6496] ppid=6368 vsize=463668 CPUtime=0 cores=0,2,4,6
/proc/6486/task/6496/stat : 6496 (python) S 6368 6368 6138 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 749103369 474796032 7177 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604441328297984 0 0 -1 6 0 0 0
[pid=6497] ppid=6368 vsize=473828 CPUtime=0.69 cores=0,2,4,6
/proc/6497/stat : 6497 (python) S 6368 6368 6138 0 -1 4202560 1149 0 0 0 37 32 0 0 19 0 2 0 749103373 485199872 7213 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6497/statm: 118457 7213 400 364 0 54339 0
[pid=6499] ppid=6497 vsize=268424 CPUtime=194.81 cores=0,2,4,6
/proc/6499/stat : 6499 (hybridGM3) T 6497 6499 6499 0 -1 4202496 6192 0 0 0 19480 1 0 0 25 0 1 0 749103373 274866176 5920 33554432000 134512640 135204385 4294595840 18446744073709551615 134537704 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6499/statm: 67106 5920 63 169 0 66936 0
[pid=6497/tid=6509] ppid=6368 vsize=473828 CPUtime=0 cores=0,2,4,6
/proc/6497/task/6509/stat : 6509 (python) S 6368 6368 6138 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749122855 485199872 7213 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604443468372032 0 0 -1 6 0 0 0
[pid=6510] ppid=6368 vsize=484416 CPUtime=0.36 cores=0,2,4,6
/proc/6510/stat : 6510 (python) S 6368 6368 6138 0 -1 4202560 1156 0 0 0 19 17 0 0 18 0 2 0 749122859 496041984 7217 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6510/statm: 121104 7217 400 364 0 56983 0
[pid=6512] ppid=6510 vsize=423400 CPUtime=99.34 cores=0,2,4,6
/proc/6512/stat : 6512 (adaptg2wsat2009) T 6510 6512 6512 0 -1 4202496 2629 0 0 0 9934 0 0 0 25 0 1 0 749122859 433561600 2358 33554432000 134512640 135107411 4289736704 18446744073709551615 134531582 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6512/statm: 105850 2358 59 146 0 104726 0
[pid=6510/tid=6520] ppid=6368 vsize=484416 CPUtime=0 cores=0,2,4,6
/proc/6510/task/6520/stat : 6520 (python) S 6368 6368 6138 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 749132795 496041984 7217 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604453769347520 0 0 -1 6 0 0 0
[pid=6521] ppid=6368 vsize=494344 CPUtime=0.72 cores=0,2,4,6
/proc/6521/stat : 6521 (python) S 6368 6368 6138 0 -1 4202560 1215 0 0 0 40 32 0 0 19 0 2 0 749132799 506208256 7287 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6521/statm: 123586 7287 407 364 0 59462 0
[pid=6523] ppid=6521 vsize=268420 CPUtime=195.79 cores=0,2,4,6
/proc/6523/stat : 6523 (hybridGM3) T 6521 6523 6523 0 -1 4202496 6192 0 0 0 19578 1 0 0 25 0 1 0 749132800 274862080 5918 33554432000 134512640 135204385 4293299248 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6523/statm: 67105 5918 63 169 0 66935 0
[pid=6521/tid=6536] ppid=6368 vsize=494344 CPUtime=0 cores=0,2,4,6
/proc/6521/task/6536/stat : 6536 (python) S 6368 6368 6138 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749152381 506208256 7287 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604442280374592 0 0 -1 6 0 0 0
[pid=6537] ppid=6368 vsize=504960 CPUtime=0.75 cores=0,2,4,6
/proc/6537/stat : 6537 (python) S 6368 6368 6138 0 -1 4202560 1227 0 0 0 40 35 0 0 19 0 2 0 749152385 517079040 7291 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6537/statm: 126240 7291 407 364 0 62113 0
[pid=6539] ppid=6537 vsize=268424 CPUtime=195.91 cores=0,2,4,6
/proc/6539/stat : 6539 (hybridGM3) T 6537 6539 6539 0 -1 4202496 6194 0 0 0 19590 1 0 0 25 0 1 0 749152386 274866176 5918 33554432000 134512640 135204385 4290546592 18446744073709551615 134537724 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6539/statm: 67106 5918 63 169 0 66936 0
[pid=6537/tid=6545] ppid=6368 vsize=504960 CPUtime=0 cores=0,2,4,6
/proc/6537/task/6545/stat : 6545 (python) S 6368 6368 6138 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 749171977 517079040 7291 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604447093065792 0 0 -1 0 0 0 0
[pid=6546] ppid=6368 vsize=515164 CPUtime=5.76 cores=0,2,4,6
/proc/6546/stat : 6546 (python) S 6368 6368 6138 0 -1 4202560 1214 0 0 0 301 275 0 0 19 0 2 0 749171981 527527936 7297 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6546/statm: 128791 7297 401 364 0 64661 0
[pid=6548] ppid=6546 vsize=461504 CPUtime=1555.23 cores=0,2,4,6
/proc/6548/stat : 6548 (TNM) T 6546 6548 6548 0 -1 4202496 2473 0 0 0 155522 1 0 0 25 0 1 0 749171981 472580096 2194 33554432000 134512640 135008748 4287225616 18446744073709551615 134528974 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6548/statm: 115376 2194 46 122 0 114276 0
[pid=6546/tid=6600] ppid=6368 vsize=515164 CPUtime=0 cores=0,2,4,6
/proc/6546/task/6600/stat : 6600 (python) S 6368 6368 6138 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 749327501 527527936 7297 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 514 18446604444659176448 0 0 -1 2 0 0 0
[pid=6601] ppid=6368 vsize=525640 CPUtime=0 cores=0,2,4,6
/proc/6601/stat : 6601 (python) S 6368 6368 6138 0 -1 4202560 829 0 0 0 0 0 0 0 19 0 1 0 749327503 538255360 7245 33554432000 4194304 5683644 140733971424320 18446744073709551615 260905414214 0 0 16781312 514 11651115430633799679 0 0 17 0 0 0 0
/proc/6601/statm: 131410 7245 346 364 0 67277 0
[pid=6603] ppid=6601 vsize=248616 CPUtime=0.43 cores=0,2,4,6
/proc/6603/stat : 6603 (FH) R 6601 6603 6603 0 -1 4202496 1489 0 0 0 43 0 0 0 21 0 1 0 749327504 254582784 1214 33554432000 134512640 134936988 4286801632 18446744073709551615 134523400 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/6603/statm: 62154 1214 37 104 0 61658 0
[pid=6368/tid=6376] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6376/stat : 6376 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 21 0 19 0 748974168 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
[pid=6368/tid=6382] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6382/stat : 6382 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 748984207 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 2 0 0 0
[pid=6368/tid=6393] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6393/stat : 6393 (python) S 6366 6368 6138 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 748994147 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 2 0 0 0
[pid=6368/tid=6404] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6404/stat : 6404 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749004183 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 4 0 0 0
[pid=6368/tid=6410] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6410/stat : 6410 (python) S 6366 6368 6138 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 749014120 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 4 0 0 0
[pid=6368/tid=6421] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6421/stat : 6421 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749024156 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
[pid=6368/tid=6428] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6428/stat : 6428 (python) S 6366 6368 6138 0 -1 4202560 26 936 0 0 0 0 0 0 20 0 19 0 749034098 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 4 0 0 0
[pid=6368/tid=6438] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6438/stat : 6438 (python) S 6366 6368 6138 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 749043936 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 4 0 0 0
[pid=6368/tid=6449] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6449/stat : 6449 (python) S 6366 6368 6138 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 19 0 749053973 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 4 0 0 0
[pid=6368/tid=6460] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6460/stat : 6460 (python) S 6366 6368 6138 0 -1 4202560 24 936 0 0 0 0 0 0 20 0 19 0 749064010 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 4 0 0 0
[pid=6368/tid=6470] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6470/stat : 6470 (python) S 6366 6368 6138 0 -1 4202560 22 936 0 0 0 0 0 0 20 0 19 0 749073948 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
[pid=6368/tid=6487] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6487/stat : 6487 (python) S 6366 6368 6138 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749093429 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 0 0 0 0
[pid=6368/tid=6498] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6498/stat : 6498 (python) S 6366 6368 6138 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749103373 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 0 0 0 0
[pid=6368/tid=6511] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6511/stat : 6511 (python) S 6366 6368 6138 0 -1 4202560 28 936 0 0 0 0 0 0 20 0 19 0 749122859 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 0 0 0 0
[pid=6368/tid=6522] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6522/stat : 6522 (python) S 6366 6368 6138 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 19 0 749132799 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 0 0 0 0
[pid=6368/tid=6538] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6538/stat : 6538 (python) S 6366 6368 6138 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 19 0 749152385 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 6 0 0 0
[pid=6368/tid=6547] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6547/stat : 6547 (python) S 6366 6368 6138 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 19 0 749171981 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 0 0 0 0
[pid=6368/tid=6602] ppid=6366 vsize=535200 CPUtime=0 cores=0,2,4,6
/proc/6368/task/6602/stat : 6602 (python) S 6366 6368 6138 0 -1 4202560 29 936 0 0 0 0 0 0 20 0 19 0 749327503 548044800 8814 33554432000 4194304 5683644 140733971424320 18446744073709551615 260917217713 0 0 16781312 2 18446604447093064896 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3548.43
Current children cumulated vsize (KiB) 16617828

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6368 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=844567
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=191970
# CPU time returned by wait4() is 3.03654
# while last known CPU time is 3548.43
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 3536.54
CPU time (s): 3548.43
CPU user time (s): 3542.34
CPU system time (s): 6.09
CPU usage (%): 100.336
Max. virtual memory (cumulated for all children) (KiB): 16617828

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.84457
system time used= 0.19197
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 31829
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2924
involuntary context switches= 48

runsolver used 31.8272 second user time and 66.9758 second system time

The end

Launcher Data 

Begin job on node116 at 2011-05-17 01:10:32
IDJOB=3386571
IDBENCH=82636
IDSOLVER=1872
FILE ID=node116/3386571-1305587432
RUNJOBID= node116-1305586147-6156
PBS_JOBID= 13350362
Free space on /tmp= 72968 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S971556279-001-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386571-1305587432/watcher-3386571-1305587432 -o /tmp/evaluation-result-3386571-1305587432/solver-3386571-1305587432 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386571-1305587432.cnf -seed 1675891500 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9224277e61359574518ca553717e02ee
RANDOM SEED=1675891500

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19661732 kB
Buffers:        188472 kB
Cached:        2462848 kB
SwapCached:      12328 kB
Active:       11773072 kB
Inactive:      1268960 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19661732 kB
SwapTotal:    67111528 kB
SwapFree:     65730892 kB
Dirty:           32620 kB
Writeback:           4 kB
AnonPages:    10389004 kB
Mapped:          26124 kB
Slab:           119584 kB
PageTables:      59508 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 116469588 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72864 MiB
End job on node116 at 2011-05-17 02:09:31