Trace number 3386626

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 3556.4 3542.04

General information on the benchmark

Namerandom/large/
unif-k7-r85-v400-c34000-S94779963-093-UNKNOWN.cnf
MD5SUMaef8bd4e13046b42a5bc777dc56e10de
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses34000
Sum of the clauses size238000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 534000

Solver Data 

0.39/0.64	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/0.64	c INFO: solving HOME/instance-3386626-1305589679.cnf
0.39/0.64	c INFO: computing task features
0.58/0.87	c INFO: computed variable-clause graph statistics
0.58/0.88	c INFO: computed clause balance statistics
1.77/2.11	c INFO: computed variable balance statistics
2.27/2.59	c INFO: computed small-clause counts
2.37/2.60	c INFO: computed Horn-clause counts
2.37/2.64	c INFO: computed variable graph statistics
2.37/2.64	c INFO: collected features for HOME/instance-3386626-1305589679.cnf in 1.99 s
2.37/2.67	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.38)
102.15/102.17	c INFO: running gnovelty+2@0 for 97 with 4898 remaining (b = 0.13)
203.04/202.66	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
301.95/301.12	c INFO: running adaptg2wsat2009++@0 for 97 with 4699 remaining (b = 0.07)
402.76/401.55	c INFO: running adaptg2wsat2009++@0 for 97 with 4599 remaining (b = 0.05)
502.64/501.00	c INFO: running gnovelty+2@0 for 97 with 4499 remaining (b = 0.04)
602.54/600.52	c INFO: running FH@0 for 97 with 4400 remaining (b = 0.03)
701.40/699.00	c INFO: running adaptg2wsat2009++@0 for 97 with 4301 remaining (b = 0.03)
800.18/797.48	c INFO: running adaptg2wsat2009++@0 for 97 with 4203 remaining (b = 0.02)
900.06/896.95	c INFO: running FH@0 for 97 with 4103 remaining (b = 0.02)
998.99/995.44	c INFO: running hybridGM3@0 for 194 with 4005 remaining (b = 0.04)
1197.81/1193.43	c INFO: running gnovelty+2@0 for 97 with 3807 remaining (b = 0.02)
1297.67/1292.95	c INFO: running hybridGM3@0 for 194 with 3708 remaining (b = 0.03)
1496.52/1490.96	c INFO: running adaptg2wsat2009++@0 for 97 with 3510 remaining (b = 0.01)
1595.47/1589.43	c INFO: running hybridGM3@0 for 194 with 3411 remaining (b = 0.03)
1796.28/1789.43	c INFO: running hybridGM3@0 for 194 with 3211 remaining (b = 0.02)
1993.15/1985.43	c INFO: running TNM@0 for 1552 with 3015 remaining (b = 0.18)
3555.54/3541.20	c INFO: running FH@0 for 97 with 1459 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386626-1305589679/watcher-3386626-1305589679 -o /tmp/evaluation-result-3386626-1305589679/solver-3386626-1305589679 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386626-1305589679.cnf -seed 1440247826 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.26 2.12 2.03 2/584 21655
/proc/meminfo: memFree=27715112/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=16244 CPUtime=0 cores=1,3,5,7
/proc/21655/stat : 21655 (python) R 21653 21655 21157 0 -1 4202496 1050 0 0 0 0 0 0 0 17 0 1 0 670566639 16633856 922 33554432000 4194304 5683644 140736143512576 18446744073709551615 251813934031 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/21655/statm: 4061 922 381 364 0 548 0

[startup+0.051909 s]
/proc/loadavg: 2.26 2.12 2.03 2/584 21655
/proc/meminfo: memFree=27715112/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=23188 CPUtime=0.03 cores=1,3,5,7
/proc/21655/stat : 21655 (python) R 21653 21655 21157 0 -1 4202496 1331 0 0 0 2 1 0 0 17 0 1 0 670566639 23744512 1179 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814238208 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/21655/statm: 5797 1179 432 364 0 724 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 23188

[startup+0.100899 s]
/proc/loadavg: 2.26 2.12 2.03 2/584 21655
/proc/meminfo: memFree=27715112/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=89772 CPUtime=0.06 cores=1,3,5,7
/proc/21655/stat : 21655 (python) R 21653 21655 21157 0 -1 4202496 2638 0 0 0 4 2 0 0 17 0 1 0 670566639 91926528 2354 33554432000 4194304 5683644 140736143512576 18446744073709551615 251809316967 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/21655/statm: 22443 2356 718 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89772

[startup+0.30078 s]
/proc/loadavg: 2.26 2.12 2.03 2/584 21655
/proc/meminfo: memFree=27715112/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=135752 CPUtime=0.2 cores=1,3,5,7
/proc/21655/stat : 21655 (python) R 21653 21655 21157 0 -1 4202496 5972 935 1 0 15 5 0 0 17 0 1 0 670566639 139010048 4149 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814236165 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/21655/statm: 33938 4149 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 135752

[startup+0.700712 s]
/proc/loadavg: 2.26 2.12 2.03 2/584 21655
/proc/meminfo: memFree=27715112/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=281260 CPUtime=0.48 cores=1,3,5,7
/proc/21655/stat : 21655 (python) R 21653 21655 21157 0 -1 4202496 9971 935 1 0 36 12 0 0 16 0 1 0 670566639 288010240 7642 33554432000 4194304 5683644 140736143512576 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/21655/statm: 70315 7642 1761 364 0 6239 0
Current children cumulated CPU time (s) 0.48
Current children cumulated vsize (KiB) 281260

[startup+1.50058 s]
/proc/loadavg: 2.26 2.12 2.03 3/585 21660
/proc/meminfo: memFree=27680508/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=289492 CPUtime=1.27 cores=1,3,5,7
/proc/21655/stat : 21655 (python) R 21653 21655 21157 0 -1 4202496 13892 935 1 0 115 12 0 0 16 0 1 0 670566639 296439808 9757 33554432000 4194304 5683644 140736143512576 18446744073709551615 4525725 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/21655/statm: 72373 9757 1826 364 0 8297 0
Current children cumulated CPU time (s) 1.27
Current children cumulated vsize (KiB) 289492

[startup+3.10031 s]
/proc/loadavg: 2.26 2.12 2.03 3/585 21660
/proc/meminfo: memFree=27679392/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=291924 CPUtime=2.44 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 14214 935 1 0 232 12 0 0 20 0 2 0 670566639 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/21655/statm: 72981 7853 1877 364 0 8899 0
Current children cumulated CPU time (s) 2.44
Current children cumulated vsize (KiB) 291924
heavy processes:

[startup+6.30077 s]
/proc/loadavg: 2.24 2.12 2.03 3/588 21663
/proc/meminfo: memFree=27680112/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=291924 CPUtime=2.44 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 14214 935 1 0 232 12 0 0 20 0 2 0 670566639 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/21655/statm: 72981 7853 1877 364 0 8899 0
[pid=21661] ppid=21655 vsize=282232 CPUtime=0 cores=1,3,5,7
/proc/21661/stat : 21661 (python) S 21655 21655 21157 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 670566907 289005568 6328 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260239 0 0 16781312 514 1718792040084373766 0 0 17 5 0 0 0
/proc/21661/statm: 70558 6328 355 364 0 6476 0
[pid=21663] ppid=21661 vsize=1020188 CPUtime=3.61 cores=1,3,5,7
/proc/21663/stat : 21663 (gnovelty+2) R 21661 21663 21663 0 -1 4202496 1140 0 0 0 361 0 0 0 25 0 1 0 670566907 1044672512 877 33554432000 134512640 135663704 4288062416 18446744073709551615 134525316 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21663/statm: 255047 877 113 282 0 254763 0
[pid=21655/tid=21662] ppid=21653 vsize=291924 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 2 0 670566907 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 6.05
Current children cumulated vsize (KiB) 1594344

[startup+12.7007 s]
/proc/loadavg: 2.22 2.11 2.03 3/588 21663
/proc/meminfo: memFree=27679244/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=291924 CPUtime=2.44 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 14214 935 1 0 232 12 0 0 20 0 2 0 670566639 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/21655/statm: 72981 7853 1877 364 0 8899 0
[pid=21661] ppid=21655 vsize=282232 CPUtime=0.01 cores=1,3,5,7
/proc/21661/stat : 21661 (python) S 21655 21655 21157 0 -1 4202560 877 0 0 0 1 0 0 0 18 0 1 0 670566907 289005568 6351 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260239 0 0 16781312 514 1718792040084373766 0 0 17 5 0 0 0
/proc/21661/statm: 70558 6351 378 364 0 6476 0
[pid=21663] ppid=21661 vsize=1020188 CPUtime=10 cores=1,3,5,7
/proc/21663/stat : 21663 (gnovelty+2) R 21661 21663 21663 0 -1 4202496 1140 0 0 0 1000 0 0 0 25 0 1 0 670566907 1044672512 877 33554432000 134512640 135663704 4288062416 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21663/statm: 255047 877 113 282 0 254763 0
[pid=21655/tid=21662] ppid=21653 vsize=291924 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 2 0 670566907 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.45
Current children cumulated vsize (KiB) 1594344

[startup+25.5006 s]
/proc/loadavg: 2.17 2.11 2.03 3/588 21663
/proc/meminfo: memFree=27680616/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=291924 CPUtime=2.44 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 14214 935 1 0 232 12 0 0 20 0 2 0 670566639 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/21655/statm: 72981 7853 1877 364 0 8899 0
[pid=21661] ppid=21655 vsize=282232 CPUtime=0.09 cores=1,3,5,7
/proc/21661/stat : 21661 (python) S 21655 21655 21157 0 -1 4202560 890 0 0 0 4 5 0 0 18 0 1 0 670566907 289005568 6351 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260239 0 0 16781312 514 6648825977998639366 0 0 17 5 0 0 0
/proc/21661/statm: 70558 6351 378 364 0 6476 0
[pid=21663] ppid=21661 vsize=1020188 CPUtime=22.78 cores=1,3,5,7
/proc/21663/stat : 21663 (gnovelty+2) R 21661 21663 21663 0 -1 4202496 1140 0 0 0 2278 0 0 0 25 0 1 0 670566907 1044672512 877 33554432000 134512640 135663704 4288062416 18446744073709551615 134524526 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21663/statm: 255047 877 113 282 0 254763 0
[pid=21655/tid=21662] ppid=21653 vsize=291924 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 2 0 670566907 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.31
Current children cumulated vsize (KiB) 1594344

[startup+51.1002 s]
/proc/loadavg: 2.11 2.10 2.03 3/588 21664
/proc/meminfo: memFree=27680644/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=291924 CPUtime=2.44 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 14214 935 1 0 232 12 0 0 20 0 2 0 670566639 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/21655/statm: 72981 7853 1877 364 0 8899 0
[pid=21661] ppid=21655 vsize=282232 CPUtime=0.18 cores=1,3,5,7
/proc/21661/stat : 21661 (python) S 21655 21655 21157 0 -1 4202560 902 0 0 0 8 10 0 0 18 0 1 0 670566907 289005568 6351 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260239 0 0 16781312 514 6648825977998639366 0 0 17 5 0 0 0
/proc/21661/statm: 70558 6351 378 364 0 6476 0
[pid=21663] ppid=21661 vsize=1020188 CPUtime=48.36 cores=1,3,5,7
/proc/21663/stat : 21663 (gnovelty+2) R 21661 21663 21663 0 -1 4202496 1140 0 0 0 4836 0 0 0 25 0 1 0 670566907 1044672512 877 33554432000 134512640 135663704 4288062416 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/21663/statm: 255047 877 113 282 0 254763 0
[pid=21655/tid=21662] ppid=21653 vsize=291924 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 2 0 670566907 298930176 7853 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.98
Current children cumulated vsize (KiB) 1594344

[startup+102.321 s]
/proc/loadavg: 2.16 2.12 2.04 3/592 21669
/proc/meminfo: memFree=27676920/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=302340 CPUtime=2.48 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 15053 935 1 0 235 13 0 0 19 0 3 0 670566639 309596160 7900 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/21655/statm: 75585 7900 1880 364 0 11500 0
[pid=21661] ppid=21655 vsize=292476 CPUtime=0.4 cores=1,3,5,7
/proc/21661/stat : 21661 (python) S 21655 21655 21157 0 -1 4202560 1082 0 0 0 20 20 0 0 19 0 2 0 670566907 299495424 6388 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21661/statm: 73119 6388 410 364 0 9037 0
[pid=21663] ppid=21661 vsize=1020188 CPUtime=99.37 cores=1,3,5,7
/proc/21663/stat : 21663 (gnovelty+2) T 21661 21663 21663 0 -1 4202496 1140 0 0 0 9937 0 0 0 25 0 1 0 670566907 1044672512 877 33554432000 134512640 135663704 4288062416 18446744073709551615 134524882 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21663/statm: 255047 877 113 282 0 254763 0
[pid=21661/tid=21666] ppid=21655 vsize=292476 CPUtime=0 cores=1,3,5,7
/proc/21661/task/21666/stat : 21666 (python) S 21655 21655 21157 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670576853 299495424 6388 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604470492749888 0 0 -1 3 0 0 0
[pid=21667] ppid=21655 vsize=293120 CPUtime=0 cores=1,3,5,7
/proc/21667/stat : 21667 (python) S 21655 21655 21157 0 -1 4202560 742 0 0 0 0 0 0 0 18 0 1 0 670576856 300154880 6366 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260294 0 0 16781312 514 5190723205647564799 0 0 17 5 0 0 0
/proc/21667/statm: 73280 6366 349 364 0 9195 0
[pid=21669] ppid=21667 vsize=1020188 CPUtime=0.14 cores=1,3,5,7
/proc/21669/stat : 21669 (gnovelty+2) R 21667 21669 21669 0 -1 4202496 1139 0 0 0 14 0 0 0 20 0 1 0 670576857 1044672512 876 33554432000 134512640 135663704 4291075200 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21669/statm: 255047 876 113 282 0 254763 0
[pid=21655/tid=21662] ppid=21653 vsize=302340 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 3 0 670566907 309596160 7900 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
[pid=21655/tid=21668] ppid=21653 vsize=302340 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21668/stat : 21668 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 3 0 670576856 309596160 7900 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 102.39
Current children cumulated vsize (KiB) 2928312

[startup+162.301 s]
/proc/loadavg: 2.10 2.11 2.04 3/592 21671
/proc/meminfo: memFree=27672652/32950928 swapFree=41934280/67111528
[pid=21655] ppid=21653 vsize=302340 CPUtime=2.48 cores=1,3,5,7
/proc/21655/stat : 21655 (python) S 21653 21655 21157 0 -1 4202496 15053 935 1 0 235 13 0 0 19 0 3 0 670566639 309596160 7900 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/21655/statm: 75585 7900 1880 364 0 11500 0
[pid=21661] ppid=21655 vsize=292476 CPUtime=0.4 cores=1,3,5,7
/proc/21661/stat : 21661 (python) S 21655 21655 21157 0 -1 4202560 1082 0 0 0 20 20 0 0 19 0 2 0 670566907 299495424 6388 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21661/statm: 73119 6388 410 364 0 9037 0
[pid=21663] ppid=21661 vsize=1020188 CPUtime=99.37 cores=1,3,5,7
/proc/21663/stat : 21663 (gnovelty+2) T 21661 21663 21663 0 -1 4202496 1140 0 0 0 9937 0 0 0 25 0 1 0 670566907 1044672512 877 33554432000 134512640 135663704 4288062416 18446744073709551615 134524882 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21663/statm: 255047 877 113 282 0 254763 0
[pid=21661/tid=21666] ppid=21655 vsize=292476 CPUtime=0 cores=1,3,5,7
/proc/21661/task/21666/stat : 21666 (python) S 21655 21655 21157 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670576853 299495424 6388 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604470492749888 0 0 -1 3 0 0 0
[pid=21667] ppid=21655 vsize=293120 CPUtime=0.23 cores=1,3,5,7
/proc/21667/stat : 21667 (python) S 21655 21655 21157 0 -1 4202560 908 0 0 0 12 11 0 0 18 0 1 0 670576856 300154880 6389 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260294 0 0 16781312 514 6648825977998639364 0 0 17 5 0 0 0
/proc/21667/statm: 73280 6389 372 364 0 9195 0
[pid=21669] ppid=21667 vsize=1020188 CPUtime=60.12 cores=1,3,5,7
/proc/21669/stat : 21669 (gnovelty+2) R 21667 21669 21669 0 -1 4202496 1139 0 0 0 6012 0 0 0 25 0 1 0 670576857 1044672512 876 33554432000 134512640 135663704 4291075200 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/21669/statm: 255047 876 113 282 0 254763 0
[pid=21655/tid=21662] ppid=21653 vsize=302340 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 3 0 670566907 309596160 7900 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0

################
# More data... #
################

[pid=21721] ppid=21655 vsize=313316 CPUtime=0.4 cores=1,3,5,7
/proc/21721/stat : 21721 (python) S 21655 21655 21157 0 -1 4202560 1117 0 0 0 20 20 0 0 19 0 2 0 670596751 320835584 6561 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21721/statm: 78329 6561 400 364 0 14238 0
[pid=21723] ppid=21721 vsize=421804 CPUtime=100.39 cores=1,3,5,7
/proc/21723/stat : 21723 (adaptg2wsat2009) T 21721 21723 21723 0 -1 4202496 2138 0 0 0 10039 0 0 0 25 0 1 0 670596752 431927296 1873 33554432000 134512640 135107411 4288574000 18446744073709551615 134539287 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21723/statm: 105451 1873 59 146 0 104327 0
[pid=21721/tid=21765] ppid=21655 vsize=313316 CPUtime=0 cores=1,3,5,7
/proc/21721/task/21765/stat : 21765 (python) S 21655 21655 21157 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670606791 320835584 6561 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604470495965440 0 0 -1 7 0 0 0
[pid=21766] ppid=21655 vsize=323768 CPUtime=0.4 cores=1,3,5,7
/proc/21766/stat : 21766 (python) S 21655 21655 21157 0 -1 4202560 1124 0 0 0 19 21 0 0 19 0 2 0 670606794 331538432 6565 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21766/statm: 80942 6565 400 364 0 16848 0
[pid=21768] ppid=21766 vsize=421804 CPUtime=99.4 cores=1,3,5,7
/proc/21768/stat : 21768 (adaptg2wsat2009) T 21766 21768 21768 0 -1 4202496 2140 0 0 0 9940 0 0 0 25 0 1 0 670606795 431927296 1872 33554432000 134512640 135107411 4293326224 18446744073709551615 134531239 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21768/statm: 105451 1872 59 146 0 104327 0
[pid=21766/tid=21776] ppid=21655 vsize=323768 CPUtime=0 cores=1,3,5,7
/proc/21766/task/21776/stat : 21776 (python) S 21655 21655 21157 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670616736 331538432 6565 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604459403997376 0 0 -1 7 0 0 0
[pid=21777] ppid=21655 vsize=334128 CPUtime=0.4 cores=1,3,5,7
/proc/21777/stat : 21777 (python) S 21655 21655 21157 0 -1 4202560 1128 0 0 0 25 15 0 0 19 0 2 0 670616740 342147072 6569 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21777/statm: 83532 6569 400 364 0 19435 0
[pid=21779] ppid=21777 vsize=1020188 CPUtime=99.47 cores=1,3,5,7
/proc/21779/stat : 21779 (gnovelty+2) T 21777 21779 21779 0 -1 4202496 1143 0 0 0 9947 0 0 0 25 0 1 0 670616740 1044672512 877 33554432000 134512640 135663704 4292314704 18446744073709551615 134524548 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21779/statm: 255047 877 113 282 0 254763 0
[pid=21777/tid=21783] ppid=21655 vsize=334128 CPUtime=0 cores=1,3,5,7
/proc/21777/task/21783/stat : 21783 (python) S 21655 21655 21157 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670626688 342147072 6569 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604468308583616 0 0 -1 7 0 0 0
[pid=21784] ppid=21655 vsize=344496 CPUtime=0.4 cores=1,3,5,7
/proc/21784/stat : 21784 (python) S 21655 21655 21157 0 -1 4202560 1170 0 0 0 21 19 0 0 19 0 2 0 670626692 352763904 6579 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21784/statm: 86124 6579 406 364 0 22024 0
[pid=21786] ppid=21784 vsize=251288 CPUtime=98.43 cores=1,3,5,7
/proc/21786/stat : 21786 (FH) T 21784 21786 21786 0 -1 4202496 1957 0 0 0 9843 0 0 0 25 0 1 0 670626692 257318912 1693 33554432000 134512640 134936988 4287415152 18446744073709551615 134552073 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21786/statm: 62822 1693 41 104 0 62326 0
[pid=21784/tid=21824] ppid=21655 vsize=344496 CPUtime=0 cores=1,3,5,7
/proc/21784/task/21824/stat : 21824 (python) S 21655 21655 21157 0 -1 4202560 98 0 0 0 0 0 0 0 19 0 2 0 670636536 352763904 6579 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604461393178816 0 0 -1 7 0 0 0
[pid=21825] ppid=21655 vsize=354872 CPUtime=0.4 cores=1,3,5,7
/proc/21825/stat : 21825 (python) S 21655 21655 21157 0 -1 4202560 1173 0 0 0 20 20 0 0 19 0 2 0 670636540 363388928 6607 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21825/statm: 88718 6607 406 364 0 24615 0
[pid=21827] ppid=21825 vsize=421804 CPUtime=98.42 cores=1,3,5,7
/proc/21827/stat : 21827 (adaptg2wsat2009) T 21825 21827 21827 0 -1 4202496 2145 0 0 0 9842 0 0 0 25 0 1 0 670636540 431927296 1872 33554432000 134512640 135107411 4289157216 18446744073709551615 134531792 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21827/statm: 105451 1872 59 146 0 104327 0
[pid=21825/tid=21844] ppid=21655 vsize=354872 CPUtime=0 cores=1,3,5,7
/proc/21825/task/21844/stat : 21844 (python) S 21655 21655 21157 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670646384 363388928 6607 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604465186242944 0 0 -1 7 0 0 0
[pid=21845] ppid=21655 vsize=365256 CPUtime=0.41 cores=1,3,5,7
/proc/21845/stat : 21845 (python) S 21655 21655 21157 0 -1 4202560 1148 0 0 0 19 22 0 0 19 0 2 0 670646388 374022144 6635 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21845/statm: 91314 6635 400 364 0 27208 0
[pid=21847] ppid=21845 vsize=421804 CPUtime=99.42 cores=1,3,5,7
/proc/21847/stat : 21847 (adaptg2wsat2009) T 21845 21847 21847 0 -1 4202496 2140 0 0 0 9942 0 0 0 25 0 1 0 670646388 431927296 1871 33554432000 134512640 135107411 4293221728 18446744073709551615 134531781 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21847/statm: 105451 1871 59 146 0 104327 0
[pid=21845/tid=21855] ppid=21655 vsize=365256 CPUtime=0 cores=1,3,5,7
/proc/21845/task/21855/stat : 21855 (python) S 21655 21655 21157 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 670656331 374022144 6635 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604466659546240 0 0 -1 7 0 0 0
[pid=21856] ppid=21655 vsize=375648 CPUtime=0.41 cores=1,3,5,7
/proc/21856/stat : 21856 (python) S 21655 21655 21157 0 -1 4202560 1158 0 0 0 23 18 0 0 19 0 2 0 670656335 384663552 6671 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21856/statm: 93912 6671 400 364 0 29803 0
[pid=21858] ppid=21856 vsize=251292 CPUtime=98.43 cores=1,3,5,7
/proc/21858/stat : 21858 (FH) T 21856 21858 21858 0 -1 4202496 1959 0 0 0 9843 0 0 0 25 0 1 0 670656335 257323008 1694 33554432000 134512640 134936988 4292870880 18446744073709551615 134552411 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21858/statm: 62823 1694 41 104 0 62327 0
[pid=21856/tid=21868] ppid=21655 vsize=375648 CPUtime=0 cores=1,3,5,7
/proc/21856/task/21868/stat : 21868 (python) S 21655 21655 21157 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670666179 384663552 6671 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604466731906112 0 0 -1 7 0 0 0
[pid=21869] ppid=21655 vsize=386044 CPUtime=0.83 cores=1,3,5,7
/proc/21869/stat : 21869 (python) S 21655 21655 21157 0 -1 4202560 1166 0 0 0 46 37 0 0 19 0 2 0 670666184 395309056 6709 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21869/statm: 96511 6709 400 364 0 32399 0
[pid=21871] ppid=21869 vsize=268428 CPUtime=197.94 cores=1,3,5,7
/proc/21871/stat : 21871 (hybridGM3) T 21869 21871 21871 0 -1 4202496 5648 0 0 0 19794 0 0 0 25 0 1 0 670666184 274870272 5378 33554432000 134512640 135204385 4294866960 18446744073709551615 134538041 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21871/statm: 67107 5378 63 169 0 66937 0
[pid=21869/tid=21886] ppid=21655 vsize=386044 CPUtime=0 cores=1,3,5,7
/proc/21869/task/21886/stat : 21886 (python) S 21655 21655 21157 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670685979 395309056 6709 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604459449497792 0 0 -1 5 0 0 0
[pid=21887] ppid=21655 vsize=396184 CPUtime=0.42 cores=1,3,5,7
/proc/21887/stat : 21887 (python) S 21655 21655 21157 0 -1 4202560 1157 0 0 0 22 20 0 0 19 0 2 0 670685983 405692416 6748 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21887/statm: 99046 6748 400 364 0 34931 0
[pid=21889] ppid=21887 vsize=1020188 CPUtime=99.47 cores=1,3,5,7
/proc/21889/stat : 21889 (gnovelty+2) T 21887 21889 21889 0 -1 4202496 1151 0 0 0 9947 0 0 0 25 0 1 0 670685983 1044672512 877 33554432000 134512640 135663704 4293730272 18446744073709551615 134525222 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21889/statm: 255047 877 113 282 0 254763 0
[pid=21887/tid=21896] ppid=21655 vsize=396184 CPUtime=0 cores=1,3,5,7
/proc/21887/task/21896/stat : 21896 (python) S 21655 21655 21157 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670695931 405692416 6748 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604459449496896 0 0 -1 1 0 0 0
[pid=21897] ppid=21655 vsize=406596 CPUtime=0.85 cores=1,3,5,7
/proc/21897/stat : 21897 (python) S 21655 21655 21157 0 -1 4202560 1179 0 0 0 46 39 0 0 19 0 2 0 670695935 416354304 6752 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21897/statm: 101649 6752 400 364 0 37531 0
[pid=21899] ppid=21897 vsize=268424 CPUtime=197.95 cores=1,3,5,7
/proc/21899/stat : 21899 (hybridGM3) T 21897 21899 21899 0 -1 4202496 5648 0 0 0 19795 0 0 0 25 0 1 0 670695935 274866176 5377 33554432000 134512640 135204385 4292288320 18446744073709551615 134537792 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21899/statm: 67106 5377 63 169 0 66936 0
[pid=21897/tid=21910] ppid=21655 vsize=406596 CPUtime=0 cores=1,3,5,7
/proc/21897/task/21910/stat : 21910 (python) S 21655 21655 21157 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670715732 416354304 6752 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604455906848896 0 0 -1 7 0 0 0
[pid=21911] ppid=21655 vsize=416732 CPUtime=0.43 cores=1,3,5,7
/proc/21911/stat : 21911 (python) S 21655 21655 21157 0 -1 4202560 1159 0 0 0 21 22 0 0 19 0 2 0 670715735 426733568 6766 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21911/statm: 104183 6766 400 364 0 40062 0
[pid=21913] ppid=21911 vsize=421804 CPUtime=98.43 cores=1,3,5,7
/proc/21913/stat : 21913 (adaptg2wsat2009) T 21911 21913 21913 0 -1 4202496 2143 0 0 0 9843 0 0 0 25 0 1 0 670715736 431927296 1872 33554432000 134512640 135107411 4292904000 18446744073709551615 134532242 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21913/statm: 105451 1872 59 146 0 104327 0
[pid=21911/tid=21916] ppid=21655 vsize=416732 CPUtime=0 cores=1,3,5,7
/proc/21911/task/21916/stat : 21916 (python) S 21655 21655 21157 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 670725579 426733568 6766 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604455906850688 0 0 -1 7 0 0 0
[pid=21917] ppid=21655 vsize=427160 CPUtime=0.85 cores=1,3,5,7
/proc/21917/stat : 21917 (python) S 21655 21655 21157 0 -1 4202560 1235 0 0 0 44 41 0 0 19 0 2 0 670725583 437411840 6778 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21917/statm: 106790 6778 408 364 0 42666 0
[pid=21919] ppid=21917 vsize=268428 CPUtime=199.94 cores=1,3,5,7
/proc/21919/stat : 21919 (hybridGM3) T 21917 21919 21919 0 -1 4202496 5650 0 0 0 19994 0 0 0 25 0 1 0 670725583 274870272 5378 33554432000 134512640 135204385 4289009088 18446744073709551615 134537911 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/21919/statm: 67107 5378 63 169 0 66937 0
[pid=21917/tid=21926] ppid=21655 vsize=427160 CPUtime=0 cores=1,3,5,7
/proc/21917/task/21926/stat : 21926 (python) S 21655 21655 21157 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670745579 437411840 6778 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604462967285056 0 0 -1 7 0 0 0
[pid=21927] ppid=21655 vsize=437092 CPUtime=0.86 cores=1,3,5,7
/proc/21927/stat : 21927 (python) S 21655 21655 21157 0 -1 4202560 1218 0 0 0 43 43 0 0 16 0 2 0 670745583 447582208 6794 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/21927/statm: 109273 6794 407 364 0 45146 0
[pid=21929] ppid=21927 vsize=268420 CPUtime=195.95 cores=1,3,5,7
/proc/21929/stat : 21929 (hybridGM3) T 21927 21929 21929 0 -1 4202496 5652 0 0 0 19595 0 0 0 25 0 1 0 670745583 274862080 5377 33554432000 134512640 135204385 4290016464 18446744073709551615 134538041 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/21929/statm: 67105 5377 63 169 0 66935 0
[pid=21927/tid=21936] ppid=21655 vsize=437092 CPUtime=0 cores=1,3,5,7
/proc/21927/task/21936/stat : 21936 (python) S 21655 21655 21157 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 670765179 447582208 6794 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604456955484096 0 0 -1 5 0 0 0
[pid=21937] ppid=21655 vsize=447980 CPUtime=6.73 cores=1,3,5,7
/proc/21937/stat : 21937 (python) S 21655 21655 21157 0 -1 4202560 1207 0 0 0 331 342 0 0 19 0 2 0 670765183 458731520 6792 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826066891 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/21937/statm: 111995 6792 401 364 0 47865 0
[pid=21939] ppid=21937 vsize=459768 CPUtime=1555.75 cores=1,3,5,7
/proc/21939/stat : 21939 (TNM) T 21937 21939 21939 0 -1 4202496 1933 0 0 0 155575 0 0 0 25 0 1 0 670765183 470802432 1657 33554432000 134512640 135008748 4290506336 18446744073709551615 134527327 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/21939/statm: 114942 1657 46 122 0 113842 0
[pid=21937/tid=21993] ppid=21655 vsize=447980 CPUtime=0 cores=1,3,5,7
/proc/21937/task/21993/stat : 21993 (python) S 21655 21655 21157 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670920757 458731520 6792 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 514 18446604456955483200 0 0 -1 7 0 0 0
[pid=21994] ppid=21655 vsize=457876 CPUtime=0 cores=1,3,5,7
/proc/21994/stat : 21994 (python) S 21655 21655 21157 0 -1 4202560 826 0 0 0 0 0 0 0 19 0 1 0 670920759 468865024 6753 33554432000 4194304 5683644 140736143512576 18446744073709551615 251814260294 0 0 16781312 514 3641178994721685503 0 0 17 5 0 0 0
/proc/21994/statm: 114469 6753 346 364 0 50336 0
[pid=21996] ppid=21994 vsize=249144 CPUtime=0.73 cores=1,3,5,7
/proc/21996/stat : 21996 (FH) R 21994 21996 21996 0 -1 4202496 1567 0 0 0 73 0 0 0 22 0 1 0 670920760 255123456 1296 33554432000 134512640 134936988 4290427024 18446744073709551615 134523529 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/21996/statm: 62286 1296 37 104 0 61790 0
[pid=21655/tid=21662] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21662/stat : 21662 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 19 0 670566907 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
[pid=21655/tid=21668] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21668/stat : 21668 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 19 0 670576856 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 7 0 0 0
[pid=21655/tid=21675] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21675/stat : 21675 (python) S 21653 21655 21157 0 -1 4202560 24 935 0 0 0 0 0 0 20 0 19 0 670586906 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21722] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21722/stat : 21722 (python) S 21653 21655 21157 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 19 0 670596752 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21767] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21767/stat : 21767 (python) S 21653 21655 21157 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 19 0 670606795 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21778] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21778/stat : 21778 (python) S 21653 21655 21157 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 19 0 670616740 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21785] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21785/stat : 21785 (python) S 21653 21655 21157 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 19 0 670626692 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
[pid=21655/tid=21826] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21826/stat : 21826 (python) S 21653 21655 21157 0 -1 4202560 32 935 0 0 0 0 0 0 20 0 19 0 670636540 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 3 0 0 0
[pid=21655/tid=21846] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21846/stat : 21846 (python) S 21653 21655 21157 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 19 0 670646388 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 3 0 0 0
[pid=21655/tid=21857] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21857/stat : 21857 (python) S 21653 21655 21157 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 19 0 670656335 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 7 0 0 0
[pid=21655/tid=21870] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21870/stat : 21870 (python) S 21653 21655 21157 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 19 0 670666184 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 7 0 0 0
[pid=21655/tid=21888] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21888/stat : 21888 (python) S 21653 21655 21157 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 19 0 670685983 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 7 0 0 0
[pid=21655/tid=21898] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21898/stat : 21898 (python) S 21653 21655 21157 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 19 0 670695935 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21912] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21912/stat : 21912 (python) S 21653 21655 21157 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 19 0 670715735 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21918] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21918/stat : 21918 (python) S 21653 21655 21157 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 19 0 670725583 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21928] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21928/stat : 21928 (python) S 21653 21655 21157 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 19 0 670745583 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21938] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21938/stat : 21938 (python) S 21653 21655 21157 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 19 0 670765183 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 1 0 0 0
[pid=21655/tid=21995] ppid=21653 vsize=467740 CPUtime=0 cores=1,3,5,7
/proc/21655/task/21995/stat : 21995 (python) S 21653 21655 21157 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 19 0 670920760 478965760 8322 33554432000 4194304 5683644 140736143512576 18446744073709551615 251826063793 0 0 16781312 2 18446604465186244736 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3556.4
Current children cumulated vsize (KiB) 16040144

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21655 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=900559
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=177972
# CPU time returned by wait4() is 3.07853
# while last known CPU time is 3556.4
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 3542.04
CPU time (s): 3556.4
CPU user time (s): 3548.86
CPU system time (s): 7.54
CPU usage (%): 100.405
Max. virtual memory (cumulated for all children) (KiB): 16040144

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.90056
system time used= 0.177972
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 32767
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2616
involuntary context switches= 73

runsolver used 35.5346 second user time and 71.3951 second system time

The end

Launcher Data 

Begin job on node126 at 2011-05-17 01:47:59
IDJOB=3386626
IDBENCH=82691
IDSOLVER=1872
FILE ID=node126/3386626-1305589679
RUNJOBID= node126-1305586146-21176
PBS_JOBID= 13350352
Free space on /tmp= 73108 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v400-c34000-S94779963-093-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386626-1305589679/watcher-3386626-1305589679 -o /tmp/evaluation-result-3386626-1305589679/solver-3386626-1305589679 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386626-1305589679.cnf -seed 1440247826 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= aef8bd4e13046b42a5bc777dc56e10de
RANDOM SEED=1440247826

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      27715392 kB
Buffers:        233680 kB
Cached:        2580052 kB
SwapCached:      49648 kB
Active:        2323936 kB
Inactive:      2606456 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      27715392 kB
SwapTotal:    67111528 kB
SwapFree:     41934280 kB
Dirty:           32756 kB
Writeback:           0 kB
AnonPages:     2079312 kB
Mapped:          28000 kB
Slab:           127592 kB
PageTables:     106884 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 164842872 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73064 MiB
End job on node126 at 2011-05-17 02:47:03