Trace number 3386481

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 1994.16 1992.13

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S1286362348-008-UNKNOWN.cnf
MD5SUM63b66954174098a654eacd27832fcf38
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.51/3.77	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.51/3.77	c INFO: solving HOME/instance-3386481-1305586148.cnf
0.51/3.77	c INFO: computing task features
0.64/3.95	c INFO: computed variable-clause graph statistics
0.64/3.95	c INFO: computed clause balance statistics
1.55/4.85	c INFO: computed variable balance statistics
1.95/5.21	c INFO: computed small-clause counts
1.95/5.21	c INFO: computed Horn-clause counts
1.95/5.24	c INFO: computed variable graph statistics
1.95/5.24	c INFO: collected features for HOME/instance-3386481-1305586148.cnf in 1.46 s
1.95/5.28	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.37)
102.61/105.61	c INFO: running gnovelty+2@0 for 97 with 4898 remaining (b = 0.13)
202.14/204.97	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
300.70/303.29	c INFO: running adaptg2wsat2009++@0 for 97 with 4700 remaining (b = 0.07)
399.41/401.61	c INFO: running adaptg2wsat2009++@0 for 97 with 4602 remaining (b = 0.05)
499.98/501.92	c INFO: running gnovelty+2@0 for 97 with 4501 remaining (b = 0.04)
600.55/602.28	c INFO: running FH@0 for 97 with 4401 remaining (b = 0.03)
700.23/701.60	c INFO: running adaptg2wsat2009++@0 for 97 with 4302 remaining (b = 0.03)
800.67/801.93	c INFO: running adaptg2wsat2009++@0 for 97 with 4202 remaining (b = 0.02)
901.11/902.28	c INFO: running FH@0 for 97 with 4102 remaining (b = 0.02)
1001.84/1002.62	c INFO: running hybridGM3@0 for 194 with 4001 remaining (b = 0.04)
1198.10/1198.32	c INFO: running gnovelty+2@0 for 97 with 3806 remaining (b = 0.02)
1298.79/1298.71	c INFO: running hybridGM3@0 for 194 with 3705 remaining (b = 0.03)
1495.09/1494.41	c INFO: running adaptg2wsat2009++@0 for 97 with 3509 remaining (b = 0.01)
1595.63/1594.77	c INFO: running hybridGM3@0 for 194 with 3409 remaining (b = 0.03)
1792.91/1791.48	c INFO: running hybridGM3@0 for 194 with 3212 remaining (b = 0.02)
1993.35/1991.23	c INFO: running TNM@0 for 1552 with 3013 remaining (b = 0.19)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386481-1305586148/watcher-3386481-1305586148 -o /tmp/evaluation-result-3386481-1305586148/solver-3386481-1305586148 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386481-1305586148.cnf -seed 564721476 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.11 2.05 1/376 5735
/proc/meminfo: memFree=22051732/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=11796 CPUtime=0 cores=0,2,4,6
/proc/5735/stat : 5735 (runsolver) S 5733 5735 5707 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 1 0 670213666 12079104 52 33554432000 4194304 5432360 140736331845856 18446744073709551615 5055511 0 2147483391 4096 24578 18446744071699215550 0 0 17 2 0 0 0
/proc/5735/statm: 2949 52 33 303 0 2643 0

[startup+0.0788801 s]
/proc/loadavg: 0.00 0.11 2.05 1/376 5735
/proc/meminfo: memFree=22051732/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 705 0 6 0 0 0 0 0 20 0 1 0 670213666 15548416 603 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594864640 0 2147483391 16781312 2 18446744071699215550 0 0 17 2 0 0 2
/proc/5735/statm: 3796 603 319 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.100819 s]
/proc/loadavg: 0.00 0.11 2.05 1/376 5735
/proc/meminfo: memFree=22051732/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 718 0 6 0 0 0 0 0 20 0 1 0 670213666 15548416 615 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594864640 0 2147483391 16781312 2 18446744071699215550 0 0 17 2 0 0 2
/proc/5735/statm: 3796 615 321 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.300784 s]
/proc/loadavg: 0.00 0.11 2.05 1/376 5735
/proc/meminfo: memFree=22051732/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=16400 CPUtime=0.01 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 1115 0 6 0 1 0 0 0 18 0 1 0 670213666 16793600 989 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594862437 0 2147483391 16781312 2 18446744071699215550 0 0 17 2 0 0 5
/proc/5735/statm: 4100 989 413 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.700713 s]
/proc/loadavg: 0.00 0.11 2.05 1/376 5735
/proc/meminfo: memFree=22051732/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=38912 CPUtime=0.05 cores=0,2,4,6
/proc/5735/stat : 5735 (python) D 5733 5735 5707 0 -1 4202496 1621 0 14 0 4 1 0 0 16 0 1 0 670213666 39845888 1464 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594866432 0 0 16781312 2 18446744071562234166 0 0 17 2 0 0 7
/proc/5735/statm: 9728 1464 455 364 0 1028 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 38912

[startup+1.50057 s]
/proc/loadavg: 0.00 0.11 2.05 1/382 5746
/proc/meminfo: memFree=21993040/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=124240 CPUtime=0.18 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 4272 0 40 0 10 8 0 0 16 0 1 0 670213666 127221760 3880 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594866432 0 0 16781312 2 18446744071562362840 0 0 17 2 0 0 38
/proc/5735/statm: 31060 3880 997 364 0 2871 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 124240

[startup+3.10029 s]
/proc/loadavg: 0.24 0.16 2.05 1/382 5754
/proc/meminfo: memFree=21958240/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=187744 CPUtime=0.35 cores=0,2,4,6
/proc/5735/stat : 5735 (python) D 5733 5735 5707 0 -1 4202496 6708 930 63 9 20 15 0 0 16 0 1 0 670213666 192249856 4827 33554432000 4194304 5683644 140735166068272 18446744073709551615 235589947075 0 0 16781312 2 18446744071562154983 0 0 17 2 0 0 89
/proc/5735/statm: 46936 4827 1349 364 0 3724 0
Current children cumulated CPU time (s) 0.35
Current children cumulated vsize (KiB) 187744
heavy processes:

[startup+6.30084 s]
/proc/loadavg: 0.24 0.16 2.05 3/385 5757
/proc/meminfo: memFree=21897332/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=290524 CPUtime=2.02 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 12745 930 110 9 182 20 0 0 18 0 2 0 670213666 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 170
/proc/5735/statm: 72631 7503 1877 364 0 8549 0
[pid=5755] ppid=5735 vsize=280764 CPUtime=0 cores=0,2,4,6
/proc/5755/stat : 5755 (python) S 5735 5735 5707 0 -1 4202560 731 0 0 0 0 0 0 0 18 0 1 0 670214194 287502336 5978 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886671 0 0 16781312 514 3521164276892958719 0 0 17 2 0 0 0
/proc/5755/statm: 70191 5978 355 364 0 6109 0
[pid=5757] ppid=5755 vsize=1019708 CPUtime=1.01 cores=0,2,4,6
/proc/5757/stat : 5757 (gnovelty+2) R 5755 5757 5757 0 -1 4202496 954 0 0 0 101 0 0 0 21 0 1 0 670214194 1044180992 689 33554432000 134512640 135663704 4289788368 18446744073709551615 134525302 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/5757/statm: 254927 689 113 282 0 254643 0
[pid=5735/tid=5756] ppid=5733 vsize=290524 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 2 0 670214194 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3.03
Current children cumulated vsize (KiB) 1590996
heavy processes:

[startup+12.7007 s]
/proc/loadavg: 0.38 0.19 2.05 3/388 5761
/proc/meminfo: memFree=21866984/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=290524 CPUtime=2.02 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 12745 930 110 9 182 20 0 0 18 0 2 0 670213666 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 170
/proc/5735/statm: 72631 7503 1877 364 0 8549 0
[pid=5755] ppid=5735 vsize=280764 CPUtime=0.01 cores=0,2,4,6
/proc/5755/stat : 5755 (python) S 5735 5735 5707 0 -1 4202560 874 0 0 0 1 0 0 0 18 0 1 0 670214194 287502336 6001 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886671 0 0 16781312 514 3521164276892926211 0 0 17 2 0 0 0
/proc/5755/statm: 70191 6001 378 364 0 6109 0
[pid=5757] ppid=5755 vsize=1019708 CPUtime=7.41 cores=0,2,4,6
/proc/5757/stat : 5757 (gnovelty+2) R 5755 5757 5757 0 -1 4202496 954 0 0 0 741 0 0 0 25 0 1 0 670214194 1044180992 689 33554432000 134512640 135663704 4289788368 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/5757/statm: 254927 689 113 282 0 254643 0
[pid=5735/tid=5756] ppid=5733 vsize=290524 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 2 0 670214194 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 9.44
Current children cumulated vsize (KiB) 1590996
heavy processes:

[startup+25.5005 s]
/proc/loadavg: 0.74 0.27 2.05 3/388 5761
/proc/meminfo: memFree=21867744/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=290524 CPUtime=2.02 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 12745 930 110 9 182 20 0 0 18 0 2 0 670213666 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 170
/proc/5735/statm: 72631 7503 1877 364 0 8549 0
[pid=5755] ppid=5735 vsize=280764 CPUtime=0.04 cores=0,2,4,6
/proc/5755/stat : 5755 (python) S 5735 5735 5707 0 -1 4202560 878 0 0 0 2 2 0 0 18 0 1 0 670214194 287502336 6001 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886671 0 0 16781312 514 5177362760685551875 0 0 17 2 0 0 0
/proc/5755/statm: 70191 6001 378 364 0 6109 0
[pid=5757] ppid=5755 vsize=1019708 CPUtime=20.21 cores=0,2,4,6
/proc/5757/stat : 5757 (gnovelty+2) R 5755 5757 5757 0 -1 4202496 954 0 0 0 2021 0 0 0 25 0 1 0 670214194 1044180992 689 33554432000 134512640 135663704 4289788368 18446744073709551615 134545635 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/5757/statm: 254927 689 113 282 0 254643 0
[pid=5735/tid=5756] ppid=5733 vsize=290524 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 2 0 670214194 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 22.27
Current children cumulated vsize (KiB) 1590996

[startup+51.101 s]
/proc/loadavg: 1.17 0.41 2.05 3/388 5761
/proc/meminfo: memFree=21867828/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=290524 CPUtime=2.02 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 12745 930 110 9 182 20 0 0 18 0 2 0 670213666 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 170
/proc/5735/statm: 72631 7503 1877 364 0 8549 0
[pid=5755] ppid=5735 vsize=280764 CPUtime=0.11 cores=0,2,4,6
/proc/5755/stat : 5755 (python) S 5735 5735 5707 0 -1 4202560 889 0 0 0 6 5 0 0 18 0 1 0 670214194 287502336 6001 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886671 0 0 16781312 514 5177362760685551875 0 0 17 2 0 0 0
/proc/5755/statm: 70191 6001 378 364 0 6109 0
[pid=5757] ppid=5755 vsize=1019708 CPUtime=45.81 cores=0,2,4,6
/proc/5757/stat : 5757 (gnovelty+2) R 5755 5757 5757 0 -1 4202496 954 0 0 0 4581 0 0 0 25 0 1 0 670214194 1044180992 689 33554432000 134512640 135663704 4289788368 18446744073709551615 134525271 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/5757/statm: 254927 689 113 282 0 254643 0
[pid=5735/tid=5756] ppid=5733 vsize=290524 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 2 0 670214194 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 47.94
Current children cumulated vsize (KiB) 1590996

[startup+102.314 s]
/proc/loadavg: 1.67 0.68 2.04 3/388 5763
/proc/meminfo: memFree=21868144/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=290524 CPUtime=2.02 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 12745 930 110 9 182 20 0 0 18 0 2 0 670213666 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 170
/proc/5735/statm: 72631 7503 1877 364 0 8549 0
[pid=5755] ppid=5735 vsize=280764 CPUtime=0.25 cores=0,2,4,6
/proc/5755/stat : 5755 (python) S 5735 5735 5707 0 -1 4202560 908 0 0 0 14 11 0 0 18 0 1 0 670214194 287502336 6001 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886671 0 0 16781312 514 5177362760685551875 0 0 17 6 0 0 0
/proc/5755/statm: 70191 6001 378 364 0 6109 0
[pid=5757] ppid=5755 vsize=1019708 CPUtime=97.02 cores=0,2,4,6
/proc/5757/stat : 5757 (gnovelty+2) R 5755 5757 5757 0 -1 4202496 954 0 0 0 9702 0 0 0 25 0 1 0 670214194 1044180992 689 33554432000 134512640 135663704 4289788368 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/5757/statm: 254927 689 113 282 0 254643 0
[pid=5735/tid=5756] ppid=5733 vsize=290524 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 2 0 670214194 297496576 7503 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 99.29
Current children cumulated vsize (KiB) 1590996

[startup+162.3 s]
/proc/loadavg: 1.87 0.90 2.04 3/396 5773
/proc/meminfo: memFree=21827100/32950928 swapFree=45972156/67111528
[pid=5735] ppid=5733 vsize=300780 CPUtime=2.05 cores=0,2,4,6
/proc/5735/stat : 5735 (python) S 5733 5735 5707 0 -1 4202496 13597 930 110 9 185 20 0 0 20 0 3 0 670213666 307998720 7510 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 170
/proc/5735/statm: 75195 7510 1880 364 0 11110 0
[pid=5755] ppid=5735 vsize=291008 CPUtime=0.27 cores=0,2,4,6
/proc/5755/stat : 5755 (python) S 5735 5735 5707 0 -1 4202560 1069 0 0 0 15 12 0 0 19 0 2 0 670214194 297992192 6038 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5755/statm: 72752 6038 410 364 0 8670 0
[pid=5757] ppid=5755 vsize=1019708 CPUtime=100.29 cores=0,2,4,6
/proc/5757/stat : 5757 (gnovelty+2) T 5755 5757 5757 0 -1 4202496 954 0 0 0 10029 0 0 0 25 0 1 0 670214194 1044180992 689 33554432000 134512640 135663704 4289788368 18446744073709551615 134524918 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5757/statm: 254927 689 113 282 0 254643 0
[pid=5755/tid=5764] ppid=5735 vsize=291008 CPUtime=0 cores=0,2,4,6
/proc/5755/task/5764/stat : 5764 (python) S 5735 5735 5707 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 670224224 297992192 6038 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604449437560896 0 0 -1 6 0 0 0
[pid=5765] ppid=5735 vsize=291096 CPUtime=0.14 cores=0,2,4,6
/proc/5765/stat : 5765 (python) S 5735 5735 5707 0 -1 4202560 912 0 0 0 8 6 0 0 18 0 1 0 670224227 298082304 5999 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886726 0 0 16781312 514 5177362760685551875 0 0 17 6 0 0 0
/proc/5765/statm: 72774 5999 372 364 0 8689 0
[pid=5767] ppid=5765 vsize=1019708 CPUtime=56.67 cores=0,2,4,6
/proc/5767/stat : 5767 (gnovelty+2) R 5765 5767 5767 0 -1 4202496 953 0 0 0 5667 0 0 0 25 0 1 0 670224228 1044180992 689 33554432000 134512640 135663704 4289597024 18446744073709551615 134525101 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/5767/statm: 254927 689 113 282 0 254643 0
[pid=5735/tid=5756] ppid=5733 vsize=300780 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 3 0 670214194 307998720 7510 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5766] ppid=5733 vsize=300780 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5766/stat : 5766 (python) S 5733 5735 5707 0 -1 4202560 30 930 0 9 0 0 0 0 20 0 3 0 670224227 307998720 7510 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 159.42
Current children cumulated vsize (KiB) 2922300

[startup+222.301 s]
/proc/loadavg: 1.95 1.11 2.03 3/404 5782
/proc/meminfo: memFree=21782980/32950928 swapFree=45972156/67111528

################
# More data... #
################

[pid=5778] ppid=5776 vsize=420220 CPUtime=98.28 cores=0,2,4,6
/proc/5778/stat : 5778 (adaptg2wsat2009) T 5776 5778 5778 0 -1 4202496 1664 0 0 0 9828 0 0 0 25 0 1 0 670234164 430305280 1400 33554432000 134512640 135107411 4287324144 18446744073709551615 134531656 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/5778/statm: 105055 1400 59 146 0 103931 0
[pid=5776/tid=5785] ppid=5735 vsize=301432 CPUtime=0 cores=0,2,4,6
/proc/5776/task/5785/stat : 5785 (python) S 5735 5735 5707 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670243992 308666368 6051 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604453985310656 0 0 -1 6 0 0 0
[pid=5786] ppid=5735 vsize=376584 CPUtime=0.28 cores=0,2,4,6
/proc/5786/stat : 5786 (python) S 5735 5735 5707 0 -1 4202560 1031 0 0 0 16 12 0 0 19 0 2 0 670243995 385622016 6226 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5786/statm: 94146 6226 399 364 0 30055 0
[pid=5788] ppid=5786 vsize=420220 CPUtime=98.28 cores=0,2,4,6
/proc/5788/stat : 5788 (adaptg2wsat2009) T 5786 5788 5788 0 -1 4202496 1656 0 0 0 9828 0 0 0 25 0 1 0 670243995 430305280 1401 33554432000 134512640 135107411 4287463568 18446744073709551615 134532326 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5788/statm: 105055 1401 59 146 0 103931 0
[pid=5786/tid=5793] ppid=5735 vsize=376584 CPUtime=0 cores=0,2,4,6
/proc/5786/task/5793/stat : 5793 (python) S 5735 5735 5707 0 -1 4202560 125 0 0 0 0 0 0 0 19 0 2 0 670253824 385622016 6226 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604453908043904 0 0 -1 6 0 0 0
[pid=5794] ppid=5735 vsize=386840 CPUtime=0.28 cores=0,2,4,6
/proc/5794/stat : 5794 (python) S 5735 5735 5707 0 -1 4202560 988 0 0 0 14 14 0 0 19 0 2 0 670253827 396124160 6233 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5794/statm: 96710 6233 399 364 0 32616 0
[pid=5796] ppid=5794 vsize=420220 CPUtime=100.27 cores=0,2,4,6
/proc/5796/stat : 5796 (adaptg2wsat2009) T 5794 5796 5796 0 -1 4202496 1658 0 0 0 10027 0 0 0 25 0 1 0 670253827 430305280 1399 33554432000 134512640 135107411 4292271056 18446744073709551615 134532371 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5796/statm: 105055 1399 59 146 0 103931 0
[pid=5794/tid=5799] ppid=5735 vsize=386840 CPUtime=0 cores=0,2,4,6
/proc/5794/task/5799/stat : 5799 (python) S 5735 5735 5707 0 -1 4202560 69 0 0 0 0 0 0 0 19 0 2 0 670263855 396124160 6233 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604441275127872 0 0 -1 6 0 0 0
[pid=5800] ppid=5735 vsize=397684 CPUtime=0.29 cores=0,2,4,6
/proc/5800/stat : 5800 (python) S 5735 5735 5707 0 -1 4202560 1095 0 0 0 16 13 0 0 19 0 2 0 670263858 407228416 6198 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5800/statm: 99421 6198 400 364 0 35324 0
[pid=5802] ppid=5800 vsize=1019708 CPUtime=100.31 cores=0,2,4,6
/proc/5802/stat : 5802 (gnovelty+2) T 5800 5802 5802 0 -1 4202496 957 0 0 0 10031 0 0 0 25 0 1 0 670263858 1044180992 690 33554432000 134512640 135663704 4293861168 18446744073709551615 134525347 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5802/statm: 254927 690 113 282 0 254643 0
[pid=5800/tid=5809] ppid=5735 vsize=397684 CPUtime=0 cores=0,2,4,6
/proc/5800/task/5809/stat : 5809 (python) S 5735 5735 5707 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670273890 407228416 6198 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604450645741632 0 0 -1 6 0 0 0
[pid=5810] ppid=5735 vsize=408548 CPUtime=0.28 cores=0,2,4,6
/proc/5810/stat : 5810 (python) S 5735 5735 5707 0 -1 4202560 1169 0 0 0 15 13 0 0 19 0 2 0 670273894 418353152 6232 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5810/statm: 102137 6232 406 364 0 38037 0
[pid=5812] ppid=5810 vsize=249872 CPUtime=99.27 cores=0,2,4,6
/proc/5812/stat : 5812 (FH) T 5810 5812 5812 0 -1 4202496 1533 0 0 0 9927 0 0 0 25 0 1 0 670273894 255868928 1263 33554432000 134512640 134936988 4291464672 18446744073709551615 134552486 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5812/statm: 62468 1263 41 104 0 61972 0
[pid=5810/tid=5817] ppid=5735 vsize=408548 CPUtime=0 cores=0,2,4,6
/proc/5810/task/5817/stat : 5817 (python) S 5735 5735 5707 0 -1 4202560 105 0 0 0 0 0 0 0 19 0 2 0 670283822 418353152 6232 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604453985309760 0 0 -1 6 0 0 0
[pid=5818] ppid=5735 vsize=418540 CPUtime=0.29 cores=0,2,4,6
/proc/5818/stat : 5818 (python) S 5735 5735 5707 0 -1 4202560 1170 0 0 0 15 14 0 0 19 0 2 0 670283826 428584960 6371 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/5818/statm: 104635 6371 406 364 0 40532 0
[pid=5820] ppid=5818 vsize=420220 CPUtime=100.12 cores=0,2,4,6
/proc/5820/stat : 5820 (adaptg2wsat2009) T 5818 5820 5820 0 -1 4202496 1677 0 0 0 10012 0 0 0 25 0 1 0 670283826 430305280 1401 33554432000 134512640 135107411 4288422768 18446744073709551615 134531411 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/5820/statm: 105055 1401 59 146 0 103931 0
[pid=5818/tid=5835] ppid=5735 vsize=418540 CPUtime=0 cores=0,2,4,6
/proc/5818/task/5835/stat : 5835 (python) S 5735 5735 5707 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 670293855 428584960 6371 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604450645744320 0 0 -1 0 0 0 0
[pid=5836] ppid=5735 vsize=428792 CPUtime=0.32 cores=0,2,4,6
/proc/5836/stat : 5836 (python) S 5735 5735 5707 0 -1 4202560 1141 0 0 0 19 13 0 0 19 0 2 0 670293859 439083008 6369 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/5836/statm: 107198 6369 400 364 0 43092 0
[pid=5838] ppid=5836 vsize=420220 CPUtime=100.18 cores=0,2,4,6
/proc/5838/stat : 5838 (adaptg2wsat2009) T 5836 5838 5838 0 -1 4202496 1673 0 0 0 10018 0 0 0 25 0 1 0 670293859 430305280 1400 33554432000 134512640 135107411 4292147264 18446744073709551615 134532088 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/5838/statm: 105055 1400 59 146 0 103931 0
[pid=5836/tid=5848] ppid=5735 vsize=428792 CPUtime=0 cores=0,2,4,6
/proc/5836/task/5848/stat : 5848 (python) S 5735 5735 5707 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670303890 439083008 6369 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604453903976256 0 0 -1 0 0 0 0
[pid=5849] ppid=5735 vsize=438832 CPUtime=0.3 cores=0,2,4,6
/proc/5849/stat : 5849 (python) S 5735 5735 5707 0 -1 4202560 1141 0 0 0 16 14 0 0 19 0 2 0 670303894 449363968 6372 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/5849/statm: 109708 6372 399 364 0 45599 0
[pid=5851] ppid=5849 vsize=249868 CPUtime=100.29 cores=0,2,4,6
/proc/5851/stat : 5851 (FH) T 5849 5851 5851 0 -1 4202496 1535 0 0 0 10029 0 0 0 25 0 1 0 670303894 255864832 1263 33554432000 134512640 134936988 4286956096 18446744073709551615 134554610 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/5851/statm: 62467 1263 41 104 0 61971 0
[pid=5849/tid=5856] ppid=5735 vsize=438832 CPUtime=0 cores=0,2,4,6
/proc/5849/task/5856/stat : 5856 (python) S 5735 5735 5707 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670313924 449363968 6372 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604453918986688 0 0 -1 4 0 0 0
[pid=5857] ppid=5735 vsize=449736 CPUtime=0.58 cores=0,2,4,6
/proc/5857/stat : 5857 (python) S 5735 5735 5707 0 -1 4202560 1158 0 0 0 34 24 0 0 19 0 2 0 670313928 460529664 6377 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5857/statm: 112434 6377 400 364 0 48322 0
[pid=5859] ppid=5857 vsize=268420 CPUtime=195.65 cores=0,2,4,6
/proc/5859/stat : 5859 (hybridGM3) T 5857 5859 5859 0 -1 4202496 5397 0 0 0 19565 0 0 0 25 0 1 0 670313928 274862080 5125 33554432000 134512640 135204385 4288121984 18446744073709551615 134538000 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/5859/statm: 67105 5125 63 169 0 66935 0
[pid=5857/tid=5869] ppid=5735 vsize=449736 CPUtime=0 cores=0,2,4,6
/proc/5857/task/5869/stat : 5869 (python) S 5735 5735 5707 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670333494 460529664 6377 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604449001166848 0 0 -1 0 0 0 0
[pid=5870] ppid=5735 vsize=459508 CPUtime=0.3 cores=0,2,4,6
/proc/5870/stat : 5870 (python) S 5735 5735 5707 0 -1 4202560 1139 0 0 0 17 13 0 0 19 0 2 0 670333498 470536192 6420 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5870/statm: 114877 6420 399 364 0 50762 0
[pid=5872] ppid=5870 vsize=1019708 CPUtime=100.34 cores=0,2,4,6
/proc/5872/stat : 5872 (gnovelty+2) T 5870 5872 5872 0 -1 4202496 964 0 0 0 10034 0 0 0 25 0 1 0 670333499 1044180992 690 33554432000 134512640 135663704 4293440624 18446744073709551615 134524530 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/5872/statm: 254927 690 113 282 0 254643 0
[pid=5870/tid=5880] ppid=5735 vsize=459508 CPUtime=0 cores=0,2,4,6
/proc/5870/task/5880/stat : 5880 (python) S 5735 5735 5707 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670343533 470536192 6420 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604446458104704 0 0 -1 6 0 0 0
[pid=5881] ppid=5735 vsize=469896 CPUtime=0.6 cores=0,2,4,6
/proc/5881/stat : 5881 (python) S 5735 5735 5707 0 -1 4202560 1164 0 0 0 32 28 0 0 19 0 2 0 670343537 481173504 6426 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5881/statm: 117474 6426 400 364 0 53356 0
[pid=5883] ppid=5881 vsize=268424 CPUtime=195.66 cores=0,2,4,6
/proc/5883/stat : 5883 (hybridGM3) T 5881 5883 5883 0 -1 4202496 5399 0 0 0 19565 1 0 0 25 0 1 0 670343538 274866176 5127 33554432000 134512640 135204385 4289513456 18446744073709551615 134537804 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5883/statm: 67106 5127 63 169 0 66936 0
[pid=5881/tid=5893] ppid=5735 vsize=469896 CPUtime=0 cores=0,2,4,6
/proc/5881/task/5893/stat : 5893 (python) S 5735 5735 5707 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670363103 481173504 6426 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604446606878528 0 0 -1 6 0 0 0
[pid=5894] ppid=5735 vsize=480484 CPUtime=0.32 cores=0,2,4,6
/proc/5894/stat : 5894 (python) S 5735 5735 5707 0 -1 4202560 1149 0 0 0 19 13 0 0 19 0 2 0 670363107 492015616 6430 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5894/statm: 120121 6430 400 364 0 56000 0
[pid=5896] ppid=5894 vsize=420220 CPUtime=100.25 cores=0,2,4,6
/proc/5896/stat : 5896 (adaptg2wsat2009) T 5894 5896 5896 0 -1 4202496 1672 0 0 0 10025 0 0 0 25 0 1 0 670363108 430305280 1400 33554432000 134512640 135107411 4293203616 18446744073709551615 134532371 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/5896/statm: 105055 1400 59 146 0 103931 0
[pid=5894/tid=5904] ppid=5735 vsize=480484 CPUtime=0 cores=0,2,4,6
/proc/5894/task/5904/stat : 5904 (python) S 5735 5735 5707 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670373140 492015616 6430 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604453983266688 0 0 -1 6 0 0 0
[pid=5905] ppid=5735 vsize=490408 CPUtime=0.6 cores=0,2,4,6
/proc/5905/stat : 5905 (python) S 5735 5735 5707 0 -1 4202560 1210 0 0 0 33 27 0 0 19 0 2 0 670373143 502177792 6441 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5905/statm: 122602 6441 407 364 0 58478 0
[pid=5907] ppid=5905 vsize=268424 CPUtime=196.66 cores=0,2,4,6
/proc/5907/stat : 5907 (hybridGM3) T 5905 5907 5907 0 -1 4202496 5403 0 0 0 19666 0 0 0 25 0 1 0 670373144 274866176 5127 33554432000 134512640 135204385 4292891856 18446744073709551615 134538264 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5907/statm: 67106 5127 63 169 0 66936 0
[pid=5905/tid=5917] ppid=5735 vsize=490408 CPUtime=0 cores=0,2,4,6
/proc/5905/task/5917/stat : 5917 (python) S 5735 5735 5707 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670392811 502177792 6441 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604452885097344 0 0 -1 2 0 0 0
[pid=5918] ppid=5735 vsize=501024 CPUtime=0.64 cores=0,2,4,6
/proc/5918/stat : 5918 (python) S 5735 5735 5707 0 -1 4202560 1224 0 0 0 33 31 0 0 19 0 2 0 670392814 513048576 6445 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606693323 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/5918/statm: 125256 6445 407 364 0 61129 0
[pid=5920] ppid=5918 vsize=268424 CPUtime=199.7 cores=0,2,4,6
/proc/5920/stat : 5920 (hybridGM3) T 5918 5920 5920 0 -1 4202496 5403 0 0 0 19970 0 0 0 25 0 1 0 670392815 274866176 5127 33554432000 134512640 135204385 4287879120 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/5920/statm: 67106 5127 63 169 0 66936 0
[pid=5918/tid=5930] ppid=5735 vsize=501024 CPUtime=0 cores=0,2,4,6
/proc/5918/task/5930/stat : 5930 (python) S 5735 5735 5707 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670412786 513048576 6445 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 514 18446604450645743424 0 0 -1 6 0 0 0
[pid=5931] ppid=5735 vsize=511468 CPUtime=0 cores=0,2,4,6
/proc/5931/stat : 5931 (python) S 5735 5735 5707 0 -1 4202560 831 0 0 0 0 0 0 0 19 0 1 0 670412789 523743232 6389 33554432000 4194304 5683644 140735166068272 18446744073709551615 235594886726 0 0 16781312 514 7458180614668156927 0 0 17 2 0 0 0
/proc/5931/statm: 127867 6389 346 364 0 63737 0
[pid=5933] ppid=5931 vsize=454088 CPUtime=0.78 cores=0,2,4,6
/proc/5933/stat : 5933 (TNM) R 5931 5933 5933 0 -1 4202496 1272 0 0 0 78 0 0 0 22 0 1 0 670412790 464986112 995 33554432000 134512640 135008748 4286862560 18446744073709551615 134523389 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/5933/statm: 113522 995 45 122 0 112422 0
[pid=5735/tid=5756] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5756/stat : 5756 (python) S 5733 5735 5707 0 -1 4202560 24 930 0 9 0 0 0 0 19 0 18 0 670214194 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5766] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5766/stat : 5766 (python) S 5733 5735 5707 0 -1 4202560 30 930 0 9 0 0 0 0 20 0 18 0 670224227 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5777] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5777/stat : 5777 (python) S 5733 5735 5707 0 -1 4202560 28 930 0 9 0 0 0 0 20 0 18 0 670234163 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 6 0 0 0
[pid=5735/tid=5787] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5787/stat : 5787 (python) S 5733 5735 5707 0 -1 4202560 92 930 0 9 0 0 0 0 20 0 18 0 670243995 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5795] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5795/stat : 5795 (python) S 5733 5735 5707 0 -1 4202560 34 930 0 9 0 0 0 0 20 0 18 0 670253827 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5801] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5801/stat : 5801 (python) S 5733 5735 5707 0 -1 4202560 28 930 0 9 0 0 0 0 20 0 18 0 670263858 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5811] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5811/stat : 5811 (python) S 5733 5735 5707 0 -1 4202560 25 930 0 9 0 0 0 0 20 0 18 0 670273894 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5819] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5819/stat : 5819 (python) S 5733 5735 5707 0 -1 4202560 30 930 0 9 0 0 0 0 20 0 18 0 670283826 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5837] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5837/stat : 5837 (python) S 5733 5735 5707 0 -1 4202560 30 930 0 9 0 0 0 0 20 0 18 0 670293859 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5850] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5850/stat : 5850 (python) S 5733 5735 5707 0 -1 4202560 29 930 0 9 0 0 0 0 20 0 18 0 670303894 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 6 0 0 0
[pid=5735/tid=5858] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5858/stat : 5858 (python) S 5733 5735 5707 0 -1 4202560 25 930 0 9 0 0 0 0 20 0 18 0 670313928 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 6 0 0 0
[pid=5735/tid=5871] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5871/stat : 5871 (python) S 5733 5735 5707 0 -1 4202560 30 930 0 9 0 0 0 0 20 0 18 0 670333498 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 0 0 0 0
[pid=5735/tid=5882] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5882/stat : 5882 (python) S 5733 5735 5707 0 -1 4202560 28 930 0 9 0 0 0 0 20 0 18 0 670343538 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 6 0 0 0
[pid=5735/tid=5895] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5895/stat : 5895 (python) S 5733 5735 5707 0 -1 4202560 26 930 0 9 0 0 0 0 20 0 18 0 670363108 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5906] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5906/stat : 5906 (python) S 5733 5735 5707 0 -1 4202560 26 930 0 9 0 0 0 0 20 0 18 0 670373144 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 6 0 0 0
[pid=5735/tid=5919] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5919/stat : 5919 (python) S 5733 5735 5707 0 -1 4202560 22 930 0 9 0 0 0 0 20 0 18 0 670392815 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 2 0 0 0
[pid=5735/tid=5932] ppid=5733 vsize=521032 CPUtime=0 cores=0,2,4,6
/proc/5735/task/5932/stat : 5932 (python) S 5733 5735 5707 0 -1 4202560 25 930 0 9 0 0 0 0 20 0 18 0 670412789 533536768 7956 33554432000 4194304 5683644 140735166068272 18446744073709551615 235606690225 0 0 16781312 2 18446604444333804544 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1994.16
Current children cumulated vsize (KiB) 16250584

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5735 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=388636
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=249962
# CPU time returned by wait4() is 2.6386
# while last known CPU time is 1994.16
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1992.13
CPU time (s): 1994.16
CPU user time (s): 1991.25
CPU system time (s): 2.91
CPU usage (%): 100.102
Max. virtual memory (cumulated for all children) (KiB): 16250584

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.38864
system time used= 0.249962
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 29012
page faults= 119
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 21428
involuntary context switches= 131

runsolver used 13.5819 second user time and 25.9231 second system time

The end

Launcher Data 

Begin job on node120 at 2011-05-17 00:49:09
IDJOB=3386481
IDBENCH=82546
IDSOLVER=1872
FILE ID=node120/3386481-1305586148
RUNJOBID= node120-1305586147-5725
PBS_JOBID= 13350358
Free space on /tmp= 73496 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S1286362348-008-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386481-1305586148/watcher-3386481-1305586148 -o /tmp/evaluation-result-3386481-1305586148/solver-3386481-1305586148 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386481-1305586148.cnf -seed 564721476 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 63b66954174098a654eacd27832fcf38
RANDOM SEED=564721476

node120.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      22052020 kB
Buffers:         85904 kB
Cached:         628356 kB
SwapCached:      63496 kB
Active:         437224 kB
Inactive:     10234272 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      22052020 kB
SwapTotal:    67111528 kB
SwapFree:     45972156 kB
Dirty:           32840 kB
Writeback:           0 kB
AnonPages:     9899588 kB
Mapped:          15364 kB
Slab:            78972 kB
PageTables:      81704 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 78714984 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73432 MiB
End job on node120 at 2011-05-17 01:22:23