Trace number 3317496

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 3539.66 890.025

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S1321406709-042.UNKNOWN.cnf
MD5SUM383bd72f29b3667b7a0d512d7fa797ad
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.32/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.46	c INFO: solving HOME/instance-3317496-1304269877.cnf
0.32/0.46	c INFO: computing task features
0.42/0.51	c INFO: computed variable-clause graph statistics
0.42/0.52	c INFO: computed clause balance statistics
0.61/0.79	c INFO: computed variable balance statistics
0.81/0.91	c INFO: computed small-clause counts
0.81/0.91	c INFO: computed Horn-clause counts
0.81/0.92	c INFO: computed variable graph statistics
0.81/0.92	c INFO: collected features for HOME/instance-3317496-1304269877.cnf in 0.45 s
0.81/0.94	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.47)
0.81/0.97	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.11)
0.91/1.00	c INFO: running adaptg2wsat2009++@0 for 97 with 4799 remaining (b = 0.06)
0.91/1.03	c INFO: running adaptg2wsat2009++@0 for 97 with 4799 remaining (b = 0.05)
400.60/101.14	c INFO: running adaptg2wsat2009++@0 for 97 with 4699 remaining (b = 0.04)
400.60/101.19	c INFO: running gnovelty+2@0 for 97 with 4699 remaining (b = 0.03)
400.98/101.22	c INFO: running FH@0 for 97 with 4699 remaining (b = 0.03)
400.98/101.26	c INFO: running adaptg2wsat2009++@0 for 97 with 4699 remaining (b = 0.02)
796.92/200.48	c INFO: running adaptg2wsat2009++@0 for 97 with 4600 remaining (b = 0.02)
800.54/201.36	c INFO: running FH@0 for 97 with 4599 remaining (b = 0.02)
800.90/201.41	c INFO: running hybridGM3@0 for 194 with 4599 remaining (b = 0.03)
800.90/201.45	c INFO: running gnovelty+2@0 for 97 with 4599 remaining (b = 0.01)
1195.91/300.70	c INFO: running hybridGM3@0 for 194 with 4499 remaining (b = 0.02)
1199.88/301.67	c INFO: running adaptg2wsat2009++@0 for 97 with 4498 remaining (b = 0.01)
1219.39/306.59	c INFO: running adaptg2wsat2009++@100 for 97 with 4493 remaining (b = 0.01)
1575.07/395.90	c INFO: running hybridGM3@100 for 97 with 4404 remaining (b = 0.01)
1587.01/398.88	c INFO: running FH@0 for 97 with 4401 remaining (b = 0.01)
1626.83/408.82	c INFO: running march_hi@0 for 388 with 4391 remaining (b = 0.04)
1978.44/497.12	c INFO: running hybridGM3@99 for 97 with 4303 remaining (b = 0.01)
1978.44/497.16	c INFO: running hybridGM3@0 for 194 with 4303 remaining (b = 0.02)
1986.36/499.11	c INFO: running hybridGM3@0 for 194 with 4301 remaining (b = 0.02)
2381.20/598.34	c INFO: running TNM@0 for 1552 with 4202 remaining (b = 0.14)
2755.43/692.60	c INFO: running hybridGM3@98 for 97 with 4107 remaining (b = 0.01)
2759.40/693.58	c INFO: running hybridGM3@98 for 97 with 4106 remaining (b = 0.01)
3157.86/793.81	c INFO: running hybridGM3@292 for 97 with 4006 remaining (b = 0.01)
3161.84/794.81	c INFO: running adaptg2wsat2009++@98 for 97 with 4005 remaining (b = 0.01)
3190.81/802.13	c INFO: running march_hi@195 for 194 with 3998 remaining (b = 0.03)
3536.03/889.02	c INFO: running march_hi@0 for 388 with 3911 remaining (b = 0.04)
3539.66/890.00	c INFO: running gnovelty+2@0 for 97 with 3910 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317496-1304269877/watcher-3317496-1304269877 -o /tmp/evaluation-result-3317496-1304269877/solver-3317496-1304269877 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317496-1304269877.cnf -seed 1320683075 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.02 8.05 7.55 5/251 16656
/proc/meminfo: memFree=23202592/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=15772 CPUtime=0 cores=1,3,5,7
/proc/16656/stat : 16656 (python) R 16654 16656 15742 0 -1 4202496 890 0 0 0 0 0 0 0 25 0 1 0 617218540 16150528 771 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988253381 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/16656/statm: 3943 772 343 364 0 430 0

[startup+0.102003 s]
/proc/loadavg: 8.02 8.05 7.55 5/251 16656
/proc/meminfo: memFree=23202592/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=104612 CPUtime=0.07 cores=1,3,5,7
/proc/16656/stat : 16656 (python) S 16654 16656 15742 0 -1 4202496 3099 0 0 0 5 2 0 0 25 0 1 0 617218540 107122688 2778 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988590592 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 0
/proc/16656/statm: 26153 2778 847 364 0 1901 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104612

[startup+0.200984 s]
/proc/loadavg: 8.02 8.05 7.55 5/251 16656
/proc/meminfo: memFree=23202592/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=131292 CPUtime=0.15 cores=1,3,5,7
/proc/16656/stat : 16656 (python) R 16654 16656 15742 0 -1 4202496 5684 933 1 0 12 3 0 0 19 0 1 0 617218540 134443008 4087 33554432000 4194304 5683644 140736313795264 18446744073709551615 5212272 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/16656/statm: 32823 4087 1015 364 0 3084 0
Current children cumulated CPU time (s) 0.15
Current children cumulated vsize (KiB) 131292

[startup+0.301005 s]
/proc/loadavg: 8.02 8.05 7.55 5/251 16656
/proc/meminfo: memFree=23202592/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=192316 CPUtime=0.23 cores=1,3,5,7
/proc/16656/stat : 16656 (python) S 16654 16656 15742 0 -1 4202496 6894 933 1 0 17 6 0 0 18 0 1 0 617218540 196931584 4948 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988590592 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/16656/statm: 48079 4948 1425 364 0 3722 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 192316

[startup+0.700895 s]
/proc/loadavg: 8.02 8.05 7.55 5/251 16656
/proc/meminfo: memFree=23202592/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=280232 CPUtime=0.61 cores=1,3,5,7
/proc/16656/stat : 16656 (python) R 16654 16656 15742 0 -1 4202496 9993 933 1 0 52 9 0 0 18 0 1 0 617218540 286957568 7443 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988248485 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/16656/statm: 70058 7443 1826 364 0 5982 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 280232

[startup+1.50075 s]
/proc/loadavg: 8.02 8.05 7.55 9/264 16673
/proc/meminfo: memFree=23158468/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=320744 CPUtime=0.95 cores=1,3,5,7
/proc/16656/stat : 16656 (python) S 16654 16656 15742 0 -1 4202496 13082 933 1 0 85 10 0 0 25 0 5 0 617218540 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/16656/statm: 80186 7385 1883 364 0 16095 0
[pid=16662] ppid=16656 vsize=279412 CPUtime=0 cores=1,3,5,7
/proc/16662/stat : 16662 (python) S 16656 16656 15742 0 -1 4202560 729 0 0 0 0 0 0 0 19 0 1 0 617218635 286117888 5792 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612623 0 0 16781312 514 916853888517144575 0 0 17 3 0 0 0
/proc/16662/statm: 69853 5792 354 364 0 5771 0
[pid=16664] ppid=16662 vsize=1018088 CPUtime=0.54 cores=1,3,5,7
/proc/16664/stat : 16664 (gnovelty+2) R 16662 16664 16664 0 -1 4202496 570 0 0 0 54 0 0 0 22 0 1 0 617218635 1042522112 307 33554432000 134512640 135663704 4290000720 18446744073709551615 134525684 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/16664/statm: 254522 307 113 282 0 254238 0
[pid=16665] ppid=16656 vsize=289988 CPUtime=0 cores=1,3,5,7
/proc/16665/stat : 16665 (python) S 16656 16656 15742 0 -1 4202560 737 0 0 0 0 0 0 0 21 0 1 0 617218638 296947712 5838 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 9986213208500731903 0 0 17 7 0 0 0
/proc/16665/statm: 72497 5838 348 364 0 8412 0
[pid=16667] ppid=16665 vsize=1018088 CPUtime=0.51 cores=1,3,5,7
/proc/16667/stat : 16667 (gnovelty+2) R 16665 16667 16667 0 -1 4202496 570 0 0 0 51 0 0 0 25 0 1 0 617218638 1042522112 306 33554432000 134512640 135663704 4288046368 18446744073709551615 134525684 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/16667/statm: 254522 306 113 282 0 254238 0
[pid=16668] ppid=16656 vsize=300244 CPUtime=0 cores=1,3,5,7
/proc/16668/stat : 16668 (python) S 16656 16656 15742 0 -1 4202560 718 0 0 0 0 0 0 0 23 0 1 0 617218641 307449856 5838 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 982119249474486271 0 0 17 3 0 0 0
/proc/16668/statm: 75061 5838 344 364 0 10973 0
[pid=16670] ppid=16668 vsize=417048 CPUtime=0.48 cores=1,3,5,7
/proc/16670/stat : 16670 (adaptg2wsat2009) R 16668 16670 16670 0 -1 4202496 684 0 0 0 48 0 0 0 25 0 1 0 617218642 427057152 418 33554432000 134512640 135107411 4290738256 18446744073709551615 134523314 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/16670/statm: 104262 418 49 146 0 103138 0
[pid=16671] ppid=16656 vsize=310500 CPUtime=0 cores=1,3,5,7
/proc/16671/stat : 16671 (python) S 16656 16656 15742 0 -1 4202560 728 0 0 0 0 0 0 0 25 0 1 0 617218644 317952000 5842 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 4601354461711958015 0 0 17 7 0 0 0
/proc/16671/statm: 77625 5842 344 364 0 13534 0
[pid=16673] ppid=16671 vsize=417048 CPUtime=0.44 cores=1,3,5,7
/proc/16673/stat : 16673 (adaptg2wsat2009) R 16671 16673 16673 0 -1 4202496 680 0 0 0 44 0 0 0 25 0 1 0 617218644 427057152 418 33554432000 134512640 135107411 4293077760 18446744073709551615 134523215 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/16673/statm: 104262 418 49 146 0 103138 0
[pid=16656/tid=16663] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16663/stat : 16663 (python) S 16654 16656 15742 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 5 0 617218635 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 1 0 0 0
[pid=16656/tid=16666] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16666/stat : 16666 (python) S 16654 16656 15742 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 5 0 617218638 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 5 0 0 0
[pid=16656/tid=16669] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16669/stat : 16669 (python) S 16654 16656 15742 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 5 0 617218641 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16672] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16672/stat : 16672 (python) S 16654 16656 15742 0 -1 4202560 28 933 0 0 0 0 0 0 25 0 5 0 617218644 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2.92
Current children cumulated vsize (KiB) 4371160

[startup+3.10048 s]
/proc/loadavg: 8.02 8.05 7.55 9/264 16673
/proc/meminfo: memFree=23157724/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=320744 CPUtime=0.95 cores=1,3,5,7
/proc/16656/stat : 16656 (python) S 16654 16656 15742 0 -1 4202496 13082 933 1 0 85 10 0 0 25 0 5 0 617218540 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/16656/statm: 80186 7385 1883 364 0 16095 0
[pid=16662] ppid=16656 vsize=279412 CPUtime=0 cores=1,3,5,7
/proc/16662/stat : 16662 (python) S 16656 16656 15742 0 -1 4202560 729 0 0 0 0 0 0 0 18 0 1 0 617218635 286117888 5792 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612623 0 0 16781312 514 916853888517112070 0 0 17 3 0 0 0
/proc/16662/statm: 69853 5792 354 364 0 5771 0
[pid=16664] ppid=16662 vsize=1018088 CPUtime=2.14 cores=1,3,5,7
/proc/16664/stat : 16664 (gnovelty+2) R 16662 16664 16664 0 -1 4202496 570 0 0 0 214 0 0 0 25 0 1 0 617218635 1042522112 307 33554432000 134512640 135663704 4290000720 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/16664/statm: 254522 307 113 282 0 254238 0
[pid=16665] ppid=16656 vsize=289988 CPUtime=0 cores=1,3,5,7
/proc/16665/stat : 16665 (python) S 16656 16656 15742 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 617218638 296947712 5838 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 9986213208500699399 0 0 17 7 0 0 0
/proc/16665/statm: 72497 5838 348 364 0 8412 0
[pid=16667] ppid=16665 vsize=1018088 CPUtime=2.11 cores=1,3,5,7
/proc/16667/stat : 16667 (gnovelty+2) R 16665 16667 16667 0 -1 4202496 570 0 0 0 211 0 0 0 25 0 1 0 617218638 1042522112 306 33554432000 134512640 135663704 4288046368 18446744073709551615 134525056 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/16667/statm: 254522 306 113 282 0 254238 0
[pid=16668] ppid=16656 vsize=300244 CPUtime=0 cores=1,3,5,7
/proc/16668/stat : 16668 (python) S 16656 16656 15742 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 617218641 307449856 5842 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 982119249474453767 0 0 17 3 0 0 0
/proc/16668/statm: 75061 5842 348 364 0 10973 0
[pid=16670] ppid=16668 vsize=417192 CPUtime=2.08 cores=1,3,5,7
/proc/16670/stat : 16670 (adaptg2wsat2009) R 16668 16670 16670 0 -1 4202496 761 0 0 0 208 0 0 0 25 0 1 0 617218642 427204608 495 33554432000 134512640 135107411 4290738256 18446744073709551615 134531942 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/16670/statm: 104298 495 59 146 0 103174 0
[pid=16671] ppid=16656 vsize=310500 CPUtime=0 cores=1,3,5,7
/proc/16671/stat : 16671 (python) S 16656 16656 15742 0 -1 4202560 746 0 0 0 0 0 0 0 18 0 1 0 617218644 317952000 5846 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 4601354461711925511 0 0 17 7 0 0 0
/proc/16671/statm: 77625 5846 348 364 0 13534 0
[pid=16673] ppid=16671 vsize=417192 CPUtime=2.03 cores=1,3,5,7
/proc/16673/stat : 16673 (adaptg2wsat2009) R 16671 16673 16673 0 -1 4202496 756 0 0 0 203 0 0 0 25 0 1 0 617218644 427204608 494 33554432000 134512640 135107411 4293077760 18446744073709551615 134531739 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/16673/statm: 104298 494 59 146 0 103174 0
[pid=16656/tid=16663] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16663/stat : 16663 (python) S 16654 16656 15742 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 5 0 617218635 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 1 0 0 0
[pid=16656/tid=16666] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16666/stat : 16666 (python) S 16654 16656 15742 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 5 0 617218638 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 5 0 0 0
[pid=16656/tid=16669] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16669/stat : 16669 (python) S 16654 16656 15742 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 5 0 617218641 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16672] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16672/stat : 16672 (python) S 16654 16656 15742 0 -1 4202560 28 933 0 0 0 0 0 0 25 0 5 0 617218644 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 9.31
Current children cumulated vsize (KiB) 4371448

[startup+6.30091 s]
/proc/loadavg: 8.02 8.05 7.55 9/264 16673
/proc/meminfo: memFree=23156656/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=320744 CPUtime=0.95 cores=1,3,5,7
/proc/16656/stat : 16656 (python) S 16654 16656 15742 0 -1 4202496 13082 933 1 0 85 10 0 0 25 0 5 0 617218540 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/16656/statm: 80186 7385 1883 364 0 16095 0
[pid=16662] ppid=16656 vsize=279412 CPUtime=0 cores=1,3,5,7
/proc/16662/stat : 16662 (python) S 16656 16656 15742 0 -1 4202560 892 0 0 0 0 0 0 0 18 0 1 0 617218635 286117888 5815 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612623 0 0 16781312 514 916853888517112070 0 0 17 3 0 0 0
/proc/16662/statm: 69853 5815 377 364 0 5771 0
[pid=16664] ppid=16662 vsize=1018088 CPUtime=5.34 cores=1,3,5,7
/proc/16664/stat : 16664 (gnovelty+2) R 16662 16664 16664 0 -1 4202496 570 0 0 0 534 0 0 0 25 0 1 0 617218635 1042522112 307 33554432000 134512640 135663704 4290000720 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/16664/statm: 254522 307 113 282 0 254238 0
[pid=16665] ppid=16656 vsize=289988 CPUtime=0 cores=1,3,5,7
/proc/16665/stat : 16665 (python) S 16656 16656 15742 0 -1 4202560 905 0 0 0 0 0 0 0 18 0 1 0 617218638 296947712 5861 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 9986213208500699399 0 0 17 7 0 0 0
/proc/16665/statm: 72497 5861 371 364 0 8412 0
[pid=16667] ppid=16665 vsize=1018088 CPUtime=5.31 cores=1,3,5,7
/proc/16667/stat : 16667 (gnovelty+2) R 16665 16667 16667 0 -1 4202496 570 0 0 0 531 0 0 0 25 0 1 0 617218638 1042522112 306 33554432000 134512640 135663704 4288046368 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/16667/statm: 254522 306 113 282 0 254238 0
[pid=16668] ppid=16656 vsize=300244 CPUtime=0 cores=1,3,5,7
/proc/16668/stat : 16668 (python) S 16656 16656 15742 0 -1 4202560 901 0 0 0 0 0 0 0 18 0 1 0 617218641 307449856 5865 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 982119249474453767 0 0 17 3 0 0 0
/proc/16668/statm: 75061 5865 371 364 0 10973 0
[pid=16670] ppid=16668 vsize=417192 CPUtime=5.26 cores=1,3,5,7
/proc/16670/stat : 16670 (adaptg2wsat2009) R 16668 16670 16670 0 -1 4202496 761 0 0 0 526 0 0 0 25 0 1 0 617218642 427204608 495 33554432000 134512640 135107411 4290738256 18446744073709551615 134531709 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/16670/statm: 104298 495 59 146 0 103174 0
[pid=16671] ppid=16656 vsize=310500 CPUtime=0 cores=1,3,5,7
/proc/16671/stat : 16671 (python) S 16656 16656 15742 0 -1 4202560 919 0 0 0 0 0 0 0 18 0 1 0 617218644 317952000 5869 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 4601354461711925511 0 0 17 7 0 0 0
/proc/16671/statm: 77625 5869 371 364 0 13534 0
[pid=16673] ppid=16671 vsize=417192 CPUtime=5.19 cores=1,3,5,7
/proc/16673/stat : 16673 (adaptg2wsat2009) R 16671 16673 16673 0 -1 4202496 756 0 0 0 519 0 0 0 25 0 1 0 617218644 427204608 494 33554432000 134512640 135107411 4293077760 18446744073709551615 134531736 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/16673/statm: 104298 494 59 146 0 103174 0
[pid=16656/tid=16663] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16663/stat : 16663 (python) S 16654 16656 15742 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 5 0 617218635 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 1 0 0 0
[pid=16656/tid=16666] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16666/stat : 16666 (python) S 16654 16656 15742 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 5 0 617218638 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 5 0 0 0
[pid=16656/tid=16669] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16669/stat : 16669 (python) S 16654 16656 15742 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 5 0 617218641 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16672] ppid=16654 vsize=320744 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16672/stat : 16672 (python) S 16654 16656 15742 0 -1 4202560 28 933 0 0 0 0 0 0 25 0 5 0 617218644 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 22.05
Current children cumulated vsize (KiB) 4371448

[startup+12.7008 s]
/proc/loadavg: 8.02 8.05 7.55 9/264 16673
/proc/meminfo: memFree=23156052/32950940 swapFree=67111364/67111528
[pid=16656] ppid=16654 vsize=320744 CPUtime=0.95 cores=1,3,5,7
/proc/16656/stat : 16656 (python) S 16654 16656 15742 0 -1 4202496 13082 933 1 0 85 10 0 0 25 0 5 0 617218540 328441856 7385 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/16656/statm: 80186 7385 1883 364 0 16095 0
[pid=16662] ppid=16656 vsize=279412 CPUtime=0.01 cores=1,3,5,7
/proc/16662/stat : 16662 (python) S 16656 16656 15742 0 -1 4202560 896 0 0 0 0 1 0 0 18 0 1 0 617218635 286117888 5815 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612623 0 0 16781312 514 7828505197579043078 0 0 17 3 0 0 0
/proc/16662/statm: 69853 5815 377 364 0 5771 0
[pid=16664] ppid=16662 vsize=1018088 CPUtime=11.74 cores=1,3,5,7
/proc/16664/stat : 16664 (gnovelty+2) R 16662 16664 16664 0 -1 4202496 570 0 0 0 1174 0 0 0 25 0 1 0 617218635 1042522112 307 33554432000 134512640 135663704 4290000720 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/16664/statm: 254522 307 113 282 0 254238 0
[pid=16665] ppid=16656 vsize=289988 CPUtime=0.01 cores=1,3,5,7

################
# More data... #
################

/proc/16682/statm: 104298 496 59 146 0 103174 0
[pid=16679/tid=16700] ppid=16656 vsize=320756 CPUtime=0 cores=1,3,5,7
/proc/16679/task/16700/stat : 16700 (python) S 16656 16656 15742 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617238673 328454144 5903 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604466011067712 0 0 -1 5 0 0 0
[pid=16683] ppid=16656 vsize=331012 CPUtime=0.17 cores=1,3,5,7
/proc/16683/stat : 16683 (python) S 16656 16656 15742 0 -1 4202560 1121 0 0 0 8 9 0 0 19 0 2 0 617228659 338956288 5907 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/16683/statm: 82753 5907 400 364 0 18656 0
[pid=16686] ppid=16683 vsize=1018088 CPUtime=100 cores=1,3,5,7
/proc/16686/stat : 16686 (gnovelty+2) T 16683 16686 16686 0 -1 4202496 571 0 0 0 10000 0 0 0 25 0 1 0 617228660 1042522112 306 33554432000 134512640 135663704 4291645808 18446744073709551615 134525224 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/16686/statm: 254522 306 113 282 0 254238 0
[pid=16683/tid=16704] ppid=16656 vsize=331012 CPUtime=0 cores=1,3,5,7
/proc/16683/task/16704/stat : 16704 (python) S 16656 16656 15742 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 617238678 338956288 5907 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604471138030720 0 0 -1 1 0 0 0
[pid=16688] ppid=16656 vsize=341268 CPUtime=0.16 cores=1,3,5,7
/proc/16688/stat : 16688 (python) S 16656 16656 15742 0 -1 4202560 1178 0 0 0 10 6 0 0 19 0 2 0 617228663 349458432 5917 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/16688/statm: 85317 5917 406 364 0 21217 0
[pid=16690] ppid=16688 vsize=247344 CPUtime=100.01 cores=1,3,5,7
/proc/16690/stat : 16690 (FH) T 16688 16690 16690 0 -1 4202496 708 0 0 0 10001 0 0 0 25 0 1 0 617228663 253280256 443 33554432000 134512640 134936988 4292321616 18446744073709551615 134553531 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/16690/statm: 61836 443 41 104 0 61340 0
[pid=16688/tid=16705] ppid=16656 vsize=341268 CPUtime=0 cores=1,3,5,7
/proc/16688/task/16705/stat : 16705 (python) S 16656 16656 15742 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617238681 349458432 5917 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604462205545472 0 0 -1 3 0 0 0
[pid=16691] ppid=16656 vsize=351524 CPUtime=0.34 cores=1,3,5,7
/proc/16691/stat : 16691 (python) S 16656 16656 15742 0 -1 4202560 1192 0 0 0 20 14 0 0 18 0 2 0 617228667 359960576 5921 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/16691/statm: 87881 5921 406 364 0 23778 0
[pid=16693] ppid=16691 vsize=417192 CPUtime=192.8 cores=1,3,5,7
/proc/16693/stat : 16693 (adaptg2wsat2009) R 16691 16693 16693 0 -1 4202496 770 0 0 0 19280 0 0 0 25 0 1 0 617228667 427204608 496 33554432000 134512640 135107411 4289761680 18446744073709551615 134531707 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/16693/statm: 104298 496 59 146 0 103174 0
[pid=16691/tid=16696] ppid=16656 vsize=351524 CPUtime=0 cores=1,3,5,7
/proc/16691/task/16696/stat : 16696 (python) S 16656 16656 15742 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617238585 359960576 5921 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604464744473728 0 0 -1 3 0 0 0
[pid=16697] ppid=16656 vsize=361780 CPUtime=0.17 cores=1,3,5,7
/proc/16697/stat : 16697 (python) S 16656 16656 15742 0 -1 4202560 1146 0 0 0 10 7 0 0 19 0 2 0 617238589 370462720 5919 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/16697/statm: 90445 5919 400 364 0 26339 0
[pid=16699] ppid=16697 vsize=417192 CPUtime=100.16 cores=1,3,5,7
/proc/16699/stat : 16699 (adaptg2wsat2009) T 16697 16699 16699 0 -1 4202496 766 0 0 0 10016 0 0 0 25 0 1 0 617238589 427204608 495 33554432000 134512640 135107411 4290175824 18446744073709551615 134532036 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/16699/statm: 104298 495 59 146 0 103174 0
[pid=16697/tid=16759] ppid=16656 vsize=361780 CPUtime=0 cores=1,3,5,7
/proc/16697/task/16759/stat : 16759 (python) S 16656 16656 15742 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617248606 370462720 5919 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604465546281088 0 0 -1 7 0 0 0
[pid=16701] ppid=16656 vsize=372036 CPUtime=0.19 cores=1,3,5,7
/proc/16701/stat : 16701 (python) S 16656 16656 15742 0 -1 4202560 1160 0 0 0 12 7 0 0 19 0 2 0 617238677 380964864 5923 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/16701/statm: 93009 5923 400 364 0 28900 0
[pid=16703] ppid=16701 vsize=247348 CPUtime=102.91 cores=1,3,5,7
/proc/16703/stat : 16703 (FH) T 16701 16703 16703 0 -1 4202496 712 0 0 0 10291 0 0 0 25 0 1 0 617238677 253284352 443 33554432000 134512640 134936988 4291045824 18446744073709551615 134552382 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/16703/statm: 61837 443 41 104 0 61341 0
[pid=16701/tid=16769] ppid=16656 vsize=372036 CPUtime=0 cores=1,3,5,7
/proc/16701/task/16769/stat : 16769 (python) S 16656 16656 15742 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 617249195 380964864 5923 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604465546282880 0 0 -1 7 0 0 0
[pid=16706] ppid=16656 vsize=382292 CPUtime=0.53 cores=1,3,5,7
/proc/16706/stat : 16706 (python) S 16656 16656 15742 0 -1 4202560 1185 0 0 0 30 23 0 0 18 0 2 0 617238682 391467008 5927 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/16706/statm: 95573 5927 400 364 0 31461 0
[pid=16708] ppid=16706 vsize=268424 CPUtime=295.1 cores=1,3,5,7
/proc/16708/stat : 16708 (hybridGM3) T 16706 16708 16708 0 -1 4202496 4944 0 0 0 29510 0 0 0 25 0 1 0 617238682 274866176 4672 33554432000 134512640 135204385 4292150208 18446744073709551615 134536986 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/16708/statm: 67106 4672 64 169 0 66936 0
[pid=16706/tid=16773] ppid=16656 vsize=382292 CPUtime=0 cores=1,3,5,7
/proc/16706/task/16773/stat : 16773 (python) S 16656 16656 15742 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 617258126 391467008 5927 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604468122690688 0 0 -1 7 0 0 0
[pid=16709] ppid=16656 vsize=392548 CPUtime=0.17 cores=1,3,5,7
/proc/16709/stat : 16709 (python) S 16656 16656 15742 0 -1 4202560 1167 0 0 0 9 8 0 0 19 0 2 0 617238686 401969152 5931 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/16709/statm: 98137 5931 400 364 0 34022 0
[pid=16711] ppid=16709 vsize=1018088 CPUtime=99.99 cores=1,3,5,7
/proc/16711/stat : 16711 (gnovelty+2) T 16709 16711 16711 0 -1 4202496 576 0 0 0 9999 0 0 0 25 0 1 0 617238686 1042522112 306 33554432000 134512640 135663704 4292143808 18446744073709551615 134525278 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/16711/statm: 254522 306 113 282 0 254238 0
[pid=16709/tid=16763] ppid=16656 vsize=392548 CPUtime=0 cores=1,3,5,7
/proc/16709/task/16763/stat : 16763 (python) S 16656 16656 15742 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 617248704 401969152 5931 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604471153409088 0 0 -1 7 0 0 0
[pid=16760] ppid=16656 vsize=402804 CPUtime=0.55 cores=1,3,5,7
/proc/16760/stat : 16760 (python) S 16656 16656 15742 0 -1 4202560 1181 0 0 0 29 26 0 0 18 0 2 0 617248610 412471296 5967 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/16760/statm: 100701 5967 400 364 0 36583 0
[pid=16762] ppid=16760 vsize=268420 CPUtime=294.84 cores=1,3,5,7
/proc/16762/stat : 16762 (hybridGM3) T 16760 16762 16762 0 -1 4202496 4939 0 0 0 29483 1 0 0 25 0 1 0 617248611 274862080 4671 33554432000 134512640 135204385 4286623312 18446744073709551615 134537272 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/16762/statm: 67105 4671 64 169 0 66935 0
[pid=16760/tid=16786] ppid=16656 vsize=402804 CPUtime=0 cores=1,3,5,7
/proc/16760/task/16786/stat : 16786 (python) S 16656 16656 15742 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 617268248 412471296 5967 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604459204106432 0 0 -1 7 0 0 0
[pid=16774] ppid=16656 vsize=413344 CPUtime=0.18 cores=1,3,5,7
/proc/16774/stat : 16774 (python) S 16656 16656 15742 0 -1 4202560 1170 0 0 0 9 9 0 0 19 0 2 0 617258429 423264256 6010 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/16774/statm: 103336 6010 400 364 0 39215 0
[pid=16776] ppid=16774 vsize=247344 CPUtime=98.43 cores=1,3,5,7
/proc/16776/stat : 16776 (FH) T 16774 16776 16776 0 -1 4202496 717 0 0 0 9843 0 0 0 25 0 1 0 617258430 253280256 443 33554432000 134512640 134936988 4291526928 18446744073709551615 134552493 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/16776/statm: 61836 443 41 104 0 61340 0
[pid=16774/tid=16790] ppid=16656 vsize=413344 CPUtime=0 cores=1,3,5,7
/proc/16774/task/16790/stat : 16790 (python) S 16656 16656 15742 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 617268447 423264256 6010 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604463286485184 0 0 -1 3 0 0 0
[pid=16781] ppid=16656 vsize=423600 CPUtime=0.93 cores=1,3,5,7
/proc/16781/stat : 16781 (python) S 16656 16656 15742 0 -1 4202560 1246 0 0 0 54 39 0 0 15 0 2 0 617259423 433766400 6022 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/16781/statm: 105900 6022 408 364 0 41776 0
[pid=16783] ppid=16781 vsize=63256 CPUtime=475.39 cores=1,3,5,7
/proc/16783/stat : 16783 (march_hi) R 16781 16783 16783 0 -1 4202496 15642 0 0 0 47536 3 0 0 25 0 1 0 617259423 64774144 14084 33554432000 4194304 4955518 140736224271472 18446744073709551615 4297337 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/16783/statm: 15814 14084 106 186 0 15627 0
[pid=16781/tid=16812] ppid=16656 vsize=423600 CPUtime=0 cores=1,3,5,7
/proc/16781/task/16812/stat : 16812 (python) S 16656 16656 15742 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 617298749 433766400 6022 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604471161566208 0 0 -1 3 0 0 0
[pid=16787] ppid=16656 vsize=433856 CPUtime=0.72 cores=1,3,5,7
/proc/16787/stat : 16787 (python) S 16656 16656 15742 0 -1 4202560 1235 0 0 0 40 32 0 0 18 0 2 0 617268257 444268544 6032 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/16787/statm: 108464 6032 408 364 0 44337 0
[pid=16789] ppid=16787 vsize=268424 CPUtime=390.72 cores=1,3,5,7
/proc/16789/stat : 16789 (hybridGM3) T 16787 16789 16789 0 -1 4202496 4952 0 0 0 39072 0 0 0 25 0 1 0 617268257 274866176 4672 33554432000 134512640 135204385 4287566848 18446744073709551615 134536961 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/16789/statm: 67106 4672 64 169 0 66936 0
[pid=16787/tid=16804] ppid=16656 vsize=433856 CPUtime=0 cores=1,3,5,7
/proc/16787/task/16804/stat : 16804 (python) S 16656 16656 15742 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 617287795 444268544 6032 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604459204104640 0 0 -1 3 0 0 0
[pid=16791] ppid=16656 vsize=444292 CPUtime=0.55 cores=1,3,5,7
/proc/16791/stat : 16791 (python) S 16656 16656 15742 0 -1 4202560 1222 0 0 0 34 21 0 0 18 0 2 0 617268452 454955008 6069 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000419275 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/16791/statm: 111073 6069 402 364 0 46943 0
[pid=16793] ppid=16791 vsize=268420 CPUtime=292.68 cores=1,3,5,7
/proc/16793/stat : 16793 (hybridGM3) T 16791 16793 16793 0 -1 4202496 4952 0 0 0 29268 0 0 0 25 0 1 0 617268452 274862080 4671 33554432000 134512640 135204385 4288221968 18446744073709551615 134536744 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/16793/statm: 67105 4671 64 169 0 66935 0
[pid=16791/tid=16805] ppid=16656 vsize=444292 CPUtime=0 cores=1,3,5,7
/proc/16791/task/16805/stat : 16805 (python) S 16656 16656 15742 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 617287894 454955008 6069 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 514 18446604459204105536 0 0 -1 7 0 0 0
[pid=16797] ppid=16656 vsize=454548 CPUtime=0.52 cores=1,3,5,7
/proc/16797/stat : 16797 (python) S 16656 16656 15742 0 -1 4202560 1060 0 0 0 31 21 0 0 18 0 1 0 617278375 465457152 6043 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 7828505197579043078 0 0 17 5 0 0 0
/proc/16797/statm: 113637 6043 373 364 0 49504 0
[pid=16799] ppid=16797 vsize=455824 CPUtime=291.03 cores=1,3,5,7
/proc/16799/stat : 16799 (TNM) R 16797 16799 16799 0 -1 4202496 708 0 0 0 29103 0 0 0 25 0 1 0 617278376 466763776 435 33554432000 134512640 135008748 4292711088 18446744073709551615 134529448 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/16799/statm: 113956 435 46 122 0 112856 0
[pid=16816] ppid=16656 vsize=464700 CPUtime=0 cores=1,3,5,7
/proc/16816/stat : 16816 (python) S 16656 16656 15742 0 -1 4202560 862 0 0 0 0 0 0 0 15 0 1 0 617307442 475852800 5997 33554432000 4194304 5683644 140736313795264 18446744073709551615 215988612678 0 0 16781312 514 12773835824726769927 0 0 17 7 0 0 0
/proc/16816/statm: 116175 5997 349 364 0 52039 0
[pid=16818] ppid=16816 vsize=3032 CPUtime=0.85 cores=1,3,5,7
/proc/16818/stat : 16818 (march_hi) R 16816 16818 16818 0 -1 4202496 2086 0 0 0 85 0 0 0 23 0 1 0 617307443 3104768 525 33554432000 4194304 4955518 140736672558304 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/16818/statm: 758 525 106 186 0 571 0
[pid=16656/tid=16663] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16663/stat : 16663 (python) S 16654 16656 15742 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 20 0 617218635 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 1 0 0 0
[pid=16656/tid=16666] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16666/stat : 16666 (python) S 16654 16656 15742 0 -1 4202560 22 933 0 0 0 0 0 0 22 0 20 0 617218638 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 5 0 0 0
[pid=16656/tid=16669] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16669/stat : 16669 (python) S 16654 16656 15742 0 -1 4202560 28 933 0 0 0 0 0 0 18 0 20 0 617218641 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16672] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16672/stat : 16672 (python) S 16654 16656 15742 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 20 0 617218644 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16680] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16680/stat : 16680 (python) S 16654 16656 15742 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 20 0 617228655 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16685] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16685/stat : 16685 (python) S 16654 16656 15742 0 -1 4202560 27 933 0 0 0 0 0 0 21 0 20 0 617228660 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 3 0 0 0
[pid=16656/tid=16689] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16689/stat : 16689 (python) S 16654 16656 15742 0 -1 4202560 35 933 0 0 0 0 0 0 23 0 20 0 617228663 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16692] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16692/stat : 16692 (python) S 16654 16656 15742 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 20 0 617228667 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16698] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16698/stat : 16698 (python) S 16654 16656 15742 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 20 0 617238589 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16702] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16702/stat : 16702 (python) S 16654 16656 15742 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 20 0 617238677 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16707] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16707/stat : 16707 (python) S 16654 16656 15742 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 20 0 617238682 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16710] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16710/stat : 16710 (python) S 16654 16656 15742 0 -1 4202560 30 933 0 0 0 0 0 0 23 0 20 0 617238686 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 3 0 0 0
[pid=16656/tid=16761] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16761/stat : 16761 (python) S 16654 16656 15742 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 20 0 617248611 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 3 0 0 0
[pid=16656/tid=16775] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16775/stat : 16775 (python) S 16654 16656 15742 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 20 0 617258429 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 3 0 0 0
[pid=16656/tid=16782] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16782/stat : 16782 (python) S 16654 16656 15742 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 20 0 617259423 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 3 0 0 0
[pid=16656/tid=16788] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16788/stat : 16788 (python) S 16654 16656 15742 0 -1 4202560 31 933 0 0 0 0 0 0 18 0 20 0 617268257 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16792] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16792/stat : 16792 (python) S 16654 16656 15742 0 -1 4202560 33 933 0 0 0 0 0 0 18 0 20 0 617268452 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16798] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16798/stat : 16798 (python) S 16654 16656 15742 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 20 0 617278375 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
[pid=16656/tid=16817] ppid=16654 vsize=474944 CPUtime=0 cores=1,3,5,7
/proc/16656/task/16817/stat : 16817 (python) S 16654 16656 15742 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 20 0 617307443 486342656 7566 33554432000 4194304 5683644 140736313795264 18446744073709551615 216000416177 0 0 16781312 2 18446604471096071488 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3539.66
Current children cumulated vsize (KiB) 16051840

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 16656 and gives
#  childrusage.ru_utime.tv_sec=197
#  childrusage.ru_utime.tv_usec=780932
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=570609
# CPU time returned by wait4() is 200.352
# while last known CPU time is 3539.66
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 890.025
CPU time (s): 3539.66
CPU user time (s): 3536.73
CPU system time (s): 2.93
CPU usage (%): 397.703
Max. virtual memory (cumulated for all children) (KiB): 16051840

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 197.781
system time used= 2.57061
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 49648
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5159
involuntary context switches= 2067

runsolver used 5.12322 second user time and 11.4693 second system time

The end

Launcher Data 

Begin job on node117 at 2011-05-01 19:11:18
IDJOB=3317496
IDBENCH=82961
IDSOLVER=1705
FILE ID=node117/3317496-1304269877
RUNJOBID= node117-1304267285-15761
PBS_JOBID= 13173962
Free space on /tmp= 72584 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S1321406709-042.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317496-1304269877/watcher-3317496-1304269877 -o /tmp/evaluation-result-3317496-1304269877/solver-3317496-1304269877 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317496-1304269877.cnf -seed 1320683075 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 383bd72f29b3667b7a0d512d7fa797ad
RANDOM SEED=1320683075

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      23202748 kB
Buffers:       1906976 kB
Cached:        5789648 kB
SwapCached:        160 kB
Active:        2657040 kB
Inactive:      6505144 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      23202748 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           32092 kB
Writeback:           0 kB
AnonPages:     1465968 kB
Mapped:          25896 kB
Slab:           507812 kB
PageTables:      17256 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS: 28246544 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72552 MiB
End job on node117 at 2011-05-01 19:26:10