Trace number 3378625

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? 4794.61 1202.27

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v500-c2130-S2061684506-048.UNKNOWN.cnf
MD5SUM96dadba6913c3cf46368b605f559e4b9
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark1288.13
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.34/0.68	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.34/0.68	c INFO: solving HOME/instance-3378625-1305466523.cnf
0.34/0.68	c INFO: computing task features
0.34/0.69	c INFO: computed variable-clause graph statistics
0.34/0.69	c INFO: computed clause balance statistics
0.42/0.73	c INFO: computed variable balance statistics
0.42/0.76	c INFO: computed small-clause counts
0.42/0.76	c INFO: computed Horn-clause counts
0.42/0.76	c INFO: computed variable graph statistics
0.42/0.76	c INFO: collected features for HOME/instance-3378625-1305466523.cnf in 0.08 s
0.42/0.77	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.57)
0.42/0.78	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.34)
0.42/0.79	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.06)
0.52/0.80	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.32)
395.99/100.00	c INFO: running march_hi@0 for 679 with 1100 remaining (b = 0.39)
399.09/100.97	c INFO: running march_hi@0 for 679 with 1099 remaining (b = 0.35)
1556.08/390.62	c INFO: running march_hi@0 for 97 with 810 remaining (b = 0.07)
1565.66/393.07	c INFO: running march_hi@99 for 291 with 807 remaining (b = 0.19)
1956.42/490.93	c INFO: running march_hi@390 for 97 with 709 remaining (b = 0.08)
2339.93/587.10	c INFO: running kcnfs-2006@0 for 613 with 613 remaining (b = 0.29)
2726.60/683.94	c INFO: running march_hi@0 for 97 with 516 remaining (b = 0.07)
3129.96/785.08	c INFO: running march_hi@199 for 194 with 415 remaining (b = 0.13)
3135.58/786.50	c INFO: running march_hi@388 for 97 with 414 remaining (b = 0.06)
3155.07/791.40	c INFO: running march_hi@99 for 409 with 409 remaining (b = 0.19)
3525.17/884.12	c INFO: running march_hi@486 for 97 with 316 remaining (b = 0.06)
3904.97/979.31	c INFO: running march_hi@581 for 221 with 221 remaining (b = 0.12)
3920.95/983.35	c INFO: running march_hi@301 for 97 with 217 remaining (b = 0.10)
4293.56/1076.80	c INFO: running march_hi@394 for 123 with 123 remaining (b = 0.07)
4794.29/1202.25	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378625-1305466523/watcher-3378625-1305466523 -o /tmp/evaluation-result-3378625-1305466523/solver-3378625-1305466523 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378625-1305466523.cnf -seed 2001714417 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.21 8.03 6.09 5/309 8239
/proc/meminfo: memFree=19571764/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=15756 CPUtime=0 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 839 0 0 0 0 0 0 0 25 0 1 0 658252059 16134144 724 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278877696 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/8239/statm: 3939 724 334 364 0 426 0

[startup+0.0127401 s]
/proc/loadavg: 8.21 8.03 6.09 5/309 8239
/proc/meminfo: memFree=19571764/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 902 0 0 0 0 0 0 0 25 0 1 0 658252059 16154624 784 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278877696 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/8239/statm: 3944 784 346 364 0 431 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15776

[startup+0.10073 s]
/proc/loadavg: 8.21 8.03 6.09 5/309 8239
/proc/meminfo: memFree=19571764/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=87704 CPUtime=0.06 cores=1,3,5,7
/proc/8239/stat : 8239 (python) R 8237 8239 7577 0 -1 4202496 2636 0 0 0 4 2 0 0 25 0 1 0 658252059 89808896 2354 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278877696 0 2147483391 16781312 2 0 0 0 17 7 0 0 0
/proc/8239/statm: 21926 2354 717 364 0 1622 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 87704

[startup+0.300694 s]
/proc/loadavg: 8.21 8.03 6.09 5/309 8239
/proc/meminfo: memFree=19571764/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=135752 CPUtime=0.18 cores=1,3,5,7
/proc/8239/stat : 8239 (python) R 8237 8239 7577 0 -1 4202496 5962 935 1 0 13 5 0 0 25 0 1 0 658252059 139010048 4152 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278875653 0 2147483391 16781312 2 0 0 0 17 7 0 0 0
/proc/8239/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135752

[startup+0.700683 s]
/proc/loadavg: 8.21 8.03 6.09 5/309 8239
/proc/meminfo: memFree=19571764/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=278304 CPUtime=0.42 cores=1,3,5,7
/proc/8239/stat : 8239 (python) R 8237 8239 7577 0 -1 4202496 9322 935 1 0 31 11 0 0 15 0 1 0 658252059 284983296 6993 33554432000 4194304 5683644 140736700223024 18446744073709551615 47116164392670 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/8239/statm: 69576 6993 1826 364 0 5500 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 278304

[startup+1.50047 s]
/proc/loadavg: 8.21 8.03 6.09 9/322 8256
/proc/meminfo: memFree=19531720/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=320200 CPUtime=0.52 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 12214 935 1 0 41 11 0 0 24 0 5 0 658252059 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8239/statm: 80050 7239 1873 364 0 15959 0
[pid=8245] ppid=8239 vsize=278512 CPUtime=0 cores=1,3,5,7
/proc/8245/stat : 8245 (python) S 8239 8239 7577 0 -1 4202560 730 0 0 0 0 0 0 0 18 0 1 0 658252137 285196288 5552 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899727 0 0 16781312 514 9855713822667440127 0 0 17 3 0 0 0
/proc/8245/statm: 69628 5552 354 364 0 5546 0
[pid=8247] ppid=8245 vsize=1017800 CPUtime=0.71 cores=1,3,5,7
/proc/8247/stat : 8247 (gnovelty+2) R 8245 8247 8247 0 -1 4202496 441 0 0 0 71 0 0 0 21 0 1 0 658252137 1042227200 178 33554432000 134512640 135663704 4288324928 18446744073709551615 134547363 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8247/statm: 254450 178 113 282 0 254166 0
[pid=8248] ppid=8239 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/8248/stat : 8248 (python) S 8239 8239 7577 0 -1 4202560 748 0 0 0 0 0 0 0 15 0 1 0 658252138 296288256 5694 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 11037355120747577608 0 0 17 1 0 0 0
/proc/8248/statm: 72336 5694 348 364 0 8251 0
[pid=8250] ppid=8248 vsize=2436 CPUtime=0.7 cores=1,3,5,7
/proc/8250/stat : 8250 (march_hi) R 8248 8250 8250 0 -1 4202496 660 0 0 0 70 0 0 0 22 0 1 0 658252138 2494464 397 33554432000 4194304 4955518 140737296968512 18446744073709551615 4224687 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8250/statm: 609 397 104 186 0 422 0
[pid=8251] ppid=8239 vsize=299092 CPUtime=0 cores=1,3,5,7
/proc/8251/stat : 8251 (python) S 8239 8239 7577 0 -1 4202560 742 0 0 0 0 0 0 0 21 0 1 0 658252139 306270208 5586 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 11541975157681684479 0 0 17 3 0 0 0
/proc/8251/statm: 74773 5586 348 364 0 10685 0
[pid=8253] ppid=8251 vsize=1017800 CPUtime=0.7 cores=1,3,5,7
/proc/8253/stat : 8253 (gnovelty+2) R 8251 8253 8253 0 -1 4202496 439 0 0 0 70 0 0 0 25 0 1 0 658252139 1042227200 178 33554432000 134512640 135663704 4287349824 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8253/statm: 254450 178 113 282 0 254166 0
[pid=8254] ppid=8239 vsize=309956 CPUtime=0 cores=1,3,5,7
/proc/8254/stat : 8254 (python) S 8239 8239 7577 0 -1 4202560 746 0 0 0 0 0 0 0 20 0 1 0 658252140 317394944 5710 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 15020586681017270534 0 0 17 3 0 0 0
/proc/8254/statm: 77489 5710 348 364 0 13398 0
[pid=8256] ppid=8254 vsize=2436 CPUtime=0.68 cores=1,3,5,7
/proc/8256/stat : 8256 (march_hi) R 8254 8256 8256 0 -1 4202496 659 0 0 0 68 0 0 0 25 0 1 0 658252140 2494464 397 33554432000 4194304 4955518 140734067536848 18446744073709551615 4238438 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8256/statm: 609 397 104 186 0 422 0
[pid=8239/tid=8246] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8246/stat : 8246 (python) S 8237 8239 7577 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 5 0 658252137 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8249] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8249/stat : 8249 (python) S 8237 8239 7577 0 -1 4202560 24 935 0 0 0 0 0 0 18 0 5 0 658252138 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8252] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8252/stat : 8252 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 22 0 5 0 658252139 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8255] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8255/stat : 8255 (python) S 8237 8239 7577 0 -1 4202560 23 935 0 0 0 0 0 0 24 0 5 0 658252140 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3.31
Current children cumulated vsize (KiB) 3537576

[startup+3.10119 s]
/proc/loadavg: 8.19 8.03 6.10 9/322 8256
/proc/meminfo: memFree=19530480/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=320200 CPUtime=0.52 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 12214 935 1 0 41 11 0 0 24 0 5 0 658252059 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8239/statm: 80050 7239 1873 364 0 15959 0
[pid=8245] ppid=8239 vsize=278512 CPUtime=0 cores=1,3,5,7
/proc/8245/stat : 8245 (python) S 8239 8239 7577 0 -1 4202560 730 0 0 0 0 0 0 0 18 0 1 0 658252137 285196288 5552 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899727 0 0 16781312 514 9855713822667407624 0 0 17 3 0 0 0
/proc/8245/statm: 69628 5552 354 364 0 5546 0
[pid=8247] ppid=8245 vsize=1017800 CPUtime=2.3 cores=1,3,5,7
/proc/8247/stat : 8247 (gnovelty+2) R 8245 8247 8247 0 -1 4202496 441 0 0 0 230 0 0 0 25 0 1 0 658252137 1042227200 178 33554432000 134512640 135663704 4288324928 18446744073709551615 134545693 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8247/statm: 254450 178 113 282 0 254166 0
[pid=8248] ppid=8239 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/8248/stat : 8248 (python) S 8239 8239 7577 0 -1 4202560 750 0 0 0 0 0 0 0 15 0 1 0 658252138 296288256 5694 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 11037355120747577608 0 0 17 1 0 0 0
/proc/8248/statm: 72336 5694 348 364 0 8251 0
[pid=8250] ppid=8248 vsize=2756 CPUtime=2.3 cores=1,3,5,7
/proc/8250/stat : 8250 (march_hi) R 8248 8250 8250 0 -1 4202496 736 0 0 0 230 0 0 0 25 0 1 0 658252138 2822144 473 33554432000 4194304 4955518 140737296968512 18446744073709551615 4224615 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8250/statm: 689 473 104 186 0 502 0
[pid=8251] ppid=8239 vsize=299092 CPUtime=0 cores=1,3,5,7
/proc/8251/stat : 8251 (python) S 8239 8239 7577 0 -1 4202560 742 0 0 0 0 0 0 0 18 0 1 0 658252139 306270208 5586 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 11541975157681651973 0 0 17 3 0 0 0
/proc/8251/statm: 74773 5586 348 364 0 10685 0
[pid=8253] ppid=8251 vsize=1017800 CPUtime=2.3 cores=1,3,5,7
/proc/8253/stat : 8253 (gnovelty+2) R 8251 8253 8253 0 -1 4202496 439 0 0 0 230 0 0 0 25 0 1 0 658252139 1042227200 178 33554432000 134512640 135663704 4287349824 18446744073709551615 134545619 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8253/statm: 254450 178 113 282 0 254166 0
[pid=8254] ppid=8239 vsize=309956 CPUtime=0 cores=1,3,5,7
/proc/8254/stat : 8254 (python) S 8239 8239 7577 0 -1 4202560 748 0 0 0 0 0 0 0 15 0 1 0 658252140 317394944 5710 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 15020586681017270534 0 0 17 3 0 0 0
/proc/8254/statm: 77489 5710 348 364 0 13398 0
[pid=8256] ppid=8254 vsize=2756 CPUtime=2.28 cores=1,3,5,7
/proc/8256/stat : 8256 (march_hi) R 8254 8256 8256 0 -1 4202496 736 0 0 0 228 0 0 0 25 0 1 0 658252140 2822144 474 33554432000 4194304 4955518 140734067536848 18446744073709551615 4206520 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8256/statm: 689 474 104 186 0 502 0
[pid=8239/tid=8246] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8246/stat : 8246 (python) S 8237 8239 7577 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 5 0 658252137 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8249] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8249/stat : 8249 (python) S 8237 8239 7577 0 -1 4202560 24 935 0 0 0 0 0 0 18 0 5 0 658252138 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8252] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8252/stat : 8252 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 22 0 5 0 658252139 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8255] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8255/stat : 8255 (python) S 8237 8239 7577 0 -1 4202560 23 935 0 0 0 0 0 0 24 0 5 0 658252140 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 9.7
Current children cumulated vsize (KiB) 3538216

[startup+6.30062 s]
/proc/loadavg: 8.19 8.03 6.10 9/323 8257
/proc/meminfo: memFree=19525192/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=320200 CPUtime=0.52 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 12214 935 1 0 41 11 0 0 24 0 5 0 658252059 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8239/statm: 80050 7239 1873 364 0 15959 0
[pid=8245] ppid=8239 vsize=278512 CPUtime=0 cores=1,3,5,7
/proc/8245/stat : 8245 (python) S 8239 8239 7577 0 -1 4202560 892 0 0 0 0 0 0 0 18 0 1 0 658252137 285196288 5575 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899727 0 0 16781312 514 9855713822667407624 0 0 17 3 0 0 0
/proc/8245/statm: 69628 5575 377 364 0 5546 0
[pid=8247] ppid=8245 vsize=1017800 CPUtime=5.47 cores=1,3,5,7
/proc/8247/stat : 8247 (gnovelty+2) R 8245 8247 8247 0 -1 4202496 441 0 0 0 547 0 0 0 25 0 1 0 658252137 1042227200 178 33554432000 134512640 135663704 4288324928 18446744073709551615 134547281 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8247/statm: 254450 178 113 282 0 254166 0
[pid=8248] ppid=8239 vsize=289344 CPUtime=0.01 cores=1,3,5,7
/proc/8248/stat : 8248 (python) S 8239 8239 7577 0 -1 4202560 925 0 0 0 0 1 0 0 15 0 1 0 658252138 296288256 5717 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 11037355120747577608 0 0 17 1 0 0 0
/proc/8248/statm: 72336 5717 371 364 0 8251 0
[pid=8250] ppid=8248 vsize=3548 CPUtime=5.49 cores=1,3,5,7
/proc/8250/stat : 8250 (march_hi) R 8248 8250 8250 0 -1 4202496 891 0 0 0 549 0 0 0 25 0 1 0 658252138 3633152 628 33554432000 4194304 4955518 140737296968512 18446744073709551615 4227760 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8250/statm: 887 628 104 186 0 700 0
[pid=8251] ppid=8239 vsize=299092 CPUtime=0 cores=1,3,5,7
/proc/8251/stat : 8251 (python) S 8239 8239 7577 0 -1 4202560 902 0 0 0 0 0 0 0 18 0 1 0 658252139 306270208 5609 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 11541975157681651973 0 0 17 3 0 0 0
/proc/8251/statm: 74773 5609 371 364 0 10685 0
[pid=8253] ppid=8251 vsize=1017800 CPUtime=5.5 cores=1,3,5,7
/proc/8253/stat : 8253 (gnovelty+2) R 8251 8253 8253 0 -1 4202496 439 0 0 0 550 0 0 0 25 0 1 0 658252139 1042227200 178 33554432000 134512640 135663704 4287349824 18446744073709551615 134545776 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8253/statm: 254450 178 113 282 0 254166 0
[pid=8254] ppid=8239 vsize=309956 CPUtime=0 cores=1,3,5,7
/proc/8254/stat : 8254 (python) S 8239 8239 7577 0 -1 4202560 902 0 0 0 0 0 0 0 15 0 1 0 658252140 317394944 5733 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 15020586681017270534 0 0 17 3 0 0 0
/proc/8254/statm: 77489 5733 371 364 0 13398 0
[pid=8256] ppid=8254 vsize=3548 CPUtime=5.44 cores=1,3,5,7
/proc/8256/stat : 8256 (march_hi) R 8254 8256 8256 0 -1 4202496 887 0 0 0 544 0 0 0 25 0 1 0 658252140 3633152 625 33554432000 4194304 4955518 140734067536848 18446744073709551615 4227647 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8256/statm: 887 625 104 186 0 700 0
[pid=8239/tid=8246] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8246/stat : 8246 (python) S 8237 8239 7577 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 5 0 658252137 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8249] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8249/stat : 8249 (python) S 8237 8239 7577 0 -1 4202560 24 935 0 0 0 0 0 0 18 0 5 0 658252138 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8252] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8252/stat : 8252 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 22 0 5 0 658252139 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8255] ppid=8237 vsize=320200 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8255/stat : 8255 (python) S 8237 8239 7577 0 -1 4202560 23 935 0 0 0 0 0 0 24 0 5 0 658252140 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 22.43
Current children cumulated vsize (KiB) 3539800

[startup+12.7005 s]
/proc/loadavg: 8.16 8.03 6.12 9/323 8257
/proc/meminfo: memFree=19520472/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=320200 CPUtime=0.52 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 12214 935 1 0 41 11 0 0 24 0 5 0 658252059 327884800 7239 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8239/statm: 80050 7239 1873 364 0 15959 0
[pid=8245] ppid=8239 vsize=278512 CPUtime=0.02 cores=1,3,5,7
/proc/8245/stat : 8245 (python) S 8239 8239 7577 0 -1 4202560 895 0 0 0 1 1 0 0 18 0 1 0 658252137 285196288 5575 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899727 0 0 16781312 514 2355224275440337160 0 0 17 3 0 0 0
/proc/8245/statm: 69628 5575 377 364 0 5546 0
[pid=8247] ppid=8245 vsize=1017800 CPUtime=11.81 cores=1,3,5,7
/proc/8247/stat : 8247 (gnovelty+2) R 8245 8247 8247 0 -1 4202496 441 0 0 0 1181 0 0 0 25 0 1 0 658252137 1042227200 178 33554432000 134512640 135663704 4288324928 18446744073709551615 134543945 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8247/statm: 254450 178 113 282 0 254166 0
[pid=8248] ppid=8239 vsize=289344 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

[pid=8239] ppid=8237 vsize=361492 CPUtime=0.63 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 16279 935 1 0 51 12 0 0 18 0 9 0 658252059 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8239/statm: 90373 7361 1882 364 0 26270 0
[pid=8245] ppid=8239 vsize=288756 CPUtime=0.2 cores=1,3,5,7
/proc/8245/stat : 8245 (python) S 8239 8239 7577 0 -1 4202560 1164 0 0 0 11 9 0 0 19 0 2 0 658252137 295686144 5619 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/8245/statm: 72189 5619 416 364 0 8107 0
[pid=8247] ppid=8245 vsize=1017800 CPUtime=99.28 cores=1,3,5,7
/proc/8247/stat : 8247 (gnovelty+2) T 8245 8247 8247 0 -1 4202496 441 0 0 0 9928 0 0 0 25 0 1 0 658252137 1042227200 178 33554432000 134512640 135663704 4288324928 18446744073709551615 134525276 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/8247/statm: 254450 178 113 282 0 254166 0
[pid=8245/tid=8265] ppid=8239 vsize=288756 CPUtime=0 cores=1,3,5,7
/proc/8245/task/8265/stat : 8265 (python) S 8239 8239 7577 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 658262156 295686144 5619 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604459502372736 0 0 -1 7 0 0 0
[pid=8248] ppid=8239 vsize=289344 CPUtime=2.01 cores=1,3,5,7
/proc/8248/stat : 8248 (python) S 8239 8239 7577 0 -1 4202560 1157 0 0 0 105 96 0 0 15 0 2 0 658252138 296288256 5747 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/8248/statm: 72336 5747 400 364 0 8251 0
[pid=8250] ppid=8248 vsize=119800 CPUtime=901.54 cores=1,3,5,7
/proc/8250/stat : 8250 (march_hi) R 8248 8250 8250 0 -1 4202496 27229 0 0 0 90148 6 0 0 25 0 1 0 658252138 122675200 26966 33554432000 4194304 4955518 140737296968512 18446744073709551615 4209733 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8250/statm: 29950 26966 104 186 0 29763 0
[pid=8248/tid=8277] ppid=8239 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/8248/task/8277/stat : 8277 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658291121 296288256 5747 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604471149740352 0 0 -1 1 0 0 0
[pid=8251] ppid=8239 vsize=299092 CPUtime=0.2 cores=1,3,5,7
/proc/8251/stat : 8251 (python) S 8239 8239 7577 0 -1 4202560 1090 0 0 0 11 9 0 0 19 0 2 0 658252139 306270208 5639 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/8251/statm: 74773 5639 400 364 0 10685 0
[pid=8253] ppid=8251 vsize=1017800 CPUtime=99.2 cores=1,3,5,7
/proc/8253/stat : 8253 (gnovelty+2) T 8251 8253 8253 0 -1 4202496 439 0 0 0 9920 0 0 0 25 0 1 0 658252139 1042227200 178 33554432000 134512640 135663704 4287349824 18446744073709551615 134547018 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/8253/statm: 254450 178 113 282 0 254166 0
[pid=8251/tid=8261] ppid=8239 vsize=299092 CPUtime=0 cores=1,3,5,7
/proc/8251/task/8261/stat : 8261 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658262059 306270208 5639 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604459894986048 0 0 -1 3 0 0 0
[pid=8254] ppid=8239 vsize=309956 CPUtime=2.47 cores=1,3,5,7
/proc/8254/stat : 8254 (python) S 8239 8239 7577 0 -1 4202560 1161 0 0 0 127 120 0 0 15 0 2 0 658252140 317394944 5764 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/8254/statm: 77489 5764 401 364 0 13398 0
[pid=8256] ppid=8254 vsize=192696 CPUtime=1097.29 cores=1,3,5,7
/proc/8256/stat : 8256 (march_hi) R 8254 8256 8256 0 -1 4202496 31913 0 0 0 109723 6 0 0 25 0 1 0 658252140 197320704 31651 33554432000 4194304 4955518 140734067536848 18446744073709551615 4227732 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8256/statm: 48174 31651 104 186 0 47987 0
[pid=8254/tid=8278] ppid=8239 vsize=309956 CPUtime=0 cores=1,3,5,7
/proc/8254/task/8278/stat : 8278 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658291366 317394944 5764 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604468713913216 0 0 -1 3 0 0 0
[pid=8262] ppid=8239 vsize=320212 CPUtime=2.5 cores=1,3,5,7
/proc/8262/stat : 8262 (python) S 8239 8239 7577 0 -1 4202560 1161 0 0 0 132 118 0 0 15 0 2 0 658262060 327897088 5767 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8262/statm: 80053 5767 400 364 0 15959 0
[pid=8264] ppid=8262 vsize=192696 CPUtime=1088.66 cores=1,3,5,7
/proc/8264/stat : 8264 (march_hi) R 8262 8264 8264 0 -1 4202496 31593 0 0 0 108858 8 0 0 25 0 1 0 658262061 197320704 31332 33554432000 4194304 4955518 140737153269680 18446744073709551615 4227970 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8264/statm: 48174 31332 104 186 0 47987 0
[pid=8262/tid=8457] ppid=8239 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/8262/task/8457/stat : 8457 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 658331199 327897088 5767 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604471145427968 0 0 -1 7 0 0 0
[pid=8266] ppid=8239 vsize=330092 CPUtime=1.79 cores=1,3,5,7
/proc/8266/stat : 8266 (python) S 8239 8239 7577 0 -1 4202560 1155 0 0 0 95 84 0 0 15 0 2 0 658262157 338014208 5677 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/8266/statm: 82523 5677 400 364 0 18426 0
[pid=8268] ppid=8266 vsize=119668 CPUtime=780.01 cores=1,3,5,7
/proc/8268/stat : 8268 (march_hi) T 8266 8268 8268 0 -1 4202496 24033 0 0 0 77997 4 0 0 25 0 1 0 658262157 122540032 23773 33554432000 4194304 4955518 140736910786432 18446744073709551615 4210637 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/8268/statm: 29917 23773 104 186 0 29730 0
[pid=8266/tid=8455] ppid=8239 vsize=330092 CPUtime=0 cores=1,3,5,7
/proc/8266/task/8455/stat : 8455 (python) S 8239 8239 7577 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 658330567 338014208 5677 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604471145429760 0 0 -1 5 0 0 0
[pid=8432] ppid=8239 vsize=340888 CPUtime=1.27 cores=1,3,5,7
/proc/8432/stat : 8432 (python) S 8239 8239 7577 0 -1 4202560 992 0 0 0 65 62 0 0 18 0 1 0 658310769 349069312 5809 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 2355224275440337157 0 0 17 3 0 0 0
/proc/8432/statm: 85222 5809 371 364 0 21122 0
[pid=8434] ppid=8432 vsize=182452 CPUtime=614.49 cores=1,3,5,7
/proc/8434/stat : 8434 (kcnfs-2006) R 8432 8434 8434 0 -1 4202496 590 0 0 0 61449 0 0 0 25 0 1 0 658310770 186830848 332 33554432000 134512640 135163436 4286636272 18446744073709551615 134569537 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8434/statm: 45613 332 77 159 0 45453 0
[pid=8445] ppid=8239 vsize=350612 CPUtime=0.25 cores=1,3,5,7
/proc/8445/stat : 8445 (python) S 8239 8239 7577 0 -1 4202560 1152 0 0 0 13 12 0 0 16 0 2 0 658320453 359026688 5725 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/8445/statm: 87653 5725 406 364 0 23550 0
[pid=8447] ppid=8445 vsize=17664 CPUtime=102.07 cores=1,3,5,7
/proc/8447/stat : 8447 (march_hi) T 8445 8447 8447 0 -1 4202496 4299 0 0 0 10207 0 0 0 25 0 1 0 658320454 18087936 4033 33554432000 4194304 4955518 140736928287488 18446744073709551615 4263698 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/8447/statm: 4416 4033 104 186 0 4229 0
[pid=8445/tid=8456] ppid=8239 vsize=350612 CPUtime=0 cores=1,3,5,7
/proc/8445/task/8456/stat : 8456 (python) S 8239 8239 7577 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 658330709 359026688 5725 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604464332169280 0 0 -1 3 0 0 0
[pid=8239/tid=8246] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8246/stat : 8246 (python) S 8237 8239 7577 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 9 0 658252137 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8249] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8249/stat : 8249 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 9 0 658252138 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8252] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8252/stat : 8252 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 22 0 9 0 658252139 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8255] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8255/stat : 8255 (python) S 8237 8239 7577 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 9 0 658252140 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8263] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8263/stat : 8263 (python) S 8237 8239 7577 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 658262060 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8267] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8267/stat : 8267 (python) S 8237 8239 7577 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 9 0 658262157 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8433] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8433/stat : 8433 (python) S 8237 8239 7577 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 658310770 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8446] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8446/stat : 8446 (python) S 8237 8239 7577 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 9 0 658320454 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4793.86
Current children cumulated vsize (KiB) 5751020

[startup+1202.2 s]
/proc/loadavg: 4.07 5.58 6.46 7/306 8486
/proc/meminfo: memFree=19582664/32950904 swapFree=67057400/67111528
[pid=8239] ppid=8237 vsize=361492 CPUtime=0.63 cores=1,3,5,7
/proc/8239/stat : 8239 (python) S 8237 8239 7577 0 -1 4202496 16279 935 1 0 51 12 0 0 18 0 9 0 658252059 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8239/statm: 90373 7361 1882 364 0 26270 0
[pid=8245] ppid=8239 vsize=288756 CPUtime=0.2 cores=1,3,5,7
/proc/8245/stat : 8245 (python) S 8239 8239 7577 0 -1 4202560 1164 0 0 0 11 9 0 0 19 0 2 0 658252137 295686144 5619 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/8245/statm: 72189 5619 416 364 0 8107 0
[pid=8247] ppid=8245 vsize=1017800 CPUtime=99.28 cores=1,3,5,7
/proc/8247/stat : 8247 (gnovelty+2) T 8245 8247 8247 0 -1 4202496 441 0 0 0 9928 0 0 0 25 0 1 0 658252137 1042227200 178 33554432000 134512640 135663704 4288324928 18446744073709551615 134525276 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/8247/statm: 254450 178 113 282 0 254166 0
[pid=8245/tid=8265] ppid=8239 vsize=288756 CPUtime=0 cores=1,3,5,7
/proc/8245/task/8265/stat : 8265 (python) S 8239 8239 7577 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 658262156 295686144 5619 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604459502372736 0 0 -1 7 0 0 0
[pid=8248] ppid=8239 vsize=289344 CPUtime=2.01 cores=1,3,5,7
/proc/8248/stat : 8248 (python) S 8239 8239 7577 0 -1 4202560 1157 0 0 0 105 96 0 0 15 0 2 0 658252138 296288256 5747 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/8248/statm: 72336 5747 400 364 0 8251 0
[pid=8250] ppid=8248 vsize=119800 CPUtime=901.65 cores=1,3,5,7
/proc/8250/stat : 8250 (march_hi) R 8248 8250 8250 0 -1 4202496 27231 0 0 0 90159 6 0 0 25 0 1 0 658252138 122675200 26968 33554432000 4194304 4955518 140737296968512 18446744073709551615 4290229 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/8250/statm: 29950 26968 104 186 0 29763 0
[pid=8248/tid=8277] ppid=8239 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/8248/task/8277/stat : 8277 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658291121 296288256 5747 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604471149740352 0 0 -1 1 0 0 0
[pid=8251] ppid=8239 vsize=299092 CPUtime=0.2 cores=1,3,5,7
/proc/8251/stat : 8251 (python) S 8239 8239 7577 0 -1 4202560 1090 0 0 0 11 9 0 0 19 0 2 0 658252139 306270208 5639 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/8251/statm: 74773 5639 400 364 0 10685 0
[pid=8253] ppid=8251 vsize=1017800 CPUtime=99.2 cores=1,3,5,7
/proc/8253/stat : 8253 (gnovelty+2) T 8251 8253 8253 0 -1 4202496 439 0 0 0 9920 0 0 0 25 0 1 0 658252139 1042227200 178 33554432000 134512640 135663704 4287349824 18446744073709551615 134547018 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/8253/statm: 254450 178 113 282 0 254166 0
[pid=8251/tid=8261] ppid=8239 vsize=299092 CPUtime=0 cores=1,3,5,7
/proc/8251/task/8261/stat : 8261 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658262059 306270208 5639 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604459894986048 0 0 -1 3 0 0 0
[pid=8254] ppid=8239 vsize=309956 CPUtime=2.47 cores=1,3,5,7
/proc/8254/stat : 8254 (python) S 8239 8239 7577 0 -1 4202560 1161 0 0 0 127 120 0 0 15 0 2 0 658252140 317394944 5764 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/8254/statm: 77489 5764 401 364 0 13398 0
[pid=8256] ppid=8254 vsize=192696 CPUtime=1097.4 cores=1,3,5,7
/proc/8256/stat : 8256 (march_hi) R 8254 8256 8256 0 -1 4202496 31916 0 0 0 109734 6 0 0 25 0 1 0 658252140 197320704 31654 33554432000 4194304 4955518 140734067536848 18446744073709551615 4209701 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8256/statm: 48174 31654 104 186 0 47987 0
[pid=8254/tid=8278] ppid=8239 vsize=309956 CPUtime=0 cores=1,3,5,7
/proc/8254/task/8278/stat : 8278 (python) S 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658291366 317394944 5764 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604468713913216 0 0 -1 3 0 0 0
[pid=8262] ppid=8239 vsize=320212 CPUtime=2.52 cores=1,3,5,7
/proc/8262/stat : 8262 (python) S 8239 8239 7577 0 -1 4202560 1161 0 0 0 133 119 0 0 15 0 2 0 658262060 327897088 5767 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8262/statm: 80053 5767 400 364 0 15959 0
[pid=8264] ppid=8262 vsize=192696 CPUtime=1088.74 cores=1,3,5,7
/proc/8264/stat : 8264 (march_hi) R 8262 8264 8264 0 -1 4202496 31594 0 0 0 108866 8 0 0 25 0 1 0 658262061 197320704 31333 33554432000 4194304 4955518 140737153269680 18446744073709551615 4286784 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/8264/statm: 48174 31333 104 186 0 47987 0
[pid=8262/tid=8457] ppid=8239 vsize=320212 CPUtime=0 cores=1,3,5,7
/proc/8262/task/8457/stat : 8457 (python) R 8239 8239 7577 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 658331199 327897088 5767 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 0 0 0 -1 7 0 0 0
[pid=8266] ppid=8239 vsize=330092 CPUtime=1.79 cores=1,3,5,7
/proc/8266/stat : 8266 (python) S 8239 8239 7577 0 -1 4202560 1155 0 0 0 95 84 0 0 15 0 2 0 658262157 338014208 5677 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/8266/statm: 82523 5677 400 364 0 18426 0
[pid=8268] ppid=8266 vsize=119668 CPUtime=780.01 cores=1,3,5,7
/proc/8268/stat : 8268 (march_hi) T 8266 8268 8268 0 -1 4202496 24033 0 0 0 77997 4 0 0 25 0 1 0 658262157 122540032 23773 33554432000 4194304 4955518 140736910786432 18446744073709551615 4210637 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/8268/statm: 29917 23773 104 186 0 29730 0
[pid=8266/tid=8455] ppid=8239 vsize=330092 CPUtime=0 cores=1,3,5,7
/proc/8266/task/8455/stat : 8455 (python) S 8239 8239 7577 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 658330567 338014208 5677 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604471145429760 0 0 -1 5 0 0 0
[pid=8432] ppid=8239 vsize=340888 CPUtime=1.27 cores=1,3,5,7
/proc/8432/stat : 8432 (python) S 8239 8239 7577 0 -1 4202560 992 0 0 0 65 62 0 0 18 0 1 0 658310769 349069312 5809 33554432000 4194304 5683644 140736700223024 18446744073709551615 237278899782 0 0 16781312 514 2355224275440337157 0 0 17 3 0 0 0
/proc/8432/statm: 85222 5809 371 364 0 21122 0
[pid=8434] ppid=8432 vsize=182452 CPUtime=614.6 cores=1,3,5,7
/proc/8434/stat : 8434 (kcnfs-2006) R 8432 8434 8434 0 -1 4202496 590 0 0 0 61460 0 0 0 25 0 1 0 658310770 186830848 332 33554432000 134512640 135163436 4286636272 18446744073709551615 134571687 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/8434/statm: 45613 332 77 159 0 45453 0
[pid=8445] ppid=8239 vsize=350612 CPUtime=0.25 cores=1,3,5,7
/proc/8445/stat : 8445 (python) S 8239 8239 7577 0 -1 4202560 1152 0 0 0 13 12 0 0 16 0 2 0 658320453 359026688 5725 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/8445/statm: 87653 5725 406 364 0 23550 0
[pid=8447] ppid=8445 vsize=17664 CPUtime=102.07 cores=1,3,5,7
/proc/8447/stat : 8447 (march_hi) T 8445 8447 8447 0 -1 4202496 4299 0 0 0 10207 0 0 0 25 0 1 0 658320454 18087936 4033 33554432000 4194304 4955518 140736928287488 18446744073709551615 4263698 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/8447/statm: 4416 4033 104 186 0 4229 0
[pid=8445/tid=8456] ppid=8239 vsize=350612 CPUtime=0 cores=1,3,5,7
/proc/8445/task/8456/stat : 8456 (python) S 8239 8239 7577 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 658330709 359026688 5725 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 514 18446604464332169280 0 0 -1 3 0 0 0
[pid=8239/tid=8246] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8246/stat : 8246 (python) S 8237 8239 7577 0 -1 4202560 27 935 0 0 0 0 0 0 18 0 9 0 658252137 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8249] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8249/stat : 8249 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 9 0 658252138 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8252] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8252/stat : 8252 (python) S 8237 8239 7577 0 -1 4202560 35 935 0 0 0 0 0 0 22 0 9 0 658252139 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8255] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8255/stat : 8255 (python) S 8237 8239 7577 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 9 0 658252140 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8263] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8263/stat : 8263 (python) S 8237 8239 7577 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 658262060 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8267] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8267/stat : 8267 (python) S 8237 8239 7577 0 -1 4202560 34 935 0 0 0 0 0 0 18 0 9 0 658262157 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8433] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8433/stat : 8433 (python) S 8237 8239 7577 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 658310770 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
[pid=8239/tid=8446] ppid=8237 vsize=361492 CPUtime=0 cores=1,3,5,7
/proc/8239/task/8446/stat : 8446 (python) S 8237 8239 7577 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 9 0 658320454 370167808 7361 33554432000 4194304 5683644 140736700223024 18446744073709551615 237290703281 0 0 16781312 2 18446604459894987840 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4794.29
Current children cumulated vsize (KiB) 5751020

Child status: 0
Real time (s): 1202.27
CPU time (s): 4794.61
CPU user time (s): 4789.01
CPU system time (s): 5.59715
CPU usage (%): 398.796
Max. virtual memory (cumulated for all children) (KiB): 5751020

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4789.01
system time used= 5.59715
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 151051
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9915
involuntary context switches= 25392

runsolver used 3.98739 second user time and 7.83981 second system time

The end

Launcher Data 

Begin job on node121 at 2011-05-15 15:35:23
IDJOB=3378625
IDBENCH=83034
IDSOLVER=1859
FILE ID=node121/3378625-1305466523
RUNJOBID= node121-1305465318-7596
PBS_JOBID= 13325054
Free space on /tmp= 73464 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v500-c2130-S2061684506-048.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378625-1305466523/watcher-3378625-1305466523 -o /tmp/evaluation-result-3378625-1305466523/solver-3378625-1305466523 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378625-1305466523.cnf -seed 2001714417 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 96dadba6913c3cf46368b605f559e4b9
RANDOM SEED=2001714417

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      19572044 kB
Buffers:        320924 kB
Cached:        6632928 kB
SwapCached:       7656 kB
Active:        7290480 kB
Inactive:      5725184 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      19572044 kB
SwapTotal:    67111528 kB
SwapFree:     67057400 kB
Dirty:           31892 kB
Writeback:           0 kB
AnonPages:     6059608 kB
Mapped:          29172 kB
Slab:           269080 kB
PageTables:      30756 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS: 34212096 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73464 MiB
End job on node121 at 2011-05-15 15:55:25