Trace number 3378329

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (TO) 2858.57 1300.06

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v600-c2556-S248192082-054.UNKNOWN.cnf
MD5SUMcf7086a47a32df0eb19a8b08d14122dd
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses2556
Sum of the clauses size7668
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32556
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.37/0.64	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/0.64	c INFO: solving HOME/instance-3378329-1305464756.cnf
0.37/0.64	c INFO: computing task features
0.37/0.65	c INFO: computed variable-clause graph statistics
0.37/0.65	c INFO: computed clause balance statistics
0.37/0.69	c INFO: computed variable balance statistics
0.46/0.73	c INFO: computed small-clause counts
0.46/0.73	c INFO: computed Horn-clause counts
0.46/0.73	c INFO: computed variable graph statistics
0.46/0.73	c INFO: collected features for HOME/instance-3378329-1305464756.cnf in 0.10 s
0.46/0.74	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.58)
0.46/0.75	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.31)
0.46/0.76	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.07)
0.46/0.78	c INFO: running march_hi@0 for 388 with 1199 remaining (b = 0.29)
334.68/134.13	c INFO: running march_hi@0 for 679 with 1066 remaining (b = 0.35)
417.45/167.15	c INFO: running march_hi@0 for 679 with 1033 remaining (b = 0.32)
1205.39/477.60	c INFO: running kcnfs-2006@0 for 582 with 723 remaining (b = 0.26)
1640.97/694.57	c INFO: running march_hi@0 for 506 with 506 remaining (b = 0.22)
2858.57/1300.05	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378329-1305464756/watcher-3378329-1305464756 -o /tmp/evaluation-result-3378329-1305464756/solver-3378329-1305464756 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378329-1305464756.cnf -seed 822640364 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.30 9.64 8.37 8/353 26935
/proc/meminfo: memFree=19939936/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=15908 CPUtime=0 cores=1,3,5,7
/proc/26935/stat : 26935 (python) R 26933 26935 25829 0 -1 4202496 940 0 0 0 0 0 0 0 25 0 1 0 658073542 16289792 819 33554432000 4194304 5683644 140736092641840 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/26935/statm: 3976 818 356 364 0 463 0

[startup+0.0540189 s]
/proc/loadavg: 10.30 9.64 8.37 8/353 26935
/proc/meminfo: memFree=19939936/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=29416 CPUtime=0.04 cores=1,3,5,7
/proc/26935/stat : 26935 (python) R 26933 26935 25829 0 -1 4202496 1364 0 0 0 3 1 0 0 25 0 1 0 658073542 30121984 1208 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550198469 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/26935/statm: 7354 1209 443 364 0 724 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 29416

[startup+0.100981 s]
/proc/loadavg: 10.30 9.64 8.37 8/353 26935
/proc/meminfo: memFree=19939936/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=91828 CPUtime=0.07 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 2678 0 0 0 5 2 0 0 17 0 1 0 658073542 94031872 2394 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550535680 0 2147483391 16781312 2 18446744071699469502 0 0 17 7 0 0 0
/proc/26935/statm: 22957 2394 755 364 0 1622 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 91828

[startup+0.300999 s]
/proc/loadavg: 10.30 9.64 8.37 8/353 26935
/proc/meminfo: memFree=19939936/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=135748 CPUtime=0.19 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 5968 934 1 0 15 4 0 0 17 0 1 0 658073542 139005952 4151 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550535680 0 2147483391 16781312 2 18446744071699469502 0 0 17 3 0 0 0
/proc/26935/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700898 s]
/proc/loadavg: 10.30 9.64 8.37 8/353 26935
/proc/meminfo: memFree=19939936/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=278448 CPUtime=0.46 cores=1,3,5,7
/proc/26935/stat : 26935 (python) R 26933 26935 25829 0 -1 4202496 9350 934 1 0 36 10 0 0 16 0 1 0 658073542 285130752 7016 33554432000 4194304 5683644 140736092641840 18446744073709551615 47859927409880 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/26935/statm: 69612 7016 1827 364 0 5536 0
Current children cumulated CPU time (s) 0.46
Current children cumulated vsize (KiB) 278448

[startup+1.50072 s]
/proc/loadavg: 10.30 9.64 8.37 12/366 26952
/proc/meminfo: memFree=19901068/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=320252 CPUtime=0.52 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 12189 934 1 0 42 10 0 0 22 0 5 0 658073542 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/26935/statm: 80063 7252 1873 364 0 15972 0
[pid=26941] ppid=26935 vsize=278472 CPUtime=0 cores=1,3,5,7
/proc/26941/stat : 26941 (python) S 26935 26935 25829 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 658073617 285155328 5549 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557711 0 0 16781312 514 16716336801379254271 0 0 17 3 0 0 0
/proc/26941/statm: 69618 5549 354 364 0 5536 0
[pid=26943] ppid=26941 vsize=1017932 CPUtime=0.44 cores=1,3,5,7
/proc/26943/stat : 26943 (gnovelty+2) R 26941 26943 26943 0 -1 4202496 448 0 0 0 44 0 0 0 20 0 1 0 658073617 1042362368 184 33554432000 134512640 135663704 4291589232 18446744073709551615 134564723 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26943/statm: 254483 184 113 282 0 254199 0
[pid=26944] ppid=26935 vsize=289364 CPUtime=0 cores=1,3,5,7
/proc/26944/stat : 26944 (python) S 26935 26935 25829 0 -1 4202560 749 0 0 0 0 0 0 0 15 0 1 0 658073618 296308736 5699 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 17543007092623769864 0 0 17 1 0 0 0
/proc/26944/statm: 72341 5699 348 364 0 8256 0
[pid=26948] ppid=26944 vsize=2640 CPUtime=0.72 cores=1,3,5,7
/proc/26948/stat : 26948 (march_hi) R 26944 26948 26948 0 -1 4202496 720 0 0 0 72 0 0 0 22 0 1 0 658073619 2703360 451 33554432000 4194304 4955518 140734632427648 18446744073709551615 4209701 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26948/statm: 660 451 104 186 0 473 0
[pid=26946] ppid=26935 vsize=299752 CPUtime=0 cores=1,3,5,7
/proc/26946/stat : 26946 (python) S 26935 26935 25829 0 -1 4202560 740 0 0 0 0 0 0 0 21 0 1 0 658073619 306946048 5719 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 8469278313431433215 0 0 17 7 0 0 0
/proc/26946/statm: 74938 5719 348 364 0 10850 0
[pid=26951] ppid=26946 vsize=1017932 CPUtime=0.32 cores=1,3,5,7
/proc/26951/stat : 26951 (gnovelty+2) R 26946 26951 26951 0 -1 4202496 447 0 0 0 32 0 0 0 24 0 1 0 658073620 1042362368 184 33554432000 134512640 135663704 4293921232 18446744073709551615 134524697 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26951/statm: 254483 184 113 282 0 254199 0
[pid=26949] ppid=26935 vsize=310008 CPUtime=0 cores=1,3,5,7
/proc/26949/stat : 26949 (python) S 26935 26935 25829 0 -1 4202560 761 0 0 0 0 0 0 0 16 0 1 0 658073620 317448192 5721 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 15983103558018892040 0 0 17 7 0 0 0
/proc/26949/statm: 77502 5721 348 364 0 13411 0
[pid=26952] ppid=26949 vsize=2504 CPUtime=0.29 cores=1,3,5,7
/proc/26952/stat : 26952 (march_hi) R 26949 26952 26952 0 -1 4202496 673 0 0 0 29 0 0 0 25 0 1 0 658073620 2564096 411 33554432000 4194304 4955518 140734713507184 18446744073709551615 4238735 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26952/statm: 626 411 104 186 0 439 0
[pid=26935/tid=26942] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26942/stat : 26942 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 5 0 658073617 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 3 0 0 0
[pid=26935/tid=26945] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26945/stat : 26945 (python) S 26933 26935 25829 0 -1 4202560 40 934 0 0 0 0 0 0 20 0 5 0 658073618 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26947] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26947/stat : 26947 (python) S 26933 26935 25829 0 -1 4202560 35 934 0 0 0 0 0 0 22 0 5 0 658073619 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26950] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26950/stat : 26950 (python) S 26933 26935 25829 0 -1 4202560 15 934 0 0 0 0 0 0 23 0 5 0 658073620 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2.29
Current children cumulated vsize (KiB) 3538856

[startup+3.10143 s]
/proc/loadavg: 10.30 9.64 8.37 12/366 26952
/proc/meminfo: memFree=19898092/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=320252 CPUtime=0.52 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 12189 934 1 0 42 10 0 0 22 0 5 0 658073542 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/26935/statm: 80063 7252 1873 364 0 15972 0
[pid=26941] ppid=26935 vsize=278472 CPUtime=0 cores=1,3,5,7
/proc/26941/stat : 26941 (python) S 26935 26935 25829 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 658073617 285155328 5549 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557711 0 0 16781312 514 16716336801379221764 0 0 17 3 0 0 0
/proc/26941/statm: 69618 5549 354 364 0 5536 0
[pid=26943] ppid=26941 vsize=1017932 CPUtime=1.24 cores=1,3,5,7
/proc/26943/stat : 26943 (gnovelty+2) R 26941 26943 26943 0 -1 4202496 448 0 0 0 124 0 0 0 22 0 1 0 658073617 1042362368 184 33554432000 134512640 135663704 4291589232 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26943/statm: 254483 184 113 282 0 254199 0
[pid=26944] ppid=26935 vsize=289364 CPUtime=0 cores=1,3,5,7
/proc/26944/stat : 26944 (python) S 26935 26935 25829 0 -1 4202560 749 0 0 0 0 0 0 0 15 0 1 0 658073618 296308736 5699 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 17543007092623769864 0 0 17 1 0 0 0
/proc/26944/statm: 72341 5699 348 364 0 8256 0
[pid=26948] ppid=26944 vsize=2960 CPUtime=2.32 cores=1,3,5,7
/proc/26948/stat : 26948 (march_hi) R 26944 26948 26948 0 -1 4202496 793 0 0 0 232 0 0 0 25 0 1 0 658073619 3031040 524 33554432000 4194304 4955518 140734632427648 18446744073709551615 4238735 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26948/statm: 740 524 104 186 0 553 0
[pid=26946] ppid=26935 vsize=299752 CPUtime=0 cores=1,3,5,7
/proc/26946/stat : 26946 (python) S 26935 26935 25829 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 658073619 306946048 5719 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 8469278313431400709 0 0 17 7 0 0 0
/proc/26946/statm: 74938 5719 348 364 0 10850 0
[pid=26951] ppid=26946 vsize=1017932 CPUtime=1.12 cores=1,3,5,7
/proc/26951/stat : 26951 (gnovelty+2) R 26946 26951 26951 0 -1 4202496 447 0 0 0 112 0 0 0 25 0 1 0 658073620 1042362368 184 33554432000 134512640 135663704 4293921232 18446744073709551615 134545518 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26951/statm: 254483 184 113 282 0 254199 0
[pid=26949] ppid=26935 vsize=310008 CPUtime=0 cores=1,3,5,7
/proc/26949/stat : 26949 (python) S 26935 26935 25829 0 -1 4202560 761 0 0 0 0 0 0 0 15 0 1 0 658073620 317448192 5721 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 15983103558018892040 0 0 17 7 0 0 0
/proc/26949/statm: 77502 5721 348 364 0 13411 0
[pid=26952] ppid=26949 vsize=2708 CPUtime=1.09 cores=1,3,5,7
/proc/26952/stat : 26952 (march_hi) R 26949 26952 26952 0 -1 4202496 726 0 0 0 109 0 0 0 25 0 1 0 658073620 2772992 464 33554432000 4194304 4955518 140734713507184 18446744073709551615 4286523 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26952/statm: 677 464 104 186 0 490 0
[pid=26935/tid=26942] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26942/stat : 26942 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 5 0 658073617 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 3 0 0 0
[pid=26935/tid=26945] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26945/stat : 26945 (python) S 26933 26935 25829 0 -1 4202560 40 934 0 0 0 0 0 0 20 0 5 0 658073618 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26947] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26947/stat : 26947 (python) S 26933 26935 25829 0 -1 4202560 35 934 0 0 0 0 0 0 22 0 5 0 658073619 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26950] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26950/stat : 26950 (python) S 26933 26935 25829 0 -1 4202560 15 934 0 0 0 0 0 0 23 0 5 0 658073620 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 3539380

[startup+6.30085 s]
/proc/loadavg: 10.52 9.70 8.40 12/366 26952
/proc/meminfo: memFree=19896728/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=320252 CPUtime=0.52 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 12189 934 1 0 42 10 0 0 22 0 5 0 658073542 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/26935/statm: 80063 7252 1873 364 0 15972 0
[pid=26941] ppid=26935 vsize=278472 CPUtime=0 cores=1,3,5,7
/proc/26941/stat : 26941 (python) S 26935 26935 25829 0 -1 4202560 897 0 0 0 0 0 0 0 18 0 1 0 658073617 285155328 5571 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557711 0 0 16781312 514 16716336801379221764 0 0 17 3 0 0 0
/proc/26941/statm: 69618 5571 376 364 0 5536 0
[pid=26943] ppid=26941 vsize=1017932 CPUtime=2.84 cores=1,3,5,7
/proc/26943/stat : 26943 (gnovelty+2) R 26941 26943 26943 0 -1 4202496 448 0 0 0 284 0 0 0 25 0 1 0 658073617 1042362368 184 33554432000 134512640 135663704 4291589232 18446744073709551615 134525688 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26943/statm: 254483 184 113 282 0 254199 0
[pid=26944] ppid=26935 vsize=289364 CPUtime=0.01 cores=1,3,5,7
/proc/26944/stat : 26944 (python) S 26935 26935 25829 0 -1 4202560 917 0 0 0 0 1 0 0 15 0 1 0 658073618 296308736 5722 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 17543007092623769864 0 0 17 1 0 0 0
/proc/26944/statm: 72341 5722 371 364 0 8256 0
[pid=26948] ppid=26944 vsize=3664 CPUtime=5.5 cores=1,3,5,7
/proc/26948/stat : 26948 (march_hi) R 26944 26948 26948 0 -1 4202496 942 0 0 0 550 0 0 0 25 0 1 0 658073619 3751936 673 33554432000 4194304 4955518 140734632427648 18446744073709551615 4261328 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26948/statm: 916 673 104 186 0 729 0
[pid=26946] ppid=26935 vsize=299752 CPUtime=0.01 cores=1,3,5,7
/proc/26946/stat : 26946 (python) S 26935 26935 25829 0 -1 4202560 903 0 0 0 1 0 0 0 18 0 1 0 658073619 306946048 5741 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 8469278313431400709 0 0 17 7 0 0 0
/proc/26946/statm: 74938 5741 370 364 0 10850 0
[pid=26951] ppid=26946 vsize=1017932 CPUtime=2.72 cores=1,3,5,7
/proc/26951/stat : 26951 (gnovelty+2) R 26946 26951 26951 0 -1 4202496 447 0 0 0 272 0 0 0 25 0 1 0 658073620 1042362368 184 33554432000 134512640 135663704 4293921232 18446744073709551615 134545646 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26951/statm: 254483 184 113 282 0 254199 0
[pid=26949] ppid=26935 vsize=310008 CPUtime=0.01 cores=1,3,5,7
/proc/26949/stat : 26949 (python) S 26935 26935 25829 0 -1 4202560 913 0 0 0 1 0 0 0 15 0 1 0 658073620 317448192 5744 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 15983103558018892040 0 0 17 7 0 0 0
/proc/26949/statm: 77502 5744 371 364 0 13411 0
[pid=26952] ppid=26949 vsize=2960 CPUtime=2.69 cores=1,3,5,7
/proc/26952/stat : 26952 (march_hi) R 26949 26952 26952 0 -1 4202496 804 0 0 0 269 0 0 0 25 0 1 0 658073620 3031040 542 33554432000 4194304 4955518 140734713507184 18446744073709551615 4227741 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26952/statm: 740 542 104 186 0 553 0
[pid=26935/tid=26942] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26942/stat : 26942 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 5 0 658073617 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 3 0 0 0
[pid=26935/tid=26945] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26945/stat : 26945 (python) S 26933 26935 25829 0 -1 4202560 40 934 0 0 0 0 0 0 20 0 5 0 658073618 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26947] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26947/stat : 26947 (python) S 26933 26935 25829 0 -1 4202560 35 934 0 0 0 0 0 0 22 0 5 0 658073619 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26950] ppid=26933 vsize=320252 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26950/stat : 26950 (python) S 26933 26935 25829 0 -1 4202560 15 934 0 0 0 0 0 0 23 0 5 0 658073620 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 14.3
Current children cumulated vsize (KiB) 3540336

[startup+12.7007 s]
/proc/loadavg: 10.56 9.72 8.41 12/366 26952
/proc/meminfo: memFree=19894532/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=320252 CPUtime=0.52 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 12189 934 1 0 42 10 0 0 22 0 5 0 658073542 327938048 7252 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/26935/statm: 80063 7252 1873 364 0 15972 0
[pid=26941] ppid=26935 vsize=278472 CPUtime=0.02 cores=1,3,5,7
/proc/26941/stat : 26941 (python) S 26935 26935 25829 0 -1 4202560 900 0 0 0 1 1 0 0 18 0 1 0 658073617 285155328 5571 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557711 0 0 16781312 514 5508025489576395012 0 0 17 3 0 0 0
/proc/26941/statm: 69618 5571 376 364 0 5536 0
[pid=26943] ppid=26941 vsize=1017932 CPUtime=6.04 cores=1,3,5,7
/proc/26943/stat : 26943 (gnovelty+2) R 26941 26943 26943 0 -1 4202496 448 0 0 0 604 0 0 0 25 0 1 0 658073617 1042362368 184 33554432000 134512640 135663704 4291589232 18446744073709551615 134544186 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/26943/statm: 254483 184 113 282 0 254199 0
[pid=26944] ppid=26935 vsize=289364 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

[pid=26944] ppid=26935 vsize=289364 CPUtime=1.91 cores=1,3,5,7
/proc/26944/stat : 26944 (python) S 26935 26935 25829 0 -1 4202560 1142 0 0 0 103 88 0 0 16 0 2 0 658073618 296308736 5752 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26944/statm: 72341 5752 400 364 0 8256 0
[pid=26948] ppid=26944 vsize=79172 CPUtime=390.52 cores=1,3,5,7
/proc/26948/stat : 26948 (march_hi) T 26944 26948 26948 0 -1 4202496 14242 0 0 0 39047 5 0 0 25 0 1 0 658073619 81072128 13973 33554432000 4194304 4955518 140734632427648 18446744073709551615 4310129 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/26948/statm: 19793 13973 104 186 0 19606 0
[pid=26944/tid=27009] ppid=26935 vsize=289364 CPUtime=0 cores=1,3,5,7
/proc/26944/task/27009/stat : 27009 (python) S 26935 26935 25829 0 -1 4202560 44 0 0 0 0 0 0 0 16 0 2 0 658142998 296308736 5752 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604468941746240 0 0 -1 1 0 0 0
[pid=26946] ppid=26935 vsize=299752 CPUtime=0.34 cores=1,3,5,7
/proc/26946/stat : 26946 (python) S 26935 26935 25829 0 -1 4202560 1101 0 0 0 18 16 0 0 19 0 2 0 658073619 306946048 5772 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/26946/statm: 74938 5772 400 364 0 10850 0
[pid=26951] ppid=26946 vsize=1017932 CPUtime=98.02 cores=1,3,5,7
/proc/26951/stat : 26951 (gnovelty+2) T 26946 26951 26951 0 -1 4202496 447 0 0 0 9802 0 0 0 25 0 1 0 658073620 1042362368 184 33554432000 134512640 135663704 4293921232 18446744073709551615 134546746 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26951/statm: 254483 184 113 282 0 254199 0
[pid=26946/tid=26972] ppid=26935 vsize=299752 CPUtime=0 cores=1,3,5,7
/proc/26946/task/26972/stat : 26972 (python) S 26935 26935 25829 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658086955 306946048 5772 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604467769502656 0 0 -1 3 0 0 0
[pid=26949] ppid=26935 vsize=310008 CPUtime=1.32 cores=1,3,5,7
/proc/26949/stat : 26949 (python) S 26935 26935 25829 0 -1 4202560 1150 0 0 0 69 63 0 0 17 0 2 0 658073620 317448192 5776 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/26949/statm: 77502 5776 400 364 0 13411 0
[pid=26952] ppid=26949 vsize=79172 CPUtime=392.59 cores=1,3,5,7
/proc/26952/stat : 26952 (march_hi) T 26949 26952 26952 0 -1 4202496 14326 0 0 0 39254 5 0 0 25 0 1 0 658073620 81072128 14064 33554432000 4194304 4955518 140734713507184 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/26952/statm: 19793 14064 104 186 0 19606 0
[pid=26949/tid=26995] ppid=26935 vsize=310008 CPUtime=0 cores=1,3,5,7
/proc/26949/task/26995/stat : 26995 (python) S 26935 26935 25829 0 -1 4202560 44 0 0 0 0 0 0 0 17 0 2 0 658121301 317448192 5776 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604469849573568 0 0 -1 5 0 0 0
[pid=26973] ppid=26935 vsize=320264 CPUtime=3.03 cores=1,3,5,7
/proc/26973/stat : 26973 (python) S 26935 26935 25829 0 -1 4202560 997 0 0 0 157 146 0 0 15 0 1 0 658086955 327950336 5767 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395016 0 0 17 7 0 0 0
/proc/26973/statm: 80066 5767 371 364 0 15972 0
[pid=26975] ppid=26973 vsize=108772 CPUtime=554.02 cores=1,3,5,7
/proc/26975/stat : 26975 (march_hi) R 26973 26975 26975 0 -1 4202496 19599 0 0 0 55393 9 0 0 25 0 1 0 658086956 111382528 19340 33554432000 4194304 4955518 140736257208912 18446744073709551615 4224584 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26975/statm: 27193 19340 104 186 0 27006 0
[pid=26978] ppid=26935 vsize=330664 CPUtime=2.93 cores=1,3,5,7
/proc/26978/stat : 26978 (python) S 26935 26935 25829 0 -1 4202560 1004 0 0 0 154 139 0 0 15 0 1 0 658090257 338599936 5790 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395012 0 0 17 5 0 0 0
/proc/26978/statm: 82666 5790 371 364 0 18569 0
[pid=26980] ppid=26978 vsize=79304 CPUtime=537.57 cores=1,3,5,7
/proc/26980/stat : 26980 (march_hi) R 26978 26980 26980 0 -1 4202496 19103 0 0 0 53747 10 0 0 25 0 1 0 658090258 81207296 18842 33554432000 4194304 4955518 140736845420752 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26980/statm: 19826 18842 104 186 0 19639 0
[pid=26996] ppid=26935 vsize=340920 CPUtime=2.04 cores=1,3,5,7
/proc/26996/stat : 26996 (python) S 26935 26935 25829 0 -1 4202560 994 0 0 0 112 92 0 0 18 0 1 0 658121302 349102080 5815 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395013 0 0 17 7 0 0 0
/proc/26996/statm: 85230 5815 371 364 0 21130 0
[pid=26998] ppid=26996 vsize=182584 CPUtime=382.09 cores=1,3,5,7
/proc/26998/stat : 26998 (kcnfs-2006) R 26996 26998 26998 0 -1 4202496 625 0 0 0 38209 0 0 0 25 0 1 0 658121302 186966016 366 33554432000 134512640 135163436 4289623904 18446744073709551615 134675556 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26998/statm: 45646 366 77 159 0 45486 0
[pid=27010] ppid=26935 vsize=351176 CPUtime=1.49 cores=1,3,5,7
/proc/27010/stat : 27010 (python) S 26935 26935 25829 0 -1 4202560 1039 0 0 0 80 69 0 0 15 0 1 0 658142999 359604224 5825 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395013 0 0 17 7 0 0 0
/proc/27010/statm: 87794 5825 377 364 0 23691 0
[pid=27012] ppid=27010 vsize=50792 CPUtime=273.85 cores=1,3,5,7
/proc/27012/stat : 27012 (march_hi) R 27010 27012 27012 0 -1 4202496 10349 0 0 0 27380 5 0 0 25 0 1 0 658143000 52011008 10072 33554432000 4194304 4955518 140733244364832 18446744073709551615 4227732 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/27012/statm: 12698 10072 104 186 0 12511 0
[pid=26935/tid=26942] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26942/stat : 26942 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 9 0 658073617 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 3 0 0 0
[pid=26935/tid=26945] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26945/stat : 26945 (python) S 26933 26935 25829 0 -1 4202560 40 934 0 0 0 0 0 0 20 0 9 0 658073618 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26947] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26947/stat : 26947 (python) S 26933 26935 25829 0 -1 4202560 35 934 0 0 0 0 0 0 22 0 9 0 658073619 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26950] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26950/stat : 26950 (python) S 26933 26935 25829 0 -1 4202560 15 934 0 0 0 0 0 0 23 0 9 0 658073620 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26974] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26974/stat : 26974 (python) S 26933 26935 25829 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 658086956 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26979] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26979/stat : 26979 (python) S 26933 26935 25829 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 658090257 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26997] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26997/stat : 26997 (python) S 26933 26935 25829 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 658121302 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=27011] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/27011/stat : 27011 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 9 0 658142999 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2742.49
Current children cumulated vsize (KiB) 5507944



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 11.22 11.25 10.55 13/394 27141
/proc/meminfo: memFree=19542080/32951124 swapFree=67056788/67111528
[pid=26935] ppid=26933 vsize=361420 CPUtime=0.56 cores=1,3,5,7
/proc/26935/stat : 26935 (python) S 26933 26935 25829 0 -1 4202496 15466 934 1 0 45 11 0 0 19 0 9 0 658073542 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/26935/statm: 90355 7342 1890 364 0 26252 0
[pid=26941] ppid=26935 vsize=288716 CPUtime=0.41 cores=1,3,5,7
/proc/26941/stat : 26941 (python) S 26935 26935 25829 0 -1 4202560 1183 0 0 0 22 19 0 0 19 0 2 0 658073617 295645184 5616 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/26941/statm: 72179 5616 416 364 0 8097 0
[pid=26943] ppid=26941 vsize=1017932 CPUtime=99.8 cores=1,3,5,7
/proc/26943/stat : 26943 (gnovelty+2) T 26941 26943 26943 0 -1 4202496 448 0 0 0 9980 0 0 0 25 0 1 0 658073617 1042362368 184 33554432000 134512640 135663704 4291589232 18446744073709551615 134524518 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/26943/statm: 254483 184 113 282 0 254199 0
[pid=26941/tid=26977] ppid=26935 vsize=288716 CPUtime=0 cores=1,3,5,7
/proc/26941/task/26977/stat : 26977 (python) S 26935 26935 25829 0 -1 4202560 44 0 0 0 0 0 0 0 19 0 2 0 658090256 295645184 5616 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604461099736000 0 0 -1 3 0 0 0
[pid=26944] ppid=26935 vsize=289364 CPUtime=1.91 cores=1,3,5,7
/proc/26944/stat : 26944 (python) S 26935 26935 25829 0 -1 4202560 1142 0 0 0 103 88 0 0 16 0 2 0 658073618 296308736 5752 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/26944/statm: 72341 5752 400 364 0 8256 0
[pid=26948] ppid=26944 vsize=79172 CPUtime=390.52 cores=1,3,5,7
/proc/26948/stat : 26948 (march_hi) T 26944 26948 26948 0 -1 4202496 14242 0 0 0 39047 5 0 0 25 0 1 0 658073619 81072128 13973 33554432000 4194304 4955518 140734632427648 18446744073709551615 4310129 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/26948/statm: 19793 13973 104 186 0 19606 0
[pid=26944/tid=27009] ppid=26935 vsize=289364 CPUtime=0 cores=1,3,5,7
/proc/26944/task/27009/stat : 27009 (python) S 26935 26935 25829 0 -1 4202560 44 0 0 0 0 0 0 0 16 0 2 0 658142998 296308736 5752 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604468941746240 0 0 -1 1 0 0 0
[pid=26946] ppid=26935 vsize=299752 CPUtime=0.34 cores=1,3,5,7
/proc/26946/stat : 26946 (python) S 26935 26935 25829 0 -1 4202560 1101 0 0 0 18 16 0 0 19 0 2 0 658073619 306946048 5772 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/26946/statm: 74938 5772 400 364 0 10850 0
[pid=26951] ppid=26946 vsize=1017932 CPUtime=98.02 cores=1,3,5,7
/proc/26951/stat : 26951 (gnovelty+2) T 26946 26951 26951 0 -1 4202496 447 0 0 0 9802 0 0 0 25 0 1 0 658073620 1042362368 184 33554432000 134512640 135663704 4293921232 18446744073709551615 134546746 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/26951/statm: 254483 184 113 282 0 254199 0
[pid=26946/tid=26972] ppid=26935 vsize=299752 CPUtime=0 cores=1,3,5,7
/proc/26946/task/26972/stat : 26972 (python) S 26935 26935 25829 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658086955 306946048 5772 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604467769502656 0 0 -1 3 0 0 0
[pid=26949] ppid=26935 vsize=310008 CPUtime=1.32 cores=1,3,5,7
/proc/26949/stat : 26949 (python) S 26935 26935 25829 0 -1 4202560 1150 0 0 0 69 63 0 0 17 0 2 0 658073620 317448192 5776 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/26949/statm: 77502 5776 400 364 0 13411 0
[pid=26952] ppid=26949 vsize=79172 CPUtime=392.59 cores=1,3,5,7
/proc/26952/stat : 26952 (march_hi) T 26949 26952 26952 0 -1 4202496 14326 0 0 0 39254 5 0 0 25 0 1 0 658073620 81072128 14064 33554432000 4194304 4955518 140734713507184 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/26952/statm: 19793 14064 104 186 0 19606 0
[pid=26949/tid=26995] ppid=26935 vsize=310008 CPUtime=0 cores=1,3,5,7
/proc/26949/task/26995/stat : 26995 (python) S 26935 26935 25829 0 -1 4202560 44 0 0 0 0 0 0 0 17 0 2 0 658121301 317448192 5776 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 514 18446604469849573568 0 0 -1 5 0 0 0
[pid=26973] ppid=26935 vsize=320264 CPUtime=3.19 cores=1,3,5,7
/proc/26973/stat : 26973 (python) S 26935 26935 25829 0 -1 4202560 997 0 0 0 165 154 0 0 15 0 1 0 658086955 327950336 5767 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 8415222198641328392 0 0 17 7 0 0 0
/proc/26973/statm: 80066 5767 371 364 0 15972 0
[pid=26975] ppid=26973 vsize=108772 CPUtime=582.9 cores=1,3,5,7
/proc/26975/stat : 26975 (march_hi) R 26973 26975 26975 0 -1 4202496 20569 0 0 0 58280 10 0 0 25 0 1 0 658086956 111382528 20310 33554432000 4194304 4955518 140736257208912 18446744073709551615 4290749 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26975/statm: 27193 20310 104 186 0 27006 0
[pid=26978] ppid=26935 vsize=330664 CPUtime=3.1 cores=1,3,5,7
/proc/26978/stat : 26978 (python) S 26935 26935 25829 0 -1 4202560 1004 0 0 0 163 147 0 0 15 0 1 0 658090257 338599936 5790 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395012 0 0 17 5 0 0 0
/proc/26978/statm: 82666 5790 371 364 0 18569 0
[pid=26980] ppid=26978 vsize=108772 CPUtime=566.41 cores=1,3,5,7
/proc/26980/stat : 26980 (march_hi) R 26978 26980 26980 0 -1 4202496 20031 0 0 0 56630 11 0 0 25 0 1 0 658090258 111382528 19770 33554432000 4194304 4955518 140736845420752 18446744073709551615 4238727 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/26980/statm: 27193 19770 104 186 0 27006 0
[pid=26996] ppid=26935 vsize=340920 CPUtime=2.2 cores=1,3,5,7
/proc/26996/stat : 26996 (python) S 26935 26935 25829 0 -1 4202560 994 0 0 0 120 100 0 0 18 0 1 0 658121302 349102080 5815 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395013 0 0 17 7 0 0 0
/proc/26996/statm: 85230 5815 371 364 0 21130 0
[pid=26998] ppid=26996 vsize=182584 CPUtime=410.95 cores=1,3,5,7
/proc/26998/stat : 26998 (kcnfs-2006) R 26996 26998 26998 0 -1 4202496 625 0 0 0 41095 0 0 0 25 0 1 0 658121302 186966016 366 33554432000 134512640 135163436 4289623904 18446744073709551615 134595854 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/26998/statm: 45646 366 77 159 0 45486 0
[pid=27010] ppid=26935 vsize=351176 CPUtime=1.65 cores=1,3,5,7
/proc/27010/stat : 27010 (python) S 26935 26935 25829 0 -1 4202560 1039 0 0 0 88 77 0 0 15 0 1 0 658142999 359604224 5825 33554432000 4194304 5683644 140736092641840 18446744073709551615 271550557766 0 0 16781312 514 5508025489576395013 0 0 17 7 0 0 0
/proc/27010/statm: 87794 5825 377 364 0 23691 0
[pid=27012] ppid=27010 vsize=50792 CPUtime=302.7 cores=1,3,5,7
/proc/27012/stat : 27012 (march_hi) R 27010 27012 27012 0 -1 4202496 11244 0 0 0 30264 6 0 0 25 0 1 0 658143000 52011008 10967 33554432000 4194304 4955518 140733244364832 18446744073709551615 4209639 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/27012/statm: 12698 10967 104 186 0 12511 0
[pid=26935/tid=26942] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26942/stat : 26942 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 9 0 658073617 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 3 0 0 0
[pid=26935/tid=26945] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26945/stat : 26945 (python) S 26933 26935 25829 0 -1 4202560 40 934 0 0 0 0 0 0 20 0 9 0 658073618 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26947] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26947/stat : 26947 (python) S 26933 26935 25829 0 -1 4202560 35 934 0 0 0 0 0 0 22 0 9 0 658073619 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26950] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26950/stat : 26950 (python) S 26933 26935 25829 0 -1 4202560 15 934 0 0 0 0 0 0 23 0 9 0 658073620 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26974] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26974/stat : 26974 (python) S 26933 26935 25829 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 658086956 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26979] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26979/stat : 26979 (python) S 26933 26935 25829 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 658090257 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=26997] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/26997/stat : 26997 (python) S 26933 26935 25829 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 658121302 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
[pid=26935/tid=27011] ppid=26933 vsize=361420 CPUtime=0 cores=1,3,5,7
/proc/26935/task/27011/stat : 27011 (python) S 26933 26935 25829 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 9 0 658142999 370094080 7342 33554432000 4194304 5683644 140736092641840 18446744073709551615 271562361265 0 0 16781312 2 18446604469485629824 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2858.57
Current children cumulated vsize (KiB) 5537412

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 26935 and gives
#  childrusage.ru_utime.tv_sec=876
#  childrusage.ru_utime.tv_usec=986677
#  childrusage.ru_stime.tv_sec=6
#  childrusage.ru_stime.tv_usec=988937
# CPU time returned by wait4() is 883.976
# while last known CPU time is 2858.57
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1300.06
CPU time (s): 2858.57
CPU user time (s): 2851.45
CPU system time (s): 7.12
CPU usage (%): 219.879
Max. virtual memory (cumulated for all children) (KiB): 5537412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 876.987
system time used= 6.98894
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 58967
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9607
involuntary context switches= 15035

runsolver used 7.06492 second user time and 13.7589 second system time

The end

Launcher Data 

Begin job on node133 at 2011-05-15 15:05:56
IDJOB=3378329
IDBENCH=82738
IDSOLVER=1859
FILE ID=node133/3378329-1305464756
RUNJOBID= node133-1305462765-25848
PBS_JOBID= 13324974
Free space on /tmp= 73596 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v600-c2556-S248192082-054.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378329-1305464756/watcher-3378329-1305464756 -o /tmp/evaluation-result-3378329-1305464756/solver-3378329-1305464756 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378329-1305464756.cnf -seed 822640364 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= cf7086a47a32df0eb19a8b08d14122dd
RANDOM SEED=822640364

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6741.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19941084 kB
Buffers:        306616 kB
Cached:        4210156 kB
SwapCached:       8724 kB
Active:        9536560 kB
Inactive:      3167396 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19941084 kB
SwapTotal:    67111528 kB
SwapFree:     67056788 kB
Dirty:           31820 kB
Writeback:           0 kB
AnonPages:     8184632 kB
Mapped:          34072 kB
Slab:           202980 kB
PageTables:      39260 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 67411468 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73564 MiB
End job on node133 at 2011-05-15 15:27:39