Trace number 3386981

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (TO) 5000.09 4994.46

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v650-c2769-S2065143904-035.UNKNOWN.cnf
MD5SUM8f7328be14ce714cebc5a66418b5924c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables650
Number of clauses2769
Sum of the clauses size8307
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32769
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.32/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.67	c INFO: solving HOME/instance-3386981-1307741611.cnf
0.32/0.67	c INFO: computing task features
0.32/0.68	c INFO: computed variable-clause graph statistics
0.32/0.68	c INFO: computed clause balance statistics
0.41/0.74	c INFO: computed variable balance statistics
0.41/0.78	c INFO: computed small-clause counts
0.41/0.78	c INFO: computed Horn-clause counts
0.41/0.79	c INFO: computed variable graph statistics
0.41/0.79	c INFO: collected features for HOME/instance-3386981-1307741611.cnf in 0.12 s
0.50/0.84	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.58)
100.68/100.98	c INFO: running march_hi@0 for 194 with 4899 remaining (b = 0.15)
295.81/295.82	c INFO: running march_hi@194 for 194 with 4704 remaining (b = 0.16)
492.62/492.43	c INFO: running march_hi@391 for 97 with 4508 remaining (b = 0.09)
587.13/586.85	c INFO: running march_hi@485 for 97 with 4413 remaining (b = 0.08)
688.05/687.64	c INFO: running march_hi@586 for 97 with 4313 remaining (b = 0.09)
783.27/782.76	c INFO: running gnovelty+2@0 for 97 with 4218 remaining (b = 0.07)
883.57/882.90	c INFO: running march_hi@681 for 97 with 4117 remaining (b = 0.06)
981.31/980.56	c INFO: running march_hi@0 for 485 with 4020 remaining (b = 0.32)
1467.49/1466.18	c INFO: running march_hi@0 for 97 with 3534 remaining (b = 0.07)
1565.21/1563.70	c INFO: running march_hi@485 for 194 with 3437 remaining (b = 0.15)
1763.56/1761.86	c INFO: running march_hi@97 for 291 with 3239 remaining (b = 0.19)
2055.01/2052.98	c INFO: running march_hi@388 for 97 with 2947 remaining (b = 0.07)
2154.83/2152.69	c INFO: running march_hi@488 for 97 with 2848 remaining (b = 0.06)
2250.15/2247.84	c INFO: running march_hi@583 for 97 with 2753 remaining (b = 0.06)
2351.37/2348.96	c INFO: running march_hi@0 for 679 with 2651 remaining (b = 0.30)
3033.00/3029.70	c INFO: running march_hi@0 for 388 with 1971 remaining (b = 0.20)
3425.89/3422.19	c INFO: running kcnfs-2006@0 for 194 with 1578 remaining (b = 0.09)
3622.42/3618.44	c INFO: running march_hi@779 for 1382 with 1382 remaining (b = 0.42)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 988)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386981-1307741611/watcher-3386981-1307741611 -o /tmp/evaluation-result-3386981-1307741611/solver-3386981-1307741611 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386981-1307741611.cnf -seed 1181408636 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.61 0.13 0.04 1/171 14801
/proc/meminfo: memFree=31661360/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=15080 CPUtime=0 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 618 0 0 0 0 0 0 0 25 0 1 0 885763776 15441920 513 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634513408 0 2147483391 16781312 2 18446744071699199166 0 0 17 2 0 0 0
/proc/14801/statm: 3770 513 300 364 0 257 0

[startup+0.0121331 s]
/proc/loadavg: 0.61 0.13 0.04 1/171 14801
/proc/meminfo: memFree=31661360/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=15080 CPUtime=0 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 651 0 0 0 0 0 0 0 25 0 1 0 885763776 15441920 545 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634513408 0 2147483391 16781312 2 18446744071699199166 0 0 17 2 0 0 0
/proc/14801/statm: 3770 545 312 364 0 257 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15080

[startup+0.101152 s]
/proc/loadavg: 0.61 0.13 0.04 1/171 14801
/proc/meminfo: memFree=31661360/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=16244 CPUtime=0.01 cores=0,2,4,6
/proc/14801/stat : 14801 (python) R 14799 14801 14492 0 -1 4202496 1065 0 0 0 1 0 0 0 17 0 1 0 885763776 16633856 935 33554432000 4194304 5683644 140733947540816 18446744073709551615 4378771 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14801/statm: 4061 935 381 364 0 548 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16244

[startup+0.301119 s]
/proc/loadavg: 0.61 0.13 0.04 1/171 14801
/proc/meminfo: memFree=31661360/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=115428 CPUtime=0.08 cores=0,2,4,6
/proc/14801/stat : 14801 (python) R 14799 14801 14492 0 -1 4202496 3606 0 0 0 6 2 0 0 16 0 1 0 885763776 118198272 3242 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634513408 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/14801/statm: 28857 3246 893 364 0 2358 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 115428

[startup+0.701066 s]
/proc/loadavg: 0.61 0.13 0.04 1/171 14801
/proc/meminfo: memFree=31661360/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=278448 CPUtime=0.41 cores=0,2,4,6
/proc/14801/stat : 14801 (python) R 14799 14801 14492 0 -1 4202496 9373 935 1 0 33 8 0 0 17 0 1 0 885763776 285130752 7028 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634173849 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/14801/statm: 69612 7028 1826 364 0 5536 0
Current children cumulated CPU time (s) 0.41
Current children cumulated vsize (KiB) 278448

[startup+1.50091 s]
/proc/loadavg: 0.61 0.13 0.04 2/175 14809
/proc/meminfo: memFree=31636948/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=288872 CPUtime=0.53 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 9847 935 1 0 45 8 0 0 19 0 2 0 885763776 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 72218 7116 1875 364 0 8136 0
[pid=14807] ppid=14801 vsize=278628 CPUtime=0 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 746 0 0 0 0 0 0 0 18 0 1 0 885763860 285315072 5591 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535439 0 0 16781312 514 8707419062990274559 0 0 17 0 0 0 0
/proc/14807/statm: 69657 5591 354 364 0 5575 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=0.65 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) R 14807 14809 14809 0 -1 4202496 458 0 0 0 65 0 0 0 21 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134547313 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14801/tid=14808] ppid=14799 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 2 0 885763860 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1.18
Current children cumulated vsize (KiB) 1585440

[startup+3.10057 s]
/proc/loadavg: 0.61 0.13 0.04 2/175 14809
/proc/meminfo: memFree=31637072/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=288872 CPUtime=0.53 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 9847 935 1 0 45 8 0 0 19 0 2 0 885763776 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 72218 7116 1875 364 0 8136 0
[pid=14807] ppid=14801 vsize=278628 CPUtime=0 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 746 0 0 0 0 0 0 0 18 0 1 0 885763860 285315072 5591 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535439 0 0 16781312 514 8707419062990242049 0 0 17 0 0 0 0
/proc/14807/statm: 69657 5591 354 364 0 5575 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=2.25 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) R 14807 14809 14809 0 -1 4202496 458 0 0 0 225 0 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134524509 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14801/tid=14808] ppid=14799 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 2 0 885763860 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2.78
Current children cumulated vsize (KiB) 1585440

[startup+6.30106 s]
/proc/loadavg: 0.73 0.16 0.05 2/175 14809
/proc/meminfo: memFree=31637756/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=288872 CPUtime=0.53 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 9847 935 1 0 45 8 0 0 19 0 2 0 885763776 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 72218 7116 1875 364 0 8136 0
[pid=14807] ppid=14801 vsize=278628 CPUtime=0 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 888 0 0 0 0 0 0 0 18 0 1 0 885763860 285315072 5614 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535439 0 0 16781312 514 8707419062990242049 0 0 17 0 0 0 0
/proc/14807/statm: 69657 5614 377 364 0 5575 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=5.45 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) R 14807 14809 14809 0 -1 4202496 458 0 0 0 544 1 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134547260 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14801/tid=14808] ppid=14799 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 2 0 885763860 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 5.98
Current children cumulated vsize (KiB) 1585440

[startup+12.7009 s]
/proc/loadavg: 0.75 0.18 0.06 2/175 14809
/proc/meminfo: memFree=31639132/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=288872 CPUtime=0.53 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 9847 935 1 0 45 8 0 0 19 0 2 0 885763776 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 72218 7116 1875 364 0 8136 0
[pid=14807] ppid=14801 vsize=278628 CPUtime=0 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 890 0 0 0 0 0 0 0 18 0 1 0 885763860 285315072 5614 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535439 0 0 16781312 514 17914545261493846273 0 0 17 0 0 0 0
/proc/14807/statm: 69657 5614 377 364 0 5575 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=11.85 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) R 14807 14809 14809 0 -1 4202496 458 0 0 0 1184 1 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134545619 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14801/tid=14808] ppid=14799 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 2 0 885763860 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 12.38
Current children cumulated vsize (KiB) 1585440

[startup+25.5006 s]
/proc/loadavg: 0.80 0.22 0.07 2/175 14809
/proc/meminfo: memFree=31641116/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=288872 CPUtime=0.53 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 9847 935 1 0 45 8 0 0 19 0 2 0 885763776 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 72218 7116 1875 364 0 8136 0
[pid=14807] ppid=14801 vsize=278628 CPUtime=0.02 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 893 0 0 0 1 1 0 0 18 0 1 0 885763860 285315072 5614 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535439 0 0 16781312 514 17914545261493846273 0 0 17 0 0 0 0
/proc/14807/statm: 69657 5614 377 364 0 5575 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=24.64 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) R 14807 14809 14809 0 -1 4202496 458 0 0 0 2463 1 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14801/tid=14808] ppid=14799 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 2 0 885763860 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 25.19
Current children cumulated vsize (KiB) 1585440

[startup+51.1011 s]
/proc/loadavg: 0.87 0.28 0.10 2/175 14810
/proc/meminfo: memFree=31641676/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=288872 CPUtime=0.53 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 9847 935 1 0 45 8 0 0 19 0 2 0 885763776 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 72218 7116 1875 364 0 8136 0
[pid=14807] ppid=14801 vsize=278628 CPUtime=0.05 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 902 0 0 0 3 2 0 0 18 0 1 0 885763860 285315072 5614 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535439 0 0 16781312 514 17914545261493846273 0 0 17 0 0 0 0
/proc/14807/statm: 69657 5614 377 364 0 5575 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=50.24 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) R 14807 14809 14809 0 -1 4202496 458 0 0 0 5023 1 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134544268 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14801/tid=14808] ppid=14799 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 2 0 885763860 295804928 7116 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 50.82
Current children cumulated vsize (KiB) 1585440

[startup+102.306 s]
/proc/loadavg: 0.94 0.39 0.14 2/179 14815
/proc/meminfo: memFree=31636784/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=299644 CPUtime=0.56 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 10667 935 1 0 48 8 0 0 20 0 3 0 885763776 306835456 7242 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14801/statm: 74911 7242 1878 364 0 10826 0
[pid=14807] ppid=14801 vsize=288872 CPUtime=0.11 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 1076 0 0 0 7 4 0 0 19 0 2 0 885763860 295804928 5652 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14807/statm: 72218 5652 410 364 0 8136 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=100.1 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) T 14807 14809 14809 0 -1 4202496 458 0 0 0 10009 1 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134524551 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14807/tid=14812] ppid=14801 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14807/task/14812/stat : 14812 (python) S 14801 14801 14492 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 885773871 295804928 5652 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604442055145344 0 0 -1 6 0 0 0
[pid=14813] ppid=14801 vsize=289400 CPUtime=0 cores=0,2,4,6
/proc/14813/stat : 14813 (python) S 14801 14801 14492 0 -1 4202560 763 0 0 0 0 0 0 0 15 0 1 0 885773874 296345600 5708 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535494 0 0 16781312 514 3551264232337015044 0 0 17 6 0 0 0
/proc/14813/statm: 72350 5708 348 364 0 8265 0
[pid=14815] ppid=14813 vsize=2728 CPUtime=1.31 cores=0,2,4,6
/proc/14815/stat : 14815 (march_hi) R 14813 14815 14815 0 -1 4202496 759 0 0 0 131 0 0 0 25 0 1 0 885773875 2793472 496 33554432000 4194304 4955518 140734731043952 18446744073709551615 4235304 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14815/statm: 682 496 104 186 0 495 0
[pid=14801/tid=14808] ppid=14799 vsize=299644 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 3 0 885763860 306835456 7242 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14814] ppid=14799 vsize=299644 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14814/stat : 14814 (python) S 14799 14801 14492 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 3 0 885773874 306835456 7242 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 102.08
Current children cumulated vsize (KiB) 1898584

[startup+162.301 s]
/proc/loadavg: 0.98 0.50 0.19 2/179 14817
/proc/meminfo: memFree=31627680/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=299644 CPUtime=0.56 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 10667 935 1 0 48 8 0 0 20 0 3 0 885763776 306835456 7242 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/14801/statm: 74911 7242 1878 364 0 10826 0
[pid=14807] ppid=14801 vsize=288872 CPUtime=0.11 cores=0,2,4,6

################
# More data... #
################

[pid=14845] ppid=14801 vsize=310388 CPUtime=0.83 cores=0,2,4,6
/proc/14845/stat : 14845 (python) S 14801 14801 14492 0 -1 4202560 1151 0 0 0 44 39 0 0 15 0 2 0 885861832 317837312 5871 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14845/statm: 77597 5871 400 364 0 13506 0
[pid=14847] ppid=14845 vsize=109972 CPUtime=683.72 cores=0,2,4,6
/proc/14847/stat : 14847 (march_hi) T 14845 14847 14847 0 -1 4202496 22225 0 0 0 68366 6 0 0 25 0 1 0 885861833 112611328 21961 33554432000 4194304 4955518 140737372174928 18446744073709551615 4227814 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14847/statm: 27493 21961 104 186 0 27306 0
[pid=14845/tid=14861] ppid=14801 vsize=310388 CPUtime=0 cores=0,2,4,6
/proc/14845/task/14861/stat : 14861 (python) S 14801 14801 14492 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 885910391 317837312 5871 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453918257472 0 0 -1 4 0 0 0
[pid=14862] ppid=14801 vsize=320644 CPUtime=0.83 cores=0,2,4,6
/proc/14862/stat : 14862 (python) S 14801 14801 14492 0 -1 4202560 1152 0 0 0 44 39 0 0 15 0 2 0 885910394 328339456 5875 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14862/statm: 80161 5875 400 364 0 16067 0
[pid=14864] ppid=14862 vsize=109972 CPUtime=684.53 cores=0,2,4,6
/proc/14864/stat : 14864 (march_hi) T 14862 14864 14864 0 -1 4202496 22162 0 0 0 68448 5 0 0 25 0 1 0 885910394 112611328 21897 33554432000 4194304 4955518 140733578021376 18446744073709551615 4229688 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14864/statm: 27493 21897 104 186 0 27306 0
[pid=14862/tid=14867] ppid=14801 vsize=320644 CPUtime=0 cores=0,2,4,6
/proc/14862/task/14867/stat : 14867 (python) S 14801 14801 14492 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 885920144 328339456 5875 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453985336000 0 0 -1 4 0 0 0
[pid=14901] ppid=14801 vsize=330900 CPUtime=0.82 cores=0,2,4,6
/proc/14901/stat : 14901 (python) S 14801 14801 14492 0 -1 4202560 1159 0 0 0 47 35 0 0 16 0 2 0 885998672 338841600 5879 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/14901/statm: 82725 5879 400 364 0 18628 0
[pid=14903] ppid=14901 vsize=109972 CPUtime=680.73 cores=0,2,4,6
/proc/14903/stat : 14903 (march_hi) T 14901 14903 14903 0 -1 4202496 22198 0 0 0 68068 5 0 0 25 0 1 0 885998673 112611328 21931 33554432000 4194304 4955518 140733219137984 18446744073709551615 4227571 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14903/statm: 27493 21931 104 186 0 27306 0
[pid=14901/tid=14922] ppid=14801 vsize=330900 CPUtime=0 cores=0,2,4,6
/proc/14901/task/14922/stat : 14922 (python) S 14801 14801 14492 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 886066745 338841600 5879 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453939695808 0 0 -1 6 0 0 0
[pid=14923] ppid=14801 vsize=341156 CPUtime=0.48 cores=0,2,4,6
/proc/14923/stat : 14923 (python) S 14801 14801 14492 0 -1 4202560 1150 0 0 0 29 19 0 0 16 0 2 0 886066746 349343744 5883 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14923/statm: 85289 5883 400 364 0 21189 0
[pid=14925] ppid=14923 vsize=71208 CPUtime=392.46 cores=0,2,4,6
/proc/14925/stat : 14925 (march_hi) T 14923 14925 14925 0 -1 4202496 13536 0 0 0 39242 4 0 0 25 0 1 0 886066747 72916992 13268 33554432000 4194304 4955518 140737052412608 18446744073709551615 4413705 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14925/statm: 17802 13268 104 186 0 17615 0
[pid=14923/tid=14937] ppid=14801 vsize=341156 CPUtime=0 cores=0,2,4,6
/proc/14923/task/14937/stat : 14937 (python) S 14801 14801 14492 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 886105993 349343744 5883 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453733576768 0 0 -1 4 0 0 0
[pid=14938] ppid=14801 vsize=351412 CPUtime=0.25 cores=0,2,4,6
/proc/14938/stat : 14938 (python) S 14801 14801 14492 0 -1 4202560 1171 0 0 0 16 9 0 0 16 0 2 0 886105995 359845888 5906 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14938/statm: 87853 5906 406 364 0 23750 0
[pid=14940] ppid=14938 vsize=182584 CPUtime=196.24 cores=0,2,4,6
/proc/14940/stat : 14940 (kcnfs-2006) T 14938 14940 14940 0 -1 4202496 653 0 0 0 19624 0 0 0 25 0 1 0 886105995 186966016 387 33554432000 134512640 135163436 4293180640 18446744073709551615 134595343 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/14940/statm: 45646 387 77 159 0 45486 0
[pid=14938/tid=14946] ppid=14801 vsize=351412 CPUtime=0 cores=0,2,4,6
/proc/14938/task/14946/stat : 14946 (python) S 14801 14801 14492 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 886125619 359845888 5906 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453976596928 0 0 -1 0 0 0 0
[pid=14801/tid=14808] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 9 0 885763860 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14814] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14814/stat : 14814 (python) S 14799 14801 14492 0 -1 4202560 37 935 0 0 0 0 0 0 18 0 9 0 885773874 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14837] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14837/stat : 14837 (python) S 14799 14801 14492 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 9 0 885842052 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14846] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14846/stat : 14846 (python) S 14799 14801 14492 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 9 0 885861832 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14863] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14863/stat : 14863 (python) S 14799 14801 14492 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 885910394 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14902] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14902/stat : 14902 (python) S 14799 14801 14492 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 9 0 885998673 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14924] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14924/stat : 14924 (python) S 14799 14801 14492 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 886066746 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14939] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14939/stat : 14939 (python) S 14799 14801 14492 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 886105995 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4967.94
Current children cumulated vsize (KiB) 5889724



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+4994.4 s]
/proc/loadavg: 1.01 1.02 0.95 2/204 14983
/proc/meminfo: memFree=31020032/32951124 swapFree=67052812/67111528
[pid=14801] ppid=14799 vsize=361656 CPUtime=0.96 cores=0,2,4,6
/proc/14801/stat : 14801 (python) S 14799 14801 14492 0 -1 4202496 17121 935 1 0 86 10 0 0 18 0 9 0 885763776 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/14801/statm: 90414 7393 1890 364 0 26311 0
[pid=14807] ppid=14801 vsize=288872 CPUtime=0.11 cores=0,2,4,6
/proc/14807/stat : 14807 (python) S 14801 14801 14492 0 -1 4202560 1076 0 0 0 7 4 0 0 19 0 2 0 885763860 295804928 5652 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14807/statm: 72218 5652 410 364 0 8136 0
[pid=14809] ppid=14807 vsize=1017940 CPUtime=100.1 cores=0,2,4,6
/proc/14809/stat : 14809 (gnovelty+2) T 14807 14809 14809 0 -1 4202496 458 0 0 0 10009 1 0 0 25 0 1 0 885763861 1042370560 192 33554432000 134512640 135663704 4293060944 18446744073709551615 134524551 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/14809/statm: 254485 192 113 282 0 254201 0
[pid=14807/tid=14812] ppid=14801 vsize=288872 CPUtime=0 cores=0,2,4,6
/proc/14807/task/14812/stat : 14812 (python) S 14801 14801 14492 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 885773871 295804928 5652 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604442055145344 0 0 -1 6 0 0 0
[pid=14813] ppid=14801 vsize=289400 CPUtime=2.54 cores=0,2,4,6
/proc/14813/stat : 14813 (python) S 14801 14801 14492 0 -1 4202560 1124 0 0 0 147 107 0 0 15 0 2 0 885773874 296345600 5761 33554432000 4194304 5683644 140733947540816 18446744073709551615 216634535494 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/14813/statm: 72350 5761 400 364 0 8265 0
[pid=14815] ppid=14813 vsize=375840 CPUtime=2155.28 cores=0,2,4,6
/proc/14815/stat : 14815 (march_hi) R 14813 14815 14815 0 -1 4202496 66277 0 0 0 215510 18 0 0 25 0 1 0 885773875 384860160 66014 33554432000 4194304 4955518 140734731043952 18446744073709551615 4238727 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/14815/statm: 93960 66014 104 186 0 93773 0
[pid=14813/tid=14822] ppid=14801 vsize=289400 CPUtime=0 cores=0,2,4,6
/proc/14813/task/14822/stat : 14822 (python) S 14801 14801 14492 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 885793355 296345600 5761 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453938677056 0 0 -1 6 0 0 0
[pid=14836] ppid=14801 vsize=299868 CPUtime=0.11 cores=0,2,4,6
/proc/14836/stat : 14836 (python) S 14801 14801 14492 0 -1 4202560 1068 0 0 0 6 5 0 0 19 0 2 0 885842052 307064832 5801 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/14836/statm: 74967 5801 400 364 0 10879 0
[pid=14838] ppid=14836 vsize=1017940 CPUtime=100.1 cores=0,2,4,6
/proc/14838/stat : 14838 (gnovelty+2) T 14836 14838 14838 0 -1 4202496 454 0 0 0 10010 0 0 0 25 0 1 0 885842053 1042370560 192 33554432000 134512640 135663704 4294361760 18446744073709551615 134546978 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/14838/statm: 254485 192 113 282 0 254201 0
[pid=14836/tid=14841] ppid=14801 vsize=299868 CPUtime=0 cores=0,2,4,6
/proc/14836/task/14841/stat : 14841 (python) S 14801 14801 14492 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 885852063 307064832 5801 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453730884608 0 0 -1 4 0 0 0
[pid=14845] ppid=14801 vsize=310388 CPUtime=0.83 cores=0,2,4,6
/proc/14845/stat : 14845 (python) S 14801 14801 14492 0 -1 4202560 1151 0 0 0 44 39 0 0 15 0 2 0 885861832 317837312 5871 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14845/statm: 77597 5871 400 364 0 13506 0
[pid=14847] ppid=14845 vsize=109972 CPUtime=683.72 cores=0,2,4,6
/proc/14847/stat : 14847 (march_hi) T 14845 14847 14847 0 -1 4202496 22225 0 0 0 68366 6 0 0 25 0 1 0 885861833 112611328 21961 33554432000 4194304 4955518 140737372174928 18446744073709551615 4227814 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14847/statm: 27493 21961 104 186 0 27306 0
[pid=14845/tid=14861] ppid=14801 vsize=310388 CPUtime=0 cores=0,2,4,6
/proc/14845/task/14861/stat : 14861 (python) S 14801 14801 14492 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 885910391 317837312 5871 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453918257472 0 0 -1 4 0 0 0
[pid=14862] ppid=14801 vsize=320644 CPUtime=0.83 cores=0,2,4,6
/proc/14862/stat : 14862 (python) S 14801 14801 14492 0 -1 4202560 1152 0 0 0 44 39 0 0 15 0 2 0 885910394 328339456 5875 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14862/statm: 80161 5875 400 364 0 16067 0
[pid=14864] ppid=14862 vsize=109972 CPUtime=684.53 cores=0,2,4,6
/proc/14864/stat : 14864 (march_hi) T 14862 14864 14864 0 -1 4202496 22162 0 0 0 68448 5 0 0 25 0 1 0 885910394 112611328 21897 33554432000 4194304 4955518 140733578021376 18446744073709551615 4229688 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14864/statm: 27493 21897 104 186 0 27306 0
[pid=14862/tid=14867] ppid=14801 vsize=320644 CPUtime=0 cores=0,2,4,6
/proc/14862/task/14867/stat : 14867 (python) S 14801 14801 14492 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 885920144 328339456 5875 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453985336000 0 0 -1 4 0 0 0
[pid=14901] ppid=14801 vsize=330900 CPUtime=0.82 cores=0,2,4,6
/proc/14901/stat : 14901 (python) S 14801 14801 14492 0 -1 4202560 1159 0 0 0 47 35 0 0 16 0 2 0 885998672 338841600 5879 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/14901/statm: 82725 5879 400 364 0 18628 0
[pid=14903] ppid=14901 vsize=109972 CPUtime=680.73 cores=0,2,4,6
/proc/14903/stat : 14903 (march_hi) T 14901 14903 14903 0 -1 4202496 22198 0 0 0 68068 5 0 0 25 0 1 0 885998673 112611328 21931 33554432000 4194304 4955518 140733219137984 18446744073709551615 4227571 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14903/statm: 27493 21931 104 186 0 27306 0
[pid=14901/tid=14922] ppid=14801 vsize=330900 CPUtime=0 cores=0,2,4,6
/proc/14901/task/14922/stat : 14922 (python) S 14801 14801 14492 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 886066745 338841600 5879 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453939695808 0 0 -1 6 0 0 0
[pid=14923] ppid=14801 vsize=341156 CPUtime=0.48 cores=0,2,4,6
/proc/14923/stat : 14923 (python) S 14801 14801 14492 0 -1 4202560 1150 0 0 0 29 19 0 0 16 0 2 0 886066746 349343744 5883 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/14923/statm: 85289 5883 400 364 0 21189 0
[pid=14925] ppid=14923 vsize=71208 CPUtime=392.46 cores=0,2,4,6
/proc/14925/stat : 14925 (march_hi) T 14923 14925 14925 0 -1 4202496 13536 0 0 0 39242 4 0 0 25 0 1 0 886066747 72916992 13268 33554432000 4194304 4955518 140737052412608 18446744073709551615 4413705 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/14925/statm: 17802 13268 104 186 0 17615 0
[pid=14923/tid=14937] ppid=14801 vsize=341156 CPUtime=0 cores=0,2,4,6
/proc/14923/task/14937/stat : 14937 (python) S 14801 14801 14492 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 886105993 349343744 5883 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453733576768 0 0 -1 4 0 0 0
[pid=14938] ppid=14801 vsize=351412 CPUtime=0.25 cores=0,2,4,6
/proc/14938/stat : 14938 (python) S 14801 14801 14492 0 -1 4202560 1171 0 0 0 16 9 0 0 16 0 2 0 886105995 359845888 5906 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646342091 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/14938/statm: 87853 5906 406 364 0 23750 0
[pid=14940] ppid=14938 vsize=182584 CPUtime=196.24 cores=0,2,4,6
/proc/14940/stat : 14940 (kcnfs-2006) T 14938 14940 14940 0 -1 4202496 653 0 0 0 19624 0 0 0 25 0 1 0 886105995 186966016 387 33554432000 134512640 135163436 4293180640 18446744073709551615 134595343 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/14940/statm: 45646 387 77 159 0 45486 0
[pid=14938/tid=14946] ppid=14801 vsize=351412 CPUtime=0 cores=0,2,4,6
/proc/14938/task/14946/stat : 14946 (python) S 14801 14801 14492 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 886125619 359845888 5906 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 514 18446604453976596928 0 0 -1 0 0 0 0
[pid=14801/tid=14808] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14808/stat : 14808 (python) S 14799 14801 14492 0 -1 4202560 32 935 0 0 0 0 0 0 19 0 9 0 885763860 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14814] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14814/stat : 14814 (python) S 14799 14801 14492 0 -1 4202560 37 935 0 0 0 0 0 0 18 0 9 0 885773874 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14837] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14837/stat : 14837 (python) S 14799 14801 14492 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 9 0 885842052 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 2 0 0 0
[pid=14801/tid=14846] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14846/stat : 14846 (python) S 14799 14801 14492 0 -1 4202560 31 935 0 0 0 0 0 0 18 0 9 0 885861832 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14863] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14863/stat : 14863 (python) S 14799 14801 14492 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 9 0 885910394 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14902] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14902/stat : 14902 (python) S 14799 14801 14492 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 9 0 885998673 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14924] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14924/stat : 14924 (python) S 14799 14801 14492 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 886066746 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
[pid=14801/tid=14939] ppid=14799 vsize=361656 CPUtime=0 cores=0,2,4,6
/proc/14801/task/14939/stat : 14939 (python) S 14799 14801 14492 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 9 0 886105995 370335744 7393 33554432000 4194304 5683644 140733947540816 18446744073709551615 216646338993 0 0 16781312 2 18446604453937848640 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5000.09
Current children cumulated vsize (KiB) 5889724

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 14801 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=868867
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=106983
# CPU time returned by wait4() is 0.97585
# while last known CPU time is 5000.09
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 4994.46
CPU time (s): 5000.09
CPU user time (s): 4997.03
CPU system time (s): 3.06
CPU usage (%): 100.113
Max. virtual memory (cumulated for all children) (KiB): 5889728

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.868867
system time used= 0.106983
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18061
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1043
involuntary context switches= 26

runsolver used 14.0879 second user time and 25.5071 second system time

The end

Launcher Data 

Begin job on node103 at 2011-06-10 23:33:31
IDJOB=3386981
IDBENCH=83046
IDSOLVER=1872
FILE ID=node103/3386981-1307741611
RUNJOBID= node103-1307741600-14510
PBS_JOBID= 13550809
Free space on /tmp= 74072 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v650-c2769-S2065143904-035.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386981-1307741611/watcher-3386981-1307741611 -o /tmp/evaluation-result-3386981-1307741611/solver-3386981-1307741611 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386981-1307741611.cnf -seed 1181408636 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8f7328be14ce714cebc5a66418b5924c
RANDOM SEED=1181408636

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31661640 kB
Buffers:        255432 kB
Cached:         791956 kB
SwapCached:      22864 kB
Active:         478384 kB
Inactive:       597844 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31661640 kB
SwapTotal:    67111528 kB
SwapFree:     67052812 kB
Dirty:           33188 kB
Writeback:           0 kB
AnonPages:       26472 kB
Mapped:           8816 kB
Slab:           146940 kB
PageTables:       4656 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   189328 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74036 MiB
End job on node103 at 2011-06-11 00:56:48