Trace number 298118

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
KCNFS SMPSAT 969.99 486.162

General information on the benchmark

Namerandom/OnTreshold/3SAT/v500/
unif-k3-r4.26-v500-c2130-S1490740667-05.SAT.shuffled.cnf
MD5SUMf01193a3f455045d150f231e0d56c100
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.012997
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	
0.00/0.00	c Computed by /tmp/evaluation/298118-1171850667/kcnfs-smp   V. 1.2
0.00/0.00	c reading file ... done
0.00/0.00	c Input file : '/tmp/evaluation/298118-1171850667/instance-298118-1171850667.cnf'
0.00/0.00	c [#vars: 500, #clauses: 2130 -> 2217 clauses (pre-processing), tautology: 0, 3-CNF]
0.00/0.01	c unit clauses propagate before processing (#clause: 2217)
0.00/0.01	c estimated distribution: 8 tasks
0.00/0.06	c <28796> [PID]: 28798 28799 28800 28801 28802 28803 28804
0.03/0.36	c <28796> [PID]: 28805
0.03/0.36	c real distribution: 8 tasks
969.99/486.15	c <28796> [28798: SAT]
969.99/486.15	c <28796> [END]: 28799 28800 28801 28802 28803 28804 28805
969.99/486.15	s SATISFIABLE
969.99/486.15	v       -1       -2        3        4       -5       -6       -7
969.99/486.15	v        8        9       10      -11      -12       13       14
969.99/486.15	v      -15       16      -17      -18       19      -20      -21
969.99/486.15	v       22      -23       24       25       26       27      -28
969.99/486.15	v      -29      -30       31      -32       33      -34       35
969.99/486.15	v      -36       37       38      -39      -40       41       42
969.99/486.15	v       43      -44       45      -46      -47      -48      -49
969.99/486.15	v      -50       51       52      -53       54       55       56
969.99/486.15	v      -57       58       59      -60       61       62       63
969.99/486.15	v       64      -65      -66       67       68       69       70
969.99/486.15	v       71       72      -73      -74      -75       76      -77
969.99/486.15	v       78      -79      -80      -81       82      -83       84
969.99/486.15	v       85       86      -87       88      -89      -90       91
969.99/486.15	v      -92      -93       94      -95       96       97       98
969.99/486.15	v       99     -100      101     -102      103     -104      105
969.99/486.15	v     -106     -107      108      109      110     -111      112
969.99/486.15	v      113      114     -115     -116      117     -118     -119
969.99/486.15	v      120     -121      122     -123      124     -125      126
969.99/486.15	v     -127     -128     -129      130     -131      132      133
969.99/486.15	v      134      135      136     -137     -138      139     -140
969.99/486.15	v      141     -142     -143     -144      145     -146     -147
969.99/486.15	v      148     -149     -150     -151     -152      153     -154
969.99/486.15	v      155     -156     -157     -158     -159      160     -161
969.99/486.15	v      162      163      164      165      166     -167      168
969.99/486.15	v     -169     -170      171     -172      173      174      175
969.99/486.15	v     -176     -177      178      179      180     -181      182
969.99/486.15	v      183      184      185      186      187      188      189
969.99/486.15	v      190     -191     -192      193      194      195     -196
969.99/486.15	v     -197      198      199     -200     -201      202      203
969.99/486.15	v      204      205     -206      207      208     -209     -210
969.99/486.15	v      211      212      213     -214     -215     -216      217
969.99/486.15	v     -218     -219      220     -221     -222      223      224
969.99/486.15	v     -225     -226     -227     -228      229     -230     -231
969.99/486.15	v      232      233     -234      235     -236     -237     -238
969.99/486.15	v     -239      240     -241      242     -243      244      245
969.99/486.15	v     -246     -247      248     -249      250      251     -252
969.99/486.15	v     -253      254      255     -256      257     -258     -259
969.99/486.15	v      260     -261      262     -263     -264      265     -266
969.99/486.15	v      267      268     -269     -270     -271      272     -273
969.99/486.15	v     -274      275      276     -277     -278     -279     -280
969.99/486.15	v     -281      282     -283      284     -285      286     -287
969.99/486.15	v     -288     -289     -290      291     -292      293     -294
969.99/486.15	v      295      296      297      298     -299      300      301
969.99/486.15	v      302      303      304     -305      306      307      308
969.99/486.15	v      309     -310     -311      312      313     -314     -315
969.99/486.15	v      316     -317     -318     -319     -320     -321     -322
969.99/486.16	v     -323     -324      325      326      327     -328      329
969.99/486.16	v     -330     -331     -332     -333      334      335      336
969.99/486.16	v     -337      338      339     -340      341     -342      343
969.99/486.16	v     -344     -345     -346     -347      348     -349     -350
969.99/486.16	v     -351     -352     -353      354      355     -356     -357
969.99/486.16	v     -358     -359      360     -361      362      363      364
969.99/486.16	v      365     -366      367      368     -369     -370      371
969.99/486.16	v     -372     -373     -374     -375      376      377     -378
969.99/486.16	v      379     -380      381      382     -383      384      385
969.99/486.16	v      386     -387     -388     -389      390     -391     -392
969.99/486.16	v      393     -394      395      396     -397     -398     -399
969.99/486.16	v     -400     -401     -402      403      404     -405      406
969.99/486.16	v      407      408     -409     -410      411      412      413
969.99/486.16	v     -414     -415      416     -417     -418     -419     -420
969.99/486.16	v     -421     -422     -423      424      425     -426     -427
969.99/486.16	v      428     -429      430     -431      432     -433     -434
969.99/486.16	v      435     -436     -437     -438     -439     -440      441
969.99/486.16	v     -442      443     -444      445     -446     -447      448
969.99/486.16	v      449      450      451     -452      453     -454     -455
969.99/486.16	v      456     -457     -458      459     -460      461      462
969.99/486.16	v     -463      464     -465      466      467      468      469
969.99/486.16	v      470      471     -472     -473     -474      475     -476
969.99/486.16	v     -477     -478      479     -480      481      482      483
969.99/486.16	v      484      485      486      487     -488      489      490
969.99/486.16	v     -491     -492     -493      494     -495     -496      497
969.99/486.16	v      498      499     -500        0
969.99/486.16	c CPU time            : 121.530 seconds
969.99/486.16	c Elapsed Time        : 486.155 seconds
969.99/486.16	c Size of search tree : 192797 nodes
969.99/486.16	c average speed       : 1586.415 nodes/sec.
969.99/486.16

Verifier Data (download as text)

OK

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node29/watcher-298118-1171850667 -o ROOT/results/node29/solver-298118-1171850667 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/298118-1171850667/kcnfs-smp /tmp/evaluation/298118-1171850667/instance-298118-1171850667.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.99 0.97 0.99 3/77 28796
/proc/meminfo: memFree=904640/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=181980 CPUtime=0
/proc/28796/stat : 28796 (kcnfs-smp) R 28794 28796 26081 0 -1 4194304 156 0 0 0 0 0 0 0 18 0 1 0 178830974 186347520 140 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134726021 0 0 4096 0 0 0 0 17 1 0 0
/proc/28796/statm: 45495 140 51 160 0 45333 0

[startup+0.107401 s]
/proc/loadavg: 0.99 0.97 0.99 3/77 28796
/proc/meminfo: memFree=904640/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.03
/proc/28796/stat : 28796 (kcnfs-smp) R 28794 28796 26081 0 -1 4194304 377 0 0 0 3 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134647674 0 0 4096 0 0 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 182448

[startup+0.517424 s]
/proc/loadavg: 0.99 0.97 0.99 3/77 28796
/proc/meminfo: memFree=904640/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 182448

[startup+1.33848 s]
/proc/loadavg: 1.55 1.09 1.03 9/86 28805
/proc/meminfo: memFree=901624/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=0.53
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 53 0 0 0 24 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683263 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=0.41
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 41 0 0 0 24 0 1 0 178830977 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134578900 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 271 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=0.38
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 38 0 0 0 23 0 1 0 178830979 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579705 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 271 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=0.36
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 36 0 0 0 25 0 1 0 178830980 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572151 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 271 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=0.24
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 24 0 0 0 25 0 1 0 178831001 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580397 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 271 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=0.22
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 22 0 0 0 25 0 1 0 178831006 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567570 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 271 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=0.2
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 20 0 0 0 25 0 1 0 178831009 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134596065 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 271 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=0.2
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 20 0 0 0 25 0 1 0 178831010 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580054 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 2.58
Current children cumulated vsize (KiB) 1642032

[startup+2.97558 s]
/proc/loadavg: 1.55 1.09 1.03 9/86 28805
/proc/meminfo: memFree=901496/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=0.93
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 93 0 0 0 25 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577530 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=0.83
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 83 0 0 0 25 0 1 0 178830977 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572793 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 271 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=0.8
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 80 0 0 0 25 0 1 0 178830979 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134589703 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 271 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=0.79
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 79 0 0 0 25 0 1 0 178830980 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567584 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 271 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=0.64
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 64 0 0 0 25 0 1 0 178831001 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577061 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 271 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=0.62
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 62 0 0 0 25 0 1 0 178831006 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134590074 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 271 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=0.6
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 60 0 0 0 25 0 1 0 178831009 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572793 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 271 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=0.6
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 60 0 0 0 25 0 1 0 178831010 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572629 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 5.85
Current children cumulated vsize (KiB) 1642032

[startup+6.27778 s]
/proc/loadavg: 1.55 1.09 1.03 9/86 28805
/proc/meminfo: memFree=901240/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=1.75
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 175 0 0 0 25 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567239 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=1.71
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 171 0 0 0 25 0 1 0 178830977 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683214 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 271 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=1.6
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 160 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567202 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=1.6
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 160 0 0 0 25 0 1 0 178830980 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134591747 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 271 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=1.44
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 144 0 0 0 25 0 1 0 178831001 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572276 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 271 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=1.5
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 150 0 0 0 25 0 1 0 178831006 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567281 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 271 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=1.4
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 140 0 0 0 25 0 1 0 178831009 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683223 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 271 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=1.4
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 140 0 0 0 25 0 1 0 178831010 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134629406 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 12.44
Current children cumulated vsize (KiB) 1642032

[startup+12.7602 s]
/proc/loadavg: 2.55 1.31 1.10 9/86 28805
/proc/meminfo: memFree=901176/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=3.43
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 343 0 0 0 25 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567586 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=3.31
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 331 0 0 0 25 0 1 0 178830977 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567521 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 271 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=3.2
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 320 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580357 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=3.2
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 320 0 0 0 25 0 1 0 178830980 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683219 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 271 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=3.04
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 304 0 0 0 25 0 1 0 178831001 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134596599 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 271 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=3.12
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 312 0 0 0 25 0 1 0 178831006 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572907 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 271 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=3.04
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 304 0 0 0 25 0 1 0 178831009 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134603387 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 271 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=3
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 300 0 0 0 25 0 1 0 178831010 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134589814 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 25.38
Current children cumulated vsize (KiB) 1642032

[startup+25.649 s]
/proc/loadavg: 3.38 1.53 1.17 9/86 28805
/proc/meminfo: memFree=901176/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=6.63
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 663 0 0 0 25 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577736 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=6.51
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 651 0 0 0 25 0 1 0 178830977 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134569459 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 271 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=6.4
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 640 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134598982 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=6.4
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 640 0 0 0 25 0 1 0 178830980 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577556 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 272 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=6.33
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 633 0 0 0 25 0 1 0 178831001 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567283 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 271 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=6.32
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 632 0 0 0 25 0 1 0 178831006 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683127 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 272 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=6.28
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 628 0 0 0 25 0 1 0 178831009 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683578 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 271 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=6.2
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 46 0 0 0 620 0 0 0 25 0 1 0 178831010 186826752 271 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572890 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 51.11

################
# More data... #
################

[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=120.5
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12050 0 0 0 25 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577447 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=120.39
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12039 0 0 0 25 0 1 0 178830977 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580277 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 272 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=120.28
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12028 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580320 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=119.78
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 11978 0 0 0 25 0 1 0 178830980 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683263 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 272 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=120.12
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12012 0 0 0 25 0 1 0 178831001 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134649742 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 272 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=119.7
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 11970 0 0 0 25 0 1 0 178831006 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683095 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 272 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=119.68
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 11968 0 0 0 25 0 1 0 178831009 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134601031 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 272 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=119.62
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 11962 0 0 0 25 0 1 0 178831010 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683195 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 960.11
Current children cumulated vsize (KiB) 1642032

[startup+484.483 s]
/proc/loadavg: 8.00 6.62 3.83 9/86 28805
/proc/meminfo: memFree=901176/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=182448 CPUtime=121.31
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12131 0 0 0 25 0 1 0 178830976 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577061 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 45612 272 78 160 0 45450 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=121.19
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12119 0 0 0 25 0 1 0 178830977 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683343 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 272 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=121.08
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12108 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683188 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=120.6
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12060 0 0 0 25 0 1 0 178830980 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134640134 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 272 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=121.01
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12101 0 0 0 25 0 1 0 178831001 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683214 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 272 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=120.5
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12050 0 0 0 25 0 1 0 178831006 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134603834 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 272 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=120.48
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12048 0 0 0 25 0 1 0 178831009 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567399 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 272 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=120.48
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12048 0 0 0 25 0 1 0 178831010 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683578 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 966.69
Current children cumulated vsize (KiB) 1642032

[startup+485.312 s]
/proc/loadavg: 8.00 6.62 3.83 9/86 28805
/proc/meminfo: memFree=901176/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=0 CPUtime=121.47
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194372 62 0 0 0 12147 0 0 0 25 0 1 0 178830976 0 0 18446744073709551615 0 0 0 0 0 0 0 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 0 0 0 0 0 0 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=121.39
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12139 0 0 0 25 0 1 0 178830977 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572771 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 272 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=121.33
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12133 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134675466 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=120.83
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12083 0 0 0 25 0 1 0 178830980 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577081 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 272 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=121.22
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12122 0 0 0 25 0 1 0 178831001 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683424 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 272 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=120.7
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12070 0 0 0 25 0 1 0 178831006 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134604440 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 272 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=120.68
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12068 0 0 0 25 0 1 0 178831009 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577551 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 272 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=120.68
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12068 0 0 0 25 0 1 0 178831010 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567283 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 968.34
Current children cumulated vsize (KiB) 1459584

[startup+485.724 s]
/proc/loadavg: 8.00 6.62 3.83 9/86 28805
/proc/meminfo: memFree=901176/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=0 CPUtime=121.47
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194372 62 0 0 0 12147 0 0 0 25 0 1 0 178830976 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 0 0 0 0 0 0 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=121.55
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12155 0 0 0 25 0 1 0 178830977 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580256 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 272 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=121.48
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12148 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134595465 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=120.94
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12094 0 0 0 25 0 1 0 178830980 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572157 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 272 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=121.32
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12132 0 0 0 25 0 1 0 178831001 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577464 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 272 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=120.8
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12080 0 0 0 25 0 1 0 178831006 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572949 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 272 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=120.78
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12078 0 0 0 25 0 1 0 178831009 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577464 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 272 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=120.78
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12078 0 0 0 25 0 1 0 178831010 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134593377 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 969.16
Current children cumulated vsize (KiB) 1459584

[startup+486.14 s]
/proc/loadavg: 8.00 6.65 3.86 9/86 28805
/proc/meminfo: memFree=901176/2055920 swapFree=4147432/4192956
[pid=28796] ppid=28794 vsize=182448 CPUtime=0.04
/proc/28796/stat : 28796 (kcnfs-smp) S 28794 28796 26081 0 -1 4194304 458 0 0 0 4 0 0 0 22 0 1 0 178830974 186826752 257 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/28796/statm: 45612 257 72 160 0 45450 0
[pid=28798] ppid=28796 vsize=0 CPUtime=121.47
/proc/28798/stat : 28798 (kcnfs-smp) R 28796 28796 26081 0 -1 4194372 62 0 0 0 12147 0 0 0 25 0 1 0 178830976 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 2 0 0 0 17 1 0 0
/proc/28798/statm: 0 0 0 0 0 0 0
[pid=28799] ppid=28796 vsize=182448 CPUtime=121.69
/proc/28799/stat : 28799 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12169 0 0 0 25 0 1 0 178830977 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134647674 0 0 4096 0 0 0 0 17 1 0 0
/proc/28799/statm: 45612 272 78 160 0 45450 0
[pid=28800] ppid=28796 vsize=182448 CPUtime=121.58
/proc/28800/stat : 28800 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12158 0 0 0 25 0 1 0 178830979 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577417 0 0 4096 2 0 0 0 17 1 0 0
/proc/28800/statm: 45612 272 78 160 0 45450 0
[pid=28801] ppid=28796 vsize=182448 CPUtime=121.05
/proc/28801/stat : 28801 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12105 0 0 0 25 0 1 0 178830980 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134601243 0 0 4096 0 0 0 0 17 0 0 0
/proc/28801/statm: 45612 272 78 160 0 45450 0
[pid=28802] ppid=28796 vsize=182448 CPUtime=121.5
/proc/28802/stat : 28802 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12150 0 0 0 25 0 1 0 178831001 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567239 0 0 4096 2 0 0 0 17 1 0 0
/proc/28802/statm: 45612 272 78 160 0 45450 0
[pid=28803] ppid=28796 vsize=182448 CPUtime=120.9
/proc/28803/stat : 28803 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12090 0 0 0 25 0 1 0 178831006 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572306 0 0 4096 0 0 0 0 17 0 0 0
/proc/28803/statm: 45612 272 78 160 0 45450 0
[pid=28804] ppid=28796 vsize=182448 CPUtime=120.88
/proc/28804/stat : 28804 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12088 0 0 0 25 0 1 0 178831009 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134628773 0 0 4096 2 0 0 0 17 0 0 0
/proc/28804/statm: 45612 272 78 160 0 45450 0
[pid=28805] ppid=28796 vsize=182448 CPUtime=120.88
/proc/28805/stat : 28805 (kcnfs-smp) R 28796 28796 26081 0 -1 4194368 47 0 0 0 12088 0 0 0 25 0 1 0 178831010 186826752 272 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580056 0 0 4096 0 0 0 0 17 0 0 0
/proc/28805/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 969.99
Current children cumulated vsize (KiB) 1459584

Child status: 10

!!! problem with CPU time !!!
wait4(...,&childrusage) returns 28796 and gives childrusage.ru_utime.tv_sec=121 childrusage.ru_utime.tv_usec=516526 childrusage.ru_stime.tv_sec=0 childrusage.ru_stime.tv_usec=11998
CPU time returned by wait4() is 121.529
while last known CPU time is 969.99

Solver probably didn't/couldn't wait for its children
Using last known CPU time as value...

Real time (s): 486.162
CPU time (s): 969.99
CPU user time (s): 969.99
CPU system time (s): 0
CPU usage (%): 199.52
Max. virtual memory (cumulated for all children) (KiB): 1642032

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 121.517
system time used= 0.011998
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 547
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 15
involuntary context switches= 1425

runsolver used 0.501923 second user time and 1.23781 second system time

The end

Launcher Data (download as text)

Begin job on node29 on Mon Feb 19 02:04:27 UTC 2007

IDJOB= 298118
IDBENCH= 19978
IDSOLVER= 112
FILE ID= node29/298118-1171850667

PBS_JOBID= 3839662

Free space on /tmp= 66561 MiB

SOLVER NAME= KCNFS SMP
BENCH NAME= HOME/pub/bench/SAT07/random/OnTreshold/3SAT/v500/unif-k3-r4.26-v500-c2130-S1490740667-05.SAT.shuffled.cnf
COMMAND LINE= /tmp/evaluation/298118-1171850667/kcnfs-smp /tmp/evaluation/298118-1171850667/instance-298118-1171850667.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node29/watcher-298118-1171850667 -o ROOT/results/node29/solver-298118-1171850667 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/298118-1171850667/kcnfs-smp /tmp/evaluation/298118-1171850667/instance-298118-1171850667.cnf            

META MD5SUM SOLVER= 23f87ccf1c743ff5b6146b444e216cd3
MD5SUM BENCH=  f01193a3f455045d150f231e0d56c100

RANDOM SEED= 268873253

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node29.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.209
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5931.00
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.209
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:        905056 kB
Buffers:         41512 kB
Cached:        1006872 kB
SwapCached:      12280 kB
Active:         820572 kB
Inactive:       250836 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:        905056 kB
SwapTotal:     4192956 kB
SwapFree:      4147432 kB
Dirty:            3200 kB
Writeback:           0 kB
Mapped:          29984 kB
Slab:            64844 kB
Committed_AS:   657536 kB
PageTables:       1864 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66561 MiB

End job on node29 on Mon Feb 19 02:12:34 UTC 2007