Trace number 299553

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
KCNFS SMPSAT 1168.09 585.439

General information on the benchmark

Namerandom/OnTreshold/3SAT/v650/
unif-k3-r4.261-v650-c2769-S1172355929-14.SAT.shuffled.cnf
MD5SUM53b587a5b98db42d3f124ef838f3e04f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.026995
Satisfiable
(Un)Satisfiability was proved
Number of variables650
Number of clauses2769
Sum of the clauses size8307
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32769
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	
0.00/0.00	c Computed by /tmp/evaluation/299553-1171873864/kcnfs-smp   V. 1.2
0.00/0.00	c reading file ... done
0.00/0.00	c Input file : '/tmp/evaluation/299553-1171873864/instance-299553-1171873864.cnf'
0.00/0.00	c [#vars: 650, #clauses: 2769 -> 2860 clauses (pre-processing), tautology: 0, 3-CNF]
0.00/0.01	c unit clauses propagate before processing (#clause: 2860)
0.00/0.01	c estimated distribution: 8 tasks
0.00/0.04	c <15024> [PID]: 15026 15027 15028 15029 15030 15031 15032
0.06/0.47	c <15024> [PID]: 15033
0.06/0.47	c real distribution: 8 tasks
1168.09/585.43	c <15024> [15026: SAT]
1168.09/585.43	c <15024> [END]: 15027 15028 15029 15030 15031 15032 15033
1168.09/585.43	s SATISFIABLE
1168.09/585.43	v       -1       -2       -3        4        5       -6        7
1168.09/585.43	v        8       -9       10       11       12       13      -14
1168.09/585.43	v       15       16       17       18       19       20      -21
1168.09/585.43	v      -22       23      -24       25       26       27       28
1168.09/585.43	v      -29      -30       31       32       33      -34       35
1168.09/585.43	v      -36      -37       38       39       40      -41       42
1168.09/585.43	v       43      -44       45      -46      -47       48       49
1168.09/585.43	v       50      -51      -52      -53       54       55       56
1168.09/585.43	v      -57      -58       59       60       61       62       63
1168.09/585.43	v      -64       65      -66      -67       68       69      -70
1168.09/585.43	v      -71       72       73       74       75       76       77
1168.09/585.43	v      -78      -79       80       81      -82       83      -84
1168.09/585.43	v       85       86      -87       88      -89      -90       91
1168.09/585.43	v       92      -93      -94       95      -96      -97      -98
1168.09/585.43	v       99      100      101      103     -104     -105      106
1168.09/585.43	v     -107      108      109      110     -111      112      113
1168.09/585.43	v      114      115      116      117      118      119      120
1168.09/585.43	v      121     -122     -123     -124     -125     -126     -127
1168.09/585.43	v     -128     -129     -130      131     -132      133     -134
1168.09/585.43	v     -135     -136     -137      138     -139     -140      141
1168.09/585.43	v      142     -143     -144     -145      146      147      148
1168.09/585.43	v     -149     -150      151     -152      153     -154     -155
1168.09/585.43	v      156     -157      158      159     -160      161     -162
1168.09/585.43	v      163     -164     -165     -166     -167      168      169
1168.09/585.43	v      170      171     -172      173      174      175      176
1168.09/585.43	v      177      178     -179      180      181     -182     -183
1168.09/585.43	v      184     -185     -186     -187      188      189      190
1168.09/585.43	v     -191     -192      193      194      195      196      197
1168.09/585.43	v     -198     -199     -200      201      202      203     -204
1168.09/585.43	v     -205     -206      207     -208      209     -210     -211
1168.09/585.43	v     -212     -213     -214      215     -216     -217     -218
1168.09/585.43	v      219     -220      221     -222      223      224      225
1168.09/585.43	v      226      227      228      229      230      231      232
1168.09/585.43	v      233      234     -235     -236      237      238     -239
1168.09/585.43	v     -240     -241      242      243      244     -245     -246
1168.09/585.43	v     -247     -248     -249     -250      251      252      253
1168.09/585.43	v     -254      255      256     -257      258     -259      260
1168.09/585.43	v     -261      262      263     -264     -265     -266      267
1168.09/585.43	v      268     -269      270     -271      272      273     -274
1168.09/585.43	v     -275     -276     -277      278      279      280     -281
1168.09/585.43	v      282      283      284      285     -286      287      288
1168.09/585.43	v      289      290     -291     -292     -293      294     -295
1168.09/585.43	v     -296     -297      298     -299     -300      301      302
1168.09/585.43	v     -303     -304      305      306      307     -308      309
1168.09/585.43	v      310      311     -312      313      314      315      316
1168.09/585.43	v      317      318      319     -320     -321      322     -323
1168.09/585.43	v     -324      325     -326      327     -328      329      330
1168.09/585.43	v     -331      332      333      334     -335      336      337
1168.09/585.43	v     -338      339     -340      341     -342      343     -344
1168.09/585.43	v      345     -346     -347     -348      349      350     -351
1168.09/585.43	v      352     -353      354     -355     -356      357     -358
1168.09/585.43	v     -359      360     -361     -362      363     -364     -365
1168.09/585.43	v      366     -367      368     -369      370      371      372
1168.09/585.43	v      373     -374     -375     -376     -377      378     -379
1168.09/585.43	v      380      381     -382      383      384     -385     -386
1168.09/585.43	v      387     -388      389      390     -391      392     -393
1168.09/585.43	v      394      395      396     -397      398     -399     -400
1168.09/585.43	v      401      402     -403      404     -405      406      407
1168.09/585.43	v     -408      409      410      411     -412      413      414
1168.09/585.43	v      415     -416      417     -418     -419      420     -421
1168.09/585.43	v      422      423     -424     -425     -426      427      428
1168.09/585.43	v     -429      430      431     -432     -433     -434      435
1168.09/585.43	v      436      437      438     -439      440      441      442
1168.09/585.43	v      443     -444      445     -446      447     -448     -449
1168.09/585.43	v     -450     -451      452      453     -454      455      456
1168.09/585.43	v     -457     -458     -459     -460      461     -462      463
1168.09/585.43	v     -464      465     -466     -467     -468     -469     -470
1168.09/585.43	v      471      472      473     -474     -475     -476     -477
1168.09/585.43	v      478     -479     -480     -481     -482     -483     -484
1168.09/585.43	v      485      486      487      488     -489     -490     -491
1168.09/585.43	v     -492     -493     -494      495      496      497      498
1168.09/585.43	v     -499     -500     -501     -502      503      504      505
1168.09/585.43	v      506     -507      508     -509     -510     -511      512
1168.09/585.43	v      513     -514     -515      516     -517     -518     -519
1168.09/585.43	v     -520      521      522     -523      524     -525     -526
1168.09/585.43	v      527     -528      529      530      531     -532     -533
1168.09/585.43	v      534      535     -536      537     -538      539     -540
1168.09/585.43	v      541      542     -543      544      545      546     -547
1168.09/585.43	v     -548      549      550     -551     -552     -553     -554
1168.09/585.43	v      555      556     -557     -558     -559     -560     -561
1168.09/585.43	v      562     -563     -564      565     -566     -567     -568
1168.09/585.43	v     -569      570     -571      572     -573     -574      575
1168.09/585.43	v      576      577      578     -579     -580     -581      582
1168.09/585.43	v      583     -584     -585      586      587     -588     -589
1168.09/585.43	v      590      591     -592     -593     -594     -595      596
1168.09/585.43	v      597     -598     -599     -600      601     -602     -603
1168.09/585.43	v      604     -605     -606     -607      608      609     -610
1168.09/585.43	v     -611     -612      613     -614      615     -616      617
1168.09/585.43	v      618      619      620     -621     -622     -623      624
1168.09/585.43	v     -625      626      627      628     -629      630      631
1168.09/585.43	v      632      633     -634     -635      636      637      638
1168.09/585.43	v     -639     -640     -641     -642     -643      644      645
1168.09/585.43	v      646     -647     -648     -649      650        0
1168.09/585.43	c CPU time            : 146.390 seconds
1168.09/585.43	c Elapsed Time        : 585.429 seconds
1168.09/585.43	c Size of search tree : 193232 nodes
1168.09/585.43	c average speed       : 1319.981 nodes/sec.
1168.09/585.43

Verifier Data (download as text)

OK

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node34/watcher-299553-1171873864 -o ROOT/results/node34/solver-299553-1171873864 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/299553-1171873864/kcnfs-smp /tmp/evaluation/299553-1171873864/instance-299553-1171873864.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.00 1.10 1.88 3/64 15024
/proc/meminfo: memFree=729072/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=180916 CPUtime=0
/proc/15024/stat : 15024 (kcnfs-smp) R 15022 15024 6093 0 -1 4194304 130 0 0 0 0 0 0 0 18 0 1 0 181150184 185257984 115 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134820679 0 0 4096 0 0 0 0 17 1 0 0
/proc/15024/statm: 45229 115 48 160 0 45067 0

[startup+0.107435 s]
/proc/loadavg: 1.00 1.10 1.88 3/64 15024
/proc/meminfo: memFree=729072/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.04
/proc/15024/stat : 15024 (kcnfs-smp) R 15022 15024 6093 0 -1 4194304 425 0 0 0 4 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134648867 0 0 4096 0 0 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 182580

[startup+0.515468 s]
/proc/loadavg: 1.00 1.10 1.88 3/64 15024
/proc/meminfo: memFree=729072/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 182580

[startup+1.33753 s]
/proc/loadavg: 1.00 1.10 1.88 9/73 15033
/proc/meminfo: memFree=725928/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=0.53
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 53 0 0 0 24 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580273 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=0.41
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 41 0 0 0 24 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134633691 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 314 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=0.36
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 36 0 0 0 23 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577752 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=0.35
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 35 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579725 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 314 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=0.24
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 48 0 0 0 24 0 0 0 25 0 1 0 181150223 186961920 313 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134629699 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 313 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=0.22
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 35 0 0 0 22 0 0 0 25 0 1 0 181150227 186961920 301 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134647674 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 301 72 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=0.2
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 35 0 0 0 20 0 0 0 25 0 1 0 181150231 186961920 301 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134649525 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 301 72 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=0.2
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 35 0 0 0 20 0 0 0 25 0 1 0 181150231 186961920 301 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134649330 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 301 72 160 0 45483 0
Current children cumulated CPU time (s) 2.58
Current children cumulated vsize (KiB) 1643220

[startup+2.98967 s]
/proc/loadavg: 1.56 1.21 1.92 9/73 15033
/proc/meminfo: memFree=725672/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=0.93
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 93 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572099 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=0.82
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 82 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567212 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 314 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=0.8
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 80 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567303 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=0.8
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 80 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572080 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 314 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=0.64
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 64 0 0 0 25 0 1 0 181150223 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134597328 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 314 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=0.62
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 62 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134569337 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=0.6
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 60 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577459 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 314 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=0.6
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 60 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580041 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 5.88
Current children cumulated vsize (KiB) 1643220

[startup+6.26893 s]
/proc/loadavg: 1.56 1.21 1.92 9/73 15033
/proc/meminfo: memFree=725416/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=1.73
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 173 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567303 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=1.7
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 170 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577451 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 314 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=1.6
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 160 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580071 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=1.6
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 160 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683188 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 314 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=1.51
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 151 0 0 0 25 0 1 0 181150223 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572621 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 314 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=1.42
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 142 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579685 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=1.4
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 140 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572267 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 314 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=1.4
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 140 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572793 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 12.43
Current children cumulated vsize (KiB) 1643220

[startup+12.7705 s]
/proc/loadavg: 2.55 1.43 1.98 9/73 15033
/proc/meminfo: memFree=725352/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=3.42
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 342 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134631744 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=3.31
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 331 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134581094 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 314 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=3.2
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 320 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567202 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=3.2
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 320 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580258 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 314 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=3.14
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 314 0 0 0 25 0 1 0 181150223 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134595431 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 314 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=3.02
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 302 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577447 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=3.04
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 304 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580190 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 314 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=3
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 300 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572890 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 25.4
Current children cumulated vsize (KiB) 1643220

[startup+25.6335 s]
/proc/loadavg: 3.39 1.65 2.05 9/73 15033
/proc/meminfo: memFree=725352/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=6.63
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 663 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134675782 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=6.51
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 651 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683214 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 314 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=6.4
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 640 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134630078 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=6.4
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 640 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577666 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 314 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=6.34
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 634 0 0 0 25 0 1 0 181150223 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579714 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 314 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=6.27
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 627 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134604503 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=6.24
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 624 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683223 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 314 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=6.2
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 620 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134640255 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 51.06

################
# More data... #
################

[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=146.01
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14601 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134593401 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=145.89
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14589 0 0 0 25 0 1 0 181150188 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572727 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 315 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=145.78
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14578 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580256 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=145.17
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14517 0 0 0 25 0 1 0 181150191 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572839 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 315 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=145.11
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14511 0 0 0 25 0 1 0 181150223 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683195 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 315 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=145.59
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14559 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572608 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=145.08
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14508 0 0 0 25 0 1 0 181150231 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579137 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 315 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=145.06
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14506 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572101 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 1163.76
Current children cumulated vsize (KiB) 1643220

[startup+584.068 s]
/proc/loadavg: 8.00 7.02 4.73 9/73 15033
/proc/meminfo: memFree=725160/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=182580 CPUtime=146.21
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14621 0 0 0 25 0 1 0 181150188 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134593419 0 0 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 45645 314 78 160 0 45483 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=146.09
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14609 0 0 0 25 0 1 0 181150188 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134605234 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 315 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=145.98
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14598 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577752 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=145.38
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14538 0 0 0 25 0 1 0 181150191 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134578994 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 315 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=145.31
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14531 0 0 0 25 0 1 0 181150223 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577544 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 315 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=145.82
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14582 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577031 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=145.28
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14528 0 0 0 25 0 1 0 181150231 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134604503 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 315 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=145.27
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14527 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567426 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 1165.41
Current children cumulated vsize (KiB) 1643220

[startup+584.896 s]
/proc/loadavg: 8.00 7.02 4.73 9/73 15033
/proc/meminfo: memFree=725224/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=0 CPUtime=146.3
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194372 68 0 0 0 14630 0 0 0 25 0 1 0 181150188 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 0 0 0 0 0 0 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=146.29
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14629 0 0 0 25 0 1 0 181150188 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134591570 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 315 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=146.24
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14624 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580258 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=145.61
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14561 0 0 0 25 0 1 0 181150191 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134594255 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 315 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=145.51
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14551 0 0 0 25 0 1 0 181150223 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134674522 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 315 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=146.09
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14609 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579525 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=145.48
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14548 0 0 0 25 0 1 0 181150231 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134589854 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 315 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=145.47
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14547 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134603272 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 1167.06
Current children cumulated vsize (KiB) 1460640

[startup+585.306 s]
/proc/loadavg: 8.00 7.02 4.73 9/73 15033
/proc/meminfo: memFree=725224/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=0 CPUtime=146.3
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194372 68 0 0 0 14630 0 0 0 25 0 1 0 181150188 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 0 0 0 0 0 0 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=146.46
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14646 0 0 0 25 0 1 0 181150188 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134604653 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 315 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=146.38
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14638 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134603834 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=145.72
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14572 0 0 0 25 0 1 0 181150191 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683132 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 315 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=145.61
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14561 0 0 0 25 0 1 0 181150223 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134580154 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 315 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=146.19
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14619 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134579327 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=145.58
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14558 0 0 0 25 0 1 0 181150231 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577103 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 315 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=145.57
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14557 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572850 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 1167.88
Current children cumulated vsize (KiB) 1460640

[startup+585.408 s]
/proc/loadavg: 8.00 7.02 4.73 9/73 15033
/proc/meminfo: memFree=725224/2055920 swapFree=4183096/4192956
[pid=15024] ppid=15022 vsize=182580 CPUtime=0.07
/proc/15024/stat : 15024 (kcnfs-smp) S 15022 15024 6093 0 -1 4194304 513 0 0 0 7 0 0 0 22 0 1 0 181150184 186961920 300 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/15024/statm: 45645 300 72 160 0 45483 0
[pid=15026] ppid=15024 vsize=0 CPUtime=146.3
/proc/15026/stat : 15026 (kcnfs-smp) R 15024 15024 6093 0 -1 4194372 68 0 0 0 14630 0 0 0 25 0 1 0 181150188 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 2 0 0 0 17 1 0 0
/proc/15026/statm: 0 0 0 0 0 0 0
[pid=15027] ppid=15024 vsize=182580 CPUtime=146.49
/proc/15027/stat : 15027 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14649 0 0 0 25 0 1 0 181150188 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577061 0 0 4096 0 0 0 0 17 1 0 0
/proc/15027/statm: 45645 315 78 160 0 45483 0
[pid=15028] ppid=15024 vsize=182580 CPUtime=146.38
/proc/15028/stat : 15028 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14638 0 0 0 25 0 1 0 181150191 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134603834 0 0 4096 2 0 0 0 17 1 0 0
/proc/15028/statm: 45645 314 78 160 0 45483 0
[pid=15029] ppid=15024 vsize=182580 CPUtime=145.77
/proc/15029/stat : 15029 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14577 0 0 0 25 0 1 0 181150191 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577666 0 0 4096 0 0 0 0 17 0 0 0
/proc/15029/statm: 45645 315 78 160 0 45483 0
[pid=15030] ppid=15024 vsize=182580 CPUtime=145.66
/proc/15030/stat : 15030 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14566 0 0 0 25 0 1 0 181150223 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134567341 0 0 4096 2 0 0 0 17 0 0 0
/proc/15030/statm: 45645 315 78 160 0 45483 0
[pid=15031] ppid=15024 vsize=182580 CPUtime=146.27
/proc/15031/stat : 15031 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14627 0 0 0 25 0 1 0 181150227 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134683149 0 0 4096 0 0 0 0 17 1 0 0
/proc/15031/statm: 45645 314 78 160 0 45483 0
[pid=15032] ppid=15024 vsize=182580 CPUtime=145.58
/proc/15032/stat : 15032 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 50 0 0 0 14558 0 0 0 25 0 1 0 181150231 186961920 315 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134577103 0 0 4096 2 0 0 0 17 0 0 0
/proc/15032/statm: 45645 315 78 160 0 45483 0
[pid=15033] ppid=15024 vsize=182580 CPUtime=145.57
/proc/15033/stat : 15033 (kcnfs-smp) R 15024 15024 6093 0 -1 4194368 49 0 0 0 14557 0 0 0 25 0 1 0 181150231 186961920 314 18446744073709551615 134512640 135172044 4294956640 18446744073709551615 134572850 0 0 4096 0 0 0 0 17 0 0 0
/proc/15033/statm: 45645 314 78 160 0 45483 0
Current children cumulated CPU time (s) 1168.09
Current children cumulated vsize (KiB) 1460640

Child status: 10

!!! problem with CPU time !!!
wait4(...,&childrusage) returns 15024 and gives childrusage.ru_utime.tv_sec=146 childrusage.ru_utime.tv_usec=375747 childrusage.ru_stime.tv_sec=0 childrusage.ru_stime.tv_usec=4999
CPU time returned by wait4() is 146.381
while last known CPU time is 1168.09

Solver probably didn't/couldn't wait for its children
Using last known CPU time as value...

Real time (s): 585.439
CPU time (s): 1168.09
CPU user time (s): 1168.09
CPU system time (s): 0
CPU usage (%): 199.524
Max. virtual memory (cumulated for all children) (KiB): 1643220

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 146.376
system time used= 0.004999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 610
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 25
involuntary context switches= 1576

runsolver used 0.599908 second user time and 1.48477 second system time

The end

Launcher Data (download as text)

Begin job on node34 on Mon Feb 19 08:31:04 UTC 2007

IDJOB= 299553
IDBENCH= 20013
IDSOLVER= 112
FILE ID= node34/299553-1171873864

PBS_JOBID= 3839651

Free space on /tmp= 66561 MiB

SOLVER NAME= KCNFS SMP
BENCH NAME= HOME/pub/bench/SAT07/random/OnTreshold/3SAT/v650/unif-k3-r4.261-v650-c2769-S1172355929-14.SAT.shuffled.cnf
COMMAND LINE= /tmp/evaluation/299553-1171873864/kcnfs-smp /tmp/evaluation/299553-1171873864/instance-299553-1171873864.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node34/watcher-299553-1171873864 -o ROOT/results/node34/solver-299553-1171873864 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/299553-1171873864/kcnfs-smp /tmp/evaluation/299553-1171873864/instance-299553-1171873864.cnf            

META MD5SUM SOLVER= 23f87ccf1c743ff5b6146b444e216cd3
MD5SUM BENCH=  53b587a5b98db42d3f124ef838f3e04f

RANDOM SEED= 63394383

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node34.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.234
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.234
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:        729544 kB
Buffers:         53616 kB
Cached:        1148696 kB
SwapCached:       4688 kB
Active:         786452 kB
Inactive:       426908 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:        729544 kB
SwapTotal:     4192956 kB
SwapFree:      4183096 kB
Dirty:            3212 kB
Writeback:           0 kB
Mapped:          16024 kB
Slab:            99080 kB
Committed_AS:   505768 kB
PageTables:       1400 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66561 MiB

End job on node34 on Mon Feb 19 08:40:49 UTC 2007