Trace number 298282

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
KCNFS SMPSAT 386.7 193.893

General information on the benchmark

Namerandom/OnTreshold/3SAT/v500/
unif-k3-r4.26-v500-c2130-S1965764400-19.SAT.shuffled.cnf
MD5SUMa8fcc21a1c405b91adf9e5bdbb6b4ddd
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.021995
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	
0.00/0.00	c Computed by /tmp/evaluation/298282-1171853215/kcnfs-smp   V. 1.2
0.00/0.00	c reading file ... done
0.00/0.00	c Input file : '/tmp/evaluation/298282-1171853215/instance-298282-1171853215.cnf'
0.00/0.00	c [#vars: 500, #clauses: 2130 -> 2199 clauses (pre-processing), tautology: 0, 3-CNF]
0.00/0.01	c unit clauses propagate before processing (#clause: 2199)
0.00/0.01	c estimated distribution: 8 tasks
0.00/0.02	c <16250> [PID]: 16252 16253 16254 16255 16256 16257 16258
0.03/0.60	c <16250> [PID]: 16259
0.03/0.60	c real distribution: 8 tasks
386.70/193.87	c <16250> [16259: SAT]
386.70/193.87	c <16250> [END]: 16252 16253 16254 16255 16256 16257 16258
386.70/193.87	s SATISFIABLE
386.70/193.87	v       -1       -2       -3       -4        5       -6       -7
386.70/193.87	v       -8        9       10       11       12      -13       14
386.70/193.87	v       15      -16       17       18      -19      -20      -21
386.70/193.87	v      -22      -23      -24      -25      -26      -27       28
386.70/193.87	v      -29       30      -31      -32       33       34       35
386.70/193.87	v       36       37       38       39       40       41       42
386.70/193.87	v      -43      -44      -45       46      -47       48       49
386.70/193.87	v       50       51       52       53       54       55       56
386.70/193.87	v       57       58       59      -60      -61      -62      -63
386.70/193.87	v       64      -65      -66      -67       68       69       70
386.70/193.87	v       71      -72      -73       74       75       76      -77
386.70/193.87	v       78       79       80       81      -82       83       84
386.70/193.87	v       85       86      -87      -88      -89      -90       91
386.70/193.87	v       92       93       94       95      -96      -97      -98
386.70/193.87	v       99     -100     -101     -102     -103     -104     -105
386.70/193.87	v     -106     -107     -108     -109     -110     -111     -112
386.70/193.87	v      113     -114      115     -116      117      118      119
386.70/193.87	v     -120     -121     -122     -123      124     -125     -126
386.70/193.87	v      127     -128     -129      130     -131      132     -133
386.70/193.87	v      134     -135      136      137     -138      139     -140
386.70/193.87	v     -141     -142      143      144     -145     -146     -147
386.70/193.87	v      148     -149     -150      151     -152      153     -154
386.70/193.87	v      155      156      157     -158      159      160      161
386.70/193.87	v     -162      163      164     -165      166     -167      168
386.70/193.87	v      169     -170      171      172     -173      174      175
386.70/193.87	v      176     -177      178     -179      180      181      182
386.70/193.87	v     -183     -184     -185     -186      187      188     -189
386.70/193.87	v      190      191     -192      193      194      195      196
386.70/193.87	v      197     -198     -199      200      201     -202     -203
386.70/193.87	v     -204     -205     -206     -207     -208      209     -210
386.70/193.87	v      211     -212      213      214      216      217     -218
386.70/193.87	v     -219      220      221      222     -223      224     -225
386.70/193.87	v     -226     -227      228      229     -230     -231      232
386.70/193.87	v      233     -234      235      236      237     -238     -239
386.70/193.87	v     -240     -241     -242     -243     -244     -245     -246
386.70/193.87	v     -247      248     -249      250      251      252      253
386.70/193.87	v     -254      255     -256     -257     -258      259     -260
386.70/193.87	v      261     -262     -263      264      265     -266     -267
386.70/193.87	v      268      269     -270     -271     -272     -273      274
386.70/193.87	v      275      276     -277     -278     -279     -280      281
386.70/193.87	v     -282      283      284      285     -286     -287      288
386.70/193.87	v     -289      290     -291      292      293      294      295
386.70/193.87	v      296      297      298     -299      300     -301      302
386.70/193.87	v      303      304      305     -306     -307      308      309
386.70/193.87	v     -310      311     -312      313      314     -315     -316
386.70/193.87	v     -317      318      319     -320     -321      322      323
386.70/193.88	v     -324     -325      326      327      328     -329     -330
386.70/193.88	v     -331      332      333      334      335      336     -337
386.70/193.88	v      338     -339     -340      341     -342      343      344
386.70/193.88	v      345     -346     -347      348     -349      350     -351
386.70/193.88	v     -352      353      354      355      356     -357     -358
386.70/193.88	v      359      360      361     -362     -363      364     -365
386.70/193.88	v     -366      367     -368      369     -370     -371      372
386.70/193.88	v      373      374      375     -376      377     -378      379
386.70/193.88	v      380      381     -382      383     -384      385     -386
386.70/193.88	v      387     -388     -389      390      391      392      393
386.70/193.88	v      394     -395      396     -397     -398     -399      400
386.70/193.88	v      401     -402     -403      404      405      406      407
386.70/193.88	v      408      409     -410      411      412     -413      414
386.70/193.88	v     -415     -416     -417     -418      419      420      421
386.70/193.88	v      422     -423     -424     -425      426     -427     -428
386.70/193.88	v      429     -430     -431      432     -433     -434     -435
386.70/193.88	v     -436      437     -438     -439      440     -441     -442
386.70/193.88	v      443      444      445      446      447      448      449
386.70/193.88	v     -450      451     -452      453      454     -455      456
386.70/193.88	v     -457     -458      459     -460      461     -462      463
386.70/193.88	v      464      465      466      467     -468      469      470
386.70/193.88	v      471      472      473     -474     -475      476     -477
386.70/193.88	v      478      479      480      481      482     -483      484
386.70/193.88	v      485     -486     -487     -488      489     -490     -491
386.70/193.88	v      492      493     -494     -495     -496      497      498
386.70/193.88	v      499     -500        0
386.70/193.88	c CPU time            : 47.890 seconds
386.70/193.88	c Elapsed Time        : 193.874 seconds
386.70/193.88	c Size of search tree : 79271 nodes
386.70/193.88	c average speed       : 1655.272 nodes/sec.
386.70/193.88

Verifier Data (download as text)

OK

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node6/watcher-298282-1171853215 -o ROOT/results/node6/solver-298282-1171853215 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/298282-1171853215/kcnfs-smp /tmp/evaluation/298282-1171853215/instance-298282-1171853215.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 16250
/proc/meminfo: memFree=1334656/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=181980 CPUtime=0
/proc/16250/stat : 16250 (kcnfs-smp) R 16248 16250 16187 0 -1 4194304 156 0 0 0 0 0 0 0 18 0 1 0 205276264 186347520 140 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134726021 0 0 4096 0 0 0 0 17 1 0 0
/proc/16250/statm: 45495 140 51 160 0 45333 0

[startup+0.106583 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 16250
/proc/meminfo: memFree=1334656/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.02
/proc/16250/stat : 16250 (kcnfs-smp) R 16248 16250 16187 0 -1 4194304 376 0 0 0 2 0 0 0 19 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134582209 0 0 4096 0 0 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 182448

[startup+0.514663 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 16250
/proc/meminfo: memFree=1334656/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) R 16248 16250 16187 0 -1 4194304 429 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134647726 0 0 4096 0 0 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 182448

[startup+1.33583 s]
/proc/loadavg: 0.92 0.98 0.99 9/86 16259
/proc/meminfo: memFree=1331640/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=0.5
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 50 0 0 0 24 0 1 0 205276266 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134593937 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 270 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=0.4
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 40 0 0 0 24 0 1 0 205276266 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134595433 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 270 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=0.44
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 44 0 0 0 25 0 1 0 205276267 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134576924 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 270 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=0.32
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 32 0 0 0 25 0 1 0 205276272 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134597629 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 270 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=0.21
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 21 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134587673 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=0.3
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 30 0 0 0 25 0 1 0 205276276 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134595574 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 270 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=0.24
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 24 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580088 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=0.14
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 14 0 0 0 25 0 1 0 205276322 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580075 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 270 78 160 0 45450 0
Current children cumulated CPU time (s) 2.58
Current children cumulated vsize (KiB) 1642032

[startup+2.98415 s]
/proc/loadavg: 1.49 1.10 1.03 9/86 16259
/proc/meminfo: memFree=1331448/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=0.9
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 90 0 0 0 25 0 1 0 205276266 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134566844 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 270 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=0.8
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 80 0 0 0 25 0 1 0 205276266 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572670 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 270 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=0.84
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 84 0 0 0 25 0 1 0 205276267 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134567429 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 270 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=0.72
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 72 0 0 0 25 0 1 0 205276272 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683095 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 270 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=0.66
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 66 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577079 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=0.7
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 70 0 0 0 25 0 1 0 205276276 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134604658 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 270 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=0.64
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 64 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134629296 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=0.58
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 58 0 0 0 25 0 1 0 205276322 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580041 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 270 78 160 0 45450 0
Current children cumulated CPU time (s) 5.87
Current children cumulated vsize (KiB) 1642032

[startup+6.2798 s]
/proc/loadavg: 1.49 1.10 1.03 9/86 16259
/proc/meminfo: memFree=1331192/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=1.75
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 175 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134636447 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=1.6
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 160 0 0 0 25 0 1 0 205276266 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134603490 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 270 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=1.64
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 164 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134579652 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=1.56
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 156 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134648674 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=1.5
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 150 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572157 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=1.5
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 150 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683062 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=1.44
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 144 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572246 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=1.42
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 142 0 0 0 25 0 1 0 205276322 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134595433 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 12.44
Current children cumulated vsize (KiB) 1642032

[startup+12.7721 s]
/proc/loadavg: 2.49 1.33 1.11 9/86 16259
/proc/meminfo: memFree=1331192/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=3.4
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 340 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134628050 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=3.24
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 46 0 0 0 324 0 0 0 25 0 1 0 205276266 186826752 270 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134651122 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 270 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=3.24
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 324 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134665427 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=3.22
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 322 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134593937 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=3.1
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 310 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683038 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=3.1
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 310 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134674464 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=3.04
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 304 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683223 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=3.02
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 302 0 0 0 25 0 1 0 205276322 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572246 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 25.39
Current children cumulated vsize (KiB) 1642032

[startup+25.6526 s]
/proc/loadavg: 3.33 1.55 1.18 9/86 16259
/proc/meminfo: memFree=1331128/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=6.59
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 659 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134579990 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=6.5
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 650 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577625 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 271 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=6.46
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 646 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683214 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=6.42
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 642 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134628340 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=6.3
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 630 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134685075 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=6.33
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 633 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572655 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=6.24
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 624 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577430 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=6.22
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 622 0 0 0 25 0 1 0 205276322 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683223 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 51.09

################
# More data... #
################

[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=47.11
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4711 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134606827 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=46.99
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4699 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580198 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 271 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=47.03
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4703 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580220 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=46.79
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4679 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577693 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=46.89
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4689 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577567 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=46.69
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4669 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683263 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=46.64
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4664 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683223 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=46.61
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4661 0 0 0 25 0 1 0 205276322 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134601252 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 374.78
Current children cumulated vsize (KiB) 1642032

[startup+191.179 s]
/proc/loadavg: 7.71 4.29 2.29 9/86 16259
/proc/meminfo: memFree=1331064/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=47.99
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4799 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134640114 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=47.8
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4780 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577545 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 271 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=47.83
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4783 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134649690 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=47.61
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4761 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683221 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=47.69
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4769 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580154 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=47.55
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4755 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134674519 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=47.44
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4744 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134595418 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=47.41
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4741 0 0 0 25 0 1 0 205276322 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134567458 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 381.35
Current children cumulated vsize (KiB) 1642032

[startup+192.835 s]
/proc/loadavg: 7.73 4.35 2.32 9/86 16259
/proc/meminfo: memFree=1331064/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=48.39
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4839 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580220 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=48.25
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4825 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580290 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 271 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=48.23
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4823 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134569510 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=48.01
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4801 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580041 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=48.09
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4809 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134567570 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=47.99
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4799 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683223 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=47.84
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4784 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134577566 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=182448 CPUtime=47.81
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4781 0 0 0 25 0 1 0 205276322 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134567577 0 0 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 384.64
Current children cumulated vsize (KiB) 1642032

[startup+193.651 s]
/proc/loadavg: 7.73 4.35 2.32 9/86 16259
/proc/meminfo: memFree=1331064/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=48.59
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4859 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572276 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=48.47
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4847 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683216 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 271 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=48.43
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4843 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572774 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=48.29
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4829 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134569468 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=48.29
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4829 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134579685 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=48.19
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4819 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134567415 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=48.14
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4814 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580317 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=0 CPUtime=47.85
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194372 62 0 0 0 4785 0 0 0 25 0 1 0 205276322 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 386.28
Current children cumulated vsize (KiB) 1459584

[startup+193.86 s]
/proc/loadavg: 7.73 4.35 2.32 9/86 16259
/proc/meminfo: memFree=1331192/2055924 swapFree=4192812/4192956
[pid=16250] ppid=16248 vsize=182448 CPUtime=0.03
/proc/16250/stat : 16250 (kcnfs-smp) S 16248 16250 16187 0 -1 4194304 458 0 0 0 3 0 0 0 24 0 1 0 205276264 186826752 256 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/16250/statm: 45612 256 72 160 0 45450 0
[pid=16252] ppid=16250 vsize=182448 CPUtime=48.59
/proc/16252/stat : 16252 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4859 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572276 0 0 4096 2 0 0 0 17 1 0 0
/proc/16252/statm: 45612 271 78 160 0 45450 0
[pid=16253] ppid=16250 vsize=182448 CPUtime=48.49
/proc/16253/stat : 16253 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4849 0 0 0 25 0 1 0 205276266 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134572933 0 0 4096 0 0 0 0 17 1 0 0
/proc/16253/statm: 45612 271 78 160 0 45450 0
[pid=16254] ppid=16250 vsize=182448 CPUtime=48.53
/proc/16254/stat : 16254 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4853 0 0 0 25 0 1 0 205276267 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134580277 0 0 4096 2 0 0 0 17 1 0 0
/proc/16254/statm: 45612 271 78 160 0 45450 0
[pid=16255] ppid=16250 vsize=182448 CPUtime=48.31
/proc/16255/stat : 16255 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4831 0 0 0 25 0 1 0 205276272 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134593065 0 0 4096 0 0 0 0 17 0 0 0
/proc/16255/statm: 45612 271 78 160 0 45450 0
[pid=16256] ppid=16250 vsize=182448 CPUtime=48.39
/proc/16256/stat : 16256 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4839 0 0 0 25 0 1 0 205276275 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134589705 0 0 4096 2 0 0 0 17 1 0 0
/proc/16256/statm: 45612 271 78 160 0 45450 0
[pid=16257] ppid=16250 vsize=182448 CPUtime=48.29
/proc/16257/stat : 16257 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4829 0 0 0 25 0 1 0 205276276 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134683448 0 0 4096 0 0 0 0 17 0 0 0
/proc/16257/statm: 45612 271 78 160 0 45450 0
[pid=16258] ppid=16250 vsize=182448 CPUtime=48.22
/proc/16258/stat : 16258 (kcnfs-smp) R 16250 16250 16187 0 -1 4194368 47 0 0 0 4822 0 0 0 25 0 1 0 205276317 186826752 271 18446744073709551615 134512640 135172044 4294956624 18446744073709551615 134593070 0 0 4096 2 0 0 0 17 0 0 0
/proc/16258/statm: 45612 271 78 160 0 45450 0
[pid=16259] ppid=16250 vsize=0 CPUtime=47.85
/proc/16259/stat : 16259 (kcnfs-smp) R 16250 16250 16187 0 -1 4194372 62 0 0 0 4785 0 0 0 25 0 1 0 205276322 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/16259/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 386.7
Current children cumulated vsize (KiB) 1459584

Child status: 10

!!! problem with CPU time !!!
wait4(...,&childrusage) returns 16250 and gives childrusage.ru_utime.tv_sec=47 childrusage.ru_utime.tv_usec=896718 childrusage.ru_stime.tv_sec=0 childrusage.ru_stime.tv_usec=5999
CPU time returned by wait4() is 47.9027
while last known CPU time is 386.7

Solver probably didn't/couldn't wait for its children
Using last known CPU time as value...

Real time (s): 193.893
CPU time (s): 386.7
CPU user time (s): 386.7
CPU system time (s): 0
CPU usage (%): 199.44
Max. virtual memory (cumulated for all children) (KiB): 1642032

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 47.8967
system time used= 0.005999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 547
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 60
involuntary context switches= 1290

runsolver used 0.198969 second user time and 0.500923 second system time

The end

Launcher Data (download as text)

Begin job on node6 on Mon Feb 19 02:46:55 UTC 2007

IDJOB= 298282
IDBENCH= 19982
IDSOLVER= 112
FILE ID= node6/298282-1171853215

PBS_JOBID= 3839406

Free space on /tmp= 66549 MiB

SOLVER NAME= KCNFS SMP
BENCH NAME= HOME/pub/bench/SAT07/random/OnTreshold/3SAT/v500/unif-k3-r4.26-v500-c2130-S1965764400-19.SAT.shuffled.cnf
COMMAND LINE= /tmp/evaluation/298282-1171853215/kcnfs-smp /tmp/evaluation/298282-1171853215/instance-298282-1171853215.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node6/watcher-298282-1171853215 -o ROOT/results/node6/solver-298282-1171853215 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/298282-1171853215/kcnfs-smp /tmp/evaluation/298282-1171853215/instance-298282-1171853215.cnf            

META MD5SUM SOLVER= 23f87ccf1c743ff5b6146b444e216cd3
MD5SUM BENCH=  a8fcc21a1c405b91adf9e5bdbb6b4ddd

RANDOM SEED= 484936480

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node6.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.277
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.277
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055924 kB
MemFree:       1335128 kB
Buffers:         68136 kB
Cached:         519588 kB
SwapCached:          0 kB
Active:         336516 kB
Inactive:       307240 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055924 kB
LowFree:       1335128 kB
SwapTotal:     4192956 kB
SwapFree:      4192812 kB
Dirty:            3356 kB
Writeback:           0 kB
Mapped:          74936 kB
Slab:            62700 kB
Committed_AS:   743756 kB
PageTables:       1744 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66549 MiB

End job on node6 on Mon Feb 19 02:50:09 UTC 2007