Trace number 298241

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
KCNFS SMPSAT 184.63 92.5975

General information on the benchmark

Namerandom/OnTreshold/3SAT/v500/
unif-k3-r4.26-v500-c2130-S44928635-12.SAT.shuffled.cnf
MD5SUMdf00c18c996450679ab1afda9ed45a72
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.150976
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	
0.00/0.00	c Computed by /tmp/evaluation/298241-1171852637/kcnfs-smp   V. 1.2
0.00/0.00	c reading file ... done
0.00/0.00	c Input file : '/tmp/evaluation/298241-1171852637/instance-298241-1171852637.cnf'
0.00/0.00	c [#vars: 500, #clauses: 2130 -> 2214 clauses (pre-processing), tautology: 0, 3-CNF]
0.00/0.01	c unit clauses propagate before processing (#clause: 2214)
0.00/0.01	c estimated distribution: 8 tasks
0.00/0.02	c <14880> [PID]: 14882 14883 14884 14885 14886 14887 14888
0.03/0.47	c <14880> [PID]: 14889
0.03/0.47	c real distribution: 8 tasks
184.63/92.59	c <14880> [14885: SAT]
184.63/92.59	c <14880> [END]: 14882 14883 14884 14886 14887 14888 14889
184.63/92.59	s SATISFIABLE
184.63/92.59	v       -1        2       -3        4        5        6        7
184.63/92.59	v       -8       -9      -10       11       12      -13      -14
184.63/92.59	v       15       16      -17      -18      -19      -20      -21
184.63/92.59	v       22       23       24      -25       26      -27      -28
184.63/92.59	v       29      -30       31      -32      -33       34       35
184.63/92.59	v       36       37       38       39      -40      -41      -42
184.63/92.59	v      -43       44      -45      -46      -47       48       49
184.63/92.59	v       50      -51      -52      -53      -54      -55      -56
184.63/92.59	v       57      -58       59       60       61      -62       63
184.63/92.59	v      -64       65      -66       67       68      -69       70
184.63/92.59	v       71       72       73       74      -75      -76      -77
184.63/92.59	v      -78      -79      -80      -81      -82       83       84
184.63/92.59	v      -85       86       87       88       89       90       91
184.63/92.59	v      -92       93      -94      -95       96       97      -98
184.63/92.59	v       99      100      101      102      103     -104     -105
184.63/92.59	v     -106     -107     -108     -109      110      112      113
184.63/92.59	v     -114     -115      116      117     -118      119     -120
184.63/92.59	v      121     -122     -123     -124     -125      126     -127
184.63/92.59	v     -128     -129     -130      131      132      133      134
184.63/92.59	v      135      136      137     -138      139      140     -141
184.63/92.59	v      142      143      144      145      146     -147     -148
184.63/92.59	v     -149      150     -151     -152      153      154      155
184.63/92.59	v      156      157      158      159      160      161     -162
184.63/92.59	v      163     -164      165     -166      167      168      169
184.63/92.59	v     -170      171     -172      173      174     -175      176
184.63/92.59	v      177     -178      179     -180      181      182      183
184.63/92.59	v     -184     -185     -186     -187     -188      189      190
184.63/92.59	v      191     -192      193     -194     -195      196     -197
184.63/92.59	v     -198      199     -200     -201     -202     -203      204
184.63/92.59	v      205     -206      207      208     -209     -210      211
184.63/92.59	v     -212     -213     -214     -215      216      217     -218
184.63/92.59	v     -219     -220     -221      222     -223      224      225
184.63/92.59	v     -226      227      228      229      230     -231      232
184.63/92.59	v     -233     -234      235      236     -237     -238     -239
184.63/92.59	v     -240      241      242     -243      244     -245     -246
184.63/92.59	v     -247     -248      249     -250      251      252     -253
184.63/92.59	v     -254     -255     -256      257      258      259      260
184.63/92.59	v     -261     -262     -263     -264     -265     -266      267
184.63/92.59	v      268      269      270     -271      272     -273      274
184.63/92.59	v     -275     -276     -277     -278     -279      280     -281
184.63/92.59	v     -282      283     -284     -285     -286     -287      288
184.63/92.59	v      289      290      291     -292     -293      294     -295
184.63/92.59	v     -296     -297      298      299      300     -301      302
184.63/92.59	v      303     -304      305      306      307      308     -309
184.63/92.59	v     -310      311      312      313      314      315      316
184.63/92.59	v     -317      318      319      320      321      322      323
184.63/92.59	v      324      325     -326      327     -328      329     -330
184.63/92.59	v     -331      332      333      334      335      336     -337
184.63/92.59	v      338     -339      340      341      342      343     -344
184.63/92.59	v      345      346     -347      348      349      350      351
184.63/92.59	v      352     -353      354      355      356      357     -358
184.63/92.59	v      359      360      361     -362     -363     -364      365
184.63/92.59	v     -366      367     -368      369     -370     -371      372
184.63/92.59	v      373      374      375      376     -377      378      379
184.63/92.59	v     -380     -381     -382      383      384      385     -386
184.63/92.59	v     -387      388      389     -390     -391     -392      393
184.63/92.59	v      394     -395     -396     -397      398     -399     -400
184.63/92.59	v     -401      402     -403      404      405      406      407
184.63/92.59	v     -408     -409      410     -411      412     -413      414
184.63/92.59	v     -415     -416     -417      418      419      420      421
184.63/92.59	v     -422     -423      424      425     -426     -427     -428
184.63/92.59	v      429      430      431     -432      433     -434     -435
184.63/92.59	v      436      437     -438     -439     -440     -441      442
184.63/92.59	v     -443     -444      445     -446      447      448      449
184.63/92.59	v      450      451      452      453     -454     -455     -456
184.63/92.59	v     -457     -458     -459     -460      461     -462     -463
184.63/92.59	v     -464      465      466      467      468      469     -470
184.63/92.59	v      471      472     -473     -474      475     -476      477
184.63/92.59	v     -478      479      480      481     -482      483     -484
184.63/92.59	v      485      486     -487      488     -489     -490      491
184.63/92.59	v     -492      493      494      495      496      497      498
184.63/92.59	v      499     -500        0
184.63/92.59	c CPU time            : 22.870 seconds
184.63/92.59	c Elapsed Time        : 92.591 seconds
184.63/92.59	c Size of search tree : 38725 nodes
184.63/92.59	c average speed       : 1693.266 nodes/sec.
184.63/92.59

Verifier Data (download as text)

OK

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node42/watcher-298241-1171852637 -o ROOT/results/node42/solver-298241-1171852637 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/298241-1171852637/kcnfs-smp /tmp/evaluation/298241-1171852637/instance-298241-1171852637.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 14880
/proc/meminfo: memFree=382160/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=181980 CPUtime=0
/proc/14880/stat : 14880 (kcnfs-smp) R 14878 14880 13615 0 -1 4194304 156 0 0 0 0 0 0 0 18 0 1 0 205219780 186347520 140 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134726194 0 0 4096 0 0 0 0 17 1 0 0
/proc/14880/statm: 45495 140 51 160 0 45333 0

[startup+0.106383 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 14880
/proc/meminfo: memFree=382160/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.03
/proc/14880/stat : 14880 (kcnfs-smp) R 14878 14880 13615 0 -1 4194304 377 0 0 0 3 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134566844 0 0 4096 0 0 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 182448

[startup+0.514461 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 14880
/proc/meminfo: memFree=382160/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 182448

[startup+1.33662 s]
/proc/loadavg: 0.92 0.98 0.99 9/73 14889
/proc/meminfo: memFree=379144/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=0.54
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 54 0 0 0 24 0 1 0 205219782 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134595458 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 271 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=0.48
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 48 0 0 0 24 0 1 0 205219783 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134600155 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 271 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=0.3
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 30 0 0 0 24 0 1 0 205219786 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134604440 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 271 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=0.36
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 36 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580290 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=0.24
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 24 0 0 0 25 0 1 0 205219818 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134606776 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 271 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=0.22
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 22 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572334 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=0.2
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 20 0 0 0 25 0 1 0 205219826 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134604503 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 271 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=0.2
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 20 0 0 0 25 0 1 0 205219827 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134628434 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 2.58
Current children cumulated vsize (KiB) 1642032

[startup+2.98893 s]
/proc/loadavg: 0.92 0.98 0.99 9/73 14889
/proc/meminfo: memFree=379016/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=0.94
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 94 0 0 0 25 0 1 0 205219782 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134628459 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 271 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=0.83
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 83 0 0 0 25 0 1 0 205219783 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577653 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 271 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=0.8
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 80 0 0 0 25 0 1 0 205219786 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580392 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 271 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=0.8
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 80 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580220 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=0.65
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 65 0 0 0 25 0 1 0 205219818 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580258 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 271 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=0.62
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 62 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683578 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=0.6
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 60 0 0 0 25 0 1 0 205219826 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134603242 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 271 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=0.6
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 60 0 0 0 25 0 1 0 205219827 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567303 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 5.88
Current children cumulated vsize (KiB) 1642032

[startup+6.27656 s]
/proc/loadavg: 1.49 1.10 1.03 9/73 14889
/proc/meminfo: memFree=378824/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=1.74
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 174 0 0 0 25 0 1 0 205219782 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572276 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 271 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=1.72
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 172 0 0 0 25 0 1 0 205219783 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580054 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 271 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=1.6
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 160 0 0 0 25 0 1 0 205219786 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580327 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 271 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=1.6
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 160 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134578193 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=1.53
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 153 0 0 0 25 0 1 0 205219818 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580397 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 271 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=1.42
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 142 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567193 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=1.4
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 140 0 0 0 25 0 1 0 205219826 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577632 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 271 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=1.4
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 140 0 0 0 25 0 1 0 205219827 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577447 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 12.45
Current children cumulated vsize (KiB) 1642032

[startup+12.7718 s]
/proc/loadavg: 2.01 1.21 1.07 9/73 14889
/proc/meminfo: memFree=378696/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=3.44
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 344 0 0 0 25 0 1 0 205219782 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577649 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 271 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=3.32
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 332 0 0 0 25 0 1 0 205219783 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683143 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 271 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=3.2
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 320 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134566835 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=3.2
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 320 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134649797 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=3.14
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 314 0 0 0 25 0 1 0 205219818 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134581717 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 271 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=3.02
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 302 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580339 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=3.05
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 305 0 0 0 25 0 1 0 205219826 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567633 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 271 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=3
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 300 0 0 0 25 0 1 0 205219827 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683214 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 25.41
Current children cumulated vsize (KiB) 1642032

[startup+25.6342 s]
/proc/loadavg: 3.33 1.55 1.18 9/73 14889
/proc/meminfo: memFree=378696/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=6.64
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 664 0 0 0 25 0 1 0 205219782 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572863 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 271 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=6.52
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 652 0 0 0 25 0 1 0 205219783 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567202 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 271 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=6.4
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 640 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683120 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=6.4
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 640 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577566 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=6.34
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 634 0 0 0 25 0 1 0 205219818 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683223 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 271 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=6.27
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 627 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579685 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=6.26
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 626 0 0 0 25 0 1 0 205219826 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580177 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 271 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=6.2
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 620 0 0 0 25 0 1 0 205219827 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134578432 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 271 78 160 0 45450 0
Current children cumulated CPU time (s) 51.07

################
# More data... #
################

[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=22.84
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2284 0 0 0 25 0 1 0 205219782 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572834 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 272 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=22.74
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2274 0 0 0 25 0 1 0 205219783 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580088 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 272 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=22.7
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2270 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579507 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=22.6
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2260 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675411 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=22.5
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2250 0 0 0 25 0 1 0 205219818 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134589865 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 272 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=22.52
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2252 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683120 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=22.4
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2240 0 0 0 25 0 1 0 205219826 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134586899 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 272 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=22.4
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2240 0 0 0 25 0 1 0 205219827 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580277 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 180.74
Current children cumulated vsize (KiB) 1642032

[startup+91.4358 s]
/proc/loadavg: 6.42 2.80 1.64 9/73 14889
/proc/meminfo: memFree=378632/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=23.04
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2304 0 0 0 25 0 1 0 205219782 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134601192 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 272 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=22.95
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2295 0 0 0 25 0 1 0 205219783 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683143 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 272 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=22.9
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2290 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567500 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=182448 CPUtime=22.8
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2280 0 0 0 25 0 1 0 205219787 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572082 0 0 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 45612 271 78 160 0 45450 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=22.72
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2272 0 0 0 25 0 1 0 205219818 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567248 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 272 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=22.72
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2272 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683113 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=22.6
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2260 0 0 0 25 0 1 0 205219826 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675448 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 272 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=22.6
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2260 0 0 0 25 0 1 0 205219827 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577653 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 182.37
Current children cumulated vsize (KiB) 1642032

[startup+92.2619 s]
/proc/loadavg: 6.42 2.80 1.64 9/73 14889
/proc/meminfo: memFree=378760/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=23.24
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2324 0 0 0 25 0 1 0 205219782 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675731 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 272 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=23.18
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2318 0 0 0 25 0 1 0 205219783 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134603900 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 272 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=23.1
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2310 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580177 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=0 CPUtime=22.83
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194372 62 0 0 0 2283 0 0 0 25 0 1 0 205219787 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 0 0 0 0 0 0 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=22.94
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2294 0 0 0 25 0 1 0 205219818 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683111 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 272 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=22.92
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2292 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572082 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=22.9
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2290 0 0 0 25 0 1 0 205219826 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683587 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 272 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=22.86
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2286 0 0 0 25 0 1 0 205219827 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580258 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 184.01
Current children cumulated vsize (KiB) 1459584

[startup+92.468 s]
/proc/loadavg: 6.42 2.80 1.64 9/73 14889
/proc/meminfo: memFree=378760/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=23.34
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2334 0 0 0 25 0 1 0 205219782 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683130 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 272 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=23.21
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2321 0 0 0 25 0 1 0 205219783 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134592555 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 272 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=23.1
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2310 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580177 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=0 CPUtime=22.83
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194372 62 0 0 0 2283 0 0 0 25 0 1 0 205219787 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 0 0 0 0 0 0 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=23.04
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2304 0 0 0 25 0 1 0 205219818 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134595659 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 272 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=22.99
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2299 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580317 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=22.97
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2297 0 0 0 25 0 1 0 205219826 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572306 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 272 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=22.9
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2290 0 0 0 25 0 1 0 205219827 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683130 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 184.42
Current children cumulated vsize (KiB) 1459584

[startup+92.571 s]
/proc/loadavg: 6.42 2.80 1.64 9/73 14889
/proc/meminfo: memFree=378760/2055920 swapFree=4183424/4192956
[pid=14880] ppid=14878 vsize=182448 CPUtime=0.04
/proc/14880/stat : 14880 (kcnfs-smp) S 14878 14880 13615 0 -1 4194304 460 0 0 0 4 0 0 0 22 0 1 0 205219780 186826752 257 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14880/statm: 45612 257 72 160 0 45450 0
[pid=14882] ppid=14880 vsize=182448 CPUtime=23.34
/proc/14882/stat : 14882 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2334 0 0 0 25 0 1 0 205219782 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683130 0 0 4096 2 0 0 0 17 1 0 0
/proc/14882/statm: 45612 272 78 160 0 45450 0
[pid=14883] ppid=14880 vsize=182448 CPUtime=23.21
/proc/14883/stat : 14883 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2321 0 0 0 25 0 1 0 205219783 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134592555 0 0 4096 0 0 0 0 17 1 0 0
/proc/14883/statm: 45612 272 78 160 0 45450 0
[pid=14884] ppid=14880 vsize=182448 CPUtime=23.18
/proc/14884/stat : 14884 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2318 0 0 0 25 0 1 0 205219786 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134650817 0 0 4096 2 0 0 0 17 1 0 0
/proc/14884/statm: 45612 272 78 160 0 45450 0
[pid=14885] ppid=14880 vsize=0 CPUtime=22.83
/proc/14885/stat : 14885 (kcnfs-smp) R 14880 14880 13615 0 -1 4194372 62 0 0 0 2283 0 0 0 25 0 1 0 205219787 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/14885/statm: 0 0 0 0 0 0 0
[pid=14886] ppid=14880 vsize=182448 CPUtime=23.04
/proc/14886/stat : 14886 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2304 0 0 0 25 0 1 0 205219818 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134595659 0 0 4096 2 0 0 0 17 0 0 0
/proc/14886/statm: 45612 272 78 160 0 45450 0
[pid=14887] ppid=14880 vsize=182448 CPUtime=23.02
/proc/14887/stat : 14887 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 46 0 0 0 2302 0 0 0 25 0 1 0 205219823 186826752 271 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134587143 0 0 4096 0 0 0 0 17 1 0 0
/proc/14887/statm: 45612 271 78 160 0 45450 0
[pid=14888] ppid=14880 vsize=182448 CPUtime=23
/proc/14888/stat : 14888 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2300 0 0 0 25 0 1 0 205219826 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134587460 0 0 4096 2 0 0 0 17 0 0 0
/proc/14888/statm: 45612 272 78 160 0 45450 0
[pid=14889] ppid=14880 vsize=182448 CPUtime=22.97
/proc/14889/stat : 14889 (kcnfs-smp) R 14880 14880 13615 0 -1 4194368 47 0 0 0 2297 0 0 0 25 0 1 0 205219827 186826752 272 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579723 0 0 4096 0 0 0 0 17 0 0 0
/proc/14889/statm: 45612 272 78 160 0 45450 0
Current children cumulated CPU time (s) 184.63
Current children cumulated vsize (KiB) 1459584

Child status: 10

!!! problem with CPU time !!!
wait4(...,&childrusage) returns 14880 and gives childrusage.ru_utime.tv_sec=22 childrusage.ru_utime.tv_usec=875522 childrusage.ru_stime.tv_sec=0 childrusage.ru_stime.tv_usec=1999
CPU time returned by wait4() is 22.8775
while last known CPU time is 184.63

Solver probably didn't/couldn't wait for its children
Using last known CPU time as value...

Real time (s): 92.5975
CPU time (s): 184.63
CPU user time (s): 184.63
CPU system time (s): 0
CPU usage (%): 199.39
Max. virtual memory (cumulated for all children) (KiB): 1642032

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 22.8755
system time used= 0.001999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 549
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12
involuntary context switches= 551

runsolver used 0.094985 second user time and 0.243962 second system time

The end

Launcher Data (download as text)

Begin job on node42 on Mon Feb 19 02:37:17 UTC 2007

IDJOB= 298241
IDBENCH= 19981
IDSOLVER= 112
FILE ID= node42/298241-1171852637

PBS_JOBID= 3839730

Free space on /tmp= 66561 MiB

SOLVER NAME= KCNFS SMP
BENCH NAME= HOME/pub/bench/SAT07/random/OnTreshold/3SAT/v500/unif-k3-r4.26-v500-c2130-S44928635-12.SAT.shuffled.cnf
COMMAND LINE= /tmp/evaluation/298241-1171852637/kcnfs-smp /tmp/evaluation/298241-1171852637/instance-298241-1171852637.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node42/watcher-298241-1171852637 -o ROOT/results/node42/solver-298241-1171852637 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/298241-1171852637/kcnfs-smp /tmp/evaluation/298241-1171852637/instance-298241-1171852637.cnf            

META MD5SUM SOLVER= 23f87ccf1c743ff5b6146b444e216cd3
MD5SUM BENCH=  df00c18c996450679ab1afda9ed45a72

RANDOM SEED= 322700350

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node42.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.261
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.261
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:        382568 kB
Buffers:         52852 kB
Cached:        1492760 kB
SwapCached:       3904 kB
Active:         829012 kB
Inactive:       727032 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:        382568 kB
SwapTotal:     4192956 kB
SwapFree:      4183424 kB
Dirty:            3180 kB
Writeback:           0 kB
Mapped:          15832 kB
Slab:           103364 kB
Committed_AS:   787560 kB
PageTables:       1408 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66561 MiB

End job on node42 on Mon Feb 19 02:38:50 UTC 2007