Trace number 309680

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
KCNFS SMPSAT 73.47 36.9227

General information on the benchmark

Namerandom/2+p/p0.9/v810/
unif2p-p0.9-v810-c2932-S1870497322-13-SAT.cnf
MD5SUMb21a160fd72c696cb1c462ee57a56514
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.302953
Satisfiable
(Un)Satisfiability was proved
Number of variables810
Number of clauses2931
Sum of the clauses size8500
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2293
Number of clauses of size 32638
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	
0.00/0.00	c Computed by /tmp/evaluation/309680-1172545437/kcnfs-smp   V. 1.2
0.00/0.00	c reading file ... done
0.00/0.00	c Input file : '/tmp/evaluation/309680-1172545437/instance-309680-1172545437.cnf'
0.00/0.00	c [#vars: 810, #clauses: 2931 , (2-3)-CNF]
0.00/0.00	c unit clauses propagate before processing (#clause: 2931)
0.00/0.00	c estimated distribution: 8 tasks
0.00/0.01	c <14766> [PID]: 14768 14769 14770 14771 14772 14773 14774
0.00/0.41	c <14766> [PID]: 14775
0.00/0.41	c real distribution: 8 tasks
73.47/36.92	c <14766> [14775: SAT]
73.47/36.92	c <14766> [END]: 14768 14769 14770 14771 14772 14773 14774
73.47/36.92	s SATISFIABLE
73.47/36.92	v       -1        2       -3       -4       -5        6       -7
73.47/36.92	v        8        9       10       11       12      -13      -14
73.47/36.92	v      -15      -16      -17      -18      -19       20      -21
73.47/36.92	v       22       23       24       25      -26       27      -28
73.47/36.92	v      -29      -30       31       32       33       34       35
73.47/36.92	v       36      -37      -38       39       40      -41      -42
73.47/36.92	v       43      -44      -45      -46       47      -48       49
73.47/36.92	v       50      -51       52       53      -54       55      -56
73.47/36.92	v      -57      -58       59      -60      -61      -62      -63
73.47/36.92	v      -64       65       66      -67      -68       69       70
73.47/36.92	v       71       72      -73      -74       75      -76       77
73.47/36.92	v       78       79      -80       81       82      -83       84
73.47/36.92	v      -85       86       87       88      -89       90      -91
73.47/36.92	v      -92      -93      -94      -95       96       97      -98
73.47/36.92	v       99      100      101      102      103     -104      105
73.47/36.92	v      106      107     -108      109      110     -111      112
73.47/36.92	v      113      114      115      116     -117      118     -119
73.47/36.92	v     -120     -121     -122      123     -124      125      126
73.47/36.92	v      127     -128     -129      130      131     -132     -133
73.47/36.92	v     -134     -135      136      137     -138     -139     -140
73.47/36.92	v     -141     -142      143     -144      145      146      147
73.47/36.92	v      148     -149     -150     -151      152     -153     -154
73.47/36.92	v     -155     -156     -157     -158     -159     -160     -161
73.47/36.92	v      162      163     -164     -165     -166      167      168
73.47/36.92	v     -169     -170     -171     -172      173     -174      175
73.47/36.92	v     -176     -177      178      179     -180      181      182
73.47/36.92	v     -183      184      185      186      187      188      189
73.47/36.92	v      190     -191      192      193     -194      195     -196
73.47/36.92	v      197      198      199      200     -201      202     -203
73.47/36.92	v      204      205      206     -207      208     -209     -210
73.47/36.92	v     -211      212      213      214      215      216      217
73.47/36.92	v      218      219      220     -221      222     -223     -224
73.47/36.92	v      225      226      227     -228      229     -230     -231
73.47/36.92	v      232     -233      234     -235      236      237      238
73.47/36.92	v      239      240     -241     -242     -243     -244      245
73.47/36.92	v      246     -247      248      249      250      251     -252
73.47/36.92	v     -253     -254     -255     -256     -257      258      259
73.47/36.92	v      260     -261      262      263     -264      265      266
73.47/36.92	v      267      268      269      270      271     -272     -273
73.47/36.92	v     -274     -275     -276     -277     -278     -279      280
73.47/36.92	v     -281      282      283     -284      285      286      287
73.47/36.92	v      288      289      290      291      292     -293     -294
73.47/36.92	v     -295     -296     -297     -299     -300      301      302
73.47/36.92	v     -303      304      305     -306     -307      308      309
73.47/36.92	v      310      311      312      313     -314     -315      316
73.47/36.92	v     -317      318     -319      320      321      322     -323
73.47/36.92	v      324      325      326     -327     -328      329      330
73.47/36.92	v     -331      332     -333     -334      335      336     -337
73.47/36.92	v      338     -339     -340     -341      342     -343     -344
73.47/36.92	v     -345     -346      347     -348      349      350      351
73.47/36.92	v      352     -353      354      355     -356      357      358
73.47/36.92	v     -359     -360     -361     -362      363     -364      365
73.47/36.92	v     -366     -367      368     -369      370     -371      372
73.47/36.92	v     -373     -374      375     -376     -377      378      379
73.47/36.92	v      380      381     -382      383     -384     -385      386
73.47/36.92	v     -387     -388      389     -390      391      392     -393
73.47/36.92	v     -394      395      396      397     -398      399      400
73.47/36.92	v     -401     -402     -403      404     -405      406      407
73.47/36.92	v      408      409      410      411     -412     -413     -414
73.47/36.92	v      415      416      417      418      419     -420     -421
73.47/36.92	v     -422      423      424     -425     -426      427      428
73.47/36.92	v      429      430      431      432     -433      434     -435
73.47/36.92	v      436     -437     -438     -439      440      441     -442
73.47/36.92	v     -443      444     -445      446      447     -448      449
73.47/36.92	v      450     -451      452     -453      454      455      456
73.47/36.92	v      457      458     -459      460     -461     -462      463
73.47/36.92	v     -464     -465      466      467     -468      469     -470
73.47/36.92	v     -471      472     -473     -474     -475      476      477
73.47/36.92	v     -478     -479     -480     -481      482      483      484
73.47/36.92	v     -485     -486     -487      488     -489     -490     -491
73.47/36.92	v      492     -493     -494     -495     -496      497      498
73.47/36.92	v     -499     -500     -501      502     -503     -504     -505
73.47/36.92	v      506     -507      508      509      510      511      512
73.47/36.92	v     -513      514      515      516     -517     -518     -519
73.47/36.92	v      520      521      522     -523      524      525      526
73.47/36.92	v     -527     -528      529      530      531      532     -533
73.47/36.92	v     -534     -535     -536     -537      538      539     -540
73.47/36.92	v      541     -542     -543     -544     -545     -546     -547
73.47/36.92	v      548     -549     -550     -551      552     -553     -554
73.47/36.92	v     -555      556      557      558      559      560     -561
73.47/36.92	v     -562     -563      564      565     -566      567      568
73.47/36.92	v      569      570     -571     -572      573      574      575
73.47/36.92	v      576      577      578     -579     -580      581     -582
73.47/36.92	v      583     -584      585      586     -587      588      589
73.47/36.92	v     -590      591     -592      593      594      595      596
73.47/36.92	v      597      598      599      600      601     -602      603
73.47/36.92	v     -604     -605      606     -607     -608      609      610
73.47/36.92	v     -611     -612     -613      614     -615     -616      617
73.47/36.92	v      618     -619      620     -621      622      623      624
73.47/36.92	v     -625      626      627     -628     -629     -630     -631
73.47/36.92	v     -632     -633      634      635      636      637     -638
73.47/36.92	v     -639      640      641     -642      643      644     -645
73.47/36.92	v     -646     -647      648     -649      650      651     -652
73.47/36.92	v     -653      654      655     -656      657     -658     -659
73.47/36.92	v      660      661      662     -663     -664      665      666
73.47/36.92	v     -667      668     -669     -670     -671     -672     -673
73.47/36.92	v     -674      675      676     -677     -678      679     -680
73.47/36.92	v      681     -682     -683     -684      685     -686     -687
73.47/36.92	v     -688     -689     -690      691      692     -693      694
73.47/36.92	v     -695     -696     -697      698      699      700      701
73.47/36.92	v     -702     -703      704      705     -706      707      708
73.47/36.92	v     -709      710      711      712      713     -714      715
73.47/36.92	v      716      717      718      719     -720      721     -722
73.47/36.92	v      723     -724     -725      726      727      728      729
73.47/36.92	v     -730     -731     -732     -733      734      735     -736
73.47/36.92	v      737     -738      739      740     -741      742     -743
73.47/36.92	v     -744     -745      746      747      748      749     -750
73.47/36.92	v     -751      752     -753     -754     -755      756     -757
73.47/36.92	v      758      759     -760      761     -762     -763     -764
73.47/36.92	v      765     -766      767      768      769     -770      771
73.47/36.92	v     -772      773      774     -775     -776     -777     -778
73.47/36.92	v     -779      780      781     -782      783     -784      785
73.47/36.92	v     -786     -787     -788     -789      790     -791     -792
73.47/36.92	v      793      794     -795      796     -797      798     -799
73.47/36.92	v      800      801      802     -803     -804      805     -806
73.47/36.92	v     -807     -808     -809      810        0
73.47/36.92	c CPU time            : 8.790 seconds
73.47/36.92	c Elapsed Time        : 36.914 seconds
73.47/36.92	c Size of search tree : 11413 nodes
73.47/36.92	c average speed       : 1298.407 nodes/sec.
73.47/36.92

Verifier Data (download as text)

OK

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node66/watcher-309680-1172545437 -o ROOT/results/node66/solver-309680-1172545437 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/309680-1172545437/kcnfs-smp /tmp/evaluation/309680-1172545437/instance-309680-1172545437.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.95 1.02 1.10 3/64 14766
/proc/meminfo: memFree=1828024/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=180916 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) R 14764 14766 13895 0 -1 4194304 129 0 0 0 0 0 0 0 18 0 1 0 274501261 185257984 114 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134820679 0 0 4096 0 0 0 0 17 1 0 0
/proc/14766/statm: 45229 115 48 160 0 45067 0

[startup+0.106959 s]
/proc/loadavg: 0.95 1.02 1.10 3/64 14766
/proc/meminfo: memFree=1828024/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) R 14764 14766 13895 0 -1 4194304 468 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836899 0 0 4096 0 0 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 181644

[startup+0.51504 s]
/proc/loadavg: 0.95 1.02 1.10 3/64 14766
/proc/meminfo: memFree=1828024/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 181644

[startup+1.33721 s]
/proc/loadavg: 0.95 1.02 1.10 9/73 14775
/proc/meminfo: memFree=1824912/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=0.44
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 44 0 0 0 23 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572122 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=0.4
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 40 0 0 0 23 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134589840 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=0.41
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 41 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577649 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=0.3
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 30 0 0 0 24 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577464 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=0.3
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 30 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683263 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=0.32
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 32 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134591895 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=0.2
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 20 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675460 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=0.24
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 24 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683130 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 2.61
Current children cumulated vsize (KiB) 1634796

[startup+2.98954 s]
/proc/loadavg: 0.95 1.02 1.10 9/73 14775
/proc/meminfo: memFree=1824720/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=0.87
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 87 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572638 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=0.82
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 82 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580909 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=0.81
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 81 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572774 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=0.7
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 70 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134674534 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=0.7
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 70 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567423 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=0.77
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 77 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577071 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=0.6
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 60 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134581914 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=0.64
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 64 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683143 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 5.91
Current children cumulated vsize (KiB) 1634796

[startup+6.2772 s]
/proc/loadavg: 1.52 1.14 1.14 9/73 14775
/proc/meminfo: memFree=1824464/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=1.74
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 174 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134633773 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=1.62
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 162 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683130 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=1.61
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 161 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577752 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=1.52
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 152 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134635226 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=1.5
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 150 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683204 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=1.6
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 160 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134589865 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=1.44
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 144 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579685 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=1.44
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 144 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580339 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 12.47
Current children cumulated vsize (KiB) 1634796

[startup+12.7705 s]
/proc/loadavg: 2.04 1.25 1.18 9/73 14775
/proc/meminfo: memFree=1824472/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=3.34
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 334 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134574154 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=3.22
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 322 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683126 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=3.21
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 321 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577430 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=3.2
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 320 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577061 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=3.11
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 311 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683587 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=3.2
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 320 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134636965 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=3.1
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 310 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580909 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=3.05
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 305 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572846 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 25.43
Current children cumulated vsize (KiB) 1634796

[startup+25.6311 s]
/proc/loadavg: 3.36 1.58 1.28 9/73 14775
/proc/meminfo: memFree=1824472/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=6.54
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 654 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675481 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=6.42
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 642 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577715 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=6.41
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 641 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572334 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=6.4
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 640 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579737 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=6.36
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 636 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683223 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=6.4
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 640 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134595448 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=6.3
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 630 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579493 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=6.25
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 625 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683195 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 51.08

################
# More data... #
################

[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=8.34
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 834 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579685 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=8.3
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 830 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683195 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=8.2
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 820 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572863 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=8.2
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 820 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572157 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=8.2
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 820 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567604 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=8.19
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 819 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675466 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=8.09
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 809 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675460 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=8.14
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 814 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683587 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 65.66
Current children cumulated vsize (KiB) 1634796

[startup+34.5949 s]
/proc/loadavg: 3.73 1.69 1.32 9/73 14775
/proc/meminfo: memFree=1824472/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=8.77
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 877 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134629419 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=8.72
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 872 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567310 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=8.64
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 864 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572623 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=8.6
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 860 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567307 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=8.6
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 860 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683127 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=8.59
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 859 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567593 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=8.49
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 849 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572655 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=181644 CPUtime=8.54
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 854 0 0 0 25 0 1 0 274501297 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134544585 0 0 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 68.95
Current children cumulated vsize (KiB) 1634796

[startup+36.2392 s]
/proc/loadavg: 4.07 1.79 1.36 9/73 14775
/proc/meminfo: memFree=1824664/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=9.22
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 922 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134627926 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=9.12
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 912 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572720 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=9.1
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 910 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577634 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=9
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 900 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572753 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=9
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 900 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572638 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=9.09
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 909 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580273 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=8.92
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 892 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572164 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=0 CPUtime=8.79
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194372 67 0 0 0 879 0 0 0 25 0 1 0 274501297 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 72.24
Current children cumulated vsize (KiB) 1453152

[startup+36.6553 s]
/proc/loadavg: 4.07 1.79 1.36 9/73 14775
/proc/meminfo: memFree=1824664/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=9.33
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 933 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580154 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=9.22
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 922 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572848 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=9.24
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 924 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683587 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=9.1
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 910 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580088 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=9.1
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 910 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567426 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=9.19
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 919 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675319 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=9.09
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 909 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134587511 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=0 CPUtime=8.79
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194372 67 0 0 0 879 0 0 0 25 0 1 0 274501297 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 73.06
Current children cumulated vsize (KiB) 1453152

[startup+36.8613 s]
/proc/loadavg: 4.07 1.79 1.36 9/73 14775
/proc/meminfo: memFree=1824664/2055920 swapFree=4179812/4192956
[pid=14766] ppid=14764 vsize=181644 CPUtime=0
/proc/14766/stat : 14766 (kcnfs-smp) S 14764 14766 13895 0 -1 4194304 498 0 0 0 0 0 0 0 24 0 1 0 274501261 186003456 296 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/14766/statm: 45411 296 65 160 0 45249 0
[pid=14768] ppid=14766 vsize=181644 CPUtime=9.34
/proc/14768/stat : 14768 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 46 0 0 0 934 0 0 0 25 0 1 0 274501261 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572802 0 0 4096 2 0 0 0 17 1 0 0
/proc/14768/statm: 45411 308 71 160 0 45249 0
[pid=14769] ppid=14766 vsize=181644 CPUtime=9.22
/proc/14769/stat : 14769 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 47 0 0 0 922 0 0 0 25 0 1 0 274501262 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572848 0 0 4096 0 0 0 0 17 1 0 0
/proc/14769/statm: 45411 308 71 160 0 45249 0
[pid=14770] ppid=14766 vsize=181644 CPUtime=9.3
/proc/14770/stat : 14770 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 930 0 0 0 25 0 1 0 274501264 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134579723 0 0 4096 2 0 0 0 17 0 0 0
/proc/14770/statm: 45411 308 71 160 0 45249 0
[pid=14771] ppid=14766 vsize=181644 CPUtime=9.2
/proc/14771/stat : 14771 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 920 0 0 0 25 0 1 0 274501265 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572276 0 0 4096 0 0 0 0 17 1 0 0
/proc/14771/statm: 45411 308 71 160 0 45249 0
[pid=14772] ppid=14766 vsize=181644 CPUtime=9.19
/proc/14772/stat : 14772 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 919 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134600485 0 0 4096 2 0 0 0 17 1 0 0
/proc/14772/statm: 45411 308 71 160 0 45249 0
[pid=14773] ppid=14766 vsize=181644 CPUtime=9.29
/proc/14773/stat : 14773 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 929 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134581126 0 0 4096 0 0 0 0 17 0 0 0
/proc/14773/statm: 45411 308 71 160 0 45249 0
[pid=14774] ppid=14766 vsize=181644 CPUtime=9.14
/proc/14774/stat : 14774 (kcnfs-smp) R 14766 14766 13895 0 -1 4194368 44 0 0 0 914 0 0 0 25 0 1 0 274501266 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683127 0 0 4096 2 0 0 0 17 0 0 0
/proc/14774/statm: 45411 308 71 160 0 45249 0
[pid=14775] ppid=14766 vsize=0 CPUtime=8.79
/proc/14775/stat : 14775 (kcnfs-smp) R 14766 14766 13895 0 -1 4194372 67 0 0 0 879 0 0 0 25 0 1 0 274501297 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 0 0 0 0 17 0 0 0
/proc/14775/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 73.47
Current children cumulated vsize (KiB) 1453152

Child status: 10

!!! problem with CPU time !!!
wait4(...,&childrusage) returns 14766 and gives childrusage.ru_utime.tv_sec=8 childrusage.ru_utime.tv_usec=804661 childrusage.ru_stime.tv_sec=0 childrusage.ru_stime.tv_usec=3999
CPU time returned by wait4() is 8.80866
while last known CPU time is 73.47

Solver probably didn't/couldn't wait for its children
Using last known CPU time as value...

Real time (s): 36.9227
CPU time (s): 73.47
CPU user time (s): 73.47
CPU system time (s): 0
CPU usage (%): 198.983
Max. virtual memory (cumulated for all children) (KiB): 1634796

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 8.80466
system time used= 0.003999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 595
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 13
involuntary context switches= 238

runsolver used 0.044993 second user time and 0.096985 second system time

The end

Launcher Data (download as text)

Begin job on node66 on Tue Feb 27 03:03:57 UTC 2007

IDJOB= 309680
IDBENCH= 20267
IDSOLVER= 112
FILE ID= node66/309680-1172545437

PBS_JOBID= 3971834

Free space on /tmp= 66559 MiB

SOLVER NAME= KCNFS SMP
BENCH NAME= HOME/pub/bench/SAT07/random/2+p/p0.9/v810/unif2p-p0.9-v810-c2932-S1870497322-13-SAT.cnf
COMMAND LINE= /tmp/evaluation/309680-1172545437/kcnfs-smp /tmp/evaluation/309680-1172545437/instance-309680-1172545437.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node66/watcher-309680-1172545437 -o ROOT/results/node66/solver-309680-1172545437 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/309680-1172545437/kcnfs-smp /tmp/evaluation/309680-1172545437/instance-309680-1172545437.cnf            

META MD5SUM SOLVER= 23f87ccf1c743ff5b6146b444e216cd3
MD5SUM BENCH=  b21a160fd72c696cb1c462ee57a56514

RANDOM SEED= 82135261

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node66.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.259
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5931.00
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.259
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1828432 kB
Buffers:         28820 kB
Cached:         146696 kB
SwapCached:       7404 kB
Active:          75876 kB
Inactive:       112896 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1828432 kB
SwapTotal:     4192956 kB
SwapFree:      4179812 kB
Dirty:            3260 kB
Writeback:           0 kB
Mapped:          15324 kB
Slab:            24792 kB
Committed_AS:   666452 kB
PageTables:       1376 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66559 MiB

End job on node66 on Tue Feb 27 03:04:34 UTC 2007