Trace number 309721

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
KCNFS SMPSAT 329.5 165.226

General information on the benchmark

Namerandom/2+p/p0.9/v810/
unif2p-p0.9-v810-c2932-S1274825698-06-SAT.cnf
MD5SUM4b942756bd3042b1996070cebc1719eb
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.029994
Satisfiable
(Un)Satisfiability was proved
Number of variables810
Number of clauses2931
Sum of the clauses size8500
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2293
Number of clauses of size 32638
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	
0.00/0.00	c Computed by /tmp/evaluation/309721-1172546084/kcnfs-smp   V. 1.2
0.00/0.00	c reading file ... done
0.00/0.00	c Input file : '/tmp/evaluation/309721-1172546084/instance-309721-1172546084.cnf'
0.00/0.00	c [#vars: 810, #clauses: 2931 , (2-3)-CNF]
0.00/0.00	c unit clauses propagate before processing (#clause: 2931)
0.00/0.00	c estimated distribution: 8 tasks
0.00/0.01	c <26389> [PID]: 26391 26392 26393 26394 26395 26396 26397
0.00/0.31	c <26389> [PID]: 26398
0.00/0.31	c real distribution: 8 tasks
29.73/14.96	c <26389> [26396: UNSAT][26398: UNSAT][26395: UNSAT][26397: UNSAT][26394: UNSAT]
329.50/165.22	c <26389> [26393: SAT]
329.50/165.22	c <26389> [END]: 26391 26392
329.50/165.22	s SATISFIABLE
329.50/165.22	v        1       -2        3        4        5        6       -7
329.50/165.22	v        8        9      -10       11       12      -13       14
329.50/165.22	v       15      -16      -17       18      -19      -20       21
329.50/165.22	v      -22       23      -24      -25       26       27      -28
329.50/165.22	v       29       30       31       32      -33       34      -35
329.50/165.22	v       36      -37      -38       39      -40       41       42
329.50/165.22	v      -43      -44       45      -46      -47      -48       49
329.50/165.22	v      -50      -51      -52       53       54      -55      -56
329.50/165.22	v      -57       58       59       60      -61       62       63
329.50/165.22	v      -64       65       66      -67       68       69      -70
329.50/165.22	v      -71      -72      -73      -74       75       76       77
329.50/165.22	v      -78       79      -80      -81      -82      -83       84
329.50/165.22	v       85       86       87      -88      -89       90       91
329.50/165.22	v       92       93      -94      -95      -96      -97      -98
329.50/165.22	v      -99     -100      101      102      103      104      105
329.50/165.22	v      106      107     -108      109      110     -111      112
329.50/165.22	v      113      114     -115     -116     -117      118      119
329.50/165.22	v      120     -121     -122      123     -124     -125      126
329.50/165.22	v      127     -128      129      130      131     -132      133
329.50/165.22	v      134      135      136     -137      138     -139     -140
329.50/165.22	v      141      142      143     -144     -145     -146      147
329.50/165.22	v      148     -149     -150      151     -152     -153      154
329.50/165.22	v      155      156      157      158     -159     -160     -161
329.50/165.22	v     -162     -163      164      165     -166      167      168
329.50/165.22	v     -169      170     -171      172      173      174     -175
329.50/165.22	v     -176      177      178     -179      180      181      182
329.50/165.22	v      183     -184     -185     -186      187     -188      189
329.50/165.22	v      190      191      192      193      194     -195      196
329.50/165.22	v     -197     -198      199     -200     -201      202     -203
329.50/165.22	v      204      205     -206     -207     -208      209      210
329.50/165.22	v      211     -212     -213      214     -215      216     -217
329.50/165.22	v     -218     -219      220      221     -222     -223     -224
329.50/165.22	v      225      226     -227      228     -229     -230     -231
329.50/165.22	v     -232      233     -234      235     -236     -237     -238
329.50/165.22	v      239      240      241      242      243     -244      245
329.50/165.22	v     -246     -247     -248     -249     -250      251     -252
329.50/165.22	v      253      254      255     -256     -257     -258      259
329.50/165.22	v     -260      261      262      263     -264     -265     -266
329.50/165.22	v     -267      268      269     -270      271      272      273
329.50/165.22	v      274      275      276     -277      278      279     -280
329.50/165.22	v      281      282     -283     -284      285     -286     -287
329.50/165.22	v     -288     -289     -290     -291      292      293     -294
329.50/165.22	v     -295     -296      297     -298     -299      300     -301
329.50/165.22	v     -302     -303      304     -305      306      307     -308
329.50/165.22	v      309     -310      311     -312      313      314     -315
329.50/165.22	v      316      317     -318      319     -320     -321     -322
329.50/165.22	v     -323      324     -325     -326     -327      328     -329
329.50/165.22	v      330     -331      332     -333     -334     -335     -336
329.50/165.22	v     -337     -338      339     -340     -341      342     -343
329.50/165.22	v     -344      345      346     -347     -348     -349      350
329.50/165.22	v     -351     -352     -353      354      355     -356     -357
329.50/165.22	v      358      359     -360      361     -362     -363     -364
329.50/165.22	v      365      366     -367      368     -369      370     -371
329.50/165.22	v      372      373      374     -375      376      377      378
329.50/165.22	v      379      380     -381     -382      383      384      385
329.50/165.22	v      386      387     -388      389     -390     -391      392
329.50/165.22	v      393      394      395      396      397      398     -399
329.50/165.22	v      400     -401      402      403     -404     -405      406
329.50/165.22	v      407     -408      409     -410     -411      412     -413
329.50/165.22	v      414     -415     -416      417      418     -419     -420
329.50/165.22	v      421      422      423     -424      425      426     -427
329.50/165.22	v     -428     -429     -430     -431     -432      433     -434
329.50/165.22	v     -435      436      437      438     -439     -440     -441
329.50/165.22	v      442      443      444     -445     -446      447      448
329.50/165.22	v      449      450     -451     -452      453     -454      455
329.50/165.22	v     -456      457      458      459      460     -461      462
329.50/165.22	v      463     -464      465     -466     -467      468     -469
329.50/165.22	v      470     -471     -472     -473      474     -475     -476
329.50/165.22	v      477     -478     -479     -480      481     -482     -483
329.50/165.22	v      484      485     -486      487      488     -489     -490
329.50/165.22	v     -491     -492      493      494     -495     -496      497
329.50/165.22	v      498     -499      500     -501     -502     -503     -504
329.50/165.22	v      505      506      507      508     -509      510      511
329.50/165.22	v     -512     -513      514     -515      516      517      518
329.50/165.22	v     -519     -520     -521     -522     -523      524      525
329.50/165.22	v      526      527      528     -529     -530      531     -532
329.50/165.22	v     -533      534     -535     -536     -537     -538      539
329.50/165.22	v      540      541     -542     -543     -544     -545     -546
329.50/165.22	v      547     -548     -549     -550      551     -552      553
329.50/165.22	v      554      555     -556     -557      558     -559     -560
329.50/165.22	v      561      562     -563     -564      565      566     -567
329.50/165.22	v     -568     -569     -570      571     -572      573     -574
329.50/165.22	v      575     -576      577      578      579     -580     -581
329.50/165.22	v      582      583     -584     -585      586     -587      588
329.50/165.22	v      589     -590      591     -592      593      594      595
329.50/165.22	v     -596      597      598     -599      600      601      602
329.50/165.22	v      603      604      605     -606      607      608     -609
329.50/165.22	v      610      611      612      613     -614     -615      616
329.50/165.22	v     -617     -618      619     -620     -621      622     -623
329.50/165.22	v     -624      625     -626      627      628      629      630
329.50/165.22	v      631     -632      633     -634     -635     -636     -637
329.50/165.22	v     -638      639      640     -641      642     -643      644
329.50/165.22	v     -645      646     -647      648      649      650      651
329.50/165.22	v     -652     -653      654     -655     -656     -657     -658
329.50/165.22	v     -659      660     -661     -662      663      664     -665
329.50/165.22	v     -666      667      668     -669     -670     -671      672
329.50/165.22	v     -673     -674      675      676      677     -678     -679
329.50/165.22	v      680      681     -682      683      684     -685      686
329.50/165.22	v     -687      688     -689     -690      691      692     -693
329.50/165.22	v      694      695     -696     -697     -698     -699     -700
329.50/165.22	v      701     -702     -703      704      705     -706      707
329.50/165.22	v     -708      709      710      711      712      713      714
329.50/165.22	v      715     -716      717     -718     -719     -720      721
329.50/165.22	v     -722     -723      724     -725      726     -727     -728
329.50/165.22	v     -729     -730      731     -732      733     -734     -735
329.50/165.22	v     -736      737      738      739     -740      741      742
329.50/165.22	v      743     -744     -745     -746     -747      748     -749
329.50/165.22	v     -750      751      752      753     -754      755      756
329.50/165.22	v     -757     -758     -759     -760      761      762      763
329.50/165.22	v      764     -765      766      767      768      769     -770
329.50/165.22	v     -771     -772     -773      774     -775     -776      777
329.50/165.22	v      778     -779      780      781      782      783     -784
329.50/165.22	v     -785      786      787      788     -789     -790     -791
329.50/165.22	v     -792      793      794      795     -796      797     -798
329.50/165.22	v      799      800      801      802      803      804     -805
329.50/165.22	v      806     -807     -808      809     -810        0
329.50/165.22	c CPU time            : 216.670 seconds
329.50/165.22	c Elapsed Time        : 165.218 seconds
329.50/165.22	c Size of search tree : 262521 nodes
329.50/165.22	c average speed       : 1211.617 nodes/sec.
329.50/165.22

Verifier Data (download as text)

OK

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node2/watcher-309721-1172546084 -o ROOT/results/node2/solver-309721-1172546084 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/309721-1172546084/kcnfs-smp /tmp/evaluation/309721-1172546084/instance-309721-1172546084.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 0.98 1.04 3/77 26389
/proc/meminfo: memFree=1672960/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=180916 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) R 26387 26389 25069 0 -1 4194304 128 0 0 0 0 0 0 0 18 0 1 0 274563364 185257984 113 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134820679 0 0 4096 0 0 0 0 17 1 0 0
/proc/26389/statm: 45229 113 48 160 0 45067 0

[startup+0.107379 s]
/proc/loadavg: 0.92 0.98 1.04 3/77 26389
/proc/meminfo: memFree=1672960/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) R 26387 26389 25069 0 -1 4194304 381 0 0 0 0 0 0 0 21 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836899 0 0 4096 0 0 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 181644

[startup+0.519422 s]
/proc/loadavg: 0.92 0.98 1.04 3/77 26389
/proc/meminfo: memFree=1672960/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 494 0 0 0 0 0 0 0 21 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 181644

[startup+1.34051 s]
/proc/loadavg: 0.92 0.98 1.04 9/86 26398
/proc/meminfo: memFree=1669872/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 494 0 0 0 0 0 0 0 21 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=0.6
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 60 0 0 0 24 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572276 0 0 4096 2 0 0 0 17 1 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=0.4
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 40 0 0 0 23 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683223 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=0.4
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 40 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567223 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=0.22
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 22 0 0 0 23 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572276 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=0.24
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 24 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567517 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26396] ppid=26389 vsize=181644 CPUtime=0.32
/proc/26396/stat : 26396 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 32 0 0 0 25 0 1 0 274563391 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577451 0 0 4096 0 0 0 0 17 0 0 0
/proc/26396/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=0.23
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 23 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683216 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
[pid=26398] ppid=26389 vsize=181644 CPUtime=0.2
/proc/26398/stat : 26398 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 20 0 0 0 25 0 1 0 274563395 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577099 0 0 4096 0 0 0 0 17 0 0 0
/proc/26398/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 2.61
Current children cumulated vsize (KiB) 1634796

[startup+2.97768 s]
/proc/loadavg: 0.92 0.98 1.04 9/86 26398
/proc/meminfo: memFree=1669680/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 494 0 0 0 0 0 0 0 21 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=1
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 100 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134675782 0 0 4096 2 0 0 0 17 1 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=0.8
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 80 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577752 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=0.8
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 80 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567399 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=0.66
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 66 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134592547 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=0.64
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 64 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567529 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26396] ppid=26389 vsize=181644 CPUtime=0.72
/proc/26396/stat : 26396 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 72 0 0 0 25 0 1 0 274563391 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580258 0 0 4096 0 0 0 0 17 0 0 0
/proc/26396/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=0.66
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 66 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580071 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
[pid=26398] ppid=26389 vsize=181644 CPUtime=0.6
/proc/26398/stat : 26398 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 60 0 0 0 25 0 1 0 274563395 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134674443 0 0 4096 0 0 0 0 17 0 0 0
/proc/26398/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 5.88
Current children cumulated vsize (KiB) 1634796

[startup+6.28002 s]
/proc/loadavg: 1.49 1.10 1.07 9/86 26398
/proc/meminfo: memFree=1669424/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 494 0 0 0 0 0 0 0 21 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=1.8
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 180 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683245 0 0 4096 2 0 0 0 17 1 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=1.6
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 160 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683195 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=1.6
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 160 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567399 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=1.5
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 150 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572958 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=1.51
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 151 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134600623 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26396] ppid=26389 vsize=181644 CPUtime=1.52
/proc/26396/stat : 26396 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 152 0 0 0 25 0 1 0 274563391 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134589849 0 0 4096 0 0 0 0 17 0 0 0
/proc/26396/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=1.51
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 151 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572276 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
[pid=26398] ppid=26389 vsize=181644 CPUtime=1.44
/proc/26398/stat : 26398 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 144 0 0 0 25 0 1 0 274563395 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580180 0 0 4096 0 0 0 0 17 0 0 0
/proc/26398/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 12.48
Current children cumulated vsize (KiB) 1634796

[startup+12.7617 s]
/proc/loadavg: 2.01 1.21 1.11 9/86 26398
/proc/meminfo: memFree=1669360/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=0
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 494 0 0 0 0 0 0 0 21 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=3.4
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 340 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683127 0 0 4096 2 0 0 0 17 1 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=3.24
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 324 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567510 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=3.2
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 320 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134589808 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=3.1
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 310 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683223 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=3.14
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 314 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683132 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26396] ppid=26389 vsize=181644 CPUtime=3.12
/proc/26396/stat : 26396 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 312 0 0 0 25 0 1 0 274563391 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134631745 0 0 4096 0 0 0 0 17 0 0 0
/proc/26396/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=3.11
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 311 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683256 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
[pid=26398] ppid=26389 vsize=181644 CPUtime=3.09
/proc/26398/stat : 26398 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 309 0 0 0 25 0 1 0 274563395 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572608 0 0 4096 0 0 0 0 17 0 0 0
/proc/26398/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 25.4
Current children cumulated vsize (KiB) 1634796

[startup+25.642 s]
/proc/loadavg: 3.03 1.48 1.20 7/84 26398
/proc/meminfo: memFree=1669824/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=7.01
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 124 0 0 0 0 701 0 15 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=7.43
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 743 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572846 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=7.3
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 730 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134636613 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=7.59
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 759 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134639250 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=7.12
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 712 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683587 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=7.14
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 714 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580277 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=7.5
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 750 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580277 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 1271508

[startup+51.2567 s]
/proc/loadavg: 4.04 1.84 1.33 7/84 26398
/proc/meminfo: memFree=1669888/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=7.01

################
# More data... #
################

/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=15.79
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 1579 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134636639 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=16.09
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 1609 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683578 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=15.69
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 1569 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683214 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=15.68
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 1568 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572774 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=16
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 1600 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683350 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 102.2
Current children cumulated vsize (KiB) 1271508

[startup+102.534 s]
/proc/loadavg: 5.15 2.48 1.57 7/84 26398
/proc/meminfo: memFree=1669888/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=7.01
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 124 0 0 0 0 701 0 15 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=33
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 3300 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134674522 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=32.89
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 3289 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577530 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=33.09
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 3309 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580220 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=32.79
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 3279 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683143 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
[pid=26395] ppid=26389 vsize=181644 CPUtime=32.75
/proc/26395/stat : 26395 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 3275 0 0 0 25 0 1 0 274563386 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134578011 0 0 4096 2 0 0 0 17 1 0 0
/proc/26395/statm: 45411 308 71 160 0 45249 0
[pid=26397] ppid=26389 vsize=181644 CPUtime=33
/proc/26397/stat : 26397 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 3300 0 0 0 25 0 1 0 274563394 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683111 0 0 4096 2 0 0 0 17 0 0 0
/proc/26397/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 204.53
Current children cumulated vsize (KiB) 1271508

[startup+162.596 s]
/proc/loadavg: 5.46 3.07 1.83 6/82 26398
/proc/meminfo: memFree=1670160/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=106.2
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 248 0 0 0 0 10620 0 16 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=53.29
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5329 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134634007 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=55.82
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5582 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683111 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=53.39
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5339 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580256 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=55.69
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5569 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683183 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 324.39
Current children cumulated vsize (KiB) 908220

Solver just ended. Dumping a history of the last processes samples

[startup+163.008 s]
/proc/loadavg: 5.46 3.07 1.83 5/82 26398
/proc/meminfo: memFree=1670168/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=106.2
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 248 0 0 0 0 10620 0 16 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 0 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=53.49
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5349 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134568118 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=56.04
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5604 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683113 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=53.6
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5360 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134578964 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
[pid=26394] ppid=26389 vsize=181644 CPUtime=55.89
/proc/26394/stat : 26394 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5589 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134636775 0 0 4096 0 0 0 0 17 1 0 0
/proc/26394/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 325.22
Current children cumulated vsize (KiB) 908220

[startup+163.828 s]
/proc/loadavg: 5.46 3.07 1.83 5/82 26398
/proc/meminfo: memFree=1670168/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=162.16
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 310 0 0 0 0 16216 0 16 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 1 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=53.89
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5389 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134567303 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=56.78
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5678 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572276 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=54.01
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5401 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134580056 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 326.84
Current children cumulated vsize (KiB) 726576

[startup+164.648 s]
/proc/loadavg: 5.27 3.07 1.83 4/81 26398
/proc/meminfo: memFree=1670368/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=162.16
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 310 0 0 0 0 16216 0 16 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 1 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=54.29
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5429 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134683111 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=57.6
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5760 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572638 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=181644 CPUtime=54.43
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5443 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572829 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 45411 308 71 160 0 45249 0
Current children cumulated CPU time (s) 328.48
Current children cumulated vsize (KiB) 726576

[startup+165.057 s]
/proc/loadavg: 5.27 3.07 1.83 4/81 26398
/proc/meminfo: memFree=1670368/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=162.16
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 310 0 0 0 0 16216 0 16 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 1 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=54.62
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5462 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134572179 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=58.01
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5801 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134569452 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=0 CPUtime=54.5
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194372 70 0 0 0 5450 0 0 0 25 0 1 0 274563366 0 0 18446744073709551615 0 0 0 0 0 0 0 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 329.29
Current children cumulated vsize (KiB) 544932

[startup+165.159 s]
/proc/loadavg: 5.27 3.07 1.83 4/81 26398
/proc/meminfo: memFree=1670368/2055920 swapFree=4192812/4192956
[pid=26389] ppid=26387 vsize=181644 CPUtime=162.16
/proc/26389/stat : 26389 (kcnfs-smp) S 26387 26389 25069 0 -1 4194304 495 310 0 0 0 0 16216 0 16 0 1 0 274563364 186003456 293 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134836797 0 0 4096 0 18446744071563356171 0 0 17 1 0 0
/proc/26389/statm: 45411 293 63 160 0 45249 0
[pid=26391] ppid=26389 vsize=181644 CPUtime=54.73
/proc/26391/stat : 26391 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5473 0 0 0 25 0 1 0 274563365 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134577577 0 0 4096 2 0 0 0 17 0 0 0
/proc/26391/statm: 45411 308 71 160 0 45249 0
[pid=26392] ppid=26389 vsize=181644 CPUtime=58.11
/proc/26392/stat : 26392 (kcnfs-smp) R 26389 26389 25069 0 -1 4194368 46 0 0 0 5811 0 0 0 25 0 1 0 274563366 186003456 308 18446744073709551615 134512640 135172044 4294956704 18446744073709551615 134674534 0 0 4096 0 0 0 0 17 1 0 0
/proc/26392/statm: 45411 308 71 160 0 45249 0
[pid=26393] ppid=26389 vsize=0 CPUtime=54.5
/proc/26393/stat : 26393 (kcnfs-smp) R 26389 26389 25069 0 -1 4194372 70 0 0 0 5450 0 0 0 25 0 1 0 274563366 0 0 18446744073709551615 0 0 0 0 0 0 2147483391 4096 2 0 0 0 17 0 0 0
/proc/26393/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 329.5
Current children cumulated vsize (KiB) 544932

Child status: 10

!!! problem with CPU time !!!
wait4(...,&childrusage) returns 26389 and gives childrusage.ru_utime.tv_sec=216 childrusage.ru_utime.tv_usec=683059 childrusage.ru_stime.tv_sec=0 childrusage.ru_stime.tv_usec=12998
CPU time returned by wait4() is 216.696
while last known CPU time is 329.5

Solver probably didn't/couldn't wait for its children
Using last known CPU time as value...

Real time (s): 165.226
CPU time (s): 329.5
CPU user time (s): 329.5
CPU system time (s): 0
CPU usage (%): 199.423
Max. virtual memory (cumulated for all children) (KiB): 1634796

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 216.683
system time used= 0.012998
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 904
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 46
involuntary context switches= 6380

runsolver used 0.159975 second user time and 0.435933 second system time

The end

Launcher Data (download as text)

Begin job on node2 on Tue Feb 27 03:14:44 UTC 2007

IDJOB= 309721
IDBENCH= 20268
IDSOLVER= 112
FILE ID= node2/309721-1172546084

PBS_JOBID= 3971638

Free space on /tmp= 66555 MiB

SOLVER NAME= KCNFS SMP
BENCH NAME= HOME/pub/bench/SAT07/random/2+p/p0.9/v810/unif2p-p0.9-v810-c2932-S1274825698-06-SAT.cnf
COMMAND LINE= /tmp/evaluation/309721-1172546084/kcnfs-smp /tmp/evaluation/309721-1172546084/instance-309721-1172546084.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node2/watcher-309721-1172546084 -o ROOT/results/node2/solver-309721-1172546084 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/309721-1172546084/kcnfs-smp /tmp/evaluation/309721-1172546084/instance-309721-1172546084.cnf            

META MD5SUM SOLVER= 23f87ccf1c743ff5b6146b444e216cd3
MD5SUM BENCH=  4b942756bd3042b1996070cebc1719eb

RANDOM SEED= 752381366

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node2.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.247
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.247
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1673368 kB
Buffers:         47260 kB
Cached:         205496 kB
SwapCached:          0 kB
Active:         249240 kB
Inactive:        63560 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1673368 kB
SwapTotal:     4192956 kB
SwapFree:      4192812 kB
Dirty:            3352 kB
Writeback:           0 kB
Mapped:          78944 kB
Slab:            55188 kB
Committed_AS:  1286120 kB
PageTables:       1812 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66555 MiB

End job on node2 on Tue Feb 27 03:17:29 UTC 2007