Trace number 3250120

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.2 1203.39

General information on the benchmark

Namecrafted/manthey/sudoku-single-hard/
sudoku_81x81_unknown.cnf
MD5SUMcbd78921480e79d133eb06cb2172caa1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables18217
Number of clauses284829
Sum of the clauses size622430
Maximum clause length28
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2275099
Number of clauses of size 3620
Number of clauses of size 4984
Number of clauses of size 51120
Number of clauses of size over 57006

Solver Data 

0.19/0.22	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250120-1304089270.cnf 
0.19/0.22	c Using SATTIMEOUT of 4800
0.19/0.22	c Using SATRAM of 15500
0.19/0.22	c Size of subset node = 32
0.19/0.22	c Variables = 18217
0.19/0.22	c Clauses = 284829
0.19/0.22	c Literals = 622430
0.19/0.22	c Reduced size:
0.19/0.22	c Variables = 18217
0.19/0.22	c Clauses = 284829
0.19/0.22	c Literals = 622430
0.19/0.22	c a2
0.39/0.46	c a1
1.19/1.25	c Independent sub-problems = 1
1.19/1.25	c Sub-problem with 18217 variables and 284829 clauses
1.19/1.25	c using permutation method 4
1.19/1.25	c count = 1
1.19/1.25	c Ordering finished 1.200 seconds
1.19/1.25	c mapped 1.240 seconds
1.19/1.25	c permuted 1.240 seconds
1.19/1.25	c about to sort
1.19/1.28	c sorted
1.89/1.93	c clauses now 284829
1.89/1.93	c init finished 1.480 seconds
1.89/1.93	c tree built 1.930 seconds
1.89/1.93	c 
1.89/1.93	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.89/1.93	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.89/1.93	c ................................ | ..........                    |                                  1.9         0.0             
1.89/1.94	c ................................ | ..........                  1 |                                  1.9         0.5             
1.89/1.94	c ................................ | ..........                  2 |                                  1.9         1.0             
1.89/1.94	c ................................ | ..........                  3 |                                  1.9         1.5             
1.89/1.94	c ................................ | ..........                  4 |                                  1.9         2.1             
1.89/1.94	c ................................ | ..........                  5 |                                  1.9         2.6             
1.89/1.95	c ................................ | ..........               1000 |              57                  1.9       512.8          632
95.94/25.55	c ................................ | ..........            1416000 | 44858        347451             96.2     14725.5      9563823
191.77/49.58	c ................................ | ..........            2095000 | 72110        513258            192.1     10906.9     15825360
288.01/73.64	c ................................ | ..........            2592000 | 92945        619667            288.2      8994.4     20323296
383.84/97.65	c ................................ | ..........            3384000 | 120042       793755            384.1      8811.1     25469301
479.69/121.70	c ................................ | *.........            4152000 | 143178       972272            480.1      8648.6     28599808
575.93/145.72	c ................................ | *.........            5029000 | 178171       1157797           576.0      8730.6     33283273
671.77/169.77	c ................................ | *.........            6091000 | 234093       1318726           672.0      9063.3     37547242
768.01/193.81	c ................................ | *.........            7001000 | 277721       1465652           768.0      9115.3     41813204
863.87/217.88	c ................................ | *.........            7581000 | 308676       1571743           864.2      8772.6     45219138
960.11/241.92	c ................................ | **........            8234000 | 344102       1685653           960.2      8575.5     49151898
1055.95/265.94	c ................................ | **........            8762000 | 371371       1765698          1056.1      8296.4     51696179
1151.81/289.95	c ................................ | **........           10314000 | 424175       2154410          1152.0      8953.0     53409522
1247.69/313.98	c ................................ | **........           12458000 | 497474       2681500          1248.0      9982.1     54605055
1343.95/338.01	c ................................ | **........           14676000 | 572498       3234729          1344.0     10919.6     55870148
1439.80/362.04	c ................................ | ***.......           17002000 | 650022       3821695          1440.0     11806.9     57095133
1535.68/386.08	c ................................ | ***.......           19290000 | 727916       4390238          1536.0     12558.5     58336433
1631.95/410.11	c ................................ | ***.......           21659000 | 806481       4988704          1632.0     13271.4     59574558
1727.81/434.15	c ................................ | ***.......           23968000 | 885759       5569738          1728.0     13870.1     60775171
1823.68/458.18	c ................................ | ***.......           26376000 | 965709       6185994          1824.0     14460.3     61945485
1919.95/482.21	c ................................ | ****......           28756000 | 1047698      6776604          1920.0     14977.0     63082573
2015.80/506.25	c ................................ | ****......           31104000 | 1128788      7367228          2016.0     15428.4     64175101
2111.70/530.28	c ................................ | ****......           33524000 | 1210192      7986877          2112.0     15873.0     65264272
2207.96/554.31	c ................................ | ****......           35992000 | 1292464      8616228          2208.0     16300.7     66175610
2303.82/578.35	c ................................ | ****......           38368000 | 1375626      9201789          2304.0     16652.6     68019811
2399.67/602.38	c ................................ | *****.....           40786000 | 1457541      9810169          2400.0     16994.2     68973369
2495.93/626.42	c ................................ | *****.....           43239000 | 1538992      10428234         2496.0     17323.2     69544471
2591.79/650.45	c ................................ | *****.....           45628000 | 1621662      11015316         2592.0     17603.3     70482405
2687.66/674.48	c ................................ | *****.....           48002000 | 1703715      11611405         2688.0     17857.9     71870276
2783.93/698.53	c ................................ | *****.....           50441000 | 1785111      12244279         2784.0     18118.0     73170613
2879.78/722.56	c ................................ | ******....           52961000 | 1864727      12908020         2880.0     18389.1     74334453
2976.03/746.60	c ................................ | ******....           55428000 | 1948323      13525972         2976.0     18624.7     75711832
3071.91/770.62	c ................................ | ******....           57828000 | 2031565      14113652         3072.0     18824.2     76779487
3167.78/794.65	c ................................ | ******....           60271000 | 2114608      14720418         3168.0     19024.9     77891191
3263.65/818.69	c ................................ | ******....           62798000 | 2197687      15366290         3264.0     19239.5     79130014
3359.90/842.73	c ................................ | *******...           65198000 | 2281894      15955253         3360.0     19404.0     80118402
3455.79/866.76	c ................................ | *******...           67661000 | 2364872      16575511         3456.0     19577.7     81159725
3551.63/890.79	c ................................ | *******...           70139000 | 2448199      17200763         3552.0     19746.3     82224080
3647.92/914.82	c ................................ | *******...           72586000 | 2531770      17803165         3648.0     19897.4     82742844
3743.79/938.86	c ................................ | *******...           74983000 | 2614445      18390813         3744.0     20027.4     83102456
3839.66/962.89	c ................................ | ********..           77407000 | 2696762      19003013         3840.0     20158.0     83392941
3935.93/986.92	c ................................ | ********..           79921000 | 2779479      19625419         3936.0     20305.0     83601302
4031.82/1010.95	c ................................ | ********..           82334000 | 2861250      20226489         4032.0     20420.1     83916215
4127.68/1034.98	c ................................ | ********..           84828000 | 2944146      20844359         4128.0     20549.3     83995440
4223.94/1059.02	c ................................ | ********..           87207000 | 3026266      21433640         4224.0     20645.4     84517291
4319.82/1083.04	c ................................ | *********.           89658000 | 3108095      22038965         4320.0     20754.1     85014740
4415.69/1107.07	c ................................ | *********.           92004000 | 3188993      22618943         4416.0     20834.2     85608577
4511.96/1131.11	c ................................ | *********.           94414000 | 3271485      23226275         4512.0     20925.0     85936497
4607.82/1155.15	c ................................ | *********.           96903000 | 3355044      23854451         4608.0     21029.2     86421057
4703.71/1179.18	c ................................ | *********.           99393000 | 3437166      24479224         4704.0     21129.3     86838778
4799.96/1203.21	c CPU limit (4800) reached
4799.96/1203.21	c Time = 4800.02 seconds
4799.96/1203.21	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250120-1304089270/watcher-3250120-1304089270 -o /tmp/evaluation-result-3250120-1304089270/solver-3250120-1304089270 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250120-1304089270.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.87 7.98 7.83 6/181 20961
/proc/meminfo: memFree=29499120/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=9544 CPUtime=0 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) R 20959 20961 20765 0 -1 4202496 455 0 0 0 0 0 0 0 25 0 1 0 599157586 9773056 397 33554432000 4194304 4338100 140733469699984 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/20961/statm: 2386 398 147 36 0 291 0

[startup+0.090531 s]
/proc/loadavg: 7.87 7.98 7.83 6/181 20961
/proc/meminfo: memFree=29499120/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=24440 CPUtime=0.08 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) R 20959 20961 20765 0 -1 4202496 4157 0 0 0 8 0 0 0 25 0 1 0 599157586 25026560 4099 33554432000 4194304 4338100 140733469699984 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/20961/statm: 6110 4100 148 36 0 4015 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 24440

[startup+0.100524 s]
/proc/loadavg: 7.87 7.98 7.83 6/181 20961
/proc/meminfo: memFree=29499120/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=26564 CPUtime=0.09 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) R 20959 20961 20765 0 -1 4202496 4593 0 0 0 9 0 0 0 25 0 1 0 599157586 27201536 4535 33554432000 4194304 4338100 140733469699984 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/20961/statm: 6641 4536 148 36 0 4546 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26564

[startup+0.30049 s]
/proc/loadavg: 7.87 7.98 7.83 6/181 20961
/proc/meminfo: memFree=29499120/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=73800 CPUtime=0.29 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) R 20959 20961 20765 0 -1 4202496 16457 0 0 0 28 1 0 0 25 0 1 0 599157586 75571200 16399 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019790864 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/20961/statm: 18450 16399 151 36 0 16355 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 73800

[startup+0.700416 s]
/proc/loadavg: 7.87 7.98 7.83 6/181 20961
/proc/meminfo: memFree=29499120/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=77368 CPUtime=0.69 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) R 20959 20961 20765 0 -1 4202496 18540 0 0 0 68 1 0 0 25 0 1 0 599157586 79224832 17331 33554432000 4194304 4338100 140733469699984 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/20961/statm: 19342 17331 156 36 0 17247 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 77368

[startup+1.50027 s]
/proc/loadavg: 7.87 7.98 7.83 6/182 20962
/proc/meminfo: memFree=29428588/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=250244 CPUtime=1.49 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) R 20959 20961 20765 0 -1 4202496 61526 0 0 0 143 6 0 0 25 0 1 0 599157586 256249856 59760 33554432000 4194304 4338100 140733469699984 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/20961/statm: 62561 59762 164 36 0 60466 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 250244

[startup+3.10094 s]
/proc/loadavg: 7.64 7.93 7.81 9/186 20966
/proc/meminfo: memFree=29144224/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=630488 CPUtime=6.54 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 96969 0 0 0 639 15 0 0 25 0 5 0 599157586 645619712 94645 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 157622 94646 176 36 0 155527 0
[pid=20961/tid=20963] ppid=20959 vsize=630488 CPUtime=1.14 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 1377 0 0 0 113 1 0 0 19 0 5 0 599157779 645619712 94647 33554432000 4194304 4338100 140733469699984 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=630488 CPUtime=1.15 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 2756 0 0 0 114 1 0 0 19 0 5 0 599157779 645619712 94647 33554432000 4194304 4338100 140733469699984 18446744073709551615 260020092459 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=630488 CPUtime=1.14 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 2238 0 0 0 113 1 0 0 18 0 5 0 599157779 645619712 94647 33554432000 4194304 4338100 140733469699984 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=630488 CPUtime=1.13 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 3250 0 0 0 112 1 0 0 18 0 5 0 599157779 645619712 94647 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 6.54
Current children cumulated vsize (KiB) 630488

[startup+6.30035 s]
/proc/loadavg: 7.64 7.93 7.81 9/186 20966
/proc/meminfo: memFree=29068868/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=690580 CPUtime=19.3 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 112508 0 0 0 1902 28 0 0 25 0 5 0 599157586 707153920 110184 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 172645 110185 176 36 0 170550 0
[pid=20961/tid=20963] ppid=20959 vsize=690580 CPUtime=4.33 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 3817 0 0 0 427 6 0 0 25 0 5 0 599157779 707153920 110185 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=690580 CPUtime=4.35 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 6968 0 0 0 432 3 0 0 25 0 5 0 599157779 707153920 110185 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019791003 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=690580 CPUtime=4.35 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 6113 0 0 0 429 6 0 0 25 0 5 0 599157779 707153920 110185 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=690580 CPUtime=4.32 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 8261 0 0 0 430 2 0 0 18 0 5 0 599157779 707153920 110185 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 19.3
Current children cumulated vsize (KiB) 690580

[startup+12.7012 s]
/proc/loadavg: 7.70 7.93 7.82 9/186 20966
/proc/meminfo: memFree=28945696/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=791864 CPUtime=44.84 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 137672 0 0 0 4433 51 0 0 25 0 5 0 599157586 810868736 135348 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 197966 135348 176 36 0 195871 0
[pid=20961/tid=20963] ppid=20959 vsize=791864 CPUtime=10.69 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 8966 0 0 0 1054 15 0 0 25 0 5 0 599157779 810868736 135348 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019791598 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=791864 CPUtime=10.75 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 12726 0 0 0 1068 7 0 0 25 0 5 0 599157779 810868736 135348 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=791864 CPUtime=10.75 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 13173 0 0 0 1060 15 0 0 25 0 5 0 599157779 810868736 135348 33554432000 4194304 4338100 140733469699984 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=791864 CPUtime=10.69 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 15457 0 0 0 1066 3 0 0 18 0 5 0 599157779 810868736 135348 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 44.84
Current children cumulated vsize (KiB) 791864

[startup+25.5008 s]
/proc/loadavg: 7.74 7.93 7.82 9/186 20966
/proc/meminfo: memFree=28805572/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=906056 CPUtime=95.94 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 167142 0 0 0 9500 94 0 0 25 0 5 0 599157586 927801344 164818 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 226514 164818 176 36 0 224419 0
[pid=20961/tid=20963] ppid=20959 vsize=906056 CPUtime=23.41 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 14821 0 0 0 2308 33 0 0 25 0 5 0 599157779 927801344 164818 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=906056 CPUtime=23.55 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 20049 0 0 0 2344 11 0 0 25 0 5 0 599157779 927801344 164818 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=906056 CPUtime=23.54 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 21438 0 0 0 2322 32 0 0 25 0 5 0 599157779 927801344 164818 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=906056 CPUtime=23.46 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 23484 0 0 0 2340 6 0 0 18 0 5 0 599157779 927801344 164818 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 95.94
Current children cumulated vsize (KiB) 906056

[startup+51.1011 s]
/proc/loadavg: 7.83 7.94 7.82 9/186 20967
/proc/meminfo: memFree=28565360/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=1111488 CPUtime=198.15 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 220409 0 0 0 19656 159 0 0 25 0 5 0 599157586 1138163712 218085 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 277872 218085 176 36 0 275777 0
[pid=20961/tid=20963] ppid=20959 vsize=1111488 CPUtime=48.93 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 25982 0 0 0 4832 61 0 0 25 0 5 0 599157779 1138163712 218085 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=1111488 CPUtime=49.15 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 33882 0 0 0 4895 20 0 0 25 0 5 0 599157779 1138163712 218085 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=1111488 CPUtime=49.09 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 35347 0 0 0 4851 58 0 0 25 0 5 0 599157779 1138163712 218085 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=1111488 CPUtime=49 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 37848 0 0 0 4891 9 0 0 20 0 5 0 599157779 1138163712 218085 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 198.15
Current children cumulated vsize (KiB) 1111488

[startup+102.307 s]
/proc/loadavg: 7.93 7.94 7.83 9/186 20968
/proc/meminfo: memFree=28190828/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=1434192 CPUtime=402.63 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 301290 0 0 0 39988 275 0 0 25 0 5 0 599157586 1468612608 298966 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 358548 298966 176 36 0 356453 0
[pid=20961/tid=20963] ppid=20959 vsize=1434192 CPUtime=100.14 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 45993 0 0 0 9905 109 0 0 25 0 5 0 599157779 1468612608 298966 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019790985 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=1434192 CPUtime=100.36 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 52838 0 0 0 10002 34 0 0 25 0 5 0 599157779 1468612608 298966 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=1434192 CPUtime=100.02 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 57581 0 0 0 9895 107 0 0 25 0 5 0 599157779 1468612608 298966 33554432000 4194304 4338100 140733469699984 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=1434192 CPUtime=100.15 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 57528 0 0 0 10001 14 0 0 18 0 5 0 599157779 1468612608 298966 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 402.63
Current children cumulated vsize (KiB) 1434192

[startup+162.301 s]
/proc/loadavg: 7.97 7.95 7.83 9/186 20970
/proc/meminfo: memFree=27782448/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=1767232 CPUtime=642.22 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 388117 0 0 0 63788 434 0 0 25 0 5 0 599157586 1809645568 385793 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 441808 385793 176 36 0 439713 0
[pid=20961/tid=20963] ppid=20959 vsize=1767232 CPUtime=159.87 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 69418 0 0 0 15803 184 0 0 25 0 5 0 599157779 1809645568 385793 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=1767232 CPUtime=160.35 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 70944 0 0 0 15987 48 0 0 25 0 5 0 599157779 1809645568 385793 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=1767232 CPUtime=159.95 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 83575 0 0 0 15824 171 0 0 25 0 5 0 599157779 1809645568 385793 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=1767232 CPUtime=160.07 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 76830 0 0 0 15988 19 0 0 19 0 5 0 599157779 1809645568 385793 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 642.22
Current children cumulated vsize (KiB) 1767232

[startup+222.301 s]
/proc/loadavg: 8.10 7.99 7.85 9/186 20972
/proc/meminfo: memFree=27472504/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=2058308 CPUtime=881.82 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 463062 0 0 0 87580 602 0 0 25 0 5 0 599157586 2107707392 460738 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 514577 460738 176 36 0 512482 0
[pid=20961/tid=20963] ppid=20959 vsize=2058308 CPUtime=219.56 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 87587 0 0 0 21699 257 0 0 25 0 5 0 599157779 2107707392 460738 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 192737 0 0 0 69907 59 0 0 20 0 5 0 599157779 3287220224 759191 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2799.09
Current children cumulated vsize (KiB) 3210176

[startup+762.307 s]
/proc/loadavg: 8.06 8.02 7.91 9/186 21018
/proc/meminfo: memFree=26497520/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3304016 CPUtime=3038.77 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 796661 0 0 0 302834 1043 0 0 25 0 5 0 599157586 3383312384 794337 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 826004 794337 176 36 0 823909 0
[pid=20961/tid=20963] ppid=20959 vsize=3304016 CPUtime=757.83 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 147731 0 0 0 75345 438 0 0 25 0 5 0 599157779 3383312384 794337 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3304016 CPUtime=760.37 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 193397 0 0 0 75880 157 0 0 25 0 5 0 599157779 3383312384 794338 33554432000 4194304 4338100 140733469699984 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3304016 CPUtime=758.96 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 163478 0 0 0 75522 374 0 0 25 0 5 0 599157779 3383312384 794338 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019791598 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3304016 CPUtime=759.64 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 204706 0 0 0 75901 63 0 0 25 0 5 0 599157779 3383312384 794338 33554432000 4194304 4338100 140733469699984 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3038.77
Current children cumulated vsize (KiB) 3304016

[startup+822.301 s]
/proc/loadavg: 8.12 8.04 7.92 9/186 21020
/proc/meminfo: memFree=26131676/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3525800 CPUtime=3278.43 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 831174 0 0 0 326766 1077 0 0 25 0 5 0 599157586 3610419200 828850 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 881450 828850 176 36 0 879355 0
[pid=20961/tid=20963] ppid=20959 vsize=3525800 CPUtime=817.75 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 153432 0 0 0 81324 451 0 0 25 0 5 0 599157779 3610419200 828850 33554432000 4194304 4338100 140733469699984 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3525800 CPUtime=820.37 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 203586 0 0 0 81871 166 0 0 25 0 5 0 599157779 3610419200 828850 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3525800 CPUtime=818.73 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 169488 0 0 0 81491 382 0 0 25 0 5 0 599157779 3610419200 828850 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019790874 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3525800 CPUtime=819.62 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 217318 0 0 0 81894 68 0 0 25 0 5 0 599157779 3610419200 828850 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3278.43
Current children cumulated vsize (KiB) 3525800

[startup+882.301 s]
/proc/loadavg: 8.08 8.05 7.92 9/186 21021
/proc/meminfo: memFree=25860940/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3615644 CPUtime=3518.07 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 864695 0 0 0 350687 1120 0 0 25 0 5 0 599157586 3702419456 862371 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 903911 862371 176 36 0 901816 0
[pid=20961/tid=20963] ppid=20959 vsize=3615644 CPUtime=877.71 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 158979 0 0 0 87303 468 0 0 25 0 5 0 599157779 3702419456 862371 33554432000 4194304 4338100 140733469699984 18446744073709551615 260020057121 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3615644 CPUtime=880.37 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 215432 0 0 0 87861 176 0 0 25 0 5 0 599157779 3702419456 862371 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3615644 CPUtime=878.47 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 175078 0 0 0 87452 395 0 0 25 0 5 0 599157779 3702419456 862371 33554432000 4194304 4338100 140733469699984 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3615644 CPUtime=879.57 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 227856 0 0 0 87886 71 0 0 22 0 5 0 599157779 3702419456 862371 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3518.07
Current children cumulated vsize (KiB) 3615644

[startup+942.301 s]
/proc/loadavg: 8.08 8.05 7.93 9/186 21023
/proc/meminfo: memFree=25643772/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3650052 CPUtime=3757.76 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 885084 0 0 0 374621 1155 0 0 25 0 5 0 599157586 3737653248 882760 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 912513 882760 176 36 0 910418 0
[pid=20961/tid=20963] ppid=20959 vsize=3650052 CPUtime=937.42 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 164533 0 0 0 93257 485 0 0 25 0 5 0 599157779 3737653248 882760 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019791112 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3650052 CPUtime=940.36 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 219649 0 0 0 93855 181 0 0 25 0 5 0 599157779 3737653248 882760 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3650052 CPUtime=938.46 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 180688 0 0 0 93441 405 0 0 25 0 5 0 599157779 3737653248 882760 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019792392 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3650052 CPUtime=939.56 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 232864 0 0 0 93882 74 0 0 19 0 5 0 599157779 3737653248 882760 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3757.76
Current children cumulated vsize (KiB) 3650052

[startup+1002.3 s]
/proc/loadavg: 8.07 8.05 7.94 9/186 21024
/proc/meminfo: memFree=25384676/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3741304 CPUtime=3997.45 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 902061 0 0 0 398560 1185 0 0 25 0 5 0 599157586 3831095296 899737 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 935326 899737 176 36 0 933231 0
[pid=20961/tid=20963] ppid=20959 vsize=3741304 CPUtime=997.21 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 170180 0 0 0 99224 497 0 0 25 0 5 0 599157779 3831095296 899737 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3741304 CPUtime=1000.37 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 224006 0 0 0 99849 188 0 0 25 0 5 0 599157779 3831095296 899737 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3741304 CPUtime=998.35 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 186136 0 0 0 99421 414 0 0 25 0 5 0 599157779 3831095296 899737 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3741304 CPUtime=999.54 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 234389 0 0 0 99879 75 0 0 23 0 5 0 599157779 3831095296 899737 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3997.45
Current children cumulated vsize (KiB) 3741304

[startup+1062.3 s]
/proc/loadavg: 8.10 8.07 7.95 9/186 21026
/proc/meminfo: memFree=25176656/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3832920 CPUtime=4237.13 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 919378 0 0 0 422496 1217 0 0 25 0 5 0 599157586 3924910080 917054 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 958230 917054 176 36 0 956135 0
[pid=20961/tid=20963] ppid=20959 vsize=3832920 CPUtime=1057.05 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 175754 0 0 0 105191 514 0 0 25 0 5 0 599157779 3924910080 917054 33554432000 4194304 4338100 140733469699984 18446744073709551615 4256760 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3832920 CPUtime=1060.37 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 227673 0 0 0 105844 193 0 0 25 0 5 0 599157779 3924910080 917054 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3832920 CPUtime=1058.22 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 192143 0 0 0 105400 422 0 0 25 0 5 0 599157779 3924910080 917054 33554432000 4194304 4338100 140733469699984 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3832920 CPUtime=1059.53 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 236458 0 0 0 105876 77 0 0 24 0 5 0 599157779 3924910080 917054 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4237.13
Current children cumulated vsize (KiB) 3832920

[startup+1122.3 s]
/proc/loadavg: 8.15 8.09 7.96 9/186 21028
/proc/meminfo: memFree=24934400/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3936520 CPUtime=4476.81 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 940120 0 0 0 446434 1247 0 0 25 0 5 0 599157586 4030996480 937796 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 984130 937796 176 36 0 982035 0
[pid=20961/tid=20963] ppid=20959 vsize=3936520 CPUtime=1116.74 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 181185 0 0 0 111143 531 0 0 25 0 5 0 599157779 4030996480 937796 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032210075 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3936520 CPUtime=1120.38 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 232897 0 0 0 111839 199 0 0 25 0 5 0 599157779 4030996480 937796 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3936520 CPUtime=1118.21 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 197872 0 0 0 111394 427 0 0 25 0 5 0 599157779 4030996480 937796 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019790971 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3936520 CPUtime=1119.51 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 240816 0 0 0 111872 79 0 0 20 0 5 0 599157779 4030996480 937796 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4476.81
Current children cumulated vsize (KiB) 3936520

[startup+1182.3 s]
/proc/loadavg: 8.16 8.10 7.97 9/186 21029
/proc/meminfo: memFree=24714344/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=3984868 CPUtime=4716.47 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) S 20959 20961 20765 0 -1 4202496 960824 0 0 0 470370 1277 0 0 25 0 5 0 599157586 4080504832 958500 33554432000 4194304 4338100 140733469699984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 996217 958500 176 36 0 994122 0
[pid=20961/tid=20963] ppid=20959 vsize=3984868 CPUtime=1176.62 cores=0,2,4,6
/proc/20961/task/20963/stat : 20963 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 186613 0 0 0 117117 545 0 0 25 0 5 0 599157779 4080504832 958500 33554432000 4194304 4338100 140733469699984 18446744073709551615 260019790864 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=3984868 CPUtime=1180.37 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 235831 0 0 0 117833 204 0 0 25 0 5 0 599157779 4080504832 958500 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20961/tid=20965] ppid=20959 vsize=3984868 CPUtime=1178.03 cores=0,2,4,6
/proc/20961/task/20965/stat : 20965 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 203725 0 0 0 117368 435 0 0 25 0 5 0 599157779 4080504832 958500 33554432000 4194304 4338100 140733469699984 18446744073709551615 4242453 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20961/tid=20966] ppid=20959 vsize=3984868 CPUtime=1179.49 cores=0,2,4,6
/proc/20961/task/20966/stat : 20966 (ttsth-5-4) R 20959 20961 20765 0 -1 4202560 247305 0 0 0 117867 82 0 0 25 0 5 0 599157779 4080504832 958500 33554432000 4194304 4338100 140733469699984 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4716.47
Current children cumulated vsize (KiB) 3984868



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.3 s]
/proc/loadavg: 8.18 8.11 7.97 6/183 21030
/proc/meminfo: memFree=26590188/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) Z 20959 20961 20765 0 -1 4203548 965818 0 0 0 478714 1297 0 0 18 0 2 0 599157586 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 0 0 0 0 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=0 CPUtime=1201.38 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4203588 236052 0 0 0 119922 216 0 0 25 0 2 0 599157779 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.3 s]
/proc/loadavg: 8.18 8.11 7.97 6/183 21030
/proc/meminfo: memFree=26590188/32951124 swapFree=67111464/67111528
[pid=20961] ppid=20959 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/20961/stat : 20961 (ttsth-5-4) Z 20959 20961 20765 0 -1 4203548 965818 0 0 0 478714 1297 0 0 18 0 2 0 599157586 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/20961/statm: 0 0 0 0 0 0 0
[pid=20961/tid=20964] ppid=20959 vsize=0 CPUtime=1201.38 cores=0,2,4,6
/proc/20961/task/20964/stat : 20964 (ttsth-5-4) R 20959 20961 20765 0 -1 4203588 236052 0 0 0 119922 216 0 0 25 0 2 0 599157779 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.39
CPU time (s): 4800.2
CPU user time (s): 4787.15
CPU system time (s): 13.051
CPU usage (%): 398.89
Max. virtual memory (cumulated for all children) (KiB): 4056700

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4787.15
system time used= 13.051
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 965818
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7910
involuntary context switches= 18317

runsolver used 1.90571 second user time and 4.17537 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 17:01:10
IDJOB=3250120
IDBENCH=83244
IDSOLVER=1654
FILE ID=node110/3250120-1304089270
RUNJOBID= node110-1304088491-20783
PBS_JOBID= 13170568
Free space on /tmp= 72836 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/manthey/sudoku-single-hard/sudoku_81x81_unknown.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250120-1304089270/watcher-3250120-1304089270 -o /tmp/evaluation-result-3250120-1304089270/solver-3250120-1304089270 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250120-1304089270.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= cbd78921480e79d133eb06cb2172caa1
RANDOM SEED=1699238759

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29499524 kB
Buffers:        895432 kB
Cached:         696832 kB
SwapCached:         64 kB
Active:        1508332 kB
Inactive:      1535080 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29499524 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:            4776 kB
Writeback:           0 kB
AnonPages:     1450820 kB
Mapped:          15328 kB
Slab:           342168 kB
PageTables:       7232 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  1663756 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72812 MiB
End job on node110 at 2011-04-29 17:21:15