Trace number 3264554

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.17 1211.71

General information on the benchmark

NameSAT04/handmade/hirsch03/
hgen8-n320-01-S1432474240.shuffled-as.sat03-890.used-as.sat04-841.cnf
MD5SUMf97e40b22ad62886042ca9677daf337d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables260
Number of clauses486
Sum of the clauses size1102
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2421
Number of clauses of size 30
Number of clauses of size 465
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264554-1307649157.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 260
0.00/0.00	c Clauses = 486
0.00/0.00	c Literals = 1102
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 260
0.00/0.00	c Clauses = 486
0.00/0.00	c Literals = 1102
0.00/0.00	c a2
0.00/0.00	c a1
2.69/2.75	c Independent sub-problems = 1
2.69/2.75	c Sub-problem with 260 variables and 486 clauses
2.69/2.75	c using permutation method 3
2.69/2.75	c Ordering finished 2.740 seconds
2.69/2.75	c mapped 2.740 seconds
2.69/2.75	c permuted 2.740 seconds
2.69/2.75	c about to sort
2.69/2.75	c sorted
2.69/2.75	c clauses now 486
2.69/2.75	c init finished 2.750 seconds
2.69/2.75	c tree built 2.750 seconds
2.69/2.75	c 
2.69/2.75	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
2.69/2.75	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
2.69/2.75	c ................................ | ..........                    |                                  2.8         0.0             
2.69/2.75	c ................................ | ..........                  1 |                                  2.8         0.4             
2.69/2.75	c ................................ | ..........                  2 |                                  2.8         0.7             
2.69/2.75	c ................................ | ..........                  3 |                                  2.8         1.1             
2.69/2.75	c ................................ | ..........                  4 |                                  2.8         1.5             
2.69/2.75	c ................................ | ..........                  5 |                                  2.8         1.8             
2.69/2.75	c ................................ | ..........               1000 | 29           119                 2.8       363.6         2472
95.98/26.21	c ................................ | ..........            2363000 | 246202       402516             96.0     24601.8      6985463
191.80/50.35	c ................................ | ..........            4288000 | 455953       711271            192.0     22333.3      9645322
287.68/74.48	c ................................ | ..........            5519000 | 598435       890506            288.0     19161.9     13382388
383.85/98.63	c ................................ | ..........            6757000 | 742533       1057105           384.0     17595.9     15019510
479.67/122.78	c ................................ | *.........            8148000 | 876807       1339061           480.0     16974.6     16636722
575.96/147.02	c ................................ | *.........            9365000 | 1008688      1564434           576.1     16256.7     17665688
671.92/171.22	c ................................ | *.........           10461000 | 1126903      1762714           672.0     15566.5     18657946
767.94/195.31	c ................................ | *.........           11444000 | 1242390      1908249           768.0     14901.0     20282368
863.91/219.53	c ................................ | *.........           12326000 | 1339712      2055452           864.0     14265.4     21654737
960.09/243.70	c ................................ | **........           13430000 | 1468460      2227617           960.1     13988.3     22015536
1056.39/267.90	c ................................ | **........           14295000 | 1584086      2376155          1056.4     13531.8     22015536
1151.91/291.93	c ................................ | **........           15305000 | 1696278      2526516          1152.0     13285.0     22212756
1247.91/316.15	c ................................ | **........           16333000 | 1812551      2696707          1248.1     13086.2     22256359
1344.01/340.30	c ................................ | **........           17340000 | 1927640      2866047          1344.0     12901.6     24041052
1439.79/364.47	c ................................ | ***.......           18266000 | 2027751      3018385          1440.1     12684.0     25014351
1535.97/388.62	c ................................ | ***.......           18999000 | 2118571      3105955          1536.1     12368.7     25292824
1631.98/412.80	c ................................ | ***.......           19605000 | 2193586      3190073          1632.0     12012.8     25878406
1727.89/437.05	c ................................ | ***.......           20610000 | 2326224      3323754          1728.1     11926.3     26420641
1823.92/461.25	c ................................ | ***.......           21607000 | 2453136      3451052          1824.1     11845.1     27407790
1919.93/485.52	c ................................ | ****......           22657000 | 2578154      3580460          1920.0     11800.5     28453213
2015.87/509.74	c ................................ | ****......           23468000 | 2679325      3673606          2016.0     11640.6     28859842
2111.81/533.97	c ................................ | ****......           24428000 | 2790274      3838804          2112.1     11565.6     30544132
2207.76/558.26	c ................................ | ****......           25345000 | 2889251      4019084          2208.0     11478.6     31289591
2304.08/582.40	c ................................ | ****......           26191000 | 2990468      4164561          2304.1     11367.2     31289591
2399.92/606.62	c ................................ | *****.....           27264000 | 3113201      4327386          2400.0     11360.0     32694352
2495.70/630.78	c ................................ | *****.....           27907000 | 3188433      4456577          2496.1     11180.5     33892055
2591.81/654.96	c ................................ | *****.....           28584000 | 3270541      4583321          2592.1     11027.5     35508727
2687.94/679.16	c ................................ | *****.....           29360000 | 3368851      4685167          2688.2     10921.8     36553971
2783.89/703.23	c ................................ | *****.....           29893000 | 3434441      4799319          2784.0     10737.4     39285190
2879.90/727.45	c ................................ | ******....           30586000 | 3516801      4928732          2880.1     10619.8     39912743
2976.11/751.70	c ................................ | ******....           31421000 | 3617152      5043001          2976.1     10557.6     41898941
3071.83/775.84	c ................................ | ******....           32267000 | 3713824      5156421          3072.0     10503.6     42849517
3167.88/800.03	c ................................ | ******....           33286000 | 3829344      5299389          3168.0     10506.9     44027726
3263.75/824.17	c ................................ | ******....           34195000 | 3931791      5427516          3264.1     10476.2     45461216
3359.67/848.38	c ................................ | *******...           35041000 | 4025390      5529617          3360.0     10428.8     45828417
3455.94/872.62	c ................................ | *******...           36006000 | 4134134      5637635          3456.1     10418.2     47015352
3551.90/896.83	c ................................ | *******...           36899000 | 4232333      5753786          3552.0     10388.1     48629995
3648.30/921.11	c ................................ | *******...           37520000 | 4299557      5912278          3648.3     10284.1     51214182
3743.75/945.28	c ................................ | *******...           38230000 | 4376937      6069509          3744.1     10210.8     52437783
3839.77/969.46	c ................................ | ********..           38864000 | 4451894      6215128          3840.0     10120.7     54197142
3936.15/993.70	c ................................ | ********..           39561000 | 4525467      6368829          3936.2     10050.7     54920651
4031.67/1017.88	c ................................ | ********..           40331000 | 4607431      6510460          4032.0     10002.7     56292500
4127.77/1042.17	c ................................ | ********..           41164000 | 4692805      6654361          4128.1      9971.8     58630260
4224.03/1066.46	c ................................ | ********..           41901000 | 4772063      6764589          4224.3      9919.1     59454146
4319.76/1090.57	c ................................ | *********.           42661000 | 4858648      6857555          4320.1      9875.1     60572157
4415.85/1114.78	c ................................ | *********.           43346000 | 4935826      6962392          4416.2      9815.3     61176162
4512.01/1138.94	c ................................ | *********.           43715000 | 4979038      7021861          4512.2      9688.2     62127474
4607.63/1163.09	c ................................ | *********.           44199000 | 5035440      7094387          4608.0      9591.8     63405342
4704.16/1187.31	c ................................ | *********.           44799000 | 5110255      7177275          4704.2      9523.2     64928432

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264554-1307649157/watcher-3264554-1307649157 -o /tmp/evaluation-result-3264554-1307649157/solver-3264554-1307649157 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264554-1307649157.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 1.23 1.63 4/176 9386
/proc/meminfo: memFree=32059004/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=8840 CPUtime=0 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) R 9382 9384 8057 0 -1 4202496 313 0 0 0 0 0 0 0 22 0 1 0 876518327 9052160 255 33554432000 4194304 4338100 140737052588480 18446744073709551615 216634183840 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/9384/statm: 2210 255 163 36 0 115 0

[startup+0.0217319 s]
/proc/loadavg: 0.92 1.23 1.63 4/176 9386
/proc/meminfo: memFree=32059004/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=8840 CPUtime=0.02 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) R 9382 9384 8057 0 -1 4202496 313 0 0 0 2 0 0 0 22 0 1 0 876518327 9052160 255 33554432000 4194304 4338100 140737052588480 18446744073709551615 4301087 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/9384/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 8840

[startup+0.100755 s]
/proc/loadavg: 0.92 1.23 1.63 4/176 9386
/proc/meminfo: memFree=32059004/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=8840 CPUtime=0.09 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) R 9382 9384 8057 0 -1 4202496 313 0 0 0 9 0 0 0 22 0 1 0 876518327 9052160 255 33554432000 4194304 4338100 140737052588480 18446744073709551615 4290782 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/9384/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8840

[startup+0.300713 s]
/proc/loadavg: 0.92 1.23 1.63 4/176 9386
/proc/meminfo: memFree=32059004/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=8840 CPUtime=0.29 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) R 9382 9384 8057 0 -1 4202496 313 0 0 0 29 0 0 0 23 0 1 0 876518327 9052160 255 33554432000 4194304 4338100 140737052588480 18446744073709551615 4290947 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/9384/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 8840

[startup+0.70057 s]
/proc/loadavg: 0.92 1.23 1.63 4/176 9386
/proc/meminfo: memFree=32059004/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=8840 CPUtime=0.69 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) R 9382 9384 8057 0 -1 4202496 324 0 0 0 69 0 0 0 25 0 1 0 876518327 9052160 266 33554432000 4194304 4338100 140737052588480 18446744073709551615 4245888 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/9384/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 8840

[startup+1.50043 s]
/proc/loadavg: 0.92 1.23 1.63 7/182 9392
/proc/meminfo: memFree=31976788/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=8840 CPUtime=1.49 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) R 9382 9384 8057 0 -1 4202496 324 0 0 0 149 0 0 0 25 0 1 0 876518327 9052160 266 33554432000 4194304 4338100 140737052588480 18446744073709551615 4290772 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/9384/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 8840

[startup+3.10011 s]
/proc/loadavg: 0.92 1.23 1.63 7/182 9392
/proc/meminfo: memFree=31865844/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=199700 CPUtime=3.99 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 3754 0 0 0 399 0 0 0 25 0 5 0 876518327 204492800 3696 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 49925 3696 196 36 0 47830 0
Current children cumulated CPU time (s) 3.99
Current children cumulated vsize (KiB) 199700

[startup+6.30055 s]
/proc/loadavg: 1.49 1.35 1.66 10/186 9396
/proc/meminfo: memFree=31532120/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=325656 CPUtime=16.77 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 18402 0 0 0 1666 11 0 0 25 0 5 0 876518327 333471744 18326 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 81414 18326 198 36 0 79319 0
[pid=9384/tid=9393] ppid=9382 vsize=325656 CPUtime=3.51 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 3338 0 0 0 348 3 0 0 25 0 5 0 876518602 333471744 18326 33554432000 4194304 4338100 140737052588480 18446744073709551615 4285989 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=325656 CPUtime=3.43 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 2040 0 0 0 340 3 0 0 20 0 5 0 876518602 333471744 18326 33554432000 4194304 4338100 140737052588480 18446744073709551615 216634211866 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=325656 CPUtime=3.53 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 7460 0 0 0 352 1 0 0 25 0 5 0 876518602 333471744 18326 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=325656 CPUtime=3.51 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 4524 0 0 0 349 2 0 0 25 0 5 0 876518602 333471744 18326 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 16.77
Current children cumulated vsize (KiB) 325656

[startup+12.7004 s]
/proc/loadavg: 2.01 1.46 1.70 10/186 9397
/proc/meminfo: memFree=30920776/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=399356 CPUtime=42.27 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 37349 0 0 0 4193 34 0 0 25 0 5 0 876518327 408940544 37273 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 99839 37273 198 36 0 97744 0
[pid=9384/tid=9393] ppid=9382 vsize=399356 CPUtime=9.86 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 4934 0 0 0 975 11 0 0 25 0 5 0 876518602 408940544 37273 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=399356 CPUtime=9.82 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 4299 0 0 0 972 10 0 0 25 0 5 0 876518602 408940544 37274 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=399356 CPUtime=9.92 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 18750 0 0 0 988 4 0 0 25 0 5 0 876518602 408940544 37274 33554432000 4194304 4338100 140737052588480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=399356 CPUtime=9.89 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 8327 0 0 0 981 8 0 0 25 0 5 0 876518602 408940544 37274 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 42.27
Current children cumulated vsize (KiB) 399356

[startup+25.5012 s]
/proc/loadavg: 3.41 1.79 1.80 9/186 9397
/proc/meminfo: memFree=29926784/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=487196 CPUtime=93.19 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 58556 0 0 0 9242 77 0 0 25 0 5 0 876518327 498888704 58480 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 121799 58480 198 36 0 119704 0
[pid=9384/tid=9393] ppid=9382 vsize=487196 CPUtime=22.53 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 5879 0 0 0 2229 24 0 0 25 0 5 0 876518602 498888704 58480 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=487196 CPUtime=22.55 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 7455 0 0 0 2232 23 0 0 25 0 5 0 876518602 498888704 58480 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=487196 CPUtime=22.7 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 31106 0 0 0 2263 7 0 0 25 0 5 0 876518602 498888704 58480 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=487196 CPUtime=22.63 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 13076 0 0 0 2241 22 0 0 25 0 5 0 876518602 498888704 58480 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 93.19
Current children cumulated vsize (KiB) 487196

[startup+51.1006 s]
/proc/loadavg: 4.98 2.29 1.97 10/186 9398
/proc/meminfo: memFree=28328748/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=590196 CPUtime=194.97 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 83444 0 0 0 19353 144 0 0 25 0 5 0 876518327 604360704 83368 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 147549 83368 198 36 0 145454 0
[pid=9384/tid=9393] ppid=9382 vsize=590196 CPUtime=47.93 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 8981 0 0 0 4749 44 0 0 25 0 5 0 876518602 604360704 83368 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=590196 CPUtime=47.99 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 8971 0 0 0 4755 44 0 0 18 0 5 0 876518602 604360704 83368 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=590196 CPUtime=48.18 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 47439 0 0 0 4807 11 0 0 25 0 5 0 876518602 604360704 83368 33554432000 4194304 4338100 140737052588480 18446744073709551615 4265570 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=590196 CPUtime=48.08 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 17013 0 0 0 4764 44 0 0 18 0 5 0 876518602 604360704 83368 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 194.97
Current children cumulated vsize (KiB) 590196

[startup+102.307 s]
/proc/loadavg: 6.69 3.17 2.28 10/186 9399
/proc/meminfo: memFree=27064148/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=789148 CPUtime=398.61 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 131067 0 0 0 39632 229 0 0 25 0 5 0 876518327 808087552 130991 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 197287 130991 198 36 0 195192 0
[pid=9384/tid=9393] ppid=9382 vsize=789148 CPUtime=98.82 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 12100 0 0 0 9812 70 0 0 19 0 5 0 876518602 808087552 130991 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=789148 CPUtime=98.91 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 19375 0 0 0 9823 68 0 0 20 0 5 0 876518602 808087552 130991 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=789148 CPUtime=99.06 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 72390 0 0 0 9889 17 0 0 25 0 5 0 876518602 808087552 130991 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=789148 CPUtime=99.02 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 26162 0 0 0 9831 71 0 0 25 0 5 0 876518602 808087552 130991 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 398.61
Current children cumulated vsize (KiB) 789148

[startup+162.301 s]
/proc/loadavg: 7.57 4.06 2.64 10/186 9401
/proc/meminfo: memFree=24988356/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=880756 CPUtime=636.57 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 154694 0 0 0 63312 345 0 0 25 0 5 0 876518327 901894144 154618 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 220189 154618 198 36 0 218094 0
[pid=9384/tid=9393] ppid=9382 vsize=880756 CPUtime=158.24 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 13099 0 0 0 15712 112 0 0 25 0 5 0 876518602 901894144 154618 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=880756 CPUtime=158.38 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 19774 0 0 0 15734 104 0 0 25 0 5 0 876518602 901894144 154618 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=880756 CPUtime=158.65 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 94220 0 0 0 15842 23 0 0 25 0 5 0 876518602 901894144 154618 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=880756 CPUtime=158.52 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 26561 0 0 0 15748 104 0 0 25 0 5 0 876518602 901894144 154618 33554432000 4194304 4338100 140737052588480 18446744073709551615 4265495 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 636.57
Current children cumulated vsize (KiB) 880756

[startup+222.3 s]
/proc/loadavg: 7.89 4.79 2.98 10/186 9402
/proc/meminfo: memFree=23810436/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=1000648 CPUtime=874.98 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 185153 0 0 0 87005 493 0 0 25 0 5 0 876518327 1024663552 185074 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 250162 185074 199 36 0 248067 0
[pid=9384/tid=9393] ppid=9382 vsize=1000648 CPUtime=217.72 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 15341 0 0 0 21610 162 0 0 25 0 5 0 876518602 1024663552 185074 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=1000648 CPUtime=217.99 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 19958 0 0 0 21650 149 0 0 25 0 5 0 876518602 1024663552 185074 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=1000648 CPUtime=218.56 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 122071 0 0 0 21825 31 0 0 25 0 5 0 876518602 1024663552 185074 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=1000648 CPUtime=217.93 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 26743 0 0 0 21643 150 0 0 25 0 5 0 876518602 1024663552 185074 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 874.98
Current children cumulated vsize (KiB) 1000648

################
# More data... #
################

[pid=9384/tid=9394] ppid=9382 vsize=1718160 CPUtime=693.71 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 34871 0 0 0 68906 465 0 0 25 0 5 0 876518602 1759395840 357001 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=1718160 CPUtime=696.86 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 261404 0 0 0 69615 71 0 0 25 0 5 0 876518602 1759395840 357001 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=1718160 CPUtime=694.13 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 33001 0 0 0 68949 464 0 0 25 0 5 0 876518602 1759395840 357001 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2780.3
Current children cumulated vsize (KiB) 1718160

[startup+762.307 s]
/proc/loadavg: 8.18 7.51 5.19 10/186 9429
/proc/meminfo: memFree=16878996/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=1948032 CPUtime=3018.19 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 394893 0 0 0 300191 1628 0 0 25 0 5 0 876518327 1994784768 394814 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 487008 394814 199 36 0 484913 0
[pid=9384/tid=9393] ppid=9382 vsize=1948032 CPUtime=752.1 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 30127 0 0 0 74678 532 0 0 25 0 5 0 876518602 1994784768 394814 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=1948032 CPUtime=752.98 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 39857 0 0 0 74789 509 0 0 25 0 5 0 876518602 1994784768 394814 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=1948032 CPUtime=756.73 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 288180 0 0 0 75595 78 0 0 25 0 5 0 876518602 1994784768 394814 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=1948032 CPUtime=753.61 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 35689 0 0 0 74853 508 0 0 25 0 5 0 876518602 1994784768 394814 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3018.19
Current children cumulated vsize (KiB) 1948032

[startup+822.3 s]
/proc/loadavg: 8.17 7.63 5.38 10/186 9430
/proc/meminfo: memFree=15548688/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2036000 CPUtime=3256.59 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 422242 0 0 0 323881 1778 0 0 25 0 5 0 876518327 2084864000 422163 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 509000 422163 199 36 0 506905 0
[pid=9384/tid=9393] ppid=9382 vsize=2036000 CPUtime=811.68 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 31868 0 0 0 80585 583 0 0 19 0 5 0 876518602 2084864000 422163 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2036000 CPUtime=812.4 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 41613 0 0 0 80684 556 0 0 25 0 5 0 876518602 2084864000 422163 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2036000 CPUtime=816.48 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 310242 0 0 0 81564 84 0 0 25 0 5 0 876518602 2084864000 422163 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2036000 CPUtime=813.24 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 37479 0 0 0 80771 553 0 0 18 0 5 0 876518602 2084864000 422163 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3256.59
Current children cumulated vsize (KiB) 2036000

[startup+882.301 s]
/proc/loadavg: 6.01 7.15 5.35 5/176 9457
/proc/meminfo: memFree=30289924/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2105072 CPUtime=3494.5 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 441801 0 0 0 347610 1840 0 0 25 0 5 0 876518327 2155593728 441722 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 526268 441722 199 36 0 524173 0
[pid=9384/tid=9393] ppid=9382 vsize=2105072 CPUtime=871 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 31934 0 0 0 86499 601 0 0 18 0 5 0 876518602 2155593728 441722 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2105072 CPUtime=872.05 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 43046 0 0 0 86625 580 0 0 25 0 5 0 876518602 2155593728 441722 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2105072 CPUtime=876 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 327739 0 0 0 87510 90 0 0 20 0 5 0 876518602 2155593728 441722 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2105072 CPUtime=872.67 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 38042 0 0 0 86700 567 0 0 18 0 5 0 876518602 2155593728 441722 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3494.5
Current children cumulated vsize (KiB) 2105072

[startup+942.3 s]
/proc/loadavg: 4.55 6.54 5.25 5/176 9458
/proc/meminfo: memFree=30110968/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2256112 CPUtime=3732.21 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 486398 0 0 0 371328 1893 0 0 25 0 5 0 876518327 2310258688 486319 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 564028 486319 199 36 0 561933 0
[pid=9384/tid=9393] ppid=9382 vsize=2256112 CPUtime=930.24 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 32462 0 0 0 92408 616 0 0 23 0 5 0 876518602 2310258688 486319 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2256112 CPUtime=931.64 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 47692 0 0 0 92566 598 0 0 19 0 5 0 876518602 2310258688 486319 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2256112 CPUtime=935.45 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 365298 0 0 0 93447 98 0 0 25 0 5 0 876518602 2310258688 486319 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2256112 CPUtime=932.09 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 39906 0 0 0 92629 580 0 0 18 0 5 0 876518602 2310258688 486319 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3732.21
Current children cumulated vsize (KiB) 2256112

[startup+1002.3 s]
/proc/loadavg: 4.07 6.03 5.16 5/176 9460
/proc/meminfo: memFree=30004180/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2341464 CPUtime=3970.41 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 513076 0 0 0 395093 1948 0 0 25 0 5 0 876518327 2397659136 512997 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 585366 512997 199 36 0 583271 0
[pid=9384/tid=9393] ppid=9382 vsize=2341464 CPUtime=989.55 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 33436 0 0 0 98326 629 0 0 19 0 5 0 876518602 2397659136 512997 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2341464 CPUtime=991.09 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 49532 0 0 0 98492 617 0 0 19 0 5 0 876518602 2397659136 512997 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2341464 CPUtime=995.34 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 386619 0 0 0 99428 106 0 0 18 0 5 0 876518602 2397659136 512997 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254332 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2341464 CPUtime=991.64 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 42449 0 0 0 98570 594 0 0 25 0 5 0 876518602 2397659136 512997 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3970.41
Current children cumulated vsize (KiB) 2341464

[startup+1062.3 s]
/proc/loadavg: 4.02 5.66 5.08 5/176 9462
/proc/meminfo: memFree=29859796/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2470352 CPUtime=4207.71 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 549493 0 0 0 418767 2004 0 0 25 0 5 0 876518327 2529640448 549414 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 617588 549414 199 36 0 615493 0
[pid=9384/tid=9393] ppid=9382 vsize=2470352 CPUtime=1048.66 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 33623 0 0 0 104224 642 0 0 19 0 5 0 876518602 2529640448 549414 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2470352 CPUtime=1050.03 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 50985 0 0 0 104373 630 0 0 19 0 5 0 876518602 2529640448 549414 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2470352 CPUtime=1055.19 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 418932 0 0 0 105404 115 0 0 23 0 5 0 876518602 2529640448 549414 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2470352 CPUtime=1051.06 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 44913 0 0 0 104490 616 0 0 25 0 5 0 876518602 2529640448 549414 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4207.71
Current children cumulated vsize (KiB) 2470352

[startup+1122.3 s]
/proc/loadavg: 4.01 5.35 5.01 5/176 9463
/proc/meminfo: memFree=29777192/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2621984 CPUtime=4446.11 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 569655 0 0 0 442558 2053 0 0 25 0 5 0 876518327 2684911616 569576 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 655496 569576 199 36 0 653401 0
[pid=9384/tid=9393] ppid=9382 vsize=2621984 CPUtime=1108.14 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 33712 0 0 0 110161 653 0 0 18 0 5 0 876518602 2684911616 569576 33554432000 4194304 4338100 140737052588480 18446744073709551615 4242574 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2621984 CPUtime=1109.44 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 51609 0 0 0 110302 642 0 0 25 0 5 0 876518602 2684911616 569576 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2621984 CPUtime=1115.05 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 436416 0 0 0 111385 120 0 0 25 0 5 0 876518602 2684911616 569576 33554432000 4194304 4338100 140737052588480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2621984 CPUtime=1110.69 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 46878 0 0 0 110433 636 0 0 18 0 5 0 876518602 2684911616 569576 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646329373 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4446.11
Current children cumulated vsize (KiB) 2621984

[startup+1182.3 s]
/proc/loadavg: 4.05 5.12 4.95 5/176 9465
/proc/meminfo: memFree=29663216/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2735232 CPUtime=4684.32 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 598505 0 0 0 466338 2094 0 0 25 0 5 0 876518327 2800877568 598426 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 683808 598426 199 36 0 681713 0
[pid=9384/tid=9393] ppid=9382 vsize=2735232 CPUtime=1167.58 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 33794 0 0 0 116095 663 0 0 18 0 5 0 876518602 2800877568 598426 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2735232 CPUtime=1169.05 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 52182 0 0 0 116248 657 0 0 25 0 5 0 876518602 2800877568 598426 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2735232 CPUtime=1174.97 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 464529 0 0 0 117370 127 0 0 25 0 5 0 876518602 2800877568 598426 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2735232 CPUtime=1169.94 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 46960 0 0 0 116348 646 0 0 19 0 5 0 876518602 2800877568 598426 33554432000 4194304 4338100 140737052588480 18446744073709551615 216633910404 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4684.32
Current children cumulated vsize (KiB) 2735232



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1211.6 s]
/proc/loadavg: 4.09 5.03 4.92 5/176 9466
/proc/meminfo: memFree=29629360/32951124 swapFree=67052812/67111528
[pid=9384] ppid=9382 vsize=2747872 CPUtime=4800.05 cores=0,2,4,6
/proc/9384/stat : 9384 (ttsth-5-4) S 9382 9384 8057 0 -1 4202496 606461 0 0 0 477884 2121 0 0 25 0 5 0 876518327 2813820928 606382 33554432000 4194304 4338100 140737052588480 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/9384/statm: 686968 606382 199 36 0 684873 0
[pid=9384/tid=9393] ppid=9382 vsize=2747872 CPUtime=1196.39 cores=0,2,4,6
/proc/9384/task/9393/stat : 9393 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 35510 0 0 0 118968 671 0 0 18 0 5 0 876518602 2813820928 606382 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9384/tid=9394] ppid=9382 vsize=2747872 CPUtime=1198.03 cores=0,2,4,6
/proc/9384/task/9394/stat : 9394 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 54274 0 0 0 119136 667 0 0 25 0 5 0 876518602 2813820928 606382 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9384/tid=9395] ppid=9382 vsize=2747872 CPUtime=1204.19 cores=0,2,4,6
/proc/9384/task/9395/stat : 9395 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 468177 0 0 0 120290 129 0 0 23 0 5 0 876518602 2813820928 606382 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9384/tid=9396] ppid=9382 vsize=2747872 CPUtime=1198.66 cores=0,2,4,6
/proc/9384/task/9396/stat : 9396 (ttsth-5-4) R 9382 9384 8057 0 -1 4202560 47460 0 0 0 119214 652 0 0 18 0 5 0 876518602 2813820928 606382 33554432000 4194304 4338100 140737052588480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 2747872

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1211.71
CPU time (s): 4800.17
CPU user time (s): 4778.87
CPU system time (s): 21.3028
CPU usage (%): 396.15
Max. virtual memory (cumulated for all children) (KiB): 2747872

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4778.87
system time used= 21.3028
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 606464
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 49765
involuntary context switches= 33240

runsolver used 1.90171 second user time and 4.37434 second system time

The end

Launcher Data 

Begin job on node103 at 2011-06-09 21:52:37
IDJOB=3264554
IDBENCH=84715
IDSOLVER=1654
FILE ID=node103/3264554-1307649157
RUNJOBID= node103-1307649157-9359
PBS_JOBID= 13549388
Free space on /tmp= 74172 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/hirsch03/hgen8-n320-01-S1432474240.shuffled-as.sat03-890.used-as.sat04-841.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264554-1307649157/watcher-3264554-1307649157 -o /tmp/evaluation-result-3264554-1307649157/solver-3264554-1307649157 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264554-1307649157.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f97e40b22ad62886042ca9677daf337d
RANDOM SEED=407245083

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      32059968 kB
Buffers:        223496 kB
Cached:         433008 kB
SwapCached:      22864 kB
Active:         317964 kB
Inactive:       367372 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      32059968 kB
SwapTotal:    67111528 kB
SwapFree:     67052812 kB
Dirty:              20 kB
Writeback:           0 kB
AnonPages:       26316 kB
Mapped:           8604 kB
Slab:           141180 kB
PageTables:       4764 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184776 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74172 MiB
End job on node103 at 2011-06-09 22:12:51