Trace number 3250956

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.48 1206.21

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q446.25-n35-p30-s33692332.cnf
MD5SUM1002234f62b5dfa01a75197c65e66327
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.921859
Satisfiable
(Un)Satisfiability was proved
Number of variables1285
Number of clauses166723
Sum of the clauses size334661
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2166688
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data 

0.09/0.11	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250956-1304105740.cnf 
0.09/0.11	c Using SATTIMEOUT of 4800
0.09/0.11	c Using SATRAM of 15500
0.09/0.11	c Size of subset node = 32
0.09/0.11	c Variables = 1285
0.09/0.11	c Clauses = 166723
0.09/0.11	c Literals = 334661
0.09/0.11	c Reduced size:
0.09/0.11	c Variables = 1285
0.09/0.11	c Clauses = 166723
0.09/0.11	c Literals = 334661
0.09/0.11	c a2
0.19/0.20	c a1
0.39/0.44	c Independent sub-problems = 1
0.39/0.44	c Sub-problem with 1285 variables and 166723 clauses
0.39/0.44	c using permutation method 4
0.39/0.44	c count = 1
0.39/0.44	c Ordering finished 0.410 seconds
0.39/0.44	c mapped 0.430 seconds
0.39/0.44	c permuted 0.430 seconds
0.39/0.44	c about to sort
0.39/0.45	c sorted
0.89/0.90	c clauses now 166723
0.89/0.90	c init finished 0.480 seconds
0.89/0.90	c tree built 0.890 seconds
0.89/0.90	c 
0.89/0.90	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.89/0.90	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.89/0.90	c ................................ | ..........                    |                                  0.9         0.0             
0.89/0.90	c ................................ | ..........                  1 |                                  0.9         1.1             
0.89/0.90	c ................................ | ..........                  2 |                                  0.9         2.2             
0.89/0.90	c ................................ | ..........                  3 |                                  0.9         3.4             
0.89/0.90	c ................................ | ..........                  4 |                                  0.9         4.5             
0.89/0.90	c ................................ | ..........                  5 |                                  0.9         5.6             
0.89/0.90	c ................................ | ..........               1000 |              124                 0.9      1098.9         7603
95.74/24.87	c ................................ | ..........            5904000 | 6773         609158             96.0     61493.6     31954095
191.76/49.06	c ................................ | ..........           11201000 | 13174        1147527           192.0     58338.5     53870295
287.88/73.23	c ................................ | ..........           16097000 | 19246        1649545           288.0     55892.4     70468200
383.63/97.39	c ................................ | ..........           20716000 | 26439        2120850           384.0     53945.1     87881241
479.70/121.57	c ................................ | *.........           24784000 | 31935        2558426           480.0     51632.3     91116810
575.80/145.75	c ................................ | *.........           28839000 | 38307        2993089           576.0     50066.8     96903574
671.91/169.92	c ................................ | *.........           32839000 | 43519        3435094           672.0     48866.1    107515422
767.66/194.10	c ................................ | *.........           37360000 | 50392        3876203           768.1     48642.0    117367057
863.79/218.25	c ................................ | *.........           41938000 | 54667        4312951           864.0     48538.8    128638947
959.91/242.42	c ................................ | **........           46454000 | 58878        4741606           960.0     48389.1    141119584
1055.63/266.59	c ................................ | **........           50849000 | 66709        5169561          1056.0     48152.0    152510626
1151.70/290.77	c ................................ | **........           55063000 | 72564        5609608          1152.0     47797.7    164015967
1247.83/314.94	c ................................ | **........           59031000 | 77808        6032053          1248.0     47300.5    174113812
1343.93/339.11	c ................................ | **........           62999000 | 84758        6457221          1344.0     46873.9    180245907
1441.27/363.64	c ................................ | ***.......           66720000 | 88784        6849022          1441.5     46286.4    185190800
1535.82/387.44	c ................................ | ***.......           70262000 | 92480        7228372          1536.0     45743.5    185622097
1631.96/411.61	c ................................ | ***.......           74025000 | 96387        7612306          1632.0     45358.2    195855636
1727.70/435.77	c ................................ | ***.......           77732000 | 100913       8014340          1728.0     44983.8    206319271
1823.80/459.95	c ................................ | ***.......           81622000 | 107490       8410359          1824.0     44748.4    216090128
1919.95/484.11	c ................................ | ****......           85645000 | 114778       8803016          1920.0     44606.3    217859351
2015.74/508.27	c ................................ | ****......           89770000 | 117720       9202280          2016.0     44528.3    227493336
2111.93/532.42	c ................................ | ****......           93809000 | 121902       9602186          2112.0     44417.1    240310108
2207.70/556.58	c ................................ | ****......           97450000 | 127777       9971183          2208.0     44135.0    241457507
2303.77/580.65	c ................................ | ****......          101135000 | 133347       10349644         2304.0     43895.4    246119471
2399.99/604.70	c ................................ | *****.....          104797000 | 139017       10720936         2400.0     43665.1    253886230
2495.82/628.74	c ................................ | *****.....          108378000 | 143897       11086893         2496.0     43420.7    259223566
2591.66/652.78	c ................................ | *****.....          111858000 | 150660       11431244         2592.0     43155.1    264172238
2687.89/676.83	c ................................ | *****.....          115229000 | 157463       11766676         2688.0     42867.6    268110279
2783.75/700.86	c ................................ | *****.....          118682000 | 161718       12107634         2784.0     42629.9    275964380
2881.58/725.34	c ................................ | ******....          122600000 | 169228       12455649         2881.8     42543.3    285718162
2975.83/748.94	c ................................ | ******....          126371000 | 173952       12798365         2976.0     42463.2    294954417
3071.67/772.98	c ................................ | ******....          129908000 | 180130       13132996         3072.0     42287.6    308101501
3167.89/797.03	c ................................ | ******....          133148000 | 186480       13449285         3168.0     42028.8    310237499
3263.71/821.07	c ................................ | ******....          136411000 | 193417       13764211         3264.0     41792.6    310237499
3359.93/845.11	c ................................ | *******...          139866000 | 200471       14123660         3360.0     41626.8    317816003
3455.77/869.15	c ................................ | *******...          143562000 | 206655       14478854         3456.0     41539.8    327627643
3551.99/893.20	c ................................ | *******...          147046000 | 209457       14806427         3552.0     41398.1    331414168
3647.78/917.25	c ................................ | *******...          150520000 | 212409       15151003         3648.0     41260.9    337657288
3743.64/941.29	c ................................ | *******...          153926000 | 217057       15502982         3744.0     41112.6    338770176
3839.90/965.32	c ................................ | ********..          157361000 | 219873       15853706         3840.0     40979.4    345719262
3935.73/989.36	c ................................ | ********..          160345000 | 223114       16186759         3936.0     40738.0    346622580
4031.94/1013.41	c ................................ | ********..          163449000 | 227057       16523574         4032.0     40537.7    353844455
4127.80/1037.45	c ................................ | ********..          166492000 | 230309       16843530         4128.0     40332.3    356974398
4223.67/1061.49	c ................................ | ********..          169661000 | 234829       17166834         4224.0     40165.8    359993838
4319.87/1085.53	c ................................ | *********.          172701000 | 238300       17484377         4320.0     39977.1    365387384
4415.69/1109.57	c ................................ | *********.          175777000 | 243567       17805768         4416.0     39804.6    368111838
4511.93/1133.62	c ................................ | *********.          178882000 | 248682       18126643         4512.0     39645.7    374111099
4607.78/1157.66	c ................................ | *********.          182472000 | 255778       18465721         4608.0     39598.8    380038749
4703.63/1181.69	c ................................ | *********.          186026000 | 260110       18794649         4704.0     39546.3    389055351
4799.85/1205.74	c CPU limit (4800) reached
4799.85/1205.74	c Time = 4800.01 seconds
4799.85/1205.74	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250956-1304105740/watcher-3250956-1304105740 -o /tmp/evaluation-result-3250956-1304105740/solver-3250956-1304105740 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250956-1304105740.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.17 8.09 8.06 5/181 3094
/proc/meminfo: memFree=28761132/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=9416 CPUtime=0 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) R 3092 3094 2052 0 -1 4202496 430 0 0 0 0 0 0 0 25 0 1 0 522174718 9641984 372 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3094/statm: 2354 373 147 36 0 259 0

[startup+0.0342121 s]
/proc/loadavg: 8.17 8.09 8.06 5/181 3094
/proc/meminfo: memFree=28761132/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=15396 CPUtime=0.03 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) R 3092 3094 2052 0 -1 4202496 1921 0 0 0 3 0 0 0 25 0 1 0 522174718 15765504 1863 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3094/statm: 3849 1865 148 36 0 1754 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 15396

[startup+0.100196 s]
/proc/loadavg: 8.17 8.09 8.06 5/181 3094
/proc/meminfo: memFree=28761132/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=28548 CPUtime=0.09 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) R 3092 3094 2052 0 -1 4202496 5134 0 0 0 9 0 0 0 25 0 1 0 522174718 29233152 5076 33554432000 4194304 4338100 140736869854704 18446744073709551615 4220779 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3094/statm: 7137 5076 150 36 0 5042 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28548

[startup+0.30119 s]
/proc/loadavg: 8.17 8.09 8.06 5/181 3094
/proc/meminfo: memFree=28761132/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=41272 CPUtime=0.29 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) R 3092 3094 2052 0 -1 4202496 11199 0 0 0 29 0 0 0 25 0 1 0 522174718 42262528 7970 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3094/statm: 10318 7971 155 36 0 8223 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 41272

[startup+0.701113 s]
/proc/loadavg: 8.17 8.09 8.06 5/181 3094
/proc/meminfo: memFree=28761132/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=78336 CPUtime=0.69 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) R 3092 3094 2052 0 -1 4202496 20861 0 0 0 68 1 0 0 25 0 1 0 522174718 80216064 17306 33554432000 4194304 4338100 140736869854704 18446744073709551615 4240510 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/3094/statm: 19584 17306 164 36 0 17489 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 78336

[startup+1.5009 s]
/proc/loadavg: 7.91 8.04 8.04 9/186 3099
/proc/meminfo: memFree=28657924/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=246728 CPUtime=3.26 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 36218 0 0 0 321 5 0 0 25 0 5 0 522174718 252649472 32336 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 61682 32336 176 36 0 59587 0
[pid=3094/tid=3096] ppid=3092 vsize=246728 CPUtime=0.58 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 2986 0 0 0 58 0 0 0 18 0 5 0 522174809 252649472 32336 33554432000 4194304 4338100 140736869854704 18446744073709551615 4256755 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=246728 CPUtime=0.57 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1629 0 0 0 57 0 0 0 18 0 5 0 522174809 252649472 32336 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=246728 CPUtime=0.58 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 3005 0 0 0 58 0 0 0 25 0 5 0 522174809 252649472 32336 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921473504 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=246728 CPUtime=0.59 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1505 0 0 0 58 1 0 0 18 0 5 0 522174809 252649472 32337 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3.26
Current children cumulated vsize (KiB) 246728

[startup+3.10061 s]
/proc/loadavg: 7.91 8.04 8.04 9/186 3099
/proc/meminfo: memFree=28591832/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=332108 CPUtime=9.62 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 56883 0 0 0 950 12 0 0 25 0 5 0 522174718 340078592 53001 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 83027 53001 176 36 0 80932 0
[pid=3094/tid=3096] ppid=3092 vsize=332108 CPUtime=2.18 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 9388 0 0 0 217 1 0 0 21 0 5 0 522174809 340078592 53001 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921171984 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=332108 CPUtime=2.14 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 5635 0 0 0 212 2 0 0 18 0 5 0 522174809 340078592 53001 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=332108 CPUtime=2.18 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 9658 0 0 0 217 1 0 0 25 0 5 0 522174809 340078592 53001 33554432000 4194304 4338100 140736869854704 18446744073709551615 4245192 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=332108 CPUtime=2.18 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 5109 0 0 0 214 4 0 0 19 0 5 0 522174809 340078592 53001 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 9.62
Current children cumulated vsize (KiB) 332108

[startup+6.30102 s]
/proc/loadavg: 7.91 8.04 8.04 9/186 3099
/proc/meminfo: memFree=28422180/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=486692 CPUtime=22.31 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 95413 0 0 0 2205 26 0 0 25 0 5 0 522174718 498372608 91531 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 121673 91531 176 36 0 119578 0
[pid=3094/tid=3096] ppid=3092 vsize=486692 CPUtime=5.39 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 22515 0 0 0 535 4 0 0 25 0 5 0 522174809 498372608 91531 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254816 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=486692 CPUtime=5.25 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 12538 0 0 0 520 5 0 0 18 0 5 0 522174809 498372608 91531 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=486692 CPUtime=5.36 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 22480 0 0 0 533 3 0 0 25 0 5 0 522174809 498372608 91531 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921171984 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=486692 CPUtime=5.38 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 10787 0 0 0 528 10 0 0 18 0 5 0 522174809 498372608 91531 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 22.31
Current children cumulated vsize (KiB) 486692

[startup+12.7009 s]
/proc/loadavg: 7.93 8.04 8.04 9/186 3099
/proc/meminfo: memFree=28114860/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=768212 CPUtime=47.72 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 165322 0 0 0 4717 55 0 0 25 0 5 0 522174718 786649088 161440 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 192053 161440 176 36 0 189958 0
[pid=3094/tid=3096] ppid=3092 vsize=768212 CPUtime=11.78 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 45829 0 0 0 1170 8 0 0 25 0 5 0 522174809 786649088 161440 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=768212 CPUtime=11.49 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 24878 0 0 0 1139 10 0 0 18 0 5 0 522174809 786649088 161440 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921172706 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=768212 CPUtime=11.73 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 45964 0 0 0 1166 7 0 0 25 0 5 0 522174809 786649088 161440 33554432000 4194304 4338100 140736869854704 18446744073709551615 4245224 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=768212 CPUtime=11.78 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 21558 0 0 0 1152 26 0 0 25 0 5 0 522174809 786649088 161441 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 47.72
Current children cumulated vsize (KiB) 768212

[startup+25.5005 s]
/proc/loadavg: 7.94 8.04 8.04 9/186 3100
/proc/meminfo: memFree=27587416/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=1317380 CPUtime=98.52 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 287427 0 0 0 9745 107 0 0 25 0 5 0 522174718 1348997120 283545 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 329345 283545 176 36 0 327250 0
[pid=3094/tid=3096] ppid=3092 vsize=1317380 CPUtime=24.58 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 89343 0 0 0 2439 19 0 0 25 0 5 0 522174809 1348997120 283545 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=1317380 CPUtime=23.96 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 47901 0 0 0 2377 19 0 0 18 0 5 0 522174809 1348997120 283545 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=1317380 CPUtime=24.46 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 86426 0 0 0 2431 15 0 0 25 0 5 0 522174809 1348997120 283545 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933589710 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=1317380 CPUtime=24.58 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 36664 0 0 0 2409 49 0 0 18 0 5 0 522174809 1348997120 283546 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 98.52
Current children cumulated vsize (KiB) 1317380

[startup+51.1008 s]
/proc/loadavg: 8.08 8.06 8.05 9/186 3100
/proc/meminfo: memFree=26770056/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=2092224 CPUtime=200.09 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 476700 0 0 0 19806 203 0 0 25 0 5 0 522174718 2142437376 472818 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 523056 472818 176 36 0 520961 0
[pid=3094/tid=3096] ppid=3092 vsize=2092224 CPUtime=50.19 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 163618 0 0 0 4982 37 0 0 25 0 5 0 522174809 2142437376 472818 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=2092224 CPUtime=48.88 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 75813 0 0 0 4852 36 0 0 18 0 5 0 522174809 2142437376 472820 33554432000 4194304 4338100 140736869854704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=2092224 CPUtime=49.94 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 152446 0 0 0 4969 25 0 0 25 0 5 0 522174809 2142437376 472821 33554432000 4194304 4338100 140736869854704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=2092224 CPUtime=50.17 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 57732 0 0 0 4915 102 0 0 25 0 5 0 522174809 2142437376 472821 33554432000 4194304 4338100 140736869854704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 200.09
Current children cumulated vsize (KiB) 2092224

[startup+102.306 s]
/proc/loadavg: 8.15 8.08 8.05 9/186 3102
/proc/meminfo: memFree=25588416/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=3219688 CPUtime=403.49 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 755549 0 0 0 39967 382 0 0 25 0 5 0 522174718 3296960512 749760 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 804922 749760 176 36 0 802827 0
[pid=3094/tid=3096] ppid=3092 vsize=3219688 CPUtime=101.38 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 269416 0 0 0 10074 64 0 0 25 0 5 0 522174809 3296960512 749760 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254097 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=3219688 CPUtime=98.95 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 123232 0 0 0 9826 69 0 0 18 0 5 0 522174809 3296960512 749760 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=3219688 CPUtime=100.87 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 252202 0 0 0 10045 42 0 0 19 0 5 0 522174809 3296960512 749760 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=3219688 CPUtime=101.35 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 83606 0 0 0 9932 203 0 0 25 0 5 0 522174809 3296960512 749760 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 403.49
Current children cumulated vsize (KiB) 3219688

[startup+162.301 s]
/proc/loadavg: 8.12 8.08 8.05 9/186 3103
/proc/meminfo: memFree=24964268/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=3767220 CPUtime=641.72 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 893001 0 0 0 63621 551 0 0 25 0 5 0 522174718 3857633280 887212 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 941805 887212 176 36 0 939710 0
[pid=3094/tid=3096] ppid=3092 vsize=3767220 CPUtime=161.37 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 329596 0 0 0 16051 86 0 0 25 0 5 0 522174809 3857633280 887212 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=3767220 CPUtime=157.5 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 134452 0 0 0 15657 93 0 0 18 0 5 0 522174809 3857633280 887212 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=3767220 CPUtime=160.57 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 318246 0 0 0 16002 55 0 0 25 0 5 0 522174809 3857633280 887212 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=3767220 CPUtime=161.33 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 83614 0 0 0 15821 312 0 0 25 0 5 0 522174809 3857633280 887212 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 641.72
Current children cumulated vsize (KiB) 3767220


################
# More data... #
################

/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 116770 0 0 0 68522 972 0 0 25 0 5 0 522174809 9432801280 2240312 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2789.74
Current children cumulated vsize (KiB) 9211720

[startup+762.306 s]
/proc/loadavg: 4.49 6.59 7.50 5/176 3145
/proc/meminfo: memFree=20944880/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=10065664 CPUtime=3029.36 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 2458476 0 0 0 300335 2601 0 0 25 0 5 0 522174718 10307239936 2452687 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 2516416 2452687 176 36 0 2514321 0
[pid=3094/tid=3096] ppid=3092 vsize=10065664 CPUtime=760.27 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1202755 0 0 0 75681 346 0 0 25 0 5 0 522174809 10307239936 2452687 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=10065664 CPUtime=754.1 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 226772 0 0 0 74371 1039 0 0 25 0 5 0 522174809 10307239936 2452687 33554432000 4194304 4338100 140736869854704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=10065664 CPUtime=759.14 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 885082 0 0 0 75757 157 0 0 20 0 5 0 522174809 10307239936 2452687 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=10065664 CPUtime=754.91 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 116774 0 0 0 74436 1055 0 0 25 0 5 0 522174809 10307239936 2452687 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3029.36
Current children cumulated vsize (KiB) 10065664

[startup+822.301 s]
/proc/loadavg: 4.23 6.13 7.29 5/176 3147
/proc/meminfo: memFree=20731000/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=10275240 CPUtime=3268.89 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 2511242 0 0 0 324118 2771 0 0 25 0 5 0 522174718 10521845760 2505453 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 2568810 2505453 176 36 0 2566715 0
[pid=3094/tid=3096] ppid=3092 vsize=10275240 CPUtime=820.18 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1234769 0 0 0 81658 360 0 0 25 0 5 0 522174809 10521845760 2505453 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921172354 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=10275240 CPUtime=814.04 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 230791 0 0 0 80290 1114 0 0 19 0 5 0 522174809 10521845760 2505453 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=10275240 CPUtime=818.88 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 896678 0 0 0 81729 159 0 0 25 0 5 0 522174809 10521845760 2505453 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=10275240 CPUtime=814.85 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 121911 0 0 0 80352 1133 0 0 25 0 5 0 522174809 10521845760 2505453 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3268.89
Current children cumulated vsize (KiB) 10275240

[startup+882.301 s]
/proc/loadavg: 4.08 5.74 7.08 5/176 3150
/proc/meminfo: memFree=20095156/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=10910224 CPUtime=3508.49 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 2670352 0 0 0 347860 2989 0 0 25 0 5 0 522174718 11172069376 2664563 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 2727556 2664563 176 36 0 2725461 0
[pid=3094/tid=3096] ppid=3092 vsize=10910224 CPUtime=879.87 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1331713 0 0 0 87600 387 0 0 25 0 5 0 522174809 11172069376 2664563 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=10910224 CPUtime=874.01 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 230792 0 0 0 86198 1203 0 0 25 0 5 0 522174809 11172069376 2664563 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=10910224 CPUtime=878.84 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 958842 0 0 0 87713 171 0 0 25 0 5 0 522174809 11172069376 2664563 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=10910224 CPUtime=874.84 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 121912 0 0 0 86260 1224 0 0 25 0 5 0 522174809 11172069376 2664563 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3508.49
Current children cumulated vsize (KiB) 10910224

[startup+942.301 s]
/proc/loadavg: 4.08 5.44 6.89 5/176 3151
/proc/meminfo: memFree=19823264/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=11179364 CPUtime=3748.04 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 2739645 0 0 0 371607 3197 0 0 25 0 5 0 522174718 11447668736 2733856 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 2794841 2733856 176 36 0 2792746 0
[pid=3094/tid=3096] ppid=3092 vsize=11179364 CPUtime=939.56 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1382422 0 0 0 93551 405 0 0 25 0 5 0 522174809 11447668736 2733856 33554432000 4194304 4338100 140736869854704 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=11179364 CPUtime=933.96 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 230793 0 0 0 92100 1296 0 0 25 0 5 0 522174809 11447668736 2733856 33554432000 4194304 4338100 140736869854704 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=11179364 CPUtime=938.81 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 976058 0 0 0 93706 175 0 0 25 0 5 0 522174809 11447668736 2733856 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=11179364 CPUtime=934.78 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 123279 0 0 0 92161 1317 0 0 25 0 5 0 522174809 11447668736 2733856 33554432000 4194304 4338100 140736869854704 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3748.04
Current children cumulated vsize (KiB) 11179364

[startup+1002.3 s]
/proc/loadavg: 4.03 5.17 6.71 5/176 3153
/proc/meminfo: memFree=19391416/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=11595388 CPUtime=3987.6 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 2846719 0 0 0 395357 3403 0 0 25 0 5 0 522174718 11873677312 2840930 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 2898847 2840930 176 36 0 2896752 0
[pid=3094/tid=3096] ppid=3092 vsize=11595388 CPUtime=999.44 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1458933 0 0 0 99515 429 0 0 25 0 5 0 522174809 11873677312 2840930 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=11595388 CPUtime=993.91 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 230793 0 0 0 98009 1382 0 0 20 0 5 0 522174809 11873677312 2840930 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=11595388 CPUtime=998.58 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1004224 0 0 0 99677 181 0 0 19 0 5 0 522174809 11873677312 2840930 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921172718 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=11595388 CPUtime=994.73 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 125676 0 0 0 98066 1407 0 0 25 0 5 0 522174809 11873677312 2840930 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3987.6
Current children cumulated vsize (KiB) 11595388

[startup+1062.3 s]
/proc/loadavg: 4.01 4.96 6.53 5/176 3154
/proc/meminfo: memFree=19140864/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=11842500 CPUtime=4227.25 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 2910645 0 0 0 419139 3586 0 0 25 0 5 0 522174718 12126720000 2904856 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 2960625 2904856 176 36 0 2958530 0
[pid=3094/tid=3096] ppid=3092 vsize=11842500 CPUtime=1059.27 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1506077 0 0 0 105479 448 0 0 25 0 5 0 522174809 12126720000 2904856 33554432000 4194304 4338100 140736869854704 18446744073709551615 222921172393 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=11842500 CPUtime=1053.9 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 231692 0 0 0 103927 1463 0 0 25 0 5 0 522174809 12126720000 2904856 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=11842500 CPUtime=1058.45 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1018308 0 0 0 105661 184 0 0 25 0 5 0 522174809 12126720000 2904856 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=11842500 CPUtime=1054.71 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 127475 0 0 0 103983 1488 0 0 25 0 5 0 522174809 12126720000 2904856 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4227.25
Current children cumulated vsize (KiB) 11842500

[startup+1122.3 s]
/proc/loadavg: 4.00 4.78 6.37 5/176 3156
/proc/meminfo: memFree=18701696/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=12327932 CPUtime=4466.81 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 3019300 0 0 0 442895 3786 0 0 25 0 5 0 522174718 12623802368 3013511 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 3081983 3013511 176 36 0 3079888 0
[pid=3094/tid=3096] ppid=3092 vsize=12327932 CPUtime=1118.97 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1562231 0 0 0 111430 467 0 0 25 0 5 0 522174809 12623802368 3013511 33554432000 4194304 4338100 140736869854704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=12327932 CPUtime=1113.83 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 235903 0 0 0 109836 1547 0 0 25 0 5 0 522174809 12623802368 3013511 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=12327932 CPUtime=1118.43 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1062628 0 0 0 111649 194 0 0 25 0 5 0 522174809 12623802368 3013511 33554432000 4194304 4338100 140736869854704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=12327932 CPUtime=1114.65 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 131445 0 0 0 109891 1574 0 0 25 0 5 0 522174809 12623802368 3013512 33554432000 4194304 4338100 140736869854704 18446744073709551615 4239575 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4466.81
Current children cumulated vsize (KiB) 12327932

[startup+1182.3 s]
/proc/loadavg: 4.04 4.65 6.23 5/176 3157
/proc/meminfo: memFree=18200416/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=12825284 CPUtime=4706.43 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) S 3092 3094 2052 0 -1 4202496 3144439 0 0 0 466658 3985 0 0 25 0 5 0 522174718 13133090816 3138650 33554432000 4194304 4338100 140736869854704 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 3206321 3138650 176 36 0 3204226 0
[pid=3094/tid=3096] ppid=3092 vsize=12825284 CPUtime=1178.67 cores=0,2,4,6
/proc/3094/task/3096/stat : 3096 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1639883 0 0 0 117377 490 0 0 25 0 5 0 522174809 13133090816 3138650 33554432000 4194304 4338100 140736869854704 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3094/tid=3097] ppid=3092 vsize=12825284 CPUtime=1173.8 cores=0,2,4,6
/proc/3094/task/3097/stat : 3097 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 235903 0 0 0 115750 1630 0 0 25 0 5 0 522174809 13133090816 3138650 33554432000 4194304 4338100 140736869854704 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3094/tid=3098] ppid=3092 vsize=12825284 CPUtime=1178.41 cores=0,2,4,6
/proc/3094/task/3098/stat : 3098 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 1110115 0 0 0 117637 204 0 0 25 0 5 0 522174809 13133090816 3138650 33554432000 4194304 4338100 140736869854704 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=12825284 CPUtime=1174.62 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4202560 131445 0 0 0 115805 1657 0 0 23 0 5 0 522174809 13133090816 3138650 33554432000 4194304 4338100 140736869854704 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4706.43
Current children cumulated vsize (KiB) 12825284



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.8 s]
/proc/loadavg: 4.02 4.60 6.17 2/173 3158
/proc/meminfo: memFree=19387428/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=0 CPUtime=4800.07 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) Z 3092 3094 2052 0 -1 4203548 3164827 0 0 0 475952 4055 0 0 18 0 2 0 522174718 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 0 0 0 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=0 CPUtime=1198.1 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4203588 134474 0 0 0 118119 1691 0 0 25 0 2 0 522174809 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1206.2 s]
/proc/loadavg: 4.02 4.60 6.17 2/173 3158
/proc/meminfo: memFree=19387428/32951124 swapFree=67110188/67111528
[pid=3094] ppid=3092 vsize=0 CPUtime=4800.47 cores=0,2,4,6
/proc/3094/stat : 3094 (ttsth-5-4) Z 3092 3094 2052 0 -1 4203548 3164827 0 0 0 475952 4095 0 0 18 0 2 0 522174718 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/3094/statm: 0 0 0 0 0 0 0
[pid=3094/tid=3099] ppid=3092 vsize=0 CPUtime=1198.5 cores=0,2,4,6
/proc/3094/task/3099/stat : 3099 (ttsth-5-4) R 3092 3094 2052 0 -1 4203588 134474 0 0 0 118119 1731 0 0 25 0 2 0 522174809 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.47
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1206.21
CPU time (s): 4800.48
CPU user time (s): 4759.53
CPU system time (s): 40.9558
CPU usage (%): 397.982
Max. virtual memory (cumulated for all children) (KiB): 12894100

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4759.53
system time used= 40.9558
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3164827
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 21724
involuntary context switches= 23399

runsolver used 1.73673 second user time and 3.89641 second system time

The end

Launcher Data 

Begin job on node109 at 2011-04-29 21:35:40
IDJOB=3250956
IDBENCH=88205
IDSOLVER=1654
FILE ID=node109/3250956-1304105740
RUNJOBID= node109-1304105160-3021
PBS_JOBID= 13170607
Free space on /tmp= 72384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q446.25-n35-p30-s33692332.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250956-1304105740/watcher-3250956-1304105740 -o /tmp/evaluation-result-3250956-1304105740/solver-3250956-1304105740 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250956-1304105740.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1002234f62b5dfa01a75197c65e66327
RANDOM SEED=710056716

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28761288 kB
Buffers:        733652 kB
Cached:         993524 kB
SwapCached:        712 kB
Active:        2144176 kB
Inactive:      1654136 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28761288 kB
SwapTotal:    67111528 kB
SwapFree:     67110188 kB
Dirty:            1952 kB
Writeback:           0 kB
AnonPages:     2070320 kB
Mapped:          15100 kB
Slab:           324488 kB
PageTables:       8700 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2271336 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72384 MiB
End job on node109 at 2011-04-29 21:55:48