Trace number 3262926

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.25 1202.33

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_26_40_6-dir.shuffled-as.sat03-163.cnf
MD5SUMc06088cac8521bfe72bbb4c9ad8d39fb
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables780
Number of clauses66810
Sum of the clauses size134348
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 266784
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 526

Solver Data 

0.04/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262926-1304095027.cnf 
0.04/0.04	c Using SATTIMEOUT of 4800
0.04/0.04	c Using SATRAM of 15500
0.04/0.04	c Size of subset node = 32
0.04/0.04	c Variables = 780
0.04/0.04	c Clauses = 66810
0.04/0.04	c Literals = 134348
0.04/0.04	c Reduced size:
0.04/0.04	c Variables = 780
0.04/0.04	c Clauses = 66810
0.04/0.04	c Literals = 134348
0.04/0.04	c a2
0.04/0.09	c a1
0.09/0.18	c Independent sub-problems = 1
0.09/0.18	c Sub-problem with 780 variables and 66810 clauses
0.09/0.18	c using permutation method 4
0.09/0.18	c count = 1
0.09/0.18	c Ordering finished 0.160 seconds
0.09/0.18	c mapped 0.170 seconds
0.09/0.18	c permuted 0.170 seconds
0.09/0.18	c about to sort
0.09/0.19	c sorted
0.29/0.35	c clauses now 66810
0.29/0.35	c init finished 0.190 seconds
0.29/0.35	c tree built 0.340 seconds
0.29/0.35	c 
0.29/0.35	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.35	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.35	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.35	c ................................ | ..........                  1 |                                  0.3         2.9             
0.29/0.35	c ................................ | ..........                  2 |                                  0.3         5.9             
0.29/0.35	c ................................ | ..........                  3 |                                  0.3         8.8             
0.29/0.35	c ................................ | ..........                  4 |                                  0.3        11.8             
0.29/0.35	c ................................ | ..........                  5 |                                  0.3        14.7             
0.29/0.35	c ................................ | ..........               1000 | 1            31                  0.3      2857.1         1830
95.98/24.31	c ................................ | ..........            6236000 | 142783       481692             96.0     64944.8     10270029
191.85/48.34	c ................................ | ..........            9717000 | 259464       813181            192.0     50606.7     15896912
287.70/72.38	c ................................ | ..........           13190000 | 346392       1080648           288.0     45793.8     21059254
383.94/96.43	c ................................ | ..........           15655000 | 426460       1293317           384.1     40762.9     24617193
479.76/120.46	c ................................ | *.........           18111000 | 512989       1482006           480.0     37730.5     26458054
575.62/144.49	c ................................ | *.........           21665000 | 600284       1708443           576.0     37612.8     30951645
671.86/168.54	c ................................ | *.........           24436000 | 685505       1979483           672.0     36360.4     34734950
767.72/192.58	c ................................ | *.........           26910000 | 758017       2181618           768.0     35038.2     39392995
863.95/216.61	c ................................ | *.........           29185000 | 837138       2367492           864.0     33778.5     40916089
959.80/240.66	c ................................ | **........           31195000 | 913901       2533634           960.1     32492.4     40916089
1055.64/264.69	c ................................ | **........           33288000 | 993602       2735709          1056.0     31522.7     41065310
1151.90/288.74	c ................................ | **........           35295000 | 1064882      2914616          1152.1     30636.4     43060921
1247.76/312.76	c ................................ | **........           36893000 | 1114216      3049307          1248.0     29561.2     44990193
1344.01/336.80	c ................................ | **........           38491000 | 1162289      3187093          1344.0     28638.7     49575252
1439.89/360.83	c ................................ | ***.......           39977000 | 1211985      3304729          1440.0     27761.8     52501836
1535.72/384.87	c ................................ | ***.......           40956000 | 1249710      3415324          1536.0     26663.9     53865878
1631.99/408.92	c ................................ | ***.......           42023000 | 1297073      3523885          1632.1     25748.0     54701484
1727.82/432.95	c ................................ | ***.......           43305000 | 1346863      3665403          1728.0     25060.3     56370493
1823.63/456.99	c ................................ | ***.......           44776000 | 1402128      3799980          1824.0     24547.8     57107998
1919.88/481.04	c ................................ | ****......           46110000 | 1452344      3912915          1920.1     24014.5     57107998
2015.75/505.07	c ................................ | ****......           47420000 | 1503162      4013542          2016.1     23521.1     57520380
2112.04/529.11	c ................................ | ****......           48682000 | 1545207      4130816          2112.1     23049.4     59389560
2207.88/553.13	c ................................ | ****......           49775000 | 1586066      4236011          2208.0     22542.9     60523773
2304.14/577.23	c ................................ | ****......           50634000 | 1625917      4346552          2304.3     21973.8     61535461
2400.00/601.24	c ................................ | *****.....           51280000 | 1663566      4405129          2400.2     21365.2     63801358
2495.86/625.24	c ................................ | *****.....           52488000 | 1716655      4492132          2496.0     21028.7     67091566
2591.71/649.29	c ................................ | *****.....           53661000 | 1759211      4585684          2592.1     20701.7     70836632
2687.96/673.32	c ................................ | *****.....           54634000 | 1803733      4668981          2688.1     20324.8     72744104
2783.84/697.34	c ................................ | *****.....           55383000 | 1843079      4754171          2784.0     19893.2     75539727
2880.06/721.41	c ................................ | ******....           56191000 | 1885377      4866476          2880.1     19509.9     78302904
2975.93/745.48	c ................................ | ******....           56692000 | 1916240      4949317          2976.2     19048.2     79850351
3071.76/769.47	c ................................ | ******....           57407000 | 1958241      5017557          3072.1     18686.8     81308950
3168.02/793.50	c ................................ | ******....           58771000 | 2023078      5113408          3168.0     18551.2     82214444
3263.86/817.55	c ................................ | ******....           59991000 | 2063525      5243529          3264.1     18379.3     85560335
3359.75/841.56	c ................................ | *******...           60955000 | 2106197      5370684          3360.0     18141.4     88530385
3455.99/865.63	c ................................ | *******...           61516000 | 2137608      5446526          3456.1     17799.2     89187683
3552.22/889.71	c ................................ | *******...           62658000 | 2190179      5535207          3552.3     17638.9     89861966
3647.69/913.68	c ................................ | *******...           63395000 | 2229198      5605801          3648.0     17377.9     91587486
3743.93/937.71	c ................................ | *******...           64442000 | 2281577      5681913          3744.0     17212.1     93087186
3839.75/961.76	c ................................ | ********..           65582000 | 2339549      5752493          3840.0     17078.6     93872946
3935.98/985.83	c ................................ | ********..           66292000 | 2373425      5816799          3936.1     16841.9     94228107
4031.86/1009.84	c ................................ | ********..           67722000 | 2419344      5894905          4032.0     16796.1     94909016
4127.67/1033.88	c ................................ | ********..           71150000 | 2507447      5993365          4128.0     17235.9     96753036
4223.92/1057.93	c ................................ | ********..           73783000 | 2579042      6080138          4224.1     17467.4     98629223
4319.77/1081.96	c ................................ | *********.           75648000 | 2651287      6147285          4320.0     17511.1    100619544
4416.00/1106.00	c ................................ | *********.           78772000 | 2729425      6233557          4416.0     17837.8    103699409
4511.84/1130.04	c ................................ | *********.           81135000 | 2799227      6327741          4512.0     17982.0    104686415
4607.69/1154.08	c ................................ | *********.           83008000 | 2862495      6407371          4608.0     18013.8    104686415
4703.97/1178.11	c ................................ | *********.           84796000 | 2929214      6476580          4704.0     18026.3    105369285
4799.76/1202.18	c CPU limit (4800) reached
4799.76/1202.18	c Time = 4800.09 seconds
4799.76/1202.18	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262926-1304095027/watcher-3262926-1304095027 -o /tmp/evaluation-result-3262926-1304095027/solver-3262926-1304095027 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262926-1304095027.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.34 7.70 7.88 3/174 899
/proc/meminfo: memFree=30880936/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=9112 CPUtime=0 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) R 897 899 32461 0 -1 4202496 358 0 0 0 0 0 0 0 22 0 1 0 599733382 9330688 301 33554432000 4194304 4338100 140735489797200 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/899/statm: 2278 302 147 36 0 183 0

[startup+0.044444 s]
/proc/loadavg: 6.34 7.70 7.88 3/174 899
/proc/meminfo: memFree=30880936/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=16684 CPUtime=0.04 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) R 897 899 32461 0 -1 4202496 2202 0 0 0 4 0 0 0 22 0 1 0 599733382 17084416 2145 33554432000 4194304 4338100 140735489797200 18446744073709551615 4221392 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/899/statm: 4171 2145 150 36 0 2076 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 16684

[startup+0.10042 s]
/proc/loadavg: 6.34 7.70 7.88 3/174 899
/proc/meminfo: memFree=30880936/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=23476 CPUtime=0.09 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) R 897 899 32461 0 -1 4202496 3859 0 0 0 9 0 0 0 22 0 1 0 599733382 24039424 3801 33554432000 4194304 4338100 140735489797200 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/899/statm: 5869 3817 153 36 0 3774 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23476

[startup+0.300375 s]
/proc/loadavg: 6.34 7.70 7.88 3/174 899
/proc/meminfo: memFree=30880936/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=43356 CPUtime=0.29 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) R 897 899 32461 0 -1 4202496 10417 0 0 0 28 1 0 0 23 0 1 0 599733382 44396544 8843 33554432000 4194304 4338100 140735489797200 18446744073709551615 218880230936 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/899/statm: 10839 8843 164 36 0 8744 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 43356

[startup+0.70023 s]
/proc/loadavg: 6.34 7.70 7.88 3/174 899
/proc/meminfo: memFree=30880936/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=238184 CPUtime=1.73 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 14801 0 0 0 170 3 0 0 23 0 5 0 599733382 243900416 13227 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 59546 13227 176 36 0 57451 0
Current children cumulated CPU time (s) 1.73
Current children cumulated vsize (KiB) 238184

[startup+1.50109 s]
/proc/loadavg: 6.34 7.70 7.88 6/182 911
/proc/meminfo: memFree=30814896/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=250960 CPUtime=4.93 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 18090 0 0 0 488 5 0 0 23 0 5 0 599733382 256983040 16516 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 62740 16516 176 36 0 60645 0
[pid=899/tid=907] ppid=897 vsize=250960 CPUtime=1.14 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 1956 0 0 0 114 0 0 0 19 0 5 0 599733417 256983040 16516 33554432000 4194304 4338100 140735489797200 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=908] ppid=897 vsize=250960 CPUtime=1.14 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 487 0 0 0 113 1 0 0 25 0 5 0 599733417 256983040 16516 33554432000 4194304 4338100 140735489797200 18446744073709551615 218879960116 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=909] ppid=897 vsize=250960 CPUtime=1.13 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 1402 0 0 0 113 0 0 0 25 0 5 0 599733417 256983040 16516 33554432000 4194304 4338100 140735489797200 18446744073709551615 218880261577 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=911] ppid=897 vsize=250960 CPUtime=1.14 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 1444 0 0 0 113 1 0 0 19 0 5 0 599733417 256983040 16516 33554432000 4194304 4338100 140735489797200 18446744073709551615 4285730 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.93
Current children cumulated vsize (KiB) 250960

[startup+3.10081 s]
/proc/loadavg: 6.34 7.70 7.88 9/186 915
/proc/meminfo: memFree=30767992/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=273516 CPUtime=11.32 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 24242 0 0 0 1121 11 0 0 23 0 5 0 599733382 280080384 22668 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 68379 22668 176 36 0 66284 0
[pid=899/tid=907] ppid=897 vsize=273516 CPUtime=2.74 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 4102 0 0 0 273 1 0 0 23 0 5 0 599733417 280080384 22668 33554432000 4194304 4338100 140735489797200 18446744073709551615 4256167 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=908] ppid=897 vsize=273516 CPUtime=2.73 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 1199 0 0 0 270 3 0 0 25 0 5 0 599733417 280080384 22668 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=909] ppid=897 vsize=273516 CPUtime=2.73 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 2881 0 0 0 273 0 0 0 25 0 5 0 599733417 280080384 22668 33554432000 4194304 4338100 140735489797200 18446744073709551615 4256294 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=911] ppid=897 vsize=273516 CPUtime=2.74 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 3259 0 0 0 270 4 0 0 19 0 5 0 599733417 280080384 22668 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.32
Current children cumulated vsize (KiB) 273516

[startup+6.30025 s]
/proc/loadavg: 6.55 7.72 7.89 9/186 915
/proc/meminfo: memFree=30484000/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=321920 CPUtime=24.09 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 36270 0 0 0 2386 23 0 0 23 0 5 0 599733382 329646080 34696 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 80480 34696 176 36 0 78385 0
[pid=899/tid=907] ppid=897 vsize=321920 CPUtime=5.94 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 8801 0 0 0 591 3 0 0 25 0 5 0 599733417 329646080 34696 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892377806 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=908] ppid=897 vsize=321920 CPUtime=5.94 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 2667 0 0 0 586 8 0 0 25 0 5 0 599733417 329646080 34696 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=909] ppid=897 vsize=321920 CPUtime=5.9 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 5773 0 0 0 590 0 0 0 25 0 5 0 599733417 329646080 34696 33554432000 4194304 4338100 140735489797200 18446744073709551615 218879960187 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=911] ppid=897 vsize=321920 CPUtime=5.93 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 6228 0 0 0 584 9 0 0 25 0 5 0 599733417 329646080 34696 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.09
Current children cumulated vsize (KiB) 321920

[startup+12.7011 s]
/proc/loadavg: 6.67 7.73 7.89 9/186 915
/proc/meminfo: memFree=30137460/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=420872 CPUtime=49.66 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 60792 0 0 0 4914 52 0 0 23 0 5 0 599733382 430972928 59218 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 105218 59218 176 36 0 103123 0
[pid=899/tid=907] ppid=897 vsize=420872 CPUtime=12.31 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 18434 0 0 0 1225 6 0 0 25 0 5 0 599733417 430972928 59218 33554432000 4194304 4338100 140735489797200 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=908] ppid=897 vsize=420872 CPUtime=12.34 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 5297 0 0 0 1214 20 0 0 25 0 5 0 599733417 430972928 59218 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=909] ppid=897 vsize=420872 CPUtime=12.29 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 14066 0 0 0 1227 2 0 0 25 0 5 0 599733417 430972928 59218 33554432000 4194304 4338100 140735489797200 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=420872 CPUtime=12.33 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 10194 0 0 0 1213 20 0 0 25 0 5 0 599733417 430972928 59218 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.66
Current children cumulated vsize (KiB) 420872

[startup+25.5009 s]
/proc/loadavg: 6.96 7.74 7.89 9/186 915
/proc/meminfo: memFree=29602280/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=576284 CPUtime=100.77 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 100440 0 0 0 9954 123 0 0 23 0 5 0 599733382 590114816 98866 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 144071 98866 176 36 0 141976 0
[pid=899/tid=907] ppid=897 vsize=576284 CPUtime=25.04 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 35743 0 0 0 2490 14 0 0 25 0 5 0 599733417 590114816 98866 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=908] ppid=897 vsize=576284 CPUtime=25.13 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 9264 0 0 0 2464 49 0 0 25 0 5 0 599733417 590114816 98866 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=909] ppid=897 vsize=576284 CPUtime=25.08 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 25391 0 0 0 2503 5 0 0 25 0 5 0 599733417 590114816 98866 33554432000 4194304 4338100 140735489797200 18446744073709551615 4230663 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=576284 CPUtime=25.13 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 17241 0 0 0 2462 51 0 0 25 0 5 0 599733417 590114816 98867 33554432000 4194304 4338100 140735489797200 18446744073709551615 4257000 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.77
Current children cumulated vsize (KiB) 576284

[startup+51.1004 s]
/proc/loadavg: 7.32 7.76 7.89 9/186 916
/proc/meminfo: memFree=29006172/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=774616 CPUtime=203.02 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 150424 0 0 0 20042 260 0 0 23 0 5 0 599733382 793206784 148850 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 193654 148850 176 36 0 191559 0
[pid=899/tid=907] ppid=897 vsize=774616 CPUtime=50.5 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 59223 0 0 0 5026 24 0 0 25 0 5 0 599733417 793206784 148850 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=908] ppid=897 vsize=774616 CPUtime=50.72 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 13970 0 0 0 4962 110 0 0 25 0 5 0 599733417 793206784 148850 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=909] ppid=897 vsize=774616 CPUtime=50.68 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 40306 0 0 0 5058 10 0 0 25 0 5 0 599733417 793206784 148850 33554432000 4194304 4338100 140735489797200 18446744073709551615 218879960489 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=774616 CPUtime=50.72 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 24124 0 0 0 4960 112 0 0 25 0 5 0 599733417 793206784 148850 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 203.02
Current children cumulated vsize (KiB) 774616

[startup+102.306 s]
/proc/loadavg: 7.76 7.81 7.90 9/186 919
/proc/meminfo: memFree=28093000/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=1049824 CPUtime=407.53 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 219563 0 0 0 40211 542 0 0 23 0 5 0 599733382 1075019776 217989 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 262456 217989 176 36 0 260361 0
[pid=899/tid=907] ppid=897 vsize=1049824 CPUtime=101.41 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 91572 0 0 0 10099 42 0 0 25 0 5 0 599733417 1075019776 217989 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=908] ppid=897 vsize=1049824 CPUtime=101.93 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 22363 0 0 0 9953 240 0 0 25 0 5 0 599733417 1075019776 217989 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=909] ppid=897 vsize=1049824 CPUtime=101.87 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 59116 0 0 0 10171 16 0 0 25 0 5 0 599733417 1075019776 217989 33554432000 4194304 4338100 140735489797200 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=1049824 CPUtime=101.93 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 33711 0 0 0 9953 240 0 0 25 0 5 0 599733417 1075019776 217989 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.53
Current children cumulated vsize (KiB) 1049824

[startup+162.301 s]
/proc/loadavg: 8.10 7.91 7.93 9/186 922
/proc/meminfo: memFree=27267376/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=1327504 CPUtime=647.1 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 291133 0 0 0 63834 876 0 0 23 0 5 0 599733382 1359364096 289559 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 331876 289559 176 36 0 329781 0
[pid=899/tid=907] ppid=897 vsize=1327504 CPUtime=161.06 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 132347 0 0 0 16041 65 0 0 25 0 5 0 599733417 1359364096 289559 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=908] ppid=897 vsize=1327504 CPUtime=161.89 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 25918 0 0 0 15796 393 0 0 25 0 5 0 599733417 1359364096 289559 33554432000 4194304 4338100 140735489797200 18446744073709551615 218879961603 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=909] ppid=897 vsize=1327504 CPUtime=161.86 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 79852 0 0 0 16162 24 0 0 25 0 5 0 599733417 1359364096 289559 33554432000 4194304 4338100 140735489797200 18446744073709551615 218879960545 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=1327504 CPUtime=161.91 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 40215 0 0 0 15799 392 0 0 25 0 5 0 599733417 1359364096 289559 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 647.1
Current children cumulated vsize (KiB) 1327504


################
# More data... #
################

/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 87141 0 0 0 68557 1446 0 0 25 0 5 0 599733417 2812268544 651214 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2803.79
Current children cumulated vsize (KiB) 2746356

[startup+762.306 s]
/proc/loadavg: 7.98 7.97 7.95 9/186 938
/proc/meminfo: memFree=22632528/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=2889096 CPUtime=3043.44 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 690201 0 0 0 301069 3275 0 0 23 0 5 0 599733382 2958434304 688627 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 722274 688627 176 36 0 720179 0
[pid=899/tid=907] ppid=897 vsize=2889096 CPUtime=760.65 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 299461 0 0 0 75897 168 0 0 25 0 5 0 599733417 2958434304 688627 33554432000 4194304 4338100 140735489797200 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=2889096 CPUtime=760.9 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 70793 0 0 0 74578 1512 0 0 25 0 5 0 599733417 2958434304 688627 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=909] ppid=897 vsize=2889096 CPUtime=761.78 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 216564 0 0 0 76107 71 0 0 25 0 5 0 599733417 2958434304 688627 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=2889096 CPUtime=759.74 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 90582 0 0 0 74453 1521 0 0 25 0 5 0 599733417 2958434304 688627 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3043.44
Current children cumulated vsize (KiB) 2889096

[startup+822.301 s]
/proc/loadavg: 8.07 8.00 7.96 9/186 939
/proc/meminfo: memFree=22164384/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3060516 CPUtime=3283.04 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 733951 0 0 0 324794 3510 0 0 23 0 5 0 599733382 3133968384 732377 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 765129 732377 176 36 0 763034 0
[pid=899/tid=907] ppid=897 vsize=3060516 CPUtime=820.61 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 319254 0 0 0 81881 180 0 0 25 0 5 0 599733417 3133968384 732377 33554432000 4194304 4338100 140735489797200 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=3060516 CPUtime=820.56 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 75616 0 0 0 80433 1623 0 0 25 0 5 0 599733417 3133968384 732377 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=909] ppid=897 vsize=3060516 CPUtime=821.77 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 230094 0 0 0 82101 76 0 0 25 0 5 0 599733417 3133968384 732377 33554432000 4194304 4338100 140735489797200 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3060516 CPUtime=819.73 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 96186 0 0 0 80344 1629 0 0 25 0 5 0 599733417 3133968384 732377 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3283.04
Current children cumulated vsize (KiB) 3060516

[startup+882.3 s]
/proc/loadavg: 7.89 7.97 7.95 9/186 942
/proc/meminfo: memFree=21751296/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3165456 CPUtime=3522.66 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 761313 0 0 0 348523 3743 0 0 23 0 5 0 599733382 3241426944 759739 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 791364 759739 176 36 0 789269 0
[pid=899/tid=907] ppid=897 vsize=3165456 CPUtime=880.6 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 328889 0 0 0 87874 186 0 0 25 0 5 0 599733417 3241426944 759739 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=3165456 CPUtime=880.21 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 81651 0 0 0 86287 1734 0 0 25 0 5 0 599733417 3241426944 759739 33554432000 4194304 4338100 140735489797200 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=909] ppid=897 vsize=3165456 CPUtime=881.75 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 235141 0 0 0 88097 78 0 0 25 0 5 0 599733417 3241426944 759739 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3165456 CPUtime=879.7 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 102831 0 0 0 86229 1741 0 0 25 0 5 0 599733417 3241426944 759739 33554432000 4194304 4338100 140735489797200 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3522.66
Current children cumulated vsize (KiB) 3165456

[startup+942.301 s]
/proc/loadavg: 8.00 7.99 7.96 9/186 944
/proc/meminfo: memFree=21351852/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3288900 CPUtime=3762.3 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 792628 0 0 0 372281 3949 0 0 23 0 5 0 599733382 3367833600 791054 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 822225 791054 176 36 0 820130 0
[pid=899/tid=907] ppid=897 vsize=3288900 CPUtime=940.6 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 348974 0 0 0 93861 199 0 0 25 0 5 0 599733417 3367833600 791054 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=3288900 CPUtime=939.87 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 87112 0 0 0 92158 1829 0 0 25 0 5 0 599733417 3367833600 791054 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=909] ppid=897 vsize=3288900 CPUtime=941.75 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 236445 0 0 0 94096 79 0 0 25 0 5 0 599733417 3367833600 791054 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3288900 CPUtime=939.69 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 107296 0 0 0 92130 1839 0 0 25 0 5 0 599733417 3367833600 791054 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3762.3
Current children cumulated vsize (KiB) 3288900

[startup+1002.3 s]
/proc/loadavg: 7.63 7.89 7.93 9/186 945
/proc/meminfo: memFree=21031996/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3327756 CPUtime=4001.88 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 802708 0 0 0 396069 4119 0 0 23 0 5 0 599733382 3407622144 801134 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 831939 801134 176 36 0 829844 0
[pid=899/tid=907] ppid=897 vsize=3327756 CPUtime=1000.59 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 356680 0 0 0 99854 205 0 0 25 0 5 0 599733417 3407622144 801134 33554432000 4194304 4338100 140735489797200 18446744073709551615 4256374 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=3327756 CPUtime=999.52 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 88426 0 0 0 98041 1911 0 0 25 0 5 0 599733417 3407622144 801134 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=909] ppid=897 vsize=3327756 CPUtime=1001.73 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 236692 0 0 0 100094 79 0 0 25 0 5 0 599733417 3407622144 801134 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3327756 CPUtime=999.66 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 108109 0 0 0 98044 1922 0 0 25 0 5 0 599733417 3407622144 801134 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4001.88
Current children cumulated vsize (KiB) 3327756

[startup+1062.3 s]
/proc/loadavg: 7.91 7.92 7.93 9/186 947
/proc/meminfo: memFree=20636160/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3470892 CPUtime=4241.49 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 838582 0 0 0 419825 4324 0 0 23 0 5 0 599733382 3554193408 837008 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 867723 837008 176 36 0 865628 0
[pid=899/tid=907] ppid=897 vsize=3470892 CPUtime=1060.56 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 391504 0 0 0 105828 228 0 0 25 0 5 0 599733417 3554193408 837008 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=3470892 CPUtime=1059.48 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 88640 0 0 0 103947 2001 0 0 25 0 5 0 599733417 3554193408 837008 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=899/tid=909] ppid=897 vsize=3470892 CPUtime=1061.71 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 237329 0 0 0 106092 79 0 0 25 0 5 0 599733417 3554193408 837008 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3470892 CPUtime=1059.34 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 108308 0 0 0 103922 2012 0 0 25 0 5 0 599733417 3554193408 837008 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4241.49
Current children cumulated vsize (KiB) 3470892

[startup+1122.3 s]
/proc/loadavg: 7.84 7.89 7.91 9/186 948
/proc/meminfo: memFree=20171868/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3648040 CPUtime=4481.08 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 888739 0 0 0 443591 4517 0 0 23 0 5 0 599733382 3735592960 887165 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 912010 887165 176 36 0 909915 0
[pid=899/tid=907] ppid=897 vsize=3648040 CPUtime=1120.54 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 428036 0 0 0 111805 249 0 0 25 0 5 0 599733417 3735592960 887165 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=908] ppid=897 vsize=3648040 CPUtime=1119.26 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 90300 0 0 0 109839 2087 0 0 19 0 5 0 599733417 3735592960 887165 33554432000 4194304 4338100 140735489797200 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=909] ppid=897 vsize=3648040 CPUtime=1121.72 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 247081 0 0 0 112088 84 0 0 25 0 5 0 599733417 3735592960 887165 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3648040 CPUtime=1119.18 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 110521 0 0 0 109823 2095 0 0 25 0 5 0 599733417 3735592960 887165 33554432000 4194304 4338100 140735489797200 18446744073709551615 4283813 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4481.08
Current children cumulated vsize (KiB) 3648040

[startup+1182.3 s]
/proc/loadavg: 7.98 7.92 7.92 9/186 950
/proc/meminfo: memFree=19905340/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=3701384 CPUtime=4720.73 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) S 897 899 32461 0 -1 4202496 899382 0 0 0 467360 4713 0 0 23 0 5 0 599733382 3790217216 897808 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 925346 897808 176 36 0 923251 0
[pid=899/tid=907] ppid=897 vsize=3701384 CPUtime=1180.22 cores=1,3,5,7
/proc/899/task/907/stat : 907 (ttsth-5-4) R 897 899 32461 0 -1 4202560 436149 0 0 0 117762 260 0 0 25 0 5 0 599733417 3790217216 897808 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=899/tid=908] ppid=897 vsize=3701384 CPUtime=1179.24 cores=1,3,5,7
/proc/899/task/908/stat : 908 (ttsth-5-4) R 897 899 32461 0 -1 4202560 91051 0 0 0 115747 2177 0 0 25 0 5 0 599733417 3790217216 897808 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=899/tid=909] ppid=897 vsize=3701384 CPUtime=1181.71 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4202560 247535 0 0 0 118087 84 0 0 25 0 5 0 599733417 3790217216 897808 33554432000 4194304 4338100 140735489797200 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=899/tid=911] ppid=897 vsize=3701384 CPUtime=1179.17 cores=1,3,5,7
/proc/899/task/911/stat : 911 (ttsth-5-4) R 897 899 32461 0 -1 4202560 111846 0 0 0 115728 2189 0 0 25 0 5 0 599733417 3790217216 897808 33554432000 4194304 4338100 140735489797200 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4720.73
Current children cumulated vsize (KiB) 3701384



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 7.98 7.92 7.92 6/183 951
/proc/meminfo: memFree=20224508/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=0 CPUtime=4800.12 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) Z 897 899 32461 0 -1 4203548 913651 0 0 0 475204 4808 0 0 18 0 2 0 599733382 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 0 0 0 0 0 0 0
[pid=899/tid=909] ppid=897 vsize=0 CPUtime=1201.62 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4203588 247656 0 0 0 120075 87 0 0 25 0 2 0 599733417 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.32 s]
/proc/loadavg: 7.98 7.92 7.92 6/183 951
/proc/meminfo: memFree=20224508/32950928 swapFree=67111524/67111528
[pid=899] ppid=897 vsize=0 CPUtime=4800.23 cores=1,3,5,7
/proc/899/stat : 899 (ttsth-5-4) Z 897 899 32461 0 -1 4203548 913651 0 0 0 475204 4819 0 0 18 0 2 0 599733382 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/899/statm: 0 0 0 0 0 0 0
[pid=899/tid=909] ppid=897 vsize=0 CPUtime=1201.72 cores=1,3,5,7
/proc/899/task/909/stat : 909 (ttsth-5-4) R 897 899 32461 0 -1 4203588 247656 0 0 0 120075 97 0 0 25 0 2 0 599733417 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.33
CPU time (s): 4800.25
CPU user time (s): 4752.05
CPU system time (s): 48.2037
CPU usage (%): 399.246
Max. virtual memory (cumulated for all children) (KiB): 3758480

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4752.05
system time used= 48.2037
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 913651
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6778
involuntary context switches= 29628

runsolver used 2.10668 second user time and 4.46632 second system time

The end

Launcher Data 

Begin job on node112 at 2011-04-29 18:37:07
IDJOB=3262926
IDBENCH=83903
IDSOLVER=1654
FILE ID=node112/3262926-1304095027
RUNJOBID= node112-1304095027-886
PBS_JOBID= 13170569
Free space on /tmp= 72564 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_26_40_6-dir.shuffled-as.sat03-163.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262926-1304095027/watcher-3262926-1304095027 -o /tmp/evaluation-result-3262926-1304095027/solver-3262926-1304095027 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262926-1304095027.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c06088cac8521bfe72bbb4c9ad8d39fb
RANDOM SEED=1308197822

node112.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5261.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30880968 kB
Buffers:        439248 kB
Cached:        1162136 kB
SwapCached:          4 kB
Active:         132196 kB
Inactive:      1537804 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30880968 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1704 kB
Writeback:           0 kB
AnonPages:       68396 kB
Mapped:          14496 kB
Slab:           334580 kB
PageTables:       4756 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   191100 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72564 MiB
End job on node112 at 2011-04-29 18:57:11