Trace number 3264334

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.32 1212.02

General information on the benchmark

NameSAT04/handmade/anton/l5k3/
lksat-n900-m3357-k3-l5-s950595579.used-as.sat04-938.cnf
MD5SUMf7a218f6e4ba6f28584266f788a5d6da
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.013997
Satisfiable
(Un)Satisfiability was proved
Number of variables895
Number of clauses3357
Sum of the clauses size10071
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 33357
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264334-1304099406.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 895
0.00/0.00	c Clauses = 3357
0.00/0.00	c Literals = 10071
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 895
0.00/0.00	c Clauses = 3255
0.00/0.00	c Literals = 9765
0.00/0.00	c a2
0.00/0.00	c a1
13.29/13.34	c Independent sub-problems = 1
13.29/13.34	c Sub-problem with 864 variables and 3255 clauses
13.29/13.34	c using permutation method 3
13.29/13.34	c Ordering finished 13.330 seconds
13.29/13.34	c mapped 13.330 seconds
13.29/13.34	c permuted 13.330 seconds
13.29/13.34	c about to sort
13.29/13.34	c sorted
13.29/13.35	c clauses now 3255
13.29/13.35	c init finished 13.340 seconds
13.29/13.35	c tree built 13.350 seconds
13.29/13.35	c 
13.29/13.35	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
13.29/13.35	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
13.29/13.35	c ................................ | ..........                    |                                 13.3         0.0             
13.29/13.35	c ................................ | ..........                  1 |                                 13.3         0.1             
13.29/13.35	c ................................ | ..........                  2 |                                 13.3         0.1             
13.29/13.35	c ................................ | ..........                  3 |                                 13.3         0.2             
13.29/13.35	c ................................ | ..........                  4 |                                 13.3         0.3             
13.29/13.35	c ................................ | ..........                  5 |                                 13.3         0.4             
13.29/13.36	c ................................ | ..........               1000 | 28           359                13.3        74.9          819
95.82/34.09	c ................................ | ..........             535000 | 56557        248868             96.2      5561.9      9406930
192.07/58.16	c ................................ | ..........             739000 | 82677        362128            192.3      3842.6     14720368
287.95/82.16	c ................................ | ..........             878000 | 101001       447227            288.2      3046.4     19105778
384.21/106.26	c ................................ | ..........            1034000 | 121058       513724            384.4      2689.6     22678286
480.48/130.33	c ................................ | *.........            1176000 | 138913       571986            480.6      2446.8     25655559
576.35/154.36	c ................................ | *.........            1302000 | 153488       626302            576.6      2258.1     28430293
671.84/178.25	c ................................ | *.........            1390000 | 165481       679828            672.0      2068.3     30930600
768.89/202.53	c ................................ | *.........            1471000 | 177052       746112            769.0      1912.8     34119157
864.36/226.42	c ................................ | *.........            1550000 | 186434       799215            864.4      1793.1     36445205
961.04/250.62	c ................................ | **........            1622000 | 196519       846289            961.1      1687.6     38382807
1055.70/274.39	c ................................ | **........            1698000 | 206744       890912           1056.1      1607.8     40558176
1152.37/298.59	c ................................ | **........            1769000 | 216954       928219           1152.8      1534.6     42449865
1247.85/322.44	c ................................ | **........            1833000 | 225998       967103           1248.0      1468.7     44171427
1344.52/346.67	c ................................ | **........            1915000 | 236979       1005206          1344.8      1424.0     45934978
1441.19/370.83	c ................................ | ***.......            1980000 | 246370       1040085          1441.3      1373.7     47367864
1537.86/395.01	c ................................ | ***.......            2045000 | 255107       1079718          1537.9      1329.7     49318400
1631.74/418.58	c ................................ | ***.......            2104000 | 263807       1111196          1632.1      1289.2     51234494
1728.01/442.66	c ................................ | ***.......            2166000 | 271362       1147083          1728.2      1253.3     52887427
1825.88/467.14	c ................................ | ***.......            2224000 | 279545       1178479          1826.1      1217.9     54277318
1921.34/491.05	c ................................ | ****......            2279000 | 288198       1213255          1921.5      1186.0     55667033
2018.01/515.21	c ................................ | ****......            2332000 | 295454       1250385          2018.0      1155.6     57156159
2111.88/538.75	c ................................ | ****......            2382000 | 302313       1295884          2112.1      1127.8     59386877
2208.95/563.04	c ................................ | ****......            2447000 | 311175       1340976          2209.1      1107.7     61317066
2306.04/587.33	c ................................ | ****......            2501000 | 318635       1384991          2306.2      1084.5     63220613
2401.49/611.25	c ................................ | *****.....            2551000 | 325406       1428089          2401.7      1062.2     65266311
2496.56/635.01	c ................................ | *****.....            2601000 | 332174       1472303          2496.6      1041.8     67338014
2593.61/659.32	c ................................ | *****.....            2653000 | 339720       1507508          2593.7      1022.9     69374683
2689.48/683.34	c ................................ | *****.....            2699000 | 346216       1540002          2689.7      1003.5     71156118
2784.16/707.00	c ................................ | *****.....            2743000 | 352598       1570241          2784.2       985.2     72909478
2880.83/731.29	c ................................ | ******....            2789000 | 359119       1600860          2881.2       968.0     74512946
2975.88/755.07	c ................................ | ******....            2833000 | 365217       1632084          2976.2       951.9     75810277
3072.15/779.17	c ................................ | ******....            2889000 | 372196       1666669          3072.4       940.3     76862473
3169.20/803.49	c ................................ | ******....            2955000 | 379399       1699227          3169.6       932.3     78172011
3263.89/827.19	c ................................ | ******....            3013000 | 386800       1730424          3264.2       923.0     79332623
3359.74/851.16	c ................................ | *******...            3071000 | 394464       1756279          3360.0       914.0     80531478
3457.35/875.73	c ................................ | *******...            3123000 | 401633       1782184          3457.5       903.3     82142696
3551.87/899.44	c ................................ | *******...            3172000 | 408775       1807110          3552.1       893.0     83101540
3648.54/923.67	c ................................ | *******...            3227000 | 416640       1834012          3648.8       884.4     84049634
3744.81/947.70	c ................................ | *******...            3276000 | 423496       1861710          3744.8       874.8     84700465
3840.28/971.67	c ................................ | ********..            3321000 | 429532       1885338          3840.6       864.7     85910404
3936.55/995.79	c ................................ | ********..            3365000 | 435615       1910294          3936.9       854.7     86826703
4032.82/1019.83	c ................................ | ********..            3413000 | 442103       1936463          4033.0       846.3     87604860
4129.12/1043.91	c ................................ | ********..            3456000 | 448478       1962127          4129.1       837.0     88566777
4224.97/1067.94	c ................................ | ********..            3492000 | 453423       1987514          4225.1       826.5     89554559
4322.02/1092.23	c ................................ | *********.            3533000 | 459436       2011709          4322.1       817.4     89944754
4415.88/1115.79	c ................................ | *********.            3573000 | 464824       2036230          4416.3       809.1     90189939
4512.95/1140.02	c ................................ | *********.            3612000 | 470288       2061010          4513.1       800.3     90552987
4608.41/1163.99	c ................................ | *********.            3665000 | 477369       2087245          4608.8       795.2     91100210
4705.09/1188.12	c ................................ | *********.            3714000 | 484152       2112786          4705.2       789.3     91868074

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264334-1304099406/watcher-3264334-1304099406 -o /tmp/evaluation-result-3264334-1304099406/solver-3264334-1304099406 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264334-1304099406.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.53 7.97 7.98 3/176 3422
/proc/meminfo: memFree=27651444/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=0 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 474 0 0 0 0 0 0 0 21 0 1 0 521538283 9871360 416 33554432000 4194304 4338100 140735377491872 18446744073709551615 218905391813 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/3422/statm: 2410 416 150 36 0 315 0

[startup+0.0301681 s]
/proc/loadavg: 7.53 7.97 7.98 3/176 3422
/proc/meminfo: memFree=27651444/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=0.02 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 514 0 0 0 2 0 0 0 21 0 1 0 521538283 9871360 456 33554432000 4194304 4338100 140735377491872 18446744073709551615 4290779 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 456 164 36 0 315 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 9640

[startup+0.101168 s]
/proc/loadavg: 7.53 7.97 7.98 3/176 3422
/proc/meminfo: memFree=27651444/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=0.09 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 514 0 0 0 9 0 0 0 21 0 1 0 521538283 9871360 456 33554432000 4194304 4338100 140735377491872 18446744073709551615 4304695 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 456 164 36 0 315 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 9640

[startup+0.30012 s]
/proc/loadavg: 7.53 7.97 7.98 3/176 3422
/proc/meminfo: memFree=27651444/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=0.29 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 514 0 0 0 29 0 0 0 22 0 1 0 521538283 9871360 456 33554432000 4194304 4338100 140735377491872 18446744073709551615 4207055 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 456 164 36 0 315 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 9640

[startup+0.701046 s]
/proc/loadavg: 7.53 7.97 7.98 3/176 3422
/proc/meminfo: memFree=27651444/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=0.69 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 525 0 0 0 69 0 0 0 25 0 1 0 521538283 9871360 467 33554432000 4194304 4338100 140735377491872 18446744073709551615 4245888 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 467 174 36 0 315 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 9640

[startup+1.5009 s]
/proc/loadavg: 7.53 7.97 7.98 7/182 3429
/proc/meminfo: memFree=27615212/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=1.49 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 525 0 0 0 149 0 0 0 25 0 1 0 521538283 9871360 467 33554432000 4194304 4338100 140735377491872 18446744073709551615 4304744 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 467 174 36 0 315 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 9640

[startup+3.10062 s]
/proc/loadavg: 7.33 7.92 7.96 7/182 3429
/proc/meminfo: memFree=27593512/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=3.09 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 525 0 0 0 309 0 0 0 25 0 1 0 521538283 9871360 467 33554432000 4194304 4338100 140735377491872 18446744073709551615 4290866 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 467 174 36 0 315 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 9640

[startup+6.30104 s]
/proc/loadavg: 7.33 7.92 7.96 7/182 3429
/proc/meminfo: memFree=27544060/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=6.29 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 527 0 0 0 629 0 0 0 25 0 1 0 521538283 9871360 469 33554432000 4194304 4338100 140735377491872 18446744073709551615 4245898 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 469 176 36 0 315 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 9640

[startup+12.7009 s]
/proc/loadavg: 6.97 7.82 7.93 7/182 3429
/proc/meminfo: memFree=27466620/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=9640 CPUtime=12.69 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) R 3420 3422 2524 0 -1 4202496 527 0 0 0 1269 0 0 0 25 0 1 0 521538283 9871360 469 33554432000 4194304 4338100 140735377491872 18446744073709551615 4308149 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3422/statm: 2410 469 176 36 0 315 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 9640

[startup+25.5005 s]
/proc/loadavg: 7.13 7.83 7.93 10/186 3434
/proc/meminfo: memFree=27136496/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=412848 CPUtime=61.85 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 57805 0 0 0 6168 17 0 0 25 0 5 0 521538283 422756352 57746 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 103212 57746 198 36 0 101117 0
[pid=3422/tid=3430] ppid=3420 vsize=412848 CPUtime=12.13 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 12928 0 0 0 1209 4 0 0 25 0 5 0 521539619 422756352 57746 33554432000 4194304 4338100 140735377491872 18446744073709551615 4257010 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=412848 CPUtime=12.13 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 11543 0 0 0 1211 2 0 0 25 0 5 0 521539619 422756352 57746 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=412848 CPUtime=12.13 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 17376 0 0 0 1209 4 0 0 25 0 5 0 521539619 422756352 57746 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=412848 CPUtime=12.07 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 12769 0 0 0 1203 4 0 0 25 0 5 0 521539619 422756352 57746 33554432000 4194304 4338100 140735377491872 18446744073709551615 4240530 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 61.85
Current children cumulated vsize (KiB) 412848

[startup+51.1009 s]
/proc/loadavg: 7.56 7.87 7.94 10/186 3435
/proc/meminfo: memFree=26793620/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=611708 CPUtime=164.11 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 108003 0 0 0 16378 33 0 0 25 0 5 0 521538283 626388992 107945 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 152927 107945 198 36 0 150832 0
[pid=3422/tid=3430] ppid=3420 vsize=611708 CPUtime=37.73 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 22607 0 0 0 3767 6 0 0 25 0 5 0 521539619 626388992 107946 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=611708 CPUtime=37.73 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 20551 0 0 0 3768 5 0 0 25 0 5 0 521539619 626388992 107946 33554432000 4194304 4338100 140735377491872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=611708 CPUtime=37.73 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 36056 0 0 0 3763 10 0 0 25 0 5 0 521539619 626388992 107946 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=611708 CPUtime=37.54 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 25602 0 0 0 3744 10 0 0 25 0 5 0 521539619 626388992 107946 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 164.11
Current children cumulated vsize (KiB) 611708

[startup+102.307 s]
/proc/loadavg: 7.96 7.92 7.95 10/186 3436
/proc/meminfo: memFree=26279616/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=898536 CPUtime=368.66 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 180389 0 0 0 36810 56 0 0 25 0 5 0 521538283 920100864 180330 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 224634 180330 198 36 0 222539 0
[pid=3422/tid=3430] ppid=3420 vsize=898536 CPUtime=88.76 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 35320 0 0 0 8866 10 0 0 25 0 5 0 521539619 920100864 180330 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=898536 CPUtime=88.94 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 34524 0 0 0 8884 10 0 0 25 0 5 0 521539619 920100864 180330 33554432000 4194304 4338100 140735377491872 18446744073709551615 4230627 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=898536 CPUtime=88.93 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 61479 0 0 0 8875 18 0 0 25 0 5 0 521539619 920100864 180330 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=898536 CPUtime=88.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 45877 0 0 0 8848 16 0 0 25 0 5 0 521539619 920100864 180330 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 368.66
Current children cumulated vsize (KiB) 898536

[startup+162.301 s]
/proc/loadavg: 8.11 7.97 7.96 10/186 3437
/proc/meminfo: memFree=25836772/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=1128736 CPUtime=608.31 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 238493 0 0 0 60756 75 0 0 25 0 5 0 521538283 1155825664 238434 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 282184 238434 198 36 0 280089 0
[pid=3422/tid=3430] ppid=3420 vsize=1128736 CPUtime=148.42 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 48275 0 0 0 14828 14 0 0 25 0 5 0 521539619 1155825664 238434 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=1128736 CPUtime=148.93 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 44574 0 0 0 14879 14 0 0 25 0 5 0 521539619 1155825664 238434 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=1128736 CPUtime=148.93 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 81699 0 0 0 14869 24 0 0 25 0 5 0 521539619 1155825664 238434 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=1128736 CPUtime=148.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 60756 0 0 0 14843 21 0 0 25 0 5 0 521539619 1155825664 238434 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 608.31
Current children cumulated vsize (KiB) 1128736

[startup+222.301 s]
/proc/loadavg: 8.15 8.00 7.98 10/186 3439
/proc/meminfo: memFree=25410936/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=1340232 CPUtime=847.99 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 291653 0 0 0 84708 91 0 0 25 0 5 0 521538283 1372397568 291594 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 335058 291594 198 36 0 332963 0
[pid=3422/tid=3430] ppid=3420 vsize=1340232 CPUtime=208.09 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 57083 0 0 0 20793 16 0 0 25 0 5 0 521539619 1372397568 291594 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=1340232 CPUtime=208.93 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 58564 0 0 0 20874 19 0 0 25 0 5 0 521539619 1372397568 291594 33554432000 4194304 4338100 140735377491872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=1340232 CPUtime=208.92 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 99365 0 0 0 20863 29 0 0 25 0 5 0 521539619 1372397568 291594 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=1340232 CPUtime=208.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 73452 0 0 0 20841 23 0 0 25 0 5 0 521539619 1372397568 291594 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 847.99
Current children cumulated vsize (KiB) 1340232

[startup+282.301 s]
/proc/loadavg: 8.16 8.03 7.99 10/186 3441
/proc/meminfo: memFree=25126948/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=1503860 CPUtime=1087.67 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 332466 0 0 0 108661 106 0 0 25 0 5 0 521538283 1539952640 332407 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 375965 332407 199 36 0 373870 0
[pid=3422/tid=3430] ppid=3420 vsize=1503860 CPUtime=267.78 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 65103 0 0 0 26759 19 0 0 25 0 5 0 521539619 1539952640 332407 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=1503860 CPUtime=268.94 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 69405 0 0 0 26871 23 0 0 25 0 5 0 521539619 1539952640 332407 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=1503860 CPUtime=268.93 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 112504 0 0 0 26858 35 0 0 25 0 5 0 521539619 1539952640 332407 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=1503860 CPUtime=268.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 82265 0 0 0 26837 27 0 0 25 0 5 0 521539619 1539952640 332407 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1087.67

################
# More data... #
################

/proc/meminfo: memFree=23172288/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=2609900 CPUtime=3005.06 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 612167 0 0 0 300295 211 0 0 25 0 5 0 521538283 2672537600 612100 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 652475 612100 199 36 0 650380 0
[pid=3422/tid=3430] ppid=3420 vsize=2609900 CPUtime=746.1 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 112959 0 0 0 74571 39 0 0 25 0 5 0 521539619 2672537600 612100 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=2609900 CPUtime=748.37 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 128895 0 0 0 74791 46 0 0 25 0 5 0 521539619 2672537600 612100 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=2609900 CPUtime=748.56 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 213382 0 0 0 74785 71 0 0 25 0 5 0 521539619 2672537600 612100 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=2609900 CPUtime=748.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 153742 0 0 0 74811 53 0 0 25 0 5 0 521539619 2672537600 612100 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3005.06
Current children cumulated vsize (KiB) 2609900

[startup+822.301 s]
/proc/loadavg: 8.12 8.10 8.02 10/186 3465
/proc/meminfo: memFree=23012300/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=2707152 CPUtime=3244.69 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 637064 0 0 0 324248 221 0 0 25 0 5 0 521538283 2772123648 636989 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 676788 636989 199 36 0 674693 0
[pid=3422/tid=3430] ppid=3420 vsize=2707152 CPUtime=806.09 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 118958 0 0 0 80567 42 0 0 25 0 5 0 521539619 2772123648 636989 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=2707152 CPUtime=808.04 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 133250 0 0 0 80756 48 0 0 25 0 5 0 521539619 2772123648 636989 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=2707152 CPUtime=808.55 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 222403 0 0 0 80780 75 0 0 25 0 5 0 521539619 2772123648 636989 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=2707152 CPUtime=808.63 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 159264 0 0 0 80809 54 0 0 25 0 5 0 521539619 2772123648 636989 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3244.69
Current children cumulated vsize (KiB) 2707152

[startup+882.3 s]
/proc/loadavg: 8.44 8.17 8.04 10/186 3470
/proc/meminfo: memFree=22839984/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=2814272 CPUtime=3483.55 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 663590 0 0 0 348124 231 0 0 25 0 5 0 521538283 2881814528 663515 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 703568 663515 199 36 0 701473 0
[pid=3422/tid=3430] ppid=3420 vsize=2814272 CPUtime=866.09 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 125150 0 0 0 86565 44 0 0 25 0 5 0 521539619 2881814528 663515 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=2814272 CPUtime=866.89 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 137964 0 0 0 86639 50 0 0 25 0 5 0 521539619 2881814528 663515 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=2814272 CPUtime=868.55 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 232576 0 0 0 86776 79 0 0 25 0 5 0 521539619 2881814528 663515 33554432000 4194304 4338100 140735377491872 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=2814272 CPUtime=868.63 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 164711 0 0 0 86807 56 0 0 25 0 5 0 521539619 2881814528 663515 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3483.55
Current children cumulated vsize (KiB) 2814272

[startup+942.3 s]
/proc/loadavg: 8.22 8.15 8.04 10/186 3471
/proc/meminfo: memFree=22705620/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=2877472 CPUtime=3723.23 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 680153 0 0 0 372083 240 0 0 25 0 5 0 521538283 2946531328 680078 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 719368 680078 199 36 0 717273 0
[pid=3422/tid=3430] ppid=3420 vsize=2877472 CPUtime=926.09 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 130268 0 0 0 92563 46 0 0 25 0 5 0 521539619 2946531328 680078 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=2877472 CPUtime=926.57 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 141925 0 0 0 92605 52 0 0 25 0 5 0 521539619 2946531328 680078 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=2877472 CPUtime=928.55 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 237852 0 0 0 92772 83 0 0 25 0 5 0 521539619 2946531328 680078 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=2877472 CPUtime=928.63 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 166919 0 0 0 92806 57 0 0 25 0 5 0 521539619 2946531328 680078 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3723.23
Current children cumulated vsize (KiB) 2877472

[startup+1002.3 s]
/proc/loadavg: 8.19 8.15 8.05 10/186 3473
/proc/meminfo: memFree=22563348/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=2956440 CPUtime=3962.91 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 699986 0 0 0 396044 247 0 0 25 0 5 0 521538283 3027394560 699911 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 739110 699911 199 36 0 737015 0
[pid=3422/tid=3430] ppid=3420 vsize=2956440 CPUtime=986.1 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 134870 0 0 0 98562 48 0 0 25 0 5 0 521539619 3027394560 699911 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=2956440 CPUtime=986.24 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 146692 0 0 0 98571 53 0 0 25 0 5 0 521539619 3027394560 699911 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=2956440 CPUtime=988.56 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 244656 0 0 0 98770 86 0 0 25 0 5 0 521539619 3027394560 699911 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=2956440 CPUtime=988.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 170579 0 0 0 98805 59 0 0 25 0 5 0 521539619 3027394560 699911 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3962.91
Current children cumulated vsize (KiB) 2956440

[startup+1062.3 s]
/proc/loadavg: 8.07 8.12 8.04 10/186 3474
/proc/meminfo: memFree=22448240/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=3029152 CPUtime=4202.6 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 718455 0 0 0 420005 255 0 0 25 0 5 0 521538283 3101851648 718380 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 757288 718380 199 36 0 755193 0
[pid=3422/tid=3430] ppid=3420 vsize=3029152 CPUtime=1046.09 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 139345 0 0 0 104560 49 0 0 25 0 5 0 521539619 3101851648 718380 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=3029152 CPUtime=1045.92 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 149120 0 0 0 104538 54 0 0 25 0 5 0 521539619 3101851648 718380 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=3029152 CPUtime=1048.55 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 252912 0 0 0 104766 89 0 0 25 0 5 0 521539619 3101851648 718380 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=3029152 CPUtime=1048.63 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 173889 0 0 0 104803 60 0 0 25 0 5 0 521539619 3101851648 718380 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4202.6
Current children cumulated vsize (KiB) 3029152

[startup+1122.3 s]
/proc/loadavg: 8.02 8.10 8.04 10/186 3476
/proc/meminfo: memFree=22328920/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=3083512 CPUtime=4442.24 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 729742 0 0 0 443964 260 0 0 25 0 5 0 521538283 3157516288 729667 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 770878 729667 199 36 0 768783 0
[pid=3422/tid=3430] ppid=3420 vsize=3083512 CPUtime=1106.09 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 141485 0 0 0 110559 50 0 0 25 0 5 0 521539619 3157516288 729667 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=3083512 CPUtime=1105.59 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 150468 0 0 0 110504 55 0 0 25 0 5 0 521539619 3157516288 729667 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=3083512 CPUtime=1108.53 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 255871 0 0 0 110763 90 0 0 25 0 5 0 521539619 3157516288 729667 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=3083512 CPUtime=1108.63 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 178729 0 0 0 110801 62 0 0 25 0 5 0 521539619 3157516288 729667 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4442.24
Current children cumulated vsize (KiB) 3083512

[startup+1182.3 s]
/proc/loadavg: 8.01 8.08 8.03 10/186 3478
/proc/meminfo: memFree=22202660/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=3128072 CPUtime=4681.93 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 741645 0 0 0 467926 267 0 0 25 0 5 0 521538283 3203145728 741570 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 782018 741570 199 36 0 779923 0
[pid=3422/tid=3430] ppid=3420 vsize=3128072 CPUtime=1166.08 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 143884 0 0 0 116557 51 0 0 25 0 5 0 521539619 3203145728 741570 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=3128072 CPUtime=1165.27 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 150542 0 0 0 116472 55 0 0 25 0 5 0 521539619 3203145728 741570 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=3128072 CPUtime=1168.54 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 261312 0 0 0 116761 93 0 0 25 0 5 0 521539619 3203145728 741570 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=3128072 CPUtime=1168.64 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 182718 0 0 0 116799 65 0 0 25 0 5 0 521539619 3203145728 741570 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4681.93
Current children cumulated vsize (KiB) 3128072



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1211.9 s]
/proc/loadavg: 7.47 7.95 7.99 5/176 3498
/proc/meminfo: memFree=24644936/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=3162740 CPUtime=4800.18 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) S 3420 3422 2524 0 -1 4202496 750163 0 0 0 479748 270 0 0 25 0 5 0 521538283 3238645760 750088 33554432000 4194304 4338100 140735377491872 18446744073709551615 218917533669 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 790685 750088 199 36 0 788590 0
[pid=3422/tid=3430] ppid=3420 vsize=3162740 CPUtime=1195.69 cores=1,3,5,7
/proc/3422/task/3430/stat : 3430 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 146082 0 0 0 119517 52 0 0 25 0 5 0 521539619 3238645760 750088 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=3162740 CPUtime=1194.71 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 151452 0 0 0 119416 55 0 0 25 0 5 0 521539619 3238645760 750088 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3422/tid=3432] ppid=3420 vsize=3162740 CPUtime=1198.15 cores=1,3,5,7
/proc/3422/task/3432/stat : 3432 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 264197 0 0 0 119720 95 0 0 25 0 5 0 521539619 3238645760 750088 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3422/tid=3433] ppid=3420 vsize=3162740 CPUtime=1198.25 cores=1,3,5,7
/proc/3422/task/3433/stat : 3433 (ttsth-5-4) R 3420 3422 2524 0 -1 4202560 185243 0 0 0 119759 66 0 0 25 0 5 0 521539619 3238645760 750088 33554432000 4194304 4338100 140735377491872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 3162740

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1212.01 s]
/proc/loadavg: 7.47 7.95 7.99 5/176 3498
/proc/meminfo: memFree=24644936/32951124 swapFree=67111356/67111528
[pid=3422] ppid=3420 vsize=0 CPUtime=4800.3 cores=1,3,5,7
/proc/3422/stat : 3422 (ttsth-5-4) Z 3420 3422 2524 0 -1 4203548 750166 0 0 0 479750 280 0 0 18 0 2 0 521538283 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3422/statm: 0 0 0 0 0 0 0
[pid=3422/tid=3431] ppid=3420 vsize=0 CPUtime=1194.81 cores=1,3,5,7
/proc/3422/task/3431/stat : 3431 (ttsth-5-4) R 3420 3422 2524 0 -1 4203588 151452 0 0 0 119416 65 0 0 25 0 2 0 521539619 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1212.02
CPU time (s): 4800.32
CPU user time (s): 4797.51
CPU system time (s): 2.81757
CPU usage (%): 396.059
Max. virtual memory (cumulated for all children) (KiB): 3162740

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.51
system time used= 2.81757
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 750166
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 675
involuntary context switches= 35442

runsolver used 1.88271 second user time and 4.71128 second system time

The end

Launcher Data 

Begin job on node137 at 2011-04-29 19:50:06
IDJOB=3264334
IDBENCH=84572
IDSOLVER=1654
FILE ID=node137/3264334-1304099406
RUNJOBID= node137-1304099406-3403
PBS_JOBID= 13170581
Free space on /tmp= 71620 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/anton/l5k3/lksat-n900-m3357-k3-l5-s950595579.used-as.sat04-938.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264334-1304099406/watcher-3264334-1304099406 -o /tmp/evaluation-result-3264334-1304099406/solver-3264334-1304099406 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264334-1304099406.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f7a218f6e4ba6f28584266f788a5d6da
RANDOM SEED=2046423411

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27651848 kB
Buffers:        921356 kB
Cached:        3849744 kB
SwapCached:         56 kB
Active:         334452 kB
Inactive:      4499664 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27651848 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:              12 kB
Writeback:           0 kB
AnonPages:       63160 kB
Mapped:          14868 kB
Slab:           401968 kB
PageTables:       4464 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   196256 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71620 MiB
End job on node137 at 2011-04-29 20:10:20