Trace number 3264686

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.36 1202.1

General information on the benchmark

NameSAT04/handmade/marijn/equilarge-l/
equilarge_l4.used-as.sat04-896.cnf
MD5SUMfc993d9544ccd8eaeaa4882599387947
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark437.97
Satisfiable
(Un)Satisfiability was proved
Number of variables4499
Number of clauses18603
Sum of the clauses size54454
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21355
Number of clauses of size 317248
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264686-1304100241.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 4499
0.00/0.01	c Clauses = 18603
0.00/0.01	c Literals = 54454
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 4499
0.00/0.01	c Clauses = 18603
0.00/0.01	c Literals = 54454
0.00/0.01	c a2
0.00/0.03	c a1
0.19/0.21	c Independent sub-problems = 1
0.19/0.21	c Sub-problem with 4499 variables and 18603 clauses
0.19/0.21	c using permutation method 4
0.19/0.21	c count = 2
0.19/0.21	c Ordering finished 0.210 seconds
0.19/0.21	c mapped 0.210 seconds
0.19/0.21	c permuted 0.210 seconds
0.19/0.21	c about to sort
0.19/0.21	c sorted
0.19/0.28	c clauses now 18603
0.19/0.28	c init finished 0.250 seconds
0.19/0.28	c tree built 0.270 seconds
0.19/0.28	c 
0.19/0.28	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.28	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.28	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.28	c ................................ | ..........                  1 |                                  0.3         3.7             
0.19/0.28	c ................................ | ..........                  2 |                                  0.3         7.4             
0.19/0.28	c ................................ | ..........                  3 |                                  0.3        11.1             
0.19/0.28	c ................................ | ..........                  4 |                                  0.3        14.8             
0.19/0.28	c ................................ | ..........                  5 |                                  0.3        18.5             
0.19/0.29	c ................................ | ..........               1000 | 25           497                 0.3      3333.3         3991
96.19/24.38	c ................................ | ..........             281000 | 20160        54512              96.5      2910.7     19404357
192.07/48.40	c ................................ | ..........             426000 | 31583        82424             192.5      2213.3     28566800
287.92/72.39	c ................................ | ..........             550000 | 41418        104789            288.3      1907.9     36166860
384.99/96.60	c ................................ | ..........             651000 | 49038        124259            385.0      1690.8     42652604
480.05/120.45	c ................................ | *.........             734000 | 55604        141419            480.3      1528.3     48035243
575.93/144.48	c ................................ | *.........             816000 | 61990        156808            576.3      1416.0     52715214
672.59/168.62	c ................................ | *.........             898000 | 68625        171959            672.7      1335.0     57901829
768.47/192.68	c ................................ | *.........             972000 | 73293        185022            768.8      1264.3     62410293
864.35/216.62	c ................................ | *.........            1046000 | 78107        197736            864.4      1210.1     66491573
960.61/240.79	c ................................ | **........            1114000 | 83451        210317            961.0      1159.2     70421437
1056.47/264.72	c ................................ | **........            1175000 | 88613        222038           1056.6      1112.1     74062864
1152.74/288.80	c ................................ | **........            1235000 | 93093        233438           1152.8      1071.3     77725477
1248.97/312.90	c ................................ | **........            1293000 | 97294        244437           1249.0      1035.2     80827000
1344.44/336.86	c ................................ | **........            1360000 | 101893       256494           1344.7      1011.4     81981965
1440.70/360.96	c ................................ | ***.......            1427000 | 106782       267941           1441.0       990.3     83069484
1536.59/385.00	c ................................ | ***.......            1489000 | 111183       278169           1537.0       968.8     84026466
1632.04/408.89	c ................................ | ***.......            1545000 | 115703       288148           1632.4       946.4     84814008
1728.71/433.09	c ................................ | ***.......            1601000 | 120740       298228           1729.1       925.9     85691333
1824.58/457.04	c ................................ | ***.......            1651000 | 124505       306905           1824.7       904.8     87207980
1921.25/481.24	c ................................ | ****......            1706000 | 128293       316005           1921.4       887.9     89493899
2017.52/505.31	c ................................ | ****......            1762000 | 131762       324954           2017.5       873.3     92001079
2112.61/529.19	c ................................ | ****......            1813000 | 135414       333100           2112.9       858.0     94130494
2208.46/553.12	c ................................ | ****......            1865000 | 139311       341545           2208.5       844.4     96231696
2304.33/577.11	c ................................ | ****......            1917000 | 143281       349673           2304.4       831.9     98326610
2401.39/601.40	c ................................ | *****.....            1968000 | 147339       358322           2401.4       819.5    100700952
2496.06/625.13	c ................................ | *****.....            2013000 | 150784       366201           2496.2       806.4    102611371
2593.54/649.52	c ................................ | *****.....            2058000 | 153996       374098           2593.6       793.5    104560819
2688.22/673.23	c ................................ | *****.....            2101000 | 157018       381623           2688.3       781.5    106914695
2784.48/697.32	c ................................ | *****.....            2145000 | 160056       389213           2784.6       770.3    109457706
2880.74/721.40	c ................................ | ******....            2187000 | 162868       396257           2880.7       759.2    111813485
2976.21/745.34	c ................................ | ******....            2230000 | 166244       403717           2976.4       749.2    114495668
3072.08/769.39	c ................................ | ******....            2270000 | 169200       410729           3072.5       738.8    116689986
3167.94/793.32	c ................................ | ******....            2311000 | 172360       417634           3168.0       729.5    118955908
3264.61/817.51	c ................................ | ******....            2352000 | 175786       424653           3264.7       720.4    121129054
3360.10/841.43	c ................................ | *******...            2391000 | 179015       431830           3360.2       711.6    123376373
3456.35/865.55	c ................................ | *******...            2427000 | 181868       438421           3456.6       702.1    125526721
3553.44/889.90	c ................................ | *******...            2463000 | 184681       445095           3553.8       693.1    127588540
3649.69/913.90	c ................................ | *******...            2502000 | 187752       451496           3649.7       685.5    129692354
3744.75/937.70	c ................................ | *******...            2539000 | 190590       457823           3744.8       678.0    131752886
3841.03/961.88	c ................................ | ********..            2577000 | 193203       464172           3841.3       670.9    133927532
3938.49/986.21	c ................................ | ********..            2612000 | 195809       470280           3938.6       663.2    135981196
4035.56/1010.51	c ................................ | ********..            2645000 | 197885       475886           4035.6       655.4    137721029
4129.05/1033.91	c ................................ | ********..            2679000 | 200518       481738           4129.1       648.8    139666185
4226.90/1058.40	c ................................ | ********..            2713000 | 203040       487845           4226.9       641.8    141573359
4322.39/1082.35	c ................................ | *********.            2747000 | 205757       493647           4322.6       635.5    143342871
4418.64/1106.45	c ................................ | *********.            2781000 | 208449       499715           4418.9       629.3    145346584
4514.51/1130.49	c ................................ | *********.            2815000 | 211236       505777           4514.9       623.5    147224723
4609.57/1154.28	c ................................ | *********.            2849000 | 213914       511861           4609.9       618.0    149270349
4704.67/1178.04	c ................................ | *********.            2882000 | 216286       517539           4704.8       612.6    151157897

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264686-1304100241/watcher-3264686-1304100241 -o /tmp/evaluation-result-3264686-1304100241/solver-3264686-1304100241 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264686-1304100241.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.83 8.04 8.01 5/181 5610
/proc/meminfo: memFree=23502908/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) R 5608 5610 5016 0 -1 4202496 424 0 0 0 0 0 0 0 25 0 1 0 600254788 9633792 366 33554432000 4194304 4338100 140733580622672 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5610/statm: 2352 367 147 36 0 257 0

[startup+0.0410741 s]
/proc/loadavg: 7.83 8.04 8.01 5/181 5610
/proc/meminfo: memFree=23502908/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=13512 CPUtime=0.03 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) R 5608 5610 5016 0 -1 4202496 1509 0 0 0 3 0 0 0 25 0 1 0 600254788 13836288 1376 33554432000 4194304 4338100 140733580622672 18446744073709551615 260905059429 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5610/statm: 3378 1377 155 36 0 1283 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 13512

[startup+0.101062 s]
/proc/loadavg: 7.83 8.04 8.01 5/181 5610
/proc/meminfo: memFree=23502908/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=14040 CPUtime=0.09 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) R 5608 5610 5016 0 -1 4202496 1646 0 0 0 9 0 0 0 25 0 1 0 600254788 14376960 1513 33554432000 4194304 4338100 140733580622672 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5610/statm: 3510 1513 156 36 0 1415 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 14040

[startup+0.301028 s]
/proc/loadavg: 7.83 8.04 8.01 5/181 5610
/proc/meminfo: memFree=23502908/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=251548 CPUtime=0.32 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 15626 0 0 0 30 2 0 0 25 0 5 0 600254788 257585152 15455 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 62887 15458 176 36 0 60792 0
Current children cumulated CPU time (s) 0.32
Current children cumulated vsize (KiB) 251548

[startup+0.700959 s]
/proc/loadavg: 7.83 8.04 8.01 5/181 5610
/proc/meminfo: memFree=23502908/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=305028 CPUtime=1.92 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 29292 0 0 0 188 4 0 0 25 0 5 0 600254788 312348672 29121 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 76257 29123 176 36 0 74162 0
Current children cumulated CPU time (s) 1.92
Current children cumulated vsize (KiB) 305028

[startup+1.50081 s]
/proc/loadavg: 7.83 8.04 8.01 9/186 5615
/proc/meminfo: memFree=23331928/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=357056 CPUtime=5.11 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 43765 0 0 0 505 6 0 0 25 0 5 0 600254788 365625344 43594 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 89264 43594 176 36 0 87169 0
[pid=5610/tid=5612] ppid=5608 vsize=357056 CPUtime=1.19 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 7566 0 0 0 118 1 0 0 25 0 5 0 600254817 365625344 43594 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=357056 CPUtime=1.2 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 6903 0 0 0 119 1 0 0 25 0 5 0 600254817 365625344 43594 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=357056 CPUtime=1.2 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 7453 0 0 0 119 1 0 0 25 0 5 0 600254817 365625344 43594 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=357056 CPUtime=1.2 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 7421 0 0 0 120 0 0 0 25 0 5 0 600254817 365625344 43594 33554432000 4194304 4338100 140733580622672 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.11
Current children cumulated vsize (KiB) 357056

[startup+3.10052 s]
/proc/loadavg: 7.83 8.04 8.01 9/186 5615
/proc/meminfo: memFree=23246988/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=426400 CPUtime=11.51 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 60376 0 0 0 1142 9 0 0 25 0 5 0 600254788 436633600 60205 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 106600 60205 176 36 0 104505 0
[pid=5610/tid=5612] ppid=5608 vsize=426400 CPUtime=2.79 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 11613 0 0 0 277 2 0 0 25 0 5 0 600254817 436633600 60206 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=426400 CPUtime=2.8 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 10954 0 0 0 279 1 0 0 25 0 5 0 600254817 436633600 60206 33554432000 4194304 4338100 140733580622672 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=426400 CPUtime=2.8 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 11983 0 0 0 279 1 0 0 25 0 5 0 600254817 436633600 60206 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=426400 CPUtime=2.8 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 11405 0 0 0 279 1 0 0 25 0 5 0 600254817 436633600 60206 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.51
Current children cumulated vsize (KiB) 426400

[startup+6.30093 s]
/proc/loadavg: 7.85 8.04 8.01 9/186 5615
/proc/meminfo: memFree=23081416/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=526892 CPUtime=24.29 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 84742 0 0 0 2417 12 0 0 25 0 5 0 600254788 539537408 84571 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 131723 84571 176 36 0 129628 0
[pid=5610/tid=5612] ppid=5608 vsize=526892 CPUtime=5.97 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 17405 0 0 0 595 2 0 0 25 0 5 0 600254817 539537408 84571 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=526892 CPUtime=6 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 17193 0 0 0 598 2 0 0 25 0 5 0 600254817 539537408 84571 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=526892 CPUtime=6 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 17765 0 0 0 597 3 0 0 25 0 5 0 600254817 539537408 84571 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=526892 CPUtime=6 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 17957 0 0 0 598 2 0 0 25 0 5 0 600254817 539537408 84571 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.29
Current children cumulated vsize (KiB) 526892

[startup+12.7008 s]
/proc/loadavg: 7.86 8.04 8.01 9/186 5615
/proc/meminfo: memFree=22860504/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=674436 CPUtime=49.85 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 122223 0 0 0 4966 19 0 0 25 0 5 0 600254788 690622464 122052 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 168609 122052 176 36 0 166514 0
[pid=5610/tid=5612] ppid=5608 vsize=674436 CPUtime=12.36 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 26591 0 0 0 1232 4 0 0 25 0 5 0 600254817 690622464 122052 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=674436 CPUtime=12.4 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 26809 0 0 0 1236 4 0 0 25 0 5 0 600254817 690622464 122052 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=674436 CPUtime=12.36 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 27769 0 0 0 1232 4 0 0 25 0 5 0 600254817 690622464 122052 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=674436 CPUtime=12.4 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 26632 0 0 0 1237 3 0 0 25 0 5 0 600254817 690622464 122052 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.85
Current children cumulated vsize (KiB) 674436

[startup+25.5004 s]
/proc/loadavg: 7.89 8.04 8.01 9/186 5616
/proc/meminfo: memFree=22544320/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=875024 CPUtime=100.98 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 172545 0 0 0 10070 28 0 0 25 0 5 0 600254788 896024576 172374 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 218756 172374 176 36 0 216661 0
[pid=5610/tid=5612] ppid=5608 vsize=875024 CPUtime=25.16 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 38599 0 0 0 2510 6 0 0 25 0 5 0 600254817 896024576 172374 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=875024 CPUtime=25.2 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 39183 0 0 0 2514 6 0 0 25 0 5 0 600254817 896024576 172374 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=875024 CPUtime=25.09 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 41529 0 0 0 2503 6 0 0 25 0 5 0 600254817 896024576 172374 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=875024 CPUtime=25.2 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 38812 0 0 0 2514 6 0 0 25 0 5 0 600254817 896024576 172374 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.98
Current children cumulated vsize (KiB) 875024

[startup+51.1007 s]
/proc/loadavg: 8.06 8.07 8.02 9/186 5616
/proc/meminfo: memFree=22144032/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=1175972 CPUtime=203.24 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 248656 0 0 0 20283 41 0 0 25 0 5 0 600254788 1204195328 248485 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 293993 248485 176 36 0 291898 0
[pid=5610/tid=5612] ppid=5608 vsize=1175972 CPUtime=50.77 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 57734 0 0 0 5067 10 0 0 25 0 5 0 600254817 1204195328 248485 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=1175972 CPUtime=50.8 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 57579 0 0 0 5071 9 0 0 25 0 5 0 600254817 1204195328 248485 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=1175972 CPUtime=50.56 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 59772 0 0 0 5046 10 0 0 25 0 5 0 600254817 1204195328 248485 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=1175972 CPUtime=50.8 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 59149 0 0 0 5071 9 0 0 25 0 5 0 600254817 1204195328 248485 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 203.24
Current children cumulated vsize (KiB) 1175972

[startup+102.306 s]
/proc/loadavg: 8.17 8.09 8.02 9/186 5618
/proc/meminfo: memFree=21522504/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=1637108 CPUtime=407.78 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 362654 0 0 0 40715 63 0 0 25 0 5 0 600254788 1676398592 362483 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 409277 362483 176 36 0 407182 0
[pid=5610/tid=5612] ppid=5608 vsize=1637108 CPUtime=101.97 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 86411 0 0 0 10181 16 0 0 25 0 5 0 600254817 1676398592 362483 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=1637108 CPUtime=102.01 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 86162 0 0 0 10187 14 0 0 25 0 5 0 600254817 1676398592 362483 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=1637108 CPUtime=101.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 89250 0 0 0 10132 16 0 0 25 0 5 0 600254817 1676398592 362483 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=1637108 CPUtime=102 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 86409 0 0 0 10186 14 0 0 25 0 5 0 600254817 1676398592 362483 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 407.78
Current children cumulated vsize (KiB) 1637108

[startup+162.3 s]
/proc/loadavg: 8.19 8.10 8.03 9/186 5619
/proc/meminfo: memFree=20897740/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=2026024 CPUtime=647.42 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 461363 0 0 0 64660 82 0 0 25 0 5 0 600254788 2074648576 461192 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 506506 461192 176 36 0 504411 0
[pid=5610/tid=5612] ppid=5608 vsize=2026024 CPUtime=161.97 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 112276 0 0 0 16176 21 0 0 25 0 5 0 600254817 2074648576 461192 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=2026024 CPUtime=161.69 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 110206 0 0 0 16151 18 0 0 25 0 5 0 600254817 2074648576 461192 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=2026024 CPUtime=161.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 113442 0 0 0 16127 21 0 0 25 0 5 0 600254817 2074648576 461192 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=2026024 CPUtime=161.98 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 111017 0 0 0 16179 19 0 0 25 0 5 0 600254817 2074648576 461192 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 647.42
Current children cumulated vsize (KiB) 2026024


################
# More data... #
################

/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 238951 0 0 0 76148 44 0 0 25 0 5 0 600254817 4284850176 969831 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3044.14
Current children cumulated vsize (KiB) 4184424

[startup+822.3 s]
/proc/loadavg: 8.11 8.08 8.03 9/186 5637
/proc/meminfo: memFree=16292828/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=4356860 CPUtime=3283.79 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1013089 0 0 0 328190 189 0 0 25 0 5 0 600254788 4461424640 1012918 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1089215 1012918 176 36 0 1087120 0
[pid=5610/tid=5612] ppid=5608 vsize=4356860 CPUtime=821.99 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 288226 0 0 0 82143 56 0 0 25 0 5 0 600254817 4461424640 1012918 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=4356860 CPUtime=818.09 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 247553 0 0 0 81765 44 0 0 25 0 5 0 600254817 4461424640 1012918 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=4356860 CPUtime=821.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 211564 0 0 0 82109 39 0 0 25 0 5 0 600254817 4461424640 1012918 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=4356860 CPUtime=821.91 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 251324 0 0 0 82145 46 0 0 25 0 5 0 600254817 4461424640 1012918 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3283.79
Current children cumulated vsize (KiB) 4356860

[startup+882.3 s]
/proc/loadavg: 8.21 8.11 8.04 9/186 5639
/proc/meminfo: memFree=15938164/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=4522908 CPUtime=3523.45 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1054696 0 0 0 352148 197 0 0 25 0 5 0 600254788 4631457792 1054525 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1130727 1054525 176 36 0 1128632 0
[pid=5610/tid=5612] ppid=5608 vsize=4522908 CPUtime=881.99 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 298229 0 0 0 88141 58 0 0 25 0 5 0 600254817 4631457792 1054525 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=4522908 CPUtime=877.77 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 258378 0 0 0 87731 46 0 0 25 0 5 0 600254817 4631457792 1054525 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=4522908 CPUtime=881.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 221898 0 0 0 88107 41 0 0 25 0 5 0 600254817 4631457792 1054525 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=4522908 CPUtime=881.9 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 261769 0 0 0 88141 49 0 0 25 0 5 0 600254817 4631457792 1054525 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3523.45
Current children cumulated vsize (KiB) 4522908

[startup+942.3 s]
/proc/loadavg: 8.21 8.12 8.05 9/186 5640
/proc/meminfo: memFree=15624908/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=4689088 CPUtime=3763.13 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1096291 0 0 0 376107 206 0 0 25 0 5 0 600254788 4801626112 1096120 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1172272 1096120 176 36 0 1170177 0
[pid=5610/tid=5612] ppid=5608 vsize=4689088 CPUtime=941.99 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 309644 0 0 0 94138 61 0 0 25 0 5 0 600254817 4801626112 1096120 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=4689088 CPUtime=937.44 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 268574 0 0 0 93696 48 0 0 25 0 5 0 600254817 4801626112 1096120 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=4689088 CPUtime=941.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 231582 0 0 0 94106 42 0 0 25 0 5 0 600254817 4801626112 1096120 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=4689088 CPUtime=941.9 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 272069 0 0 0 94139 51 0 0 25 0 5 0 600254817 4801626112 1096120 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3763.13
Current children cumulated vsize (KiB) 4689088

[startup+1002.3 s]
/proc/loadavg: 8.13 8.11 8.05 9/186 5642
/proc/meminfo: memFree=15387056/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=4842492 CPUtime=4002.8 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1135562 0 0 0 400067 213 0 0 25 0 5 0 600254788 4958711808 1135391 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1210623 1135391 176 36 0 1208528 0
[pid=5610/tid=5612] ppid=5608 vsize=4842492 CPUtime=1001.99 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 319424 0 0 0 100136 63 0 0 25 0 5 0 600254817 4958711808 1135391 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=4842492 CPUtime=997.12 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 278354 0 0 0 99662 50 0 0 25 0 5 0 600254817 4958711808 1135391 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=4842492 CPUtime=1001.49 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 240873 0 0 0 100105 44 0 0 25 0 5 0 600254817 4958711808 1135391 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=4842492 CPUtime=1001.89 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 282489 0 0 0 100136 53 0 0 25 0 5 0 600254817 4958711808 1135391 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4002.8
Current children cumulated vsize (KiB) 4842492

[startup+1062.3 s]
/proc/loadavg: 8.04 8.09 8.04 9/186 5643
/proc/meminfo: memFree=15162084/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=4993032 CPUtime=4242.47 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1172384 0 0 0 424028 219 0 0 25 0 5 0 600254788 5112864768 1172213 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1248258 1172213 176 36 0 1246163 0
[pid=5610/tid=5612] ppid=5608 vsize=4993032 CPUtime=1061.99 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 328720 0 0 0 106135 64 0 0 25 0 5 0 600254817 5112864768 1172213 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=4993032 CPUtime=1056.79 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 288111 0 0 0 105628 51 0 0 25 0 5 0 600254817 5112864768 1172213 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=4993032 CPUtime=1061.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 249817 0 0 0 106103 45 0 0 25 0 5 0 600254817 5112864768 1172213 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=4993032 CPUtime=1061.88 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 291314 0 0 0 106133 55 0 0 25 0 5 0 600254817 5112864768 1172213 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4242.47
Current children cumulated vsize (KiB) 4993032

[startup+1122.3 s]
/proc/loadavg: 8.08 8.08 8.04 9/186 5645
/proc/meminfo: memFree=14908364/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=5137184 CPUtime=4482.15 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1208854 0 0 0 447988 227 0 0 25 0 5 0 600254788 5260476416 1208683 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1284296 1208683 176 36 0 1282201 0
[pid=5610/tid=5612] ppid=5608 vsize=5137184 CPUtime=1122 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 338969 0 0 0 112134 66 0 0 25 0 5 0 600254817 5260476416 1208683 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=5137184 CPUtime=1116.46 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 296904 0 0 0 111593 53 0 0 25 0 5 0 600254817 5260476416 1208683 33554432000 4194304 4338100 140733580622672 18446744073709551615 4285828 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=5137184 CPUtime=1121.48 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 258414 0 0 0 112101 47 0 0 25 0 5 0 600254817 5260476416 1208683 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=5137184 CPUtime=1121.88 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 300145 0 0 0 112131 57 0 0 25 0 5 0 600254817 5260476416 1208683 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4482.15
Current children cumulated vsize (KiB) 5137184

[startup+1182.3 s]
/proc/loadavg: 8.09 8.08 8.04 9/186 5647
/proc/meminfo: memFree=14646568/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=5293848 CPUtime=4721.83 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1247677 0 0 0 471948 235 0 0 25 0 5 0 600254788 5420900352 1247506 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1323462 1247506 176 36 0 1321367 0
[pid=5610/tid=5612] ppid=5608 vsize=5293848 CPUtime=1182 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 350466 0 0 0 118132 68 0 0 25 0 5 0 600254817 5420900352 1247506 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=5293848 CPUtime=1176.14 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 306823 0 0 0 117558 56 0 0 25 0 5 0 600254817 5420900352 1247506 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=5293848 CPUtime=1181.49 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 267157 0 0 0 118100 49 0 0 25 0 5 0 600254817 5420900352 1247506 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=5293848 CPUtime=1181.88 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 308809 0 0 0 118130 58 0 0 25 0 5 0 600254817 5420900352 1247506 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4721.83
Current children cumulated vsize (KiB) 5293848



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.9 s]
/proc/loadavg: 8.06 8.08 8.04 5/180 5648
/proc/meminfo: memFree=18578520/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=5337640 CPUtime=4800.13 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) S 5608 5610 5016 0 -1 4202496 1258799 0 0 0 479777 236 0 0 25 0 5 0 600254788 5465743360 1258628 33554432000 4194304 4338100 140733580622672 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 1334410 1258628 176 36 0 1332315 0
[pid=5610/tid=5612] ppid=5608 vsize=5337640 CPUtime=1201.61 cores=0,2,4,6
/proc/5610/task/5612/stat : 5612 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 353966 0 0 0 120092 69 0 0 25 0 5 0 600254817 5465743360 1258628 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=5337640 CPUtime=1195.64 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 309226 0 0 0 119507 57 0 0 25 0 5 0 600254817 5465743360 1258628 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=5610/tid=5614] ppid=5608 vsize=5337640 CPUtime=1201.09 cores=0,2,4,6
/proc/5610/task/5614/stat : 5614 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 269274 0 0 0 120060 49 0 0 25 0 5 0 600254817 5465743360 1258628 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=5610/tid=5615] ppid=5608 vsize=5337640 CPUtime=1201.47 cores=0,2,4,6
/proc/5610/task/5615/stat : 5615 (ttsth-5-4) R 5608 5610 5016 0 -1 4202560 311911 0 0 0 120089 58 0 0 25 0 5 0 600254817 5465743360 1258628 33554432000 4194304 4338100 140733580622672 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 5337640

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.01 s]
/proc/loadavg: 8.06 8.08 8.04 5/180 5648
/proc/meminfo: memFree=18578520/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) Z 5608 5610 5016 0 -1 4203548 1258799 0 0 0 479779 246 0 0 18 0 2 0 600254788 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 0 0 0 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=0 CPUtime=1195.75 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4203588 309226 0 0 0 119508 67 0 0 25 0 2 0 600254817 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

[startup+1202.1 s]
/proc/loadavg: 8.06 8.08 8.04 5/180 5648
/proc/meminfo: memFree=18578520/32950928 swapFree=67071984/67111528
[pid=5610] ppid=5608 vsize=0 CPUtime=4800.34 cores=0,2,4,6
/proc/5610/stat : 5610 (ttsth-5-4) Z 5608 5610 5016 0 -1 4203548 1258799 0 0 0 479779 255 0 0 18 0 2 0 600254788 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/5610/statm: 0 0 0 0 0 0 0
[pid=5610/tid=5613] ppid=5608 vsize=0 CPUtime=1195.83 cores=0,2,4,6
/proc/5610/task/5613/stat : 5613 (ttsth-5-4) R 5608 5610 5016 0 -1 4203588 309226 0 0 0 119508 75 0 0 25 0 2 0 600254817 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.1
CPU time (s): 4800.36
CPU user time (s): 4797.8
CPU system time (s): 2.56061
CPU usage (%): 399.329
Max. virtual memory (cumulated for all children) (KiB): 5337640

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.8
system time used= 2.56061
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1258799
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 869
involuntary context switches= 32480

runsolver used 1.90471 second user time and 4.65829 second system time

The end

Launcher Data 

Begin job on node116 at 2011-04-29 20:04:01
IDJOB=3264686
IDBENCH=84762
IDSOLVER=1654
FILE ID=node116/3264686-1304100241
RUNJOBID= node116-1304099035-5481
PBS_JOBID= 13170597
Free space on /tmp= 72516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/marijn/equilarge-l/equilarge_l4.used-as.sat04-896.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264686-1304100241/watcher-3264686-1304100241 -o /tmp/evaluation-result-3264686-1304100241/solver-3264686-1304100241 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264686-1304100241.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= fc993d9544ccd8eaeaa4882599387947
RANDOM SEED=1593962888

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23503064 kB
Buffers:       1905092 kB
Cached:        6953424 kB
SwapCached:       9732 kB
Active:        4521048 kB
Inactive:      4497988 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23503064 kB
SwapTotal:    67111528 kB
SwapFree:     67071984 kB
Dirty:             916 kB
Writeback:           0 kB
AnonPages:      154524 kB
Mapped:          14212 kB
Slab:           365768 kB
PageTables:       4868 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   377052 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72516 MiB
End job on node116 at 2011-04-29 20:24:05