Trace number 3258262

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1202.51

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v1375c111739g4.cnf
MD5SUMb2f1edd9e4045a375bcfedb2d5e5bd0e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark51.2682
Satisfiable
(Un)Satisfiability was proved
Number of variables1375
Number of clauses111739
Sum of the clauses size224743
Maximum clause length25
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2111684
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 555

Solver Data 

0.07/0.07	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258262-1304050116.cnf 
0.07/0.07	c Using SATTIMEOUT of 4800
0.07/0.07	c Using SATRAM of 15500
0.07/0.07	c Size of subset node = 32
0.07/0.07	c Variables = 1375
0.07/0.07	c Clauses = 111739
0.07/0.07	c Literals = 224743
0.07/0.07	c Reduced size:
0.07/0.07	c Variables = 1375
0.07/0.07	c Clauses = 111739
0.07/0.07	c Literals = 224743
0.07/0.07	c a2
0.09/0.14	c a1
0.19/0.27	c Independent sub-problems = 1
0.19/0.27	c Sub-problem with 1375 variables and 111739 clauses
0.19/0.27	c using permutation method 4
0.19/0.27	c count = 1
0.19/0.27	c Ordering finished 0.260 seconds
0.19/0.27	c mapped 0.270 seconds
0.19/0.27	c permuted 0.270 seconds
0.19/0.27	c about to sort
0.19/0.28	c sorted
0.49/0.53	c clauses now 111739
0.49/0.53	c init finished 0.300 seconds
0.49/0.53	c tree built 0.520 seconds
0.49/0.53	c 
0.49/0.53	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.53	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.53	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.53	c ................................ | ..........                  1 |                                  0.5         1.9             
0.49/0.53	c ................................ | ..........                  2 |                                  0.5         3.8             
0.49/0.53	c ................................ | ..........                  3 |                                  0.5         5.8             
0.49/0.53	c ................................ | ..........                  4 |                                  0.5         7.7             
0.49/0.53	c ................................ | ..........                  5 |                                  0.5         9.6             
0.49/0.53	c ................................ | ..........               1000 | 6            101                 0.5      1886.8         5236
95.87/24.44	c ................................ | ..........            3554000 | 88604        350719             96.0     37013.1     18474311
191.73/48.47	c ................................ | ..........            6256000 | 159052       596577            192.0     32578.2     28707015
287.98/72.50	c ................................ | ..........            8541000 | 216985       801955            288.0     29656.2     31956256
383.84/96.54	c ................................ | ..........           10588000 | 264271       1002601           384.0     27571.5     34917344
479.70/120.57	c ................................ | *.........           12995000 | 319236       1210031           480.0     27072.9     39354110
575.96/144.61	c ................................ | *.........           15101000 | 370833       1389836           576.0     26216.1     41334477
671.84/168.64	c ................................ | *.........           17046000 | 419772       1553057           672.0     25366.1     46906217
767.69/192.68	c ................................ | *.........           18862000 | 465574       1713110           768.0     24559.6     53140555
863.94/216.72	c ................................ | *.........           20598000 | 507988       1856974           864.0     23839.5     59318811
959.80/240.77	c ................................ | **........           22154000 | 547940       1995025           960.1     23075.4     63550250
1055.63/264.79	c ................................ | **........           23569000 | 582163       2142051          1056.0     22318.9     67949521
1151.88/288.83	c ................................ | **........           25068000 | 621151       2284826          1152.0     21760.2     67981575
1247.75/312.86	c ................................ | **........           26599000 | 661158       2435505          1248.0     21313.1     68194042
1344.04/336.90	c ................................ | **........           28090000 | 696276       2596283          1344.0     20899.5     68359440
1439.89/360.93	c ................................ | ***.......           29662000 | 736735       2740864          1440.0     20598.0     72923070
1535.71/384.99	c ................................ | ***.......           31385000 | 783234       2878931          1536.1     20432.1     78456011
1631.95/409.01	c ................................ | ***.......           32895000 | 823695       2997393          1632.0     20156.1     79285711
1727.78/433.06	c ................................ | ***.......           34332000 | 861606       3120893          1728.0     19867.6     81516416
1824.03/457.10	c ................................ | ***.......           35863000 | 904364       3247363          1824.0     19661.2     86034711
1919.86/481.14	c ................................ | ****......           37276000 | 945992       3372081          1920.1     19414.0     88235665
2015.71/505.19	c ................................ | ****......           38668000 | 984636       3487211          2016.0     19180.1     90038810
2111.95/529.21	c ................................ | ****......           40066000 | 1019533      3599448          2112.0     18970.5     94380568
2207.77/553.26	c ................................ | ****......           41327000 | 1052226      3705687          2208.0     18716.8     94805194
2304.01/577.30	c ................................ | ****......           42515000 | 1082886      3816360          2304.0     18452.4     95327921
2399.87/601.34	c ................................ | *****.....           43659000 | 1114939      3914796          2400.1     18190.9     97050086
2495.71/625.37	c ................................ | *****.....           44803000 | 1143700      4021951          2496.0     17949.8     99178568
2591.98/649.42	c ................................ | *****.....           45904000 | 1172769      4113122          2592.1     17709.5    102912343
2687.86/673.46	c ................................ | *****.....           46940000 | 1200550      4202007          2688.1     17462.1    103602920
2783.69/697.47	c ................................ | *****.....           48057000 | 1231407      4298235          2784.0     17261.9    103659552
2879.96/721.51	c ................................ | ******....           49269000 | 1263188      4402779          2880.0     17107.2    106272081
2975.81/745.55	c ................................ | ******....           50414000 | 1293391      4500986          2976.0     16940.1    107111726
3071.64/769.59	c ................................ | ******....           51510000 | 1323692      4600456          3072.0     16767.5    108669515
3167.88/793.63	c ................................ | ******....           52515000 | 1350805      4699874          3168.0     16576.7    108669515
3263.74/817.68	c ................................ | ******....           53505000 | 1377191      4799986          3264.1     16392.2    110252337
3359.60/841.70	c ................................ | *******...           54626000 | 1405833      4899150          3360.0     16257.7    110993447
3455.81/865.74	c ................................ | *******...           55729000 | 1434406      4987410          3456.0     16125.2    110993447
3551.66/889.79	c ................................ | *******...           56841000 | 1466602      5071709          3552.0     16002.4    112818283
3647.89/913.83	c ................................ | *******...           57909000 | 1496774      5158038          3648.0     15874.1    116695325
3743.72/937.88	c ................................ | *******...           58932000 | 1526182      5240885          3744.1     15740.1    118393426
3839.96/961.91	c ................................ | ********..           59948000 | 1552833      5340812          3840.0     15611.3    120994656
3935.82/985.97	c ................................ | ********..           60891000 | 1579708      5440901          3936.1     15469.9    121215464
4031.65/1009.98	c ................................ | ********..           61797000 | 1605550      5529941          4032.0     15326.6    121559861
4127.81/1034.05	c ................................ | ********..           62746000 | 1631514      5618661          4128.0     15200.0    122756595
4224.03/1058.10	c ................................ | ********..           63632000 | 1657451      5701200          4224.0     15064.3    125458135
4319.87/1082.15	c ................................ | *********.           64514000 | 1684787      5790141          4320.1     14933.5    127554280
4415.73/1106.16	c ................................ | *********.           65357000 | 1709751      5879946          4416.0     14800.0    130877441
4512.00/1130.20	c ................................ | *********.           66230000 | 1733854      5960935          4512.0     14678.5    132503458
4607.87/1154.25	c ................................ | *********.           67119000 | 1760330      6040639          4608.1     14565.6    135109841
4703.71/1178.29	c ................................ | *********.           67969000 | 1786212      6116802          4704.1     14449.0    137451875
4799.94/1202.32	c CPU limit (4800) reached
4799.94/1202.32	c Time = 4800.03 seconds
4799.94/1202.32	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258262-1304050116/watcher-3258262-1304050116 -o /tmp/evaluation-result-3258262-1304050116/solver-3258262-1304050116 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258262-1304050116.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.02 7.31 1/175 2113
/proc/meminfo: memFree=18304108/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=9244 CPUtime=0 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) R 2111 2113 1994 0 -1 4202496 383 0 0 0 0 0 0 0 25 0 1 0 516612071 9465856 326 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2113/statm: 2311 327 147 36 0 216 0

[startup+0.0761721 s]
/proc/loadavg: 8.10 8.02 7.31 1/175 2113
/proc/meminfo: memFree=18304108/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=22064 CPUtime=0.07 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) R 2111 2113 1994 0 -1 4202496 3523 0 0 0 7 0 0 0 25 0 1 0 516612071 22593536 3466 33554432000 4194304 4338100 140736207262272 18446744073709551615 4221704 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2113/statm: 5516 3466 150 36 0 3421 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 22064

[startup+0.10117 s]
/proc/loadavg: 8.10 8.02 7.31 1/175 2113
/proc/meminfo: memFree=18304108/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=31540 CPUtime=0.09 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) R 2111 2113 1994 0 -1 4202496 5905 0 0 0 9 0 0 0 25 0 1 0 516612071 32296960 5847 33554432000 4194304 4338100 140736207262272 18446744073709551615 4206987 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2113/statm: 7885 5847 153 36 0 5790 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31540

[startup+0.301145 s]
/proc/loadavg: 8.10 8.02 7.31 1/175 2113
/proc/meminfo: memFree=18304108/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=33188 CPUtime=0.29 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) R 2111 2113 1994 0 -1 4202496 8655 0 0 0 28 1 0 0 25 0 1 0 516612071 33984512 6333 33554432000 4194304 4338100 140736207262272 18446744073709551615 4273326 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2113/statm: 8297 6333 163 36 0 6202 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 33188

[startup+0.95603 s]
/proc/loadavg: 8.10 8.02 7.31 1/175 2113
/proc/meminfo: memFree=18304108/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=280284 CPUtime=2.22 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 26171 0 0 0 216 6 0 0 25 0 5 0 516612071 287010816 23630 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 70071 23630 176 36 0 67976 0
Current children cumulated CPU time (s) 2.22
Current children cumulated vsize (KiB) 280284

[startup+1.50087 s]
/proc/loadavg: 8.10 8.02 7.31 10/186 2149
/proc/meminfo: memFree=18115468/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=302444 CPUtime=4.38 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 31629 0 0 0 428 10 0 0 25 0 5 0 516612071 309702656 29088 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 75611 29089 176 36 0 73516 0
[pid=2113/tid=2142] ppid=2111 vsize=302444 CPUtime=0.96 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 3411 0 0 0 96 0 0 0 19 0 5 0 516612124 309702656 29089 33554432000 4194304 4338100 140736207262272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=302444 CPUtime=0.96 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 1992 0 0 0 93 3 0 0 19 0 5 0 516612124 309702656 29089 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254041 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=302444 CPUtime=0.95 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 3601 0 0 0 95 0 0 0 19 0 5 0 516612124 309702656 29090 33554432000 4194304 4338100 140736207262272 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=302444 CPUtime=0.95 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 1671 0 0 0 92 3 0 0 25 0 5 0 516612124 309702656 29090 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808560870 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.38
Current children cumulated vsize (KiB) 302444

[startup+3.10058 s]
/proc/loadavg: 8.09 8.02 7.31 9/186 2150
/proc/meminfo: memFree=18034620/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=356016 CPUtime=10.78 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 46122 0 0 0 1052 26 0 0 25 0 5 0 516612071 364560384 43581 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 89004 43581 176 36 0 86909 0
[pid=2113/tid=2142] ppid=2111 vsize=356016 CPUtime=2.56 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 8738 0 0 0 254 2 0 0 22 0 5 0 516612124 364560384 43581 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808560777 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=356016 CPUtime=2.56 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 4734 0 0 0 247 9 0 0 22 0 5 0 516612124 364560384 43581 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=356016 CPUtime=2.55 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 7746 0 0 0 254 1 0 0 22 0 5 0 516612124 364560384 43581 33554432000 4194304 4338100 140736207262272 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=356016 CPUtime=2.55 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 3949 0 0 0 245 10 0 0 25 0 5 0 516612124 364560384 43581 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.78
Current children cumulated vsize (KiB) 356016

[startup+6.301 s]
/proc/loadavg: 8.09 8.02 7.31 9/186 2150
/proc/meminfo: memFree=17843112/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=453788 CPUtime=23.56 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 70053 0 0 0 2300 56 0 0 25 0 5 0 516612071 464678912 67512 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 113447 67512 176 36 0 111352 0
[pid=2113/tid=2142] ppid=2111 vsize=453788 CPUtime=5.76 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 16710 0 0 0 573 3 0 0 25 0 5 0 516612124 464678912 67513 33554432000 4194304 4338100 140736207262272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=453788 CPUtime=5.76 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 8869 0 0 0 553 23 0 0 25 0 5 0 516612124 464678912 67513 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=453788 CPUtime=5.73 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 15679 0 0 0 571 2 0 0 25 0 5 0 516612124 464678912 67513 33554432000 4194304 4338100 140736207262272 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=453788 CPUtime=5.75 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 7841 0 0 0 551 24 0 0 25 0 5 0 516612124 464678912 67513 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.56
Current children cumulated vsize (KiB) 453788

[startup+12.7008 s]
/proc/loadavg: 8.08 8.01 7.32 9/186 2150
/proc/meminfo: memFree=17542828/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=610716 CPUtime=49.12 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 108869 0 0 0 4799 113 0 0 25 0 5 0 516612071 625373184 106328 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 152679 106328 176 36 0 150584 0
[pid=2113/tid=2142] ppid=2111 vsize=610716 CPUtime=12.16 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 30080 0 0 0 1210 6 0 0 25 0 5 0 516612124 625373184 106329 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808561046 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=610716 CPUtime=12.16 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 15536 0 0 0 1167 49 0 0 25 0 5 0 516612124 625373184 106329 33554432000 4194304 4338100 140736207262272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=610716 CPUtime=12.09 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 28071 0 0 0 1205 4 0 0 25 0 5 0 516612124 625373184 106329 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808561065 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=610716 CPUtime=12.16 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 14228 0 0 0 1166 50 0 0 25 0 5 0 516612124 625373184 106329 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.12
Current children cumulated vsize (KiB) 610716

[startup+25.5005 s]
/proc/loadavg: 8.14 8.03 7.33 9/186 2150
/proc/meminfo: memFree=17066400/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=871692 CPUtime=100.27 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 173514 0 0 0 9799 228 0 0 25 0 5 0 516612071 892612608 170973 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 217923 170973 176 36 0 215828 0
[pid=2113/tid=2142] ppid=2111 vsize=871692 CPUtime=24.96 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 54130 0 0 0 2483 13 0 0 25 0 5 0 516612124 892612608 170983 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=871692 CPUtime=24.96 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 25323 0 0 0 2395 101 0 0 25 0 5 0 516612124 892612608 170983 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=871692 CPUtime=24.82 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 50369 0 0 0 2474 8 0 0 25 0 5 0 516612124 892612608 170983 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=871692 CPUtime=24.96 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 22747 0 0 0 2395 101 0 0 25 0 5 0 516612124 892612608 170983 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.27
Current children cumulated vsize (KiB) 871692

[startup+51.1008 s]
/proc/loadavg: 8.16 8.04 7.36 9/186 2151
/proc/meminfo: memFree=16360064/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=1200068 CPUtime=202.51 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 256725 0 0 0 19817 434 0 0 25 0 5 0 516612071 1228869632 252666 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 300017 252666 176 36 0 297922 0
[pid=2113/tid=2142] ppid=2111 vsize=1200068 CPUtime=50.56 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 87719 0 0 0 5033 23 0 0 25 0 5 0 516612124 1228869632 252666 33554432000 4194304 4338100 140736207262272 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=1200068 CPUtime=50.56 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 38189 0 0 0 4858 198 0 0 25 0 5 0 516612124 1228869632 252666 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=1200068 CPUtime=50.29 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 77805 0 0 0 5014 15 0 0 25 0 5 0 516612124 1228869632 252666 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808561068 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=1200068 CPUtime=50.55 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 32057 0 0 0 4861 194 0 0 25 0 5 0 516612124 1228869632 252666 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.51
Current children cumulated vsize (KiB) 1200068

[startup+102.3 s]
/proc/loadavg: 8.07 8.03 7.39 9/186 2152
/proc/meminfo: memFree=15764472/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=1461964 CPUtime=407 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 318219 0 0 0 39920 780 0 0 25 0 5 0 516612071 1497051136 314160 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 365491 314160 176 36 0 363396 0
[pid=2113/tid=2142] ppid=2111 vsize=1461964 CPUtime=101.75 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 122218 0 0 0 10137 38 0 0 25 0 5 0 516612124 1497051136 314160 33554432000 4194304 4338100 140736207262272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=1461964 CPUtime=101.75 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 42017 0 0 0 9813 362 0 0 25 0 5 0 516612124 1497051136 314160 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=1461964 CPUtime=101.2 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 95881 0 0 0 10100 20 0 0 25 0 5 0 516612124 1497051136 314160 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808560692 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=1461964 CPUtime=101.74 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 37148 0 0 0 9818 356 0 0 25 0 5 0 516612124 1497051136 314160 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407
Current children cumulated vsize (KiB) 1461964

[startup+162.3 s]
/proc/loadavg: 8.02 8.02 7.42 9/186 2154
/proc/meminfo: memFree=15081960/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=1765644 CPUtime=646.66 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 394742 0 0 0 63515 1151 0 0 25 0 5 0 516612071 1808019456 390683 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 441411 390683 176 36 0 439316 0
[pid=2113/tid=2142] ppid=2111 vsize=1765644 CPUtime=161.63 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 167355 0 0 0 16105 58 0 0 25 0 5 0 516612124 1808019456 390683 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=1765644 CPUtime=161.75 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 42060 0 0 0 15643 532 0 0 25 0 5 0 516612124 1808019456 390683 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=1765644 CPUtime=160.99 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 127110 0 0 0 16071 28 0 0 25 0 5 0 516612124 1808019456 390683 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=1765644 CPUtime=161.74 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 37262 0 0 0 15645 529 0 0 25 0 5 0 516612124 1808019456 390683 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.66
Current children cumulated vsize (KiB) 1765644


################
# More data... #
################

/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 63650 0 0 0 68383 1781 0 0 25 0 5 0 516612124 3850178560 867901 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991348 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2803.26
Current children cumulated vsize (KiB) 3759940

[startup+762.301 s]
/proc/loadavg: 8.12 8.10 7.73 9/186 2170
/proc/meminfo: memFree=11272876/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=3869552 CPUtime=3042.9 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 899747 0 0 0 300256 4034 0 0 25 0 5 0 516612071 3962421248 895688 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 967388 895688 176 36 0 965293 0
[pid=2113/tid=2142] ppid=2111 vsize=3869552 CPUtime=759.65 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 433908 0 0 0 75783 182 0 0 25 0 5 0 516612124 3962421248 895688 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=3869552 CPUtime=760.69 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 62503 0 0 0 74191 1878 0 0 25 0 5 0 516612124 3962421248 895688 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=3869552 CPUtime=760.36 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 318649 0 0 0 75954 82 0 0 25 0 5 0 516612124 3962421248 895688 33554432000 4194304 4338100 140736207262272 18446744073709551615 230808561026 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=3869552 CPUtime=761.63 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 63732 0 0 0 74275 1888 0 0 25 0 5 0 516612124 3962421248 895688 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.9
Current children cumulated vsize (KiB) 3869552

[startup+822.301 s]
/proc/loadavg: 8.20 8.12 7.76 9/186 2171
/proc/meminfo: memFree=11038368/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4029804 CPUtime=3282.5 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 926064 0 0 0 323983 4267 0 0 25 0 5 0 516612071 4126519296 922005 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1007451 922005 176 36 0 1005356 0
[pid=2113/tid=2142] ppid=2111 vsize=4029804 CPUtime=819.31 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 442351 0 0 0 81743 188 0 0 25 0 5 0 516612124 4126519296 922005 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4029804 CPUtime=820.68 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 65187 0 0 0 80080 1988 0 0 25 0 5 0 516612124 4126519296 922005 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4029804 CPUtime=820.35 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 329486 0 0 0 81950 85 0 0 25 0 5 0 516612124 4126519296 922005 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4029804 CPUtime=821.6 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 68085 0 0 0 80158 2002 0 0 25 0 5 0 516612124 4126519296 922005 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.5
Current children cumulated vsize (KiB) 4029804

[startup+882.301 s]
/proc/loadavg: 8.22 8.13 7.78 9/186 2173
/proc/meminfo: memFree=10929244/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4079852 CPUtime=3522.1 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 944188 0 0 0 347733 4477 0 0 25 0 5 0 516612071 4177768448 940129 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1019963 940129 176 36 0 1017868 0
[pid=2113/tid=2142] ppid=2111 vsize=4079852 CPUtime=879.18 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 449413 0 0 0 87723 195 0 0 25 0 5 0 516612124 4177768448 940129 33554432000 4194304 4338100 140736207262272 18446744073709551615 4235674 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4079852 CPUtime=880.66 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 67776 0 0 0 85978 2088 0 0 25 0 5 0 516612124 4177768448 940129 33554432000 4194304 4338100 140736207262272 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4079852 CPUtime=880.33 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 334932 0 0 0 87947 86 0 0 25 0 5 0 516612124 4177768448 940129 33554432000 4194304 4338100 140736207262272 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4079852 CPUtime=881.35 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 71112 0 0 0 86033 2102 0 0 25 0 5 0 516612124 4177768448 940129 33554432000 4194304 4338100 140736207262272 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3522.1
Current children cumulated vsize (KiB) 4079852

[startup+942.301 s]
/proc/loadavg: 8.12 8.12 7.80 9/186 2175
/proc/meminfo: memFree=10500804/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4308328 CPUtime=3761.69 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 1000930 0 0 0 371471 4698 0 0 25 0 5 0 516612071 4411727872 996871 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1077082 996871 176 36 0 1074987 0
[pid=2113/tid=2142] ppid=2111 vsize=4308328 CPUtime=939.15 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 479617 0 0 0 93709 206 0 0 25 0 5 0 516612124 4411727872 996871 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4308328 CPUtime=940.66 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 67792 0 0 0 91879 2187 0 0 25 0 5 0 516612124 4411727872 996871 33554432000 4194304 4338100 140736207262272 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4308328 CPUtime=940.32 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 361436 0 0 0 93937 95 0 0 25 0 5 0 516612124 4411727872 996871 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4308328 CPUtime=941.01 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 71130 0 0 0 91895 2206 0 0 25 0 5 0 516612124 4411727872 996871 33554432000 4194304 4338100 140736207262272 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3761.69
Current children cumulated vsize (KiB) 4308328

[startup+1002.3 s]
/proc/loadavg: 8.04 8.09 7.81 9/186 2176
/proc/meminfo: memFree=10366252/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4374016 CPUtime=4001.31 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 1022457 0 0 0 395195 4936 0 0 25 0 5 0 516612071 4478992384 1018398 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1093504 1018398 176 36 0 1091409 0
[pid=2113/tid=2142] ppid=2111 vsize=4374016 CPUtime=999.14 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 488439 0 0 0 99700 214 0 0 25 0 5 0 516612124 4478992384 1018398 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820979867 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4374016 CPUtime=1000.63 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 69017 0 0 0 97757 2306 0 0 25 0 5 0 516612124 4478992384 1018398 33554432000 4194304 4338100 140736207262272 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4374016 CPUtime=1000.32 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 369587 0 0 0 99935 97 0 0 25 0 5 0 516612124 4478992384 1018398 33554432000 4194304 4338100 140736207262272 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4374016 CPUtime=1000.66 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 74459 0 0 0 97751 2315 0 0 25 0 5 0 516612124 4478992384 1018398 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4001.31
Current children cumulated vsize (KiB) 4374016

[startup+1062.3 s]
/proc/loadavg: 8.06 8.09 7.83 9/186 2178
/proc/meminfo: memFree=10177976/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4599956 CPUtime=4240.81 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 1059826 0 0 0 418918 5163 0 0 25 0 5 0 516612071 4710354944 1055767 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1149989 1055767 176 36 0 1147894 0
[pid=2113/tid=2142] ppid=2111 vsize=4599956 CPUtime=1059.08 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 507612 0 0 0 105683 225 0 0 25 0 5 0 516612124 4710354944 1055767 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4599956 CPUtime=1060.6 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 69478 0 0 0 103650 2410 0 0 25 0 5 0 516612124 4710354944 1055767 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4599956 CPUtime=1060.29 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 387021 0 0 0 105927 102 0 0 25 0 5 0 516612124 4710354944 1055767 33554432000 4194304 4338100 140736207262272 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4599956 CPUtime=1060.27 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 74760 0 0 0 103606 2421 0 0 25 0 5 0 516612124 4710354944 1055767 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4240.81
Current children cumulated vsize (KiB) 4599956

[startup+1122.31 s]
/proc/loadavg: 8.06 8.08 7.84 9/186 2179
/proc/meminfo: memFree=9880620/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4809664 CPUtime=4480.46 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 1111416 0 0 0 442668 5378 0 0 25 0 5 0 516612071 4925095936 1107357 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1202416 1107357 176 36 0 1200321 0
[pid=2113/tid=2142] ppid=2111 vsize=4809664 CPUtime=1119.07 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 532194 0 0 0 111670 237 0 0 25 0 5 0 516612124 4925095936 1107357 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4809664 CPUtime=1120.61 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 69496 0 0 0 109553 2508 0 0 25 0 5 0 516612124 4925095936 1107357 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4809664 CPUtime=1120.29 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 413913 0 0 0 111918 111 0 0 25 0 5 0 516612124 4925095936 1107357 33554432000 4194304 4338100 140736207262272 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4809664 CPUtime=1119.95 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 74858 0 0 0 109476 2519 0 0 25 0 5 0 516612124 4925095936 1107357 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4480.46
Current children cumulated vsize (KiB) 4809664

[startup+1182.3 s]
/proc/loadavg: 8.06 8.08 7.85 9/186 2181
/proc/meminfo: memFree=9703652/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=4962936 CPUtime=4720.09 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) S 2111 2113 1994 0 -1 4202496 1150463 0 0 0 466435 5574 0 0 25 0 5 0 516612071 5082046464 1146404 33554432000 4194304 4338100 140736207262272 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 1240734 1146404 176 36 0 1238639 0
[pid=2113/tid=2142] ppid=2111 vsize=4962936 CPUtime=1179.05 cores=1,3,5,7
/proc/2113/task/2142/stat : 2142 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 552819 0 0 0 117656 249 0 0 25 0 5 0 516612124 5082046464 1146404 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=4962936 CPUtime=1180.59 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 69516 0 0 0 115463 2596 0 0 25 0 5 0 516612124 5082046464 1146404 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2113/tid=2144] ppid=2111 vsize=4962936 CPUtime=1180.27 cores=1,3,5,7
/proc/2113/task/2144/stat : 2144 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 432296 0 0 0 117910 117 0 0 25 0 5 0 516612124 5082046464 1146404 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2113/tid=2145] ppid=2111 vsize=4962936 CPUtime=1179.62 cores=1,3,5,7
/proc/2113/task/2145/stat : 2145 (ttsth-5-4) R 2111 2113 1994 0 -1 4202560 74877 0 0 0 115354 2608 0 0 25 0 5 0 516612124 5082046464 1146404 33554432000 4194304 4338100 140736207262272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4720.09
Current children cumulated vsize (KiB) 4962936



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.4 s]
/proc/loadavg: 8.11 8.09 7.86 6/183 2182
/proc/meminfo: memFree=11400560/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) Z 2111 2113 1994 0 -1 4203548 1163168 0 0 0 474358 5653 0 0 18 0 2 0 516612071 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 0 0 0 0 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=0 CPUtime=1200.7 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4203588 69522 0 0 0 117431 2639 0 0 25 0 2 0 516612124 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.4 s]
/proc/loadavg: 8.11 8.09 7.86 6/183 2182
/proc/meminfo: memFree=11400560/32951124 swapFree=67111528/67111528
[pid=2113] ppid=2111 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/2113/stat : 2113 (ttsth-5-4) Z 2111 2113 1994 0 -1 4203548 1163168 0 0 0 474358 5653 0 0 18 0 2 0 516612071 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/2113/statm: 0 0 0 0 0 0 0
[pid=2113/tid=2143] ppid=2111 vsize=0 CPUtime=1200.7 cores=1,3,5,7
/proc/2113/task/2143/stat : 2143 (ttsth-5-4) R 2111 2113 1994 0 -1 4203588 69522 0 0 0 117431 2639 0 0 25 0 2 0 516612124 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.51
CPU time (s): 4800.22
CPU user time (s): 4743.58
CPU system time (s): 56.6384
CPU usage (%): 399.184
Max. virtual memory (cumulated for all children) (KiB): 5016112

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4743.58
system time used= 56.6384
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1163168
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4674
involuntary context switches= 31712

runsolver used 2.07168 second user time and 4.50031 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 06:08:36
IDJOB=3258262
IDBENCH=71317
IDSOLVER=1654
FILE ID=node104/3258262-1304050116
RUNJOBID= node104-1304048911-2013
PBS_JOBID= 13170409
Free space on /tmp= 71548 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v1375c111739g4.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258262-1304050116/watcher-3258262-1304050116 -o /tmp/evaluation-result-3258262-1304050116/solver-3258262-1304050116 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258262-1304050116.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b2f1edd9e4045a375bcfedb2d5e5bd0e
RANDOM SEED=504088123

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18304388 kB
Buffers:       2195868 kB
Cached:       11798596 kB
SwapCached:          0 kB
Active:        2336452 kB
Inactive:     11724088 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18304388 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            1192 kB
Writeback:           0 kB
AnonPages:       65492 kB
Mapped:          15072 kB
Slab:           522644 kB
PageTables:       4628 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   220156 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71548 MiB
End job on node104 at 2011-04-29 06:28:41