Trace number 3258042

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.41 1203.73

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v945c61409g6.cnf
MD5SUM2094ee8d4e443e860293b947e469bd81
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61409
Sum of the clauses size123673
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261364
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258042-1304048911.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c a2
0.00/0.08	c a1
0.09/0.15	c Independent sub-problems = 1
0.09/0.15	c Sub-problem with 945 variables and 61409 clauses
0.09/0.15	c using permutation method 4
0.09/0.15	c count = 1
0.09/0.15	c Ordering finished 0.140 seconds
0.09/0.15	c mapped 0.150 seconds
0.09/0.15	c permuted 0.150 seconds
0.09/0.15	c about to sort
0.09/0.16	c sorted
0.19/0.29	c clauses now 61409
0.19/0.29	c init finished 0.170 seconds
0.19/0.29	c tree built 0.290 seconds
0.19/0.29	c 
0.19/0.29	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.29	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.29	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.29	c ................................ | ..........                  1 |                                  0.3         3.4             
0.19/0.29	c ................................ | ..........                  2 |                                  0.3         6.9             
0.19/0.29	c ................................ | ..........                  3 |                                  0.3        10.3             
0.19/0.29	c ................................ | ..........                  4 |                                  0.3        13.8             
0.19/0.29	c ................................ | ..........                  5 |                                  0.3        17.2             
0.30/0.30	c ................................ | ..........               1000 | 7            108                 0.3      3333.3         5575
95.75/24.36	c ................................ | ..........            4191000 | 141653       596926             96.0     43656.2     20722206
191.62/48.49	c ................................ | ..........            7602000 | 259599       1017868           192.0     39593.8     31232971
288.01/72.60	c ................................ | ..........           10427000 | 352191       1372068           288.0     36203.6     40894112
383.87/96.64	c ................................ | ..........           13150000 | 436484       1697048           384.0     34242.1     47793356
479.70/120.68	c ................................ | *.........           15697000 | 517501       2019734           480.0     32701.4     60146925
575.88/144.73	c ................................ | *.........           18350000 | 609920       2350649           576.0     31857.1     65591382
671.86/168.83	c ................................ | *.........           21133000 | 712589       2653103           672.0     31447.0     74635780
767.91/192.93	c ................................ | *.........           23567000 | 810704       2990135           768.0     30685.0     83757953
863.74/216.96	c ................................ | *.........           25868000 | 900458       3289277           864.0     29939.8     91225248
959.92/241.02	c ................................ | **........           27935000 | 964729       3590691           960.0     29098.4     99096709
1055.78/265.05	c ................................ | **........           29941000 | 1028351      3898381          1056.0     28353.2    110954059
1152.01/289.10	c ................................ | **........           31953000 | 1101884      4172278          1152.0     27736.3    120704632
1247.85/313.15	c ................................ | **........           34117000 | 1188565      4422832          1248.0     27336.7    125392749
1344.00/337.20	c ................................ | **........           36400000 | 1287281      4653251          1344.0     27083.3    125728984
1439.83/361.24	c ................................ | ***.......           38644000 | 1371246      4889359          1440.0     26836.1    127971615
1535.62/385.29	c ................................ | ***.......           40850000 | 1461295      5123525          1536.0     26594.9    135565865
1631.77/409.36	c ................................ | ***.......           43089000 | 1559717      5383748          1632.0     26402.2    136573379
1727.95/433.42	c ................................ | ***.......           45111000 | 1644258      5608512          1728.0     26105.6    137216081
1823.74/457.46	c ................................ | ***.......           47095000 | 1725591      5843817          1824.0     25819.3    141748695
1919.92/481.52	c ................................ | ****......           49158000 | 1800825      6073734          1920.0     25602.9    144897224
2015.70/505.58	c ................................ | ****......           51312000 | 1873560      6293046          2016.0     25452.4    146556650
2111.93/529.61	c ................................ | ****......           53466000 | 1954733      6525026          2112.0     25315.2    153424409
2207.69/553.68	c ................................ | ****......           55390000 | 2030609      6749782          2208.0     25085.6    154771930
2303.93/577.72	c ................................ | ****......           57321000 | 2096072      6968927          2304.0     24878.5    161692521
2399.73/601.77	c ................................ | *****.....           59258000 | 2171870      7186235          2400.0     24690.4    163066953
2495.95/625.82	c ................................ | *****.....           61321000 | 2266261      7385574          2496.0     24567.4    165605014
2591.68/649.88	c ................................ | *****.....           63281000 | 2349506      7603813          2592.0     24414.0    171840852
2687.80/673.96	c ................................ | *****.....           65027000 | 2416794      7822454          2688.0     24191.4    177892617
2783.68/698.09	c ................................ | *****.....           66571000 | 2469774      8044410          2784.1     23911.5    178776792
2879.72/722.18	c ................................ | ******....           68137000 | 2529451      8263616          2880.0     23658.4    181623859
2975.88/746.24	c ................................ | ******....           69658000 | 2583819      8479196          2976.1     23406.2    184575546
3071.72/770.28	c ................................ | ******....           71147000 | 2639797      8696274          3072.0     23159.5    187858183
3167.87/794.33	c ................................ | ******....           72742000 | 2711592      8891917          3168.0     22961.4    189604791
3264.04/818.40	c ................................ | ******....           74269000 | 2771172      9098797          3264.1     22753.6    195605493
3359.85/842.45	c ................................ | *******...           75731000 | 2824574      9297362          3360.1     22538.6    197382216
3456.02/866.50	c ................................ | *******...           77160000 | 2872159      9484298          3456.1     22326.1    200413489
3551.79/890.57	c ................................ | *******...           78543000 | 2921023      9652901          3552.1     22112.0    202899869
3647.97/914.61	c ................................ | *******...           79859000 | 2971611      9833589          3648.0     21891.1    205239496
3743.64/938.68	c ................................ | *******...           81290000 | 3027654      10007536         3744.0     21712.1    207976815
3839.77/962.76	c ................................ | ********..           82768000 | 3084684      10182198         3840.1     21553.9    208954009
3936.00/986.80	c ................................ | ********..           84198000 | 3135456      10353145         3936.0     21391.7    211271889
4031.83/1010.85	c ................................ | ********..           85564000 | 3188261      10520800         4032.0     21221.0    213810388
4127.66/1034.90	c ................................ | ********..           86955000 | 3247921      10722281         4128.1     21064.4    218118047
4223.86/1058.94	c ................................ | ********..           88452000 | 3308877      10931872         4224.0     20940.2    223276586
4319.70/1082.98	c ................................ | *********.           90291000 | 3382135      11146326         4320.1     20900.5    229109288
4415.88/1107.04	c ................................ | *********.           91988000 | 3451930      11339634         4416.1     20830.4    233805866
4511.70/1131.08	c ................................ | *********.           93533000 | 3515173      11517051         4512.0     20729.8    237589831
4607.81/1155.15	c ................................ | *********.           94989000 | 3572758      11686798         4608.0     20613.8    239803957
4703.76/1179.26	c ................................ | *********.           96379000 | 3622085      11860098         4704.0     20488.7    239803957
4799.74/1203.38	c CPU limit (4800) reached
4799.74/1203.38	c Time = 4800.05 seconds
4799.74/1203.38	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258042-1304048911/watcher-3258042-1304048911 -o /tmp/evaluation-result-3258042-1304048911/solver-3258042-1304048911 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258042-1304048911.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.40 4.19 5.33 3/177 2032
/proc/meminfo: memFree=18299888/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=9112 CPUtime=0 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) R 2030 2032 1994 0 -1 4202496 359 0 0 0 0 0 0 0 25 0 1 0 516491553 9330688 301 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/2032/statm: 2278 302 147 36 0 183 0

[startup+0.0902841 s]
/proc/loadavg: 3.40 4.19 5.33 3/177 2032
/proc/meminfo: memFree=18299888/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=21376 CPUtime=0.08 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) R 2030 2032 1994 0 -1 4202496 3386 0 0 0 8 0 0 0 25 0 1 0 516491553 21889024 3327 33554432000 4194304 4338100 140734837043536 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/2032/statm: 5344 3344 153 36 0 3249 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 21376

[startup+0.101269 s]
/proc/loadavg: 3.40 4.19 5.33 3/177 2032
/proc/meminfo: memFree=18299888/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=19448 CPUtime=0.09 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) R 2030 2032 1994 0 -1 4202496 3513 0 0 0 9 0 0 0 25 0 1 0 516491553 19914752 2890 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/2032/statm: 4862 2892 155 36 0 2767 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19448

[startup+0.301235 s]
/proc/loadavg: 3.40 4.19 5.33 3/177 2032
/proc/meminfo: memFree=18299888/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=232040 CPUtime=0.3 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 11741 0 0 0 29 1 0 0 25 0 5 0 516491553 237608960 10878 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 58010 10880 176 36 0 55915 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 232040

[startup+1.13894 s]
/proc/loadavg: 3.40 4.19 5.33 3/177 2032
/proc/meminfo: memFree=18299888/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=264244 CPUtime=3.63 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 20904 0 0 0 359 4 0 0 25 0 5 0 516491553 270585856 20041 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 66061 20041 176 36 0 63966 0
Current children cumulated CPU time (s) 3.63
Current children cumulated vsize (KiB) 264244

[startup+1.50092 s]
/proc/loadavg: 3.40 4.19 5.33 3/177 2032
/proc/meminfo: memFree=18299888/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=280016 CPUtime=5.08 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 24409 0 0 0 503 5 0 0 25 0 5 0 516491553 286736384 23546 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 70004 23546 176 36 0 67909 0
Current children cumulated CPU time (s) 5.08
Current children cumulated vsize (KiB) 280016

[startup+3.10062 s]
/proc/loadavg: 3.85 4.27 5.35 9/186 2041
/proc/meminfo: memFree=17972348/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=339712 CPUtime=11.44 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 39708 0 0 0 1134 10 0 0 25 0 5 0 516491553 347865088 38845 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 84928 38845 176 36 0 82833 0
[pid=2032/tid=2034] ppid=2030 vsize=339712 CPUtime=2.79 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 9452 0 0 0 278 1 0 0 23 0 5 0 516491582 347865088 38845 33554432000 4194304 4338100 140734837043536 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=339712 CPUtime=2.79 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 3671 0 0 0 276 3 0 0 21 0 5 0 516491582 347865088 38845 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=339712 CPUtime=2.77 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 9196 0 0 0 276 1 0 0 23 0 5 0 516491582 347865088 38845 33554432000 4194304 4338100 140734837043536 18446744073709551615 4263517 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=339712 CPUtime=2.75 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 5895 0 0 0 273 2 0 0 18 0 5 0 516491582 347865088 38846 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.44
Current children cumulated vsize (KiB) 339712

[startup+6.30103 s]
/proc/loadavg: 3.85 4.27 5.35 9/186 2041
/proc/meminfo: memFree=17792556/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=453396 CPUtime=24.17 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 67675 0 0 0 2398 19 0 0 25 0 5 0 516491553 464277504 66812 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 113349 66812 176 36 0 111254 0
[pid=2032/tid=2034] ppid=2030 vsize=453396 CPUtime=6 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 19279 0 0 0 596 4 0 0 25 0 5 0 516491582 464277504 66812 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808561056 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=453396 CPUtime=5.99 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 7194 0 0 0 593 6 0 0 25 0 5 0 516491582 464277504 66812 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808560737 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=453396 CPUtime=5.96 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 19894 0 0 0 593 3 0 0 25 0 5 0 516491582 464277504 66812 33554432000 4194304 4338100 140734837043536 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=453396 CPUtime=5.89 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 9814 0 0 0 586 3 0 0 18 0 5 0 516491582 464277504 66812 33554432000 4194304 4338100 140734837043536 18446744073709551615 4263856 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.17
Current children cumulated vsize (KiB) 453396

[startup+12.7009 s]
/proc/loadavg: 4.49 4.39 5.37 9/186 2042
/proc/meminfo: memFree=17481552/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=638908 CPUtime=49.62 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 114582 0 0 0 4922 40 0 0 25 0 5 0 516491553 654241792 113351 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 159727 113351 176 36 0 157632 0
[pid=2032/tid=2034] ppid=2030 vsize=638908 CPUtime=12.4 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 37351 0 0 0 1232 8 0 0 25 0 5 0 516491582 654241792 113351 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=638908 CPUtime=12.4 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 12334 0 0 0 1224 16 0 0 25 0 5 0 516491582 654241792 113351 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=638908 CPUtime=12.33 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 38147 0 0 0 1227 6 0 0 25 0 5 0 516491582 654241792 113351 33554432000 4194304 4338100 140734837043536 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=638908 CPUtime=12.18 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 15256 0 0 0 1210 8 0 0 18 0 5 0 516491582 654241792 113351 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.62
Current children cumulated vsize (KiB) 638908

[startup+25.5005 s]
/proc/loadavg: 5.10 4.52 5.41 9/186 2042
/proc/meminfo: memFree=17019052/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=928900 CPUtime=100.53 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 185291 0 0 0 9975 78 0 0 25 0 5 0 516491553 951193600 184060 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 232225 184060 176 36 0 230130 0
[pid=2032/tid=2034] ppid=2030 vsize=928900 CPUtime=25.19 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 62188 0 0 0 2505 14 0 0 25 0 5 0 516491582 951193600 184060 33554432000 4194304 4338100 140734837043536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=928900 CPUtime=25.19 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 19771 0 0 0 2484 35 0 0 25 0 5 0 516491582 951193600 184060 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=928900 CPUtime=25.06 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 70799 0 0 0 2495 11 0 0 25 0 5 0 516491582 951193600 184060 33554432000 4194304 4338100 140734837043536 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=928900 CPUtime=24.77 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 21039 0 0 0 2462 15 0 0 18 0 5 0 516491582 951193600 184060 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.53
Current children cumulated vsize (KiB) 928900

[startup+51.1008 s]
/proc/loadavg: 6.09 4.80 5.47 9/186 2043
/proc/meminfo: memFree=16378476/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=1309020 CPUtime=202.37 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 279975 0 0 0 20097 140 0 0 25 0 5 0 516491553 1340436480 278744 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 327255 278744 176 36 0 325160 0
[pid=2032/tid=2034] ppid=2030 vsize=1309020 CPUtime=50.77 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 101814 0 0 0 5052 25 0 0 21 0 5 0 516491582 1340436480 278744 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=1309020 CPUtime=50.77 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 27094 0 0 0 5008 69 0 0 25 0 5 0 516491582 1340436480 278744 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=1309020 CPUtime=50.53 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 107918 0 0 0 5036 17 0 0 25 0 5 0 516491582 1340436480 278744 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808561402 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=1309020 CPUtime=49.97 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 31655 0 0 0 4971 26 0 0 18 0 5 0 516491582 1340436480 278744 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.37
Current children cumulated vsize (KiB) 1309020

[startup+102.3 s]
/proc/loadavg: 7.26 5.32 5.61 9/186 2044
/proc/meminfo: memFree=15447088/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=1932252 CPUtime=406.61 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 431078 0 0 0 40242 419 0 0 25 0 5 0 516491553 1978626048 429848 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 483063 429848 176 36 0 480968 0
[pid=2032/tid=2034] ppid=2030 vsize=1932252 CPUtime=101.95 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 169103 0 0 0 10150 45 0 0 25 0 5 0 516491582 1978626048 429849 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=1932252 CPUtime=101.76 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 36683 0 0 0 9978 198 0 0 25 0 5 0 516491582 1978626048 429849 33554432000 4194304 4338100 140734837043536 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=1932252 CPUtime=101.63 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 171241 0 0 0 10133 30 0 0 25 0 5 0 516491582 1978626048 429849 33554432000 4194304 4338100 140734837043536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=1932252 CPUtime=100.94 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 42559 0 0 0 9951 143 0 0 25 0 5 0 516491582 1978626048 429849 33554432000 4194304 4338100 140734837043536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 406.61
Current children cumulated vsize (KiB) 1932252

[startup+162.3 s]
/proc/loadavg: 7.76 5.84 5.77 9/186 2046
/proc/meminfo: memFree=14392564/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=2556044 CPUtime=645.99 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 587898 0 0 0 63845 754 0 0 25 0 5 0 516491553 2617389056 586667 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 639011 586667 176 36 0 636916 0
[pid=2032/tid=2034] ppid=2030 vsize=2556044 CPUtime=161.93 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 241407 0 0 0 16129 64 0 0 25 0 5 0 516491582 2617389056 586670 33554432000 4194304 4338100 140734837043536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=2556044 CPUtime=161.49 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 42657 0 0 0 15814 335 0 0 18 0 5 0 516491582 2617389056 586670 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=2556044 CPUtime=161.48 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 245316 0 0 0 16103 45 0 0 25 0 5 0 516491582 2617389056 586670 33554432000 4194304 4338100 140734837043536 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=2556044 CPUtime=160.76 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 47027 0 0 0 15770 306 0 0 25 0 5 0 516491582 2617389056 586670 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 645.99
Current children cumulated vsize (KiB) 2556044

[startup+222.3 s]
/proc/loadavg: 7.95 6.22 5.90 9/186 2047
/proc/meminfo: memFree=13511288/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=3248784 CPUtime=885.28 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 761272 0 0 0 87424 1104 0 0 25 0 5 0 516491553 3326754816 760041 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 812196 760041 176 36 0 810101 0
[pid=2032/tid=2034] ppid=2030 vsize=3248784 CPUtime=221.82 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 312173 0 0 0 22100 82 0 0 25 0 5 0 516491582 3326754816 760041 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 104247 0 0 0 68172 1823 0 0 25 0 5 0 516491582 6385430528 1478048 33554432000 4194304 4338100 140734837043536 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2800.83
Current children cumulated vsize (KiB) 6235772

[startup+762.301 s]
/proc/loadavg: 8.07 7.80 6.87 9/186 2071
/proc/meminfo: memFree=8319900/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=6439884 CPUtime=3040.17 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1531915 0 0 0 299728 4289 0 0 25 0 5 0 516491553 6594441216 1530684 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 1609971 1530684 176 36 0 1607876 0
[pid=2032/tid=2034] ppid=2030 vsize=6439884 CPUtime=760.61 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 673784 0 0 0 75865 196 0 0 25 0 5 0 516491582 6594441216 1530684 33554432000 4194304 4338100 140734837043536 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=6439884 CPUtime=759.68 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 113851 0 0 0 73992 1976 0 0 25 0 5 0 516491582 6594441216 1530684 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=6439884 CPUtime=760.02 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 627643 0 0 0 75883 119 0 0 25 0 5 0 516491582 6594441216 1530684 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=6439884 CPUtime=759.53 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 105143 0 0 0 73958 1995 0 0 25 0 5 0 516491582 6594441216 1530684 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3040.17
Current children cumulated vsize (KiB) 6439884

[startup+822.301 s]
/proc/loadavg: 8.07 7.86 6.94 9/186 2072
/proc/meminfo: memFree=7774120/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=6750216 CPUtime=3279.62 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1610883 0 0 0 323343 4619 0 0 25 0 5 0 516491553 6912221184 1609652 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 1687554 1609652 176 36 0 1685459 0
[pid=2032/tid=2034] ppid=2030 vsize=6750216 CPUtime=820.25 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 707384 0 0 0 81819 206 0 0 25 0 5 0 516491582 6912221184 1609654 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254077 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=6750216 CPUtime=819.63 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 114593 0 0 0 79830 2133 0 0 25 0 5 0 516491582 6912221184 1609654 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=6750216 CPUtime=819.97 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 671812 0 0 0 81868 129 0 0 25 0 5 0 516491582 6912221184 1609654 33554432000 4194304 4338100 140734837043536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=6750216 CPUtime=819.45 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 105602 0 0 0 79797 2148 0 0 25 0 5 0 516491582 6912221184 1609654 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3279.62
Current children cumulated vsize (KiB) 6750216

[startup+882.301 s]
/proc/loadavg: 7.92 7.85 6.99 9/186 2074
/proc/meminfo: memFree=7326824/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=6941552 CPUtime=3519.06 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1662029 0 0 0 346994 4912 0 0 25 0 5 0 516491553 7108149248 1660798 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 1735388 1660798 176 36 0 1733293 0
[pid=2032/tid=2034] ppid=2030 vsize=6941552 CPUtime=879.95 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 722068 0 0 0 87782 213 0 0 25 0 5 0 516491582 7108149248 1660798 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=6941552 CPUtime=879.53 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 116258 0 0 0 85683 2270 0 0 25 0 5 0 516491582 7108149248 1660798 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=6941552 CPUtime=879.93 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 704971 0 0 0 87855 138 0 0 25 0 5 0 516491582 7108149248 1660798 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=6941552 CPUtime=879.32 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 107238 0 0 0 85644 2288 0 0 25 0 5 0 516491582 7108149248 1660798 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3519.06
Current children cumulated vsize (KiB) 6941552

[startup+942.307 s]
/proc/loadavg: 8.11 7.91 7.07 9/186 2078
/proc/meminfo: memFree=7003092/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=7132888 CPUtime=3758.44 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1710908 0 0 0 370668 5176 0 0 25 0 5 0 516491553 7304077312 1709677 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 1783222 1709677 176 36 0 1781127 0
[pid=2032/tid=2034] ppid=2030 vsize=7132888 CPUtime=939.81 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 746195 0 0 0 93760 221 0 0 25 0 5 0 516491582 7304077312 1709677 33554432000 4194304 4338100 140734837043536 18446744073709551615 4284214 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=7132888 CPUtime=939.14 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 116425 0 0 0 91516 2398 0 0 25 0 5 0 516491582 7304077312 1709677 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808862251 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=7132888 CPUtime=939.92 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 729483 0 0 0 93850 142 0 0 25 0 5 0 516491582 7304077312 1709677 33554432000 4194304 4338100 140734837043536 18446744073709551615 4263529 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=7132888 CPUtime=939.25 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 107311 0 0 0 91513 2412 0 0 25 0 5 0 516491582 7304077312 1709677 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3758.44
Current children cumulated vsize (KiB) 7132888

[startup+1002.3 s]
/proc/loadavg: 8.08 7.93 7.12 9/186 2080
/proc/meminfo: memFree=6746624/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=7308584 CPUtime=3997.88 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1757394 0 0 0 394345 5443 0 0 25 0 5 0 516491553 7483990016 1756163 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 1827146 1756163 176 36 0 1825051 0
[pid=2032/tid=2034] ppid=2030 vsize=7308584 CPUtime=999.41 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 778775 0 0 0 99709 232 0 0 25 0 5 0 516491582 7483990016 1756163 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=7308584 CPUtime=999.1 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 117948 0 0 0 97386 2524 0 0 25 0 5 0 516491582 7483990016 1756163 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=7308584 CPUtime=999.86 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 740507 0 0 0 99842 144 0 0 25 0 5 0 516491582 7483990016 1756163 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808862176 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=7308584 CPUtime=999.18 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 108670 0 0 0 97378 2540 0 0 25 0 5 0 516491582 7483990016 1756163 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3997.88
Current children cumulated vsize (KiB) 7308584

[startup+1062.3 s]
/proc/loadavg: 8.11 7.97 7.19 9/186 2082
/proc/meminfo: memFree=6331320/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=7646936 CPUtime=4237.41 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1842308 0 0 0 417957 5784 0 0 25 0 5 0 516491553 7830462464 1841077 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 1911734 1841077 176 36 0 1909639 0
[pid=2032/tid=2034] ppid=2030 vsize=7646936 CPUtime=1059.23 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 825150 0 0 0 105676 247 0 0 25 0 5 0 516491582 7830462464 1841077 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=7646936 CPUtime=1059.03 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 118038 0 0 0 103219 2684 0 0 25 0 5 0 516491582 7830462464 1841077 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=7646936 CPUtime=1059.69 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 778853 0 0 0 105817 152 0 0 25 0 5 0 516491582 7830462464 1841077 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=7646936 CPUtime=1059.12 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 108773 0 0 0 103215 2697 0 0 25 0 5 0 516491582 7830462464 1841078 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4237.41
Current children cumulated vsize (KiB) 7646936

[startup+1122.3 s]
/proc/loadavg: 8.08 7.99 7.24 9/186 2083
/proc/meminfo: memFree=5825748/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=8010576 CPUtime=4476.96 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1933961 0 0 0 441614 6082 0 0 25 0 5 0 516491553 8202829824 1932730 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 2002644 1932730 176 36 0 2000549 0
[pid=2032/tid=2034] ppid=2030 vsize=8010576 CPUtime=1119.19 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 871143 0 0 0 111659 260 0 0 25 0 5 0 516491582 8202829824 1932730 33554432000 4194304 4338100 140734837043536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=8010576 CPUtime=1119.02 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 118122 0 0 0 109077 2825 0 0 25 0 5 0 516491582 8202829824 1932730 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=8010576 CPUtime=1119.36 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 824349 0 0 0 111775 161 0 0 25 0 5 0 516491582 8202829824 1932730 33554432000 4194304 4338100 140734837043536 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=8010576 CPUtime=1119.08 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 108853 0 0 0 109075 2833 0 0 25 0 5 0 516491582 8202829824 1932730 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4476.96
Current children cumulated vsize (KiB) 8010576

[startup+1182.3 s]
/proc/loadavg: 8.05 8.00 7.29 9/186 2085
/proc/meminfo: memFree=5474012/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=8133096 CPUtime=4716.13 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) S 2030 2032 1994 0 -1 4202496 1968721 0 0 0 465354 6259 0 0 25 0 5 0 516491553 8328290304 1967490 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 2033274 1967490 176 36 0 2031179 0
[pid=2032/tid=2034] ppid=2030 vsize=8133096 CPUtime=1178.9 cores=0,2,4,6
/proc/2032/task/2034/stat : 2034 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 892175 0 0 0 117619 271 0 0 25 0 5 0 516491582 8328290304 1967490 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808561074 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=8133096 CPUtime=1178.85 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 118567 0 0 0 114965 2920 0 0 25 0 5 0 516491582 8328290304 1967490 33554432000 4194304 4338100 140734837043536 18446744073709551615 230808561121 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2032/tid=2036] ppid=2030 vsize=8133096 CPUtime=1179.24 cores=0,2,4,6
/proc/2032/task/2036/stat : 2036 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 833525 0 0 0 117761 163 0 0 25 0 5 0 516491582 8328290304 1967490 33554432000 4194304 4338100 140734837043536 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2032/tid=2037] ppid=2030 vsize=8133096 CPUtime=1178.82 cores=0,2,4,6
/proc/2032/task/2037/stat : 2037 (ttsth-5-4) R 2030 2032 1994 0 -1 4202560 112960 0 0 0 114980 2902 0 0 18 0 5 0 516491582 8328290304 1967490 33554432000 4194304 4338100 140734837043536 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4716.13
Current children cumulated vsize (KiB) 8133096



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.4 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2086
/proc/meminfo: memFree=10702288/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=0 CPUtime=4800.08 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) Z 2030 2032 1994 0 -1 4203548 1991509 0 0 0 473715 6293 0 0 18 0 2 0 516491553 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 0 0 0 0 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=0 CPUtime=1199.92 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4203588 123049 0 0 0 117047 2945 0 0 25 0 2 0 516491582 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.7 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2086
/proc/meminfo: memFree=10702288/32951124 swapFree=67111528/67111528
[pid=2032] ppid=2030 vsize=0 CPUtime=4800.37 cores=0,2,4,6
/proc/2032/stat : 2032 (ttsth-5-4) Z 2030 2032 1994 0 -1 4203548 1991509 0 0 0 473715 6322 0 0 18 0 2 0 516491553 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2032/statm: 0 0 0 0 0 0 0
[pid=2032/tid=2035] ppid=2030 vsize=0 CPUtime=1200.22 cores=0,2,4,6
/proc/2032/task/2035/stat : 2035 (ttsth-5-4) R 2030 2032 1994 0 -1 4203588 123049 0 0 0 117047 2975 0 0 25 0 2 0 516491582 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.73
CPU time (s): 4800.41
CPU user time (s): 4737.16
CPU system time (s): 63.2514
CPU usage (%): 398.796
Max. virtual memory (cumulated for all children) (KiB): 8197500

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4737.16
system time used= 63.2514
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1991509
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10514
involuntary context switches= 22403

runsolver used 1.90671 second user time and 4.15837 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 05:48:31
IDJOB=3258042
IDBENCH=71300
IDSOLVER=1654
FILE ID=node104/3258042-1304048911
RUNJOBID= node104-1304048911-2012
PBS_JOBID= 13170409
Free space on /tmp= 71548 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v945c61409g6.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258042-1304048911/watcher-3258042-1304048911 -o /tmp/evaluation-result-3258042-1304048911/solver-3258042-1304048911 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258042-1304048911.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 2094ee8d4e443e860293b947e469bd81
RANDOM SEED=1217715903

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18300416 kB
Buffers:       2195808 kB
Cached:       11799524 kB
SwapCached:          0 kB
Active:        2337436 kB
Inactive:     11724844 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18300416 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            3568 kB
Writeback:           0 kB
AnonPages:       66508 kB
Mapped:          14812 kB
Slab:           522708 kB
PageTables:       4456 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   213584 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71548 MiB
End job on node104 at 2011-04-29 06:08:36