Trace number 3259450

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.26 1203.41

General information on the benchmark

NameCRAFTED/rbsat/crafted/forced/
rbsat-v760c43649gyes8.cnf
MD5SUMca42736a2fe4e8cf2cba6dd1668e0ad6
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark3.9384
Satisfiable
(Un)Satisfiability was proved
Number of variables760
Number of clauses43649
Sum of the clauses size87978
Maximum clause length19
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243609
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259450-1304055275.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 760
0.00/0.03	c Clauses = 43649
0.00/0.03	c Literals = 87978
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 760
0.00/0.03	c Clauses = 43649
0.00/0.03	c Literals = 87978
0.00/0.03	c a2
0.03/0.05	c a1
0.09/0.10	c Independent sub-problems = 1
0.09/0.10	c Sub-problem with 760 variables and 43649 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 1
0.09/0.10	c Ordering finished 0.100 seconds
0.09/0.10	c mapped 0.100 seconds
0.09/0.10	c permuted 0.100 seconds
0.09/0.10	c about to sort
0.09/0.11	c sorted
0.19/0.20	c clauses now 43649
0.19/0.20	c init finished 0.120 seconds
0.19/0.20	c tree built 0.190 seconds
0.19/0.20	c 
0.19/0.20	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.20	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.20	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.20	c ................................ | ..........                  1 |                                  0.2         5.3             
0.19/0.20	c ................................ | ..........                  2 |                                  0.2        10.5             
0.19/0.20	c ................................ | ..........                  3 |                                  0.2        15.8             
0.19/0.20	c ................................ | ..........                  4 |                                  0.2        21.1             
0.19/0.20	c ................................ | ..........                  5 |                                  0.2        26.3             
0.19/0.20	c ................................ | ..........               1000 | 8            87                  0.2      5000.0         3335
95.78/24.55	c ................................ | ..........            3176000 | 72322        708745             96.0     33083.3     18180876
191.75/48.66	c ................................ | ..........            5428000 | 129865       1199507           192.0     28269.4     26247205
287.73/72.77	c ................................ | ..........            7366000 | 178448       1671850           288.0     25575.5     34456530
383.69/96.88	c ................................ | ..........            9396000 | 225661       2105569           384.0     24466.8     43889789
479.82/120.95	c ................................ | *.........           11199000 | 267877       2469422           480.0     23329.8     49907778
575.97/145.02	c ................................ | *.........           12774000 | 308562       2824517           576.1     22174.8     54648095
671.73/169.07	c ................................ | *.........           14158000 | 348319       3172068           672.0     21068.1     60414717
767.92/193.12	c ................................ | *.........           15538000 | 384370       3483987           768.0     20231.8     61577243
863.70/217.18	c ................................ | *.........           17205000 | 427136       3785253           864.0     19912.0     67189752
959.90/241.23	c ................................ | **........           18717000 | 464437       4087387           960.0     19496.5     71812012
1055.74/265.28	c ................................ | **........           20002000 | 499393       4371755          1056.1     18939.9     76706893
1151.91/289.33	c ................................ | **........           21277000 | 532214       4659621          1152.0     18469.0     81800247
1248.04/313.40	c ................................ | **........           22505000 | 563230       4919469          1248.0     18032.3     85757642
1343.88/337.43	c ................................ | **........           23694000 | 595383       5176965          1344.0     17629.5     92199756
1439.68/361.48	c ................................ | ***.......           24864000 | 627071       5421378          1440.0     17266.7     98201112
1535.85/385.54	c ................................ | ***.......           25879000 | 654836       5644113          1536.0     16848.3     98201112
1632.04/409.60	c ................................ | ***.......           26885000 | 683810       5864497          1632.1     16472.9     98201112
1727.85/433.65	c ................................ | ***.......           27827000 | 713835       6070743          1728.1     16103.0     99133870
1824.07/457.70	c ................................ | ***.......           28833000 | 741365       6294151          1824.1     15807.0    103326978
1919.91/481.72	c ................................ | ****......           29864000 | 771540       6523513          1920.0     15554.0    105577832
2015.72/505.78	c ................................ | ****......           30838000 | 801297       6741572          2016.1     15296.1    107811834
2111.92/529.82	c ................................ | ****......           31740000 | 829796       6941353          2112.0     15028.3    108479571
2207.74/553.87	c ................................ | ****......           32823000 | 858710       7152211          2208.0     14865.4    110756145
2303.92/577.92	c ................................ | ****......           34027000 | 892146       7358284          2304.0     14768.6    111545967
2399.71/601.97	c ................................ | *****.....           35316000 | 924478       7581229          2400.0     14714.9    114487597
2495.91/626.04	c ................................ | *****.....           36522000 | 954882       7799111          2496.1     14631.8    117359580
2591.74/650.07	c ................................ | *****.....           37651000 | 982777       7993538          2592.0     14525.6    119013437
2687.88/674.15	c ................................ | *****.....           38783000 | 1008601      8202784          2688.1     14427.8    120659582
2783.65/698.19	c ................................ | *****.....           39871000 | 1034909      8428346          2784.0     14321.4    121564083
2879.89/722.25	c ................................ | ******....           40898000 | 1059792      8662201          2880.1     14200.3    122920682
2975.65/746.30	c ................................ | ******....           41877000 | 1084280      8883761          2976.1     14071.3    124274142
3071.91/770.32	c ................................ | ******....           42853000 | 1110761      9094540          3072.0     13949.5    125674344
3167.73/794.38	c ................................ | ******....           43892000 | 1137046      9307404          3168.1     13854.5    127922601
3263.98/818.42	c ................................ | ******....           44970000 | 1170387      9503478          3264.1     13777.3    127931939
3359.81/842.45	c ................................ | *******...           45943000 | 1200417      9708286          3360.0     13673.3    130741505
3455.64/866.49	c ................................ | *******...           47030000 | 1231370      9923275          3456.0     13608.1    133710240
3551.90/890.53	c ................................ | *******...           48107000 | 1259355      10128710         3552.0     13543.6    135374979
3648.08/914.60	c ................................ | *******...           49095000 | 1286238      10324901         3648.1     13457.7    138572070
3743.90/938.63	c ................................ | *******...           50111000 | 1314359      10518880         3744.0     13384.3    140912443
3839.72/962.67	c ................................ | ********..           51099000 | 1343253      10707857         3840.0     13307.0    141068436
3935.87/986.73	c ................................ | ********..           52006000 | 1369650      10890901         3936.0     13212.9    142574426
4031.69/1010.78	c ................................ | ********..           52980000 | 1396551      11070139         4032.0     13139.8    145065619
4127.81/1034.87	c ................................ | ********..           53915000 | 1421487      11246258         4128.1     13060.5    145447477
4224.01/1058.90	c ................................ | ********..           54820000 | 1444844      11414489         4224.0     12978.2    147962460
4319.84/1082.96	c ................................ | *********.           55765000 | 1468344      11587241         4320.1     12908.3    149548377
4416.03/1107.00	c ................................ | *********.           56671000 | 1492274      11758546         4416.1     12832.9    152359389
4511.80/1131.06	c ................................ | *********.           57586000 | 1516958      11924767         4512.0     12762.7    154690624
4607.66/1155.08	c ................................ | *********.           58458000 | 1540798      12096748         4608.0     12686.2    157280255
4703.90/1179.13	c ................................ | *********.           59341000 | 1566625      12254604         4704.1     12614.9    159712085
4799.68/1203.18	c CPU limit (4800) reached
4799.68/1203.18	c Time = 4800.04 seconds
4799.68/1203.18	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259450-1304055275/watcher-3259450-1304055275 -o /tmp/evaluation-result-3259450-1304055275/solver-3259450-1304055275 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259450-1304055275.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 8.03 7.64 2/175 21493
/proc/meminfo: memFree=25067864/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=9544 CPUtime=0 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) R 21491 21493 18837 0 -1 4202496 464 0 0 0 0 0 0 0 25 0 1 0 517127168 9773056 406 33554432000 4194304 4338100 140735086229024 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21493/statm: 2386 407 147 36 0 291 0

[startup+0.035305 s]
/proc/loadavg: 8.11 8.03 7.64 2/175 21493
/proc/meminfo: memFree=25067864/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=16844 CPUtime=0.03 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) R 21491 21493 18837 0 -1 4202496 2248 0 0 0 3 0 0 0 25 0 1 0 517127168 17248256 2190 33554432000 4194304 4338100 140735086229024 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21493/statm: 4211 2198 151 36 0 2116 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 16844

[startup+0.100283 s]
/proc/loadavg: 8.11 8.03 7.64 2/175 21493
/proc/meminfo: memFree=25067864/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=18576 CPUtime=0.09 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) R 21491 21493 18837 0 -1 4202496 3534 0 0 0 9 0 0 0 25 0 1 0 517127168 19021824 2663 33554432000 4194304 4338100 140735086229024 18446744073709551615 274664466194 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21493/statm: 4644 2663 156 36 0 2549 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18576

[startup+0.313131 s]
/proc/loadavg: 8.11 8.03 7.64 2/175 21493
/proc/meminfo: memFree=25067864/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=319288 CPUtime=3.02 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 17954 0 0 0 299 3 0 0 25 0 5 0 517127168 326950912 16911 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 80604 18037 176 36 0 78509 0
Current children cumulated CPU time (s) 3.02
Current children cumulated vsize (KiB) 319288

[startup+1.11404 s]
/proc/loadavg: 8.11 8.03 7.64 2/175 21493
/proc/meminfo: memFree=25067864/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=325676 CPUtime=3.82 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 20108 0 0 0 378 4 0 0 25 0 5 0 517127168 333492224 19065 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 81419 19065 176 36 0 79324 0
Current children cumulated CPU time (s) 3.82
Current children cumulated vsize (KiB) 325676

[startup+1.50097 s]
/proc/loadavg: 8.11 8.03 7.64 2/175 21493
/proc/meminfo: memFree=25067864/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=344444 CPUtime=5.37 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 24087 0 0 0 530 7 0 0 25 0 5 0 517127168 352710656 23044 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 86111 23045 176 36 0 84016 0
Current children cumulated CPU time (s) 5.37
Current children cumulated vsize (KiB) 344444

[startup+3.10953 s]
/proc/loadavg: 8.19 8.04 7.65 9/184 21502
/proc/meminfo: memFree=24857616/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=398164 CPUtime=10.69 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 38416 0 0 0 1052 17 0 0 25 0 5 0 517127168 407719936 37373 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 99541 37373 176 36 0 97446 0
[pid=21493/tid=21495] ppid=21491 vsize=398164 CPUtime=2.49 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 8698 0 0 0 247 2 0 0 22 0 5 0 517127189 407719936 37373 33554432000 4194304 4338100 140735086229024 18446744073709551615 274664203373 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=398164 CPUtime=2.89 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 6719 0 0 0 284 5 0 0 18 0 5 0 517127189 407719936 37373 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=398164 CPUtime=2.85 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 10753 0 0 0 283 2 0 0 24 0 5 0 517127189 407719936 37374 33554432000 4194304 4338100 140735086229024 18446744073709551615 4284509 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=398164 CPUtime=2.22 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 3214 0 0 0 217 5 0 0 21 0 5 0 517127189 407719936 37374 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 10.69
Current children cumulated vsize (KiB) 398164

[startup+6.30114 s]
/proc/loadavg: 8.17 8.04 7.65 9/184 21502
/proc/meminfo: memFree=24607384/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=494152 CPUtime=23.38 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 61459 0 0 0 2312 26 0 0 25 0 5 0 517127168 506011648 59894 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 123538 59894 176 36 0 121443 0
[pid=21493/tid=21495] ppid=21491 vsize=494152 CPUtime=5.68 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 17142 0 0 0 564 4 0 0 25 0 5 0 517127189 506011648 59894 33554432000 4194304 4338100 140735086229024 18446744073709551615 274664204294 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=494152 CPUtime=6.04 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 9931 0 0 0 597 7 0 0 18 0 5 0 517127189 506011648 59895 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=494152 CPUtime=6.02 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 19557 0 0 0 598 4 0 0 25 0 5 0 517127189 506011648 59895 33554432000 4194304 4338100 140735086229024 18446744073709551615 4286323 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=494152 CPUtime=5.41 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 5797 0 0 0 532 9 0 0 25 0 5 0 517127189 506011648 59895 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 23.38
Current children cumulated vsize (KiB) 494152

[startup+12.707 s]
/proc/loadavg: 8.16 8.04 7.65 9/184 21502
/proc/meminfo: memFree=24263840/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=666456 CPUtime=48.86 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 105111 0 0 0 4841 45 0 0 25 0 5 0 517127168 682450944 103546 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 166614 103546 176 36 0 164519 0
[pid=21493/tid=21495] ppid=21491 vsize=666456 CPUtime=12.08 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 31160 0 0 0 1200 8 0 0 25 0 5 0 517127189 682450944 103546 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=666456 CPUtime=12.36 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 16385 0 0 0 1225 11 0 0 18 0 5 0 517127189 682450944 103546 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=666456 CPUtime=12.39 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 36290 0 0 0 1233 6 0 0 25 0 5 0 517127189 682450944 103546 33554432000 4194304 4338100 140735086229024 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=666456 CPUtime=11.81 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 12244 0 0 0 1163 18 0 0 25 0 5 0 517127189 682450944 103546 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 48.86
Current children cumulated vsize (KiB) 666456

[startup+25.5008 s]
/proc/loadavg: 8.13 8.04 7.65 9/184 21503
/proc/meminfo: memFree=23783712/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=925020 CPUtime=99.76 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 166285 0 0 0 9898 78 0 0 25 0 5 0 517127168 947220480 164720 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 231255 164720 176 36 0 229160 0
[pid=21493/tid=21495] ppid=21491 vsize=925020 CPUtime=24.85 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 53471 0 0 0 2472 13 0 0 25 0 5 0 517127189 947220480 164720 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=925020 CPUtime=24.96 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 27529 0 0 0 2479 17 0 0 18 0 5 0 517127189 947220480 164720 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=925020 CPUtime=25.11 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 54071 0 0 0 2502 9 0 0 25 0 5 0 517127189 947220480 164720 33554432000 4194304 4338100 140735086229024 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=925020 CPUtime=24.6 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 22182 0 0 0 2424 36 0 0 18 0 5 0 517127189 947220480 164720 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.76
Current children cumulated vsize (KiB) 925020

[startup+51.1004 s]
/proc/loadavg: 8.14 8.05 7.67 9/184 21503
/proc/meminfo: memFree=23066236/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=1176976 CPUtime=201.71 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 230324 0 0 0 20039 132 0 0 25 0 5 0 517127168 1205223424 228759 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 294244 228759 176 36 0 292149 0
[pid=21493/tid=21495] ppid=21491 vsize=1176976 CPUtime=50.44 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 85402 0 0 0 5022 22 0 0 25 0 5 0 517127189 1205223424 228759 33554432000 4194304 4338100 140735086229024 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=1176976 CPUtime=50.28 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 27755 0 0 0 5006 22 0 0 18 0 5 0 517127189 1205223424 228759 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=1176976 CPUtime=50.56 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 85706 0 0 0 5041 15 0 0 25 0 5 0 517127189 1205223424 228759 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=1176976 CPUtime=50.2 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 22429 0 0 0 4949 71 0 0 25 0 5 0 517127189 1205223424 228759 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 201.71
Current children cumulated vsize (KiB) 1176976

[startup+102.301 s]
/proc/loadavg: 8.32 8.10 7.70 10/184 21505
/proc/meminfo: memFree=21915988/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=1761952 CPUtime=405.62 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 377688 0 0 0 40317 245 0 0 25 0 5 0 517127168 1804238848 376123 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 440488 376123 176 36 0 438393 0
[pid=21493/tid=21495] ppid=21491 vsize=1761952 CPUtime=101.58 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 150093 0 0 0 10118 40 0 0 25 0 5 0 517127189 1804238848 376123 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=1761952 CPUtime=100.95 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 41069 0 0 0 10059 36 0 0 18 0 5 0 517127189 1804238848 376123 33554432000 4194304 4338100 140735086229024 18446744073709551615 4279536 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=1761952 CPUtime=101.48 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 142207 0 0 0 10120 28 0 0 25 0 5 0 517127189 1804238848 376123 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=1761952 CPUtime=101.37 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 35287 0 0 0 9998 139 0 0 25 0 5 0 517127189 1804238848 376123 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 405.62
Current children cumulated vsize (KiB) 1761952

[startup+162.306 s]
/proc/loadavg: 8.22 8.11 7.73 9/184 21506
/proc/meminfo: memFree=20756192/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=2205508 CPUtime=645.02 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 490076 0 0 0 63865 637 0 0 25 0 5 0 517127168 2258440192 488511 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 551377 488511 176 36 0 549282 0
[pid=21493/tid=21495] ppid=21491 vsize=2205508 CPUtime=161.48 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 200543 0 0 0 16094 54 0 0 25 0 5 0 517127189 2258440192 488511 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=2205508 CPUtime=160.9 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 51639 0 0 0 15868 222 0 0 25 0 5 0 517127189 2258440192 488511 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676621006 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=2205508 CPUtime=161.43 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 187424 0 0 0 16106 37 0 0 25 0 5 0 517127189 2258440192 488511 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=2205508 CPUtime=160.98 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 41438 0 0 0 15776 322 0 0 25 0 5 0 517127189 2258440192 488511 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676633976 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 645.02
Current children cumulated vsize (KiB) 2205508

[startup+222.301 s]
/proc/loadavg: 8.14 8.11 7.75 9/184 21508
/proc/meminfo: memFree=19982912/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=2504352 CPUtime=884.44 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 563299 0 0 0 87480 964 0 0 25 0 5 0 517127168 2564456448 561734 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 626088 561734 176 36 0 623993 0
[pid=21493/tid=21495] ppid=21491 vsize=2504352 CPUtime=221.35 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 239179 0 0 0 22068 67 0 0 25 0 5 0 517127189 2564456448 561734 33554432000 4194304 4338100 140735086229024 18446744073709551615 4284592 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 488021 0 0 0 75939 152 0 0 25 0 5 0 517127189 4672344064 1067297 33554432000 4194304 4338100 140735086229024 18446744073709551615 274664504875 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=4562836 CPUtime=759.88 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 88916 0 0 0 74422 1566 0 0 25 0 5 0 517127189 4672344064 1067297 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=4562836 CPUtime=759.12 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 395368 0 0 0 75823 89 0 0 25 0 5 0 517127189 4672344064 1067297 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=4562836 CPUtime=759.79 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 87525 0 0 0 74313 1666 0 0 25 0 5 0 517127189 4672344064 1067297 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3039.96
Current children cumulated vsize (KiB) 4562836

[startup+822.306 s]
/proc/loadavg: 8.11 8.06 7.87 9/184 21524
/proc/meminfo: memFree=15610068/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=4667512 CPUtime=3279.58 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1096758 0 0 0 324210 3748 0 0 25 0 5 0 517127168 4779532288 1095193 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1166878 1095193 176 36 0 1164783 0
[pid=21493/tid=21495] ppid=21491 vsize=4667512 CPUtime=820.91 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 503291 0 0 0 81932 159 0 0 25 0 5 0 517127189 4779532288 1095193 33554432000 4194304 4338100 140735086229024 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=4667512 CPUtime=819.87 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 89716 0 0 0 80288 1699 0 0 25 0 5 0 517127189 4779532288 1095193 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=4667512 CPUtime=818.82 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 405890 0 0 0 81791 91 0 0 25 0 5 0 517127189 4779532288 1095193 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=4667512 CPUtime=819.75 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 88829 0 0 0 80178 1797 0 0 25 0 5 0 517127189 4779532288 1095193 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3279.58
Current children cumulated vsize (KiB) 4667512

[startup+882.301 s]
/proc/loadavg: 8.09 8.06 7.88 9/184 21526
/proc/meminfo: memFree=15215716/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=4872152 CPUtime=3519.13 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1148526 0 0 0 347900 4013 0 0 25 0 5 0 517127168 4989083648 1146961 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1218038 1146961 176 36 0 1215943 0
[pid=21493/tid=21495] ppid=21491 vsize=4872152 CPUtime=880.89 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 533282 0 0 0 87922 167 0 0 25 0 5 0 517127189 4989083648 1146961 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=4872152 CPUtime=879.84 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 89939 0 0 0 86158 1826 0 0 25 0 5 0 517127189 4989083648 1146961 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=4872152 CPUtime=878.49 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 427236 0 0 0 87754 95 0 0 25 0 5 0 517127189 4989083648 1146961 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=4872152 CPUtime=879.67 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 89037 0 0 0 86045 1922 0 0 25 0 5 0 517127189 4989083648 1146961 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3519.13
Current children cumulated vsize (KiB) 4872152

[startup+942.3 s]
/proc/loadavg: 8.07 8.06 7.89 9/184 21527
/proc/meminfo: memFree=14846792/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=5057892 CPUtime=3758.67 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1195959 0 0 0 371608 4259 0 0 25 0 5 0 517127168 5179281408 1194394 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1264473 1194394 176 36 0 1262378 0
[pid=21493/tid=21495] ppid=21491 vsize=5057892 CPUtime=940.85 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 560377 0 0 0 93908 177 0 0 25 0 5 0 517127189 5179281408 1194394 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=5057892 CPUtime=939.79 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 90164 0 0 0 92034 1945 0 0 25 0 5 0 517127189 5179281408 1194394 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=5057892 CPUtime=938.19 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 447123 0 0 0 93720 99 0 0 25 0 5 0 517127189 5179281408 1194394 33554432000 4194304 4338100 140735086229024 18446744073709551615 4240513 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=5057892 CPUtime=939.6 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 89263 0 0 0 91925 2035 0 0 25 0 5 0 517127189 5179281408 1194394 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3758.67
Current children cumulated vsize (KiB) 5057892

[startup+1002.3 s]
/proc/loadavg: 8.13 8.08 7.91 9/184 21529
/proc/meminfo: memFree=14474632/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=5159044 CPUtime=3998.16 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1223107 0 0 0 395315 4501 0 0 25 0 5 0 517127168 5282861056 1221542 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1289761 1221542 176 36 0 1287666 0
[pid=21493/tid=21495] ppid=21491 vsize=5159044 CPUtime=1000.82 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 577345 0 0 0 99897 185 0 0 25 0 5 0 517127189 5282861056 1221542 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=5159044 CPUtime=999.69 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 90388 0 0 0 97906 2063 0 0 25 0 5 0 517127189 5282861056 1221542 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=5159044 CPUtime=998.12 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 455976 0 0 0 99712 100 0 0 25 0 5 0 517127189 5282861056 1221542 33554432000 4194304 4338100 140735086229024 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=5159044 CPUtime=999.3 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 90366 0 0 0 97779 2151 0 0 25 0 5 0 517127189 5282861056 1221542 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3998.16
Current children cumulated vsize (KiB) 5159044

[startup+1062.3 s]
/proc/loadavg: 8.15 8.09 7.92 9/184 21530
/proc/meminfo: memFree=14119720/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=5382912 CPUtime=4237.57 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1262905 0 0 0 419024 4733 0 0 25 0 5 0 517127168 5512101888 1261340 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1345728 1261340 176 36 0 1343633 0
[pid=21493/tid=21495] ppid=21491 vsize=5382912 CPUtime=1060.78 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 596819 0 0 0 105886 192 0 0 25 0 5 0 517127189 5512101888 1261340 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=5382912 CPUtime=1059.57 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 92200 0 0 0 103784 2173 0 0 25 0 5 0 517127189 5512101888 1261340 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=5382912 CPUtime=1057.87 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 471581 0 0 0 105683 104 0 0 25 0 5 0 517127189 5512101888 1261340 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=5382912 CPUtime=1059.1 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 93273 0 0 0 103649 2261 0 0 25 0 5 0 517127189 5512101888 1261340 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4237.57
Current children cumulated vsize (KiB) 5382912

[startup+1122.3 s]
/proc/loadavg: 8.14 8.10 7.93 9/184 21532
/proc/meminfo: memFree=13654708/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=5571516 CPUtime=4477.09 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1313010 0 0 0 442753 4956 0 0 25 0 5 0 517127168 5705232384 1311445 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1392879 1311445 176 36 0 1390784 0
[pid=21493/tid=21495] ppid=21491 vsize=5571516 CPUtime=1120.67 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 624787 0 0 0 111864 203 0 0 25 0 5 0 517127189 5705232384 1311445 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=5571516 CPUtime=1119.5 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 92912 0 0 0 109674 2276 0 0 25 0 5 0 517127189 5705232384 1311445 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254072 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=5571516 CPUtime=1117.6 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 491457 0 0 0 111651 109 0 0 25 0 5 0 517127189 5705232384 1311445 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=5571516 CPUtime=1119.08 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 94822 0 0 0 109543 2365 0 0 25 0 5 0 517127189 5705232384 1311445 33554432000 4194304 4338100 140735086229024 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4477.09
Current children cumulated vsize (KiB) 5571516

[startup+1182.3 s]
/proc/loadavg: 8.14 8.11 7.94 9/184 21534
/proc/meminfo: memFree=13357332/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=5744612 CPUtime=4716.68 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) S 21491 21493 18837 0 -1 4202496 1357126 0 0 0 466489 5179 0 0 25 0 5 0 517127168 5882482688 1355561 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 1436153 1355561 176 36 0 1434058 0
[pid=21493/tid=21495] ppid=21491 vsize=5744612 CPUtime=1180.35 cores=0,2,4,6
/proc/21493/task/21495/stat : 21495 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 650284 0 0 0 117823 212 0 0 25 0 5 0 517127189 5882482688 1355561 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=5744612 CPUtime=1179.46 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 93105 0 0 0 115565 2381 0 0 25 0 5 0 517127189 5882482688 1355561 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21493/tid=21497] ppid=21491 vsize=5744612 CPUtime=1177.59 cores=0,2,4,6
/proc/21493/task/21497/stat : 21497 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 509696 0 0 0 117644 115 0 0 25 0 5 0 517127189 5882482688 1355561 33554432000 4194304 4338100 140735086229024 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21493/tid=21498] ppid=21491 vsize=5744612 CPUtime=1179.05 cores=0,2,4,6
/proc/21493/task/21498/stat : 21498 (ttsth-5-4) R 21491 21493 18837 0 -1 4202560 95009 0 0 0 115436 2469 0 0 25 0 5 0 517127189 5882482688 1355561 33554432000 4194304 4338100 140735086229024 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4716.68
Current children cumulated vsize (KiB) 5744612



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.2 s]
/proc/loadavg: 8.17 8.12 7.95 2/175 21535
/proc/meminfo: memFree=19909160/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=0 CPUtime=4800.05 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) Z 21491 21493 18837 0 -1 4203548 1371786 0 0 0 474754 5251 0 0 18 0 2 0 517127168 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 0 0 0 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=0 CPUtime=1200.27 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4203588 94826 0 0 0 117611 2416 0 0 25 0 2 0 517127189 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.31 s]
/proc/loadavg: 8.17 8.12 7.95 2/175 21535
/proc/meminfo: memFree=19909160/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=0 CPUtime=4800.16 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) Z 21491 21493 18837 0 -1 4203548 1371786 0 0 0 474754 5262 0 0 18 0 2 0 517127168 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 0 0 0 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=0 CPUtime=1200.37 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4203588 94826 0 0 0 117611 2426 0 0 25 0 2 0 517127189 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.16
Current children cumulated vsize (KiB) 0

[startup+1203.4 s]
/proc/loadavg: 8.17 8.12 7.95 2/175 21535
/proc/meminfo: memFree=19909160/32951132 swapFree=67111524/67111528
[pid=21493] ppid=21491 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/21493/stat : 21493 (ttsth-5-4) Z 21491 21493 18837 0 -1 4203548 1371786 0 0 0 474754 5271 0 0 18 0 2 0 517127168 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21493/statm: 0 0 0 0 0 0 0
[pid=21493/tid=21496] ppid=21491 vsize=0 CPUtime=1200.46 cores=0,2,4,6
/proc/21493/task/21496/stat : 21496 (ttsth-5-4) R 21491 21493 18837 0 -1 4203588 94826 0 0 0 117611 2435 0 0 25 0 2 0 517127189 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.41
CPU time (s): 4800.26
CPU user time (s): 4747.54
CPU system time (s): 52.719
CPU usage (%): 398.889
Max. virtual memory (cumulated for all children) (KiB): 5794924

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4747.54
system time used= 52.719
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1371786
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7822
involuntary context switches= 20966

runsolver used 1.9407 second user time and 4.53631 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 07:34:35
IDJOB=3259450
IDBENCH=71572
IDSOLVER=1654
FILE ID=node131/3259450-1304055275
RUNJOBID= node131-1304054069-21377
PBS_JOBID= 13170364
Free space on /tmp= 71384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v760c43649gyes8.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259450-1304055275/watcher-3259450-1304055275 -o /tmp/evaluation-result-3259450-1304055275/solver-3259450-1304055275 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259450-1304055275.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ca42736a2fe4e8cf2cba6dd1668e0ad6
RANDOM SEED=1885763758

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      25068252 kB
Buffers:        956840 kB
Cached:        6288120 kB
SwapCached:          4 kB
Active:         357836 kB
Inactive:      7030264 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      25068252 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            3508 kB
Writeback:           0 kB
AnonPages:      142268 kB
Mapped:          15244 kB
Slab:           430960 kB
PageTables:       4600 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   287116 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71380 MiB
End job on node131 at 2011-04-29 07:54:41