Trace number 3249152

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.57 1203.88

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-6x34.cnf
MD5SUM1633295f58c28a5b4150be8640bdf9df
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables816
Number of clauses35088
Sum of the clauses size137904
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21224
Number of clauses of size 30
Number of clauses of size 433864
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249152-1304060330.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 816
0.00/0.03	c Clauses = 35088
0.00/0.03	c Literals = 137904
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 816
0.00/0.03	c Clauses = 35088
0.00/0.03	c Literals = 137904
0.00/0.03	c a2
0.09/0.10	c a1
0.19/0.20	c Independent sub-problems = 1
0.19/0.20	c Sub-problem with 816 variables and 35088 clauses
0.19/0.20	c using permutation method 4
0.19/0.20	c count = 1
0.19/0.20	c Ordering finished 0.190 seconds
0.19/0.20	c mapped 0.200 seconds
0.19/0.20	c permuted 0.200 seconds
0.19/0.20	c about to sort
0.19/0.21	c sorted
0.19/0.28	c clauses now 35088
0.19/0.28	c init finished 0.210 seconds
0.19/0.28	c tree built 0.270 seconds
0.19/0.28	c 
0.19/0.28	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.28	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.28	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.28	c ................................ | ..........                  1 |                                  0.3         3.7             
0.19/0.28	c ................................ | ..........                  2 |                                  0.3         7.1             
0.19/0.28	c ................................ | ..........                  3 |                                  0.3        10.7             
0.19/0.28	c ................................ | ..........                  4 |                                  0.3        14.3             
0.19/0.28	c ................................ | ..........                  5 |                                  0.3        17.9             
0.19/0.28	c ................................ | ..........               1000 | 4            272                 0.3      3571.4         1277
95.77/24.27	c ................................ | ..........             757000 | 115325       332783             96.1      7878.0     25123756
191.96/48.35	c ................................ | ..........            1290000 | 201952       531193            192.2      6712.5     44418897
288.05/72.41	c ................................ | ..........            1756000 | 274567       719224            288.1      6095.1     54800168
384.25/96.54	c ................................ | ..........            2065000 | 322646       858760            384.4      5371.9     62684472
479.94/120.55	c ................................ | *.........            2444000 | 381229       1002165           480.1      5090.1     68000059
576.07/144.64	c ................................ | *.........            2789000 | 436948       1165295           576.3      4839.8     75190397
671.83/168.68	c ................................ | *.........            3134000 | 491801       1298051           672.1      4662.6     85105242
767.98/192.74	c ................................ | *.........            3493000 | 550988       1441581           768.1      4547.3     96615286
864.07/216.89	c ................................ | *.........            3840000 | 604279       1581647           864.4      4442.2    107039784
960.18/240.96	c ................................ | **........            4119000 | 646704       1678484           960.4      4288.7    108808638
1055.97/264.92	c ................................ | **........            4357000 | 681266       1790654          1056.0      4125.8    110980176
1152.12/289.01	c ................................ | **........            4595000 | 715507       1905377          1152.2      3988.2    118185909
1247.84/313.06	c ................................ | **........            4818000 | 749329       1994236          1248.0      3860.4    125914584
1344.03/337.15	c ................................ | **........            5138000 | 800320       2105296          1344.2      3822.2    136030300
1440.18/361.24	c ................................ | ***.......            5435000 | 848184       2218456          1440.4      3773.4    144999235
1535.73/385.28	c ................................ | ***.......            5717000 | 893389       2315972          1536.1      3721.8    151260850
1631.86/409.36	c ................................ | ***.......            5932000 | 927040       2396236          1632.1      3634.5    155076612
1727.81/433.45	c ................................ | ***.......            6148000 | 959709       2490084          1728.0      3557.8    159638573
1824.01/457.57	c ................................ | ***.......            6394000 | 997424       2572957          1824.3      3504.8    164673484
1920.19/481.67	c ................................ | ****......            6649000 | 1040517      2654212          1920.5      3462.1    169377516
2016.02/505.66	c ................................ | ****......            6905000 | 1083283      2733214          2016.2      3424.7    172531078
2111.77/529.69	c ................................ | ****......            7099000 | 1113828      2796581          2112.2      3361.0    174306157
2207.98/553.73	c ................................ | ****......            7298000 | 1143370      2857225          2208.1      3305.1    174306157
2304.58/577.90	c ................................ | ****......            7524000 | 1179413      2922446          2304.6      3264.8    174848082
2399.85/601.86	c ................................ | *****.....            7780000 | 1221104      3014137          2400.1      3241.5    177118554
2496.18/626.04	c ................................ | *****.....            7966000 | 1250020      3096242          2496.4      3191.0    177923060
2592.34/650.15	c ................................ | *****.....            8209000 | 1286622      3166779          2592.6      3166.4    177923060
2688.06/674.18	c ................................ | *****.....            8404000 | 1316100      3237165          2688.4      3126.0    177923060
2783.75/698.18	c ................................ | *****.....            8584000 | 1342339      3309263          2784.1      3083.2    178146933
2879.92/722.23	c ................................ | ******....            8747000 | 1367978      3393708          2880.0      3037.1    179467955
2976.54/746.43	c ................................ | ******....            8894000 | 1392280      3448318          2976.7      2987.9    180166548
3071.76/770.38	c ................................ | ******....            9063000 | 1418562      3505874          3072.1      2950.1    181353379
3168.39/794.58	c ................................ | ******....            9239000 | 1447041      3572370          3168.7      2915.7    182140535
3263.67/818.49	c ................................ | ******....            9384000 | 1468561      3646069          3264.1      2874.9    184726329
3359.86/842.59	c ................................ | *******...            9531000 | 1491481      3702605          3360.2      2836.4    184726329
3455.73/866.59	c ................................ | *******...            9677000 | 1514702      3757767          3456.1      2800.0    184726329
3552.12/890.72	c ................................ | *******...            9828000 | 1536964      3822614          3552.2      2766.7    184891894
3647.94/914.76	c ................................ | *******...            9962000 | 1557876      3884037          3648.2      2730.7    184891894
3743.76/938.77	c ................................ | *******...           10094000 | 1579592      3928491          3744.1      2696.0    184891894
3840.38/962.90	c ................................ | ********..           10325000 | 1616433      3988214          3840.4      2688.5    184891894
3935.73/986.87	c ................................ | ********..           10501000 | 1644941      4059561          3936.0      2667.9    184891894
4032.08/1011.03	c ................................ | ********..           10709000 | 1678104      4134386          4032.2      2655.9    184891894
4128.10/1035.18	c ................................ | ********..           10850000 | 1699621      4192789          4128.3      2628.2    184891894
4224.20/1059.24	c ................................ | ********..           11029000 | 1727283      4255625          4224.4      2610.8    184891894
4319.91/1083.24	c ................................ | *********.           11231000 | 1758209      4322013          4320.1      2599.7    184891894
4416.04/1107.33	c ................................ | *********.           11431000 | 1790786      4381987          4416.1      2588.5    184891894
4511.75/1131.39	c ................................ | *********.           11642000 | 1824553      4446700          4512.1      2580.2    184891894
4608.24/1155.51	c ................................ | *********.           11785000 | 1846496      4499593          4608.3      2557.3    184891894
4704.09/1179.50	c ................................ | *********.           11957000 | 1872637      4549478          4704.1      2541.8    184891894

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249152-1304060330/watcher-3249152-1304060330 -o /tmp/evaluation-result-3249152-1304060330/solver-3249152-1304060330 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249152-1304060330.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.07 7.99 7.93 6/181 28434
/proc/meminfo: memFree=12740480/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) R 28432 28434 28330 0 -1 4202496 406 0 0 0 0 0 0 0 21 0 1 0 596263674 9633792 348 33554432000 4194304 4338100 140734133581952 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28434/statm: 2352 348 147 36 0 257 0

[startup+0.0706649 s]
/proc/loadavg: 8.07 7.99 7.93 6/181 28434
/proc/meminfo: memFree=12740480/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=19816 CPUtime=0.06 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) R 28432 28434 28330 0 -1 4202496 3006 0 0 0 6 0 0 0 21 0 1 0 596263674 20291584 2947 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960249 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28434/statm: 4954 2947 153 36 0 2859 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 19816

[startup+0.100658 s]
/proc/loadavg: 8.07 7.99 7.93 6/181 28434
/proc/meminfo: memFree=12740480/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=19816 CPUtime=0.09 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) R 28432 28434 28330 0 -1 4202496 3006 0 0 0 9 0 0 0 21 0 1 0 596263674 20291584 2947 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960201 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28434/statm: 4954 2947 153 36 0 2859 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19816

[startup+0.307255 s]
/proc/loadavg: 8.07 7.99 7.93 6/181 28434
/proc/meminfo: memFree=12740480/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=256848 CPUtime=2.88 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 19050 0 0 0 286 2 0 0 22 0 5 0 596263674 263012352 17684 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 65776 19446 176 36 0 63681 0
Current children cumulated CPU time (s) 2.88
Current children cumulated vsize (KiB) 256848

[startup+1.18455 s]
/proc/loadavg: 8.07 7.99 7.93 6/181 28434
/proc/meminfo: memFree=12740480/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=269492 CPUtime=3.87 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 22187 0 0 0 384 3 0 0 22 0 5 0 596263674 275959808 20821 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 67373 20821 176 36 0 65278 0
Current children cumulated CPU time (s) 3.87
Current children cumulated vsize (KiB) 269492

[startup+1.50035 s]
/proc/loadavg: 8.07 7.99 7.93 6/181 28434
/proc/meminfo: memFree=12740480/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=282004 CPUtime=5.13 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 26127 0 0 0 509 4 0 0 22 0 5 0 596263674 288772096 24761 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 70501 24761 176 36 0 68406 0
Current children cumulated CPU time (s) 5.13
Current children cumulated vsize (KiB) 282004

[startup+3.10107 s]
/proc/loadavg: 8.06 7.99 7.93 9/186 28439
/proc/meminfo: memFree=12582840/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=347956 CPUtime=11.52 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 42724 0 0 0 1144 8 0 0 22 0 5 0 596263674 356306944 41358 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 86989 41360 176 36 0 84894 0
[pid=28434/tid=28436] ppid=28432 vsize=347956 CPUtime=2.8 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 6286 0 0 0 279 1 0 0 24 0 5 0 596263703 356306944 41360 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=347956 CPUtime=2.8 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 10910 0 0 0 279 1 0 0 21 0 5 0 596263703 356306944 41360 33554432000 4194304 4338100 140734133581952 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=347956 CPUtime=2.8 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 8451 0 0 0 279 1 0 0 23 0 5 0 596263703 356306944 41360 33554432000 4194304 4338100 140734133581952 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=347956 CPUtime=2.79 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 9436 0 0 0 278 1 0 0 24 0 5 0 596263703 356306944 41361 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960341 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.52
Current children cumulated vsize (KiB) 347956

[startup+6.30052 s]
/proc/loadavg: 8.06 7.99 7.93 9/186 28439
/proc/meminfo: memFree=12479884/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=448448 CPUtime=24.31 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 66379 0 0 0 2416 15 0 0 22 0 5 0 596263674 459210752 65013 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 112112 65013 176 36 0 110017 0
[pid=28434/tid=28436] ppid=28432 vsize=448448 CPUtime=6.01 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 10444 0 0 0 598 3 0 0 25 0 5 0 596263703 459210752 65014 33554432000 4194304 4338100 140734133581952 18446744073709551615 218880261561 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=448448 CPUtime=6 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 18449 0 0 0 598 2 0 0 18 0 5 0 596263703 459210752 65014 33554432000 4194304 4338100 140734133581952 18446744073709551615 4239840 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=448448 CPUtime=6 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 14436 0 0 0 597 3 0 0 25 0 5 0 596263703 459210752 65015 33554432000 4194304 4338100 140734133581952 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=448448 CPUtime=5.97 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 15408 0 0 0 594 3 0 0 25 0 5 0 596263703 459210752 65016 33554432000 4194304 4338100 140734133581952 18446744073709551615 4263559 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.31
Current children cumulated vsize (KiB) 448448

[startup+12.7004 s]
/proc/loadavg: 8.05 7.99 7.93 9/186 28440
/proc/meminfo: memFree=12281140/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=630268 CPUtime=49.87 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 113007 0 0 0 4959 28 0 0 22 0 5 0 596263674 645394432 111641 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 157567 111641 176 36 0 155472 0
[pid=28434/tid=28436] ppid=28432 vsize=630268 CPUtime=12.4 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 18737 0 0 0 1235 5 0 0 25 0 5 0 596263703 645394432 111641 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=630268 CPUtime=12.39 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 32705 0 0 0 1234 5 0 0 18 0 5 0 596263703 645394432 111643 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=630268 CPUtime=12.4 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 28196 0 0 0 1233 7 0 0 25 0 5 0 596263703 645394432 111643 33554432000 4194304 4338100 140734133581952 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=630268 CPUtime=12.35 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 25728 0 0 0 1228 7 0 0 25 0 5 0 596263703 645394432 111643 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.87
Current children cumulated vsize (KiB) 630268

[startup+25.5012 s]
/proc/loadavg: 7.72 7.93 7.91 5/177 28458
/proc/meminfo: memFree=13695052/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=1106232 CPUtime=100.96 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 215999 0 0 0 10045 51 0 0 22 0 5 0 596263674 1132781568 214633 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 276558 214633 176 36 0 274463 0
[pid=28434/tid=28436] ppid=28432 vsize=1106232 CPUtime=25.21 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 35446 0 0 0 2509 12 0 0 25 0 5 0 596263703 1132781568 214633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=1106232 CPUtime=25.17 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 59121 0 0 0 2507 10 0 0 18 0 5 0 596263703 1132781568 214633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=1106232 CPUtime=25.2 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 58734 0 0 0 2507 13 0 0 25 0 5 0 596263703 1132781568 214633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=1106232 CPUtime=25.09 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 55055 0 0 0 2495 14 0 0 18 0 5 0 596263703 1132781568 214633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.96
Current children cumulated vsize (KiB) 1106232

[startup+51.1007 s]
/proc/loadavg: 7.66 7.88 7.89 9/186 28473
/proc/meminfo: memFree=12725260/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=1763384 CPUtime=203.14 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 379791 0 0 0 20226 88 0 0 22 0 5 0 596263674 1805705216 378425 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 440846 378425 176 36 0 438751 0
[pid=28434/tid=28436] ppid=28432 vsize=1763384 CPUtime=50.77 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 65683 0 0 0 5055 22 0 0 25 0 5 0 596263703 1805705216 378425 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=1763384 CPUtime=50.71 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 104968 0 0 0 5053 18 0 0 18 0 5 0 596263703 1805705216 378425 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=1763384 CPUtime=50.78 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 103049 0 0 0 5058 20 0 0 25 0 5 0 596263703 1805705216 378425 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=1763384 CPUtime=50.55 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 98448 0 0 0 5032 23 0 0 25 0 5 0 596263703 1805705216 378425 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.14
Current children cumulated vsize (KiB) 1763384

[startup+102.301 s]
/proc/loadavg: 7.89 7.91 7.91 9/186 28474
/proc/meminfo: memFree=11863576/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=2333940 CPUtime=407.36 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 520999 0 0 0 40597 139 0 0 22 0 5 0 596263674 2389954560 519633 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 583485 519633 176 36 0 581390 0
[pid=28434/tid=28436] ppid=28432 vsize=2333940 CPUtime=101.87 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 90560 0 0 0 10153 34 0 0 25 0 5 0 596263703 2389954560 519633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=2333940 CPUtime=101.8 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 151827 0 0 0 10151 29 0 0 25 0 5 0 596263703 2389954560 519633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=2333940 CPUtime=101.93 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 135753 0 0 0 10158 35 0 0 25 0 5 0 596263703 2389954560 519633 33554432000 4194304 4338100 140734133581952 18446744073709551615 218880261632 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=2333940 CPUtime=101.45 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 135216 0 0 0 10107 38 0 0 25 0 5 0 596263703 2389954560 519633 33554432000 4194304 4338100 140734133581952 18446744073709551615 4240530 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 407.36
Current children cumulated vsize (KiB) 2333940

[startup+162.3 s]
/proc/loadavg: 7.99 7.94 7.91 9/186 28485
/proc/meminfo: memFree=11025236/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=2914516 CPUtime=646.72 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 666329 0 0 0 64471 201 0 0 22 0 5 0 596263674 2984464384 664963 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 728629 664963 176 36 0 726534 0
[pid=28434/tid=28436] ppid=28432 vsize=2914516 CPUtime=161.56 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 108815 0 0 0 16103 53 0 0 25 0 5 0 596263703 2984464384 664963 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960158 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=2914516 CPUtime=161.72 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 188625 0 0 0 16124 48 0 0 25 0 5 0 596263703 2984464384 664963 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=2914516 CPUtime=161.88 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 189062 0 0 0 16141 47 0 0 25 0 5 0 596263703 2984464384 664963 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=2914516 CPUtime=161.25 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 172184 0 0 0 16075 50 0 0 25 0 5 0 596263703 2984464384 664963 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.72
Current children cumulated vsize (KiB) 2914516

[startup+222.301 s]
/proc/loadavg: 8.06 7.96 7.92 9/186 28486
/proc/meminfo: memFree=9970960/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=3760264 CPUtime=886.04 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 878104 0 0 0 88326 278 0 0 22 0 5 0 596263674 3850510336 876738 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 940066 876739 176 36 0 937971 0
[pid=28434/tid=28436] ppid=28432 vsize=3760264 CPUtime=221.2 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 141166 0 0 0 22040 80 0 0 25 0 5 0 596263703 3850510336 876739 33554432000 4194304 4338100 140734133581952 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0

################
# More data... #
################

/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 360170 0 0 0 75934 102 0 0 25 0 5 0 596263703 6263988224 1467210 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3039.96
Current children cumulated vsize (KiB) 6117176

[startup+822.301 s]
/proc/loadavg: 8.05 8.05 7.97 9/186 28502
/proc/meminfo: memFree=6654752/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6327872 CPUtime=3279.26 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1506140 0 0 0 327294 632 0 0 22 0 5 0 596263674 6479740928 1504774 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1581968 1504774 176 36 0 1579873 0
[pid=28434/tid=28436] ppid=28432 vsize=6327872 CPUtime=819.16 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 279758 0 0 0 81696 220 0 0 25 0 5 0 596263703 6479740928 1504774 33554432000 4194304 4338100 140734133581952 18446744073709551615 218880383895 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6327872 CPUtime=819.98 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 375863 0 0 0 81808 190 0 0 25 0 5 0 596263703 6479740928 1504774 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6327872 CPUtime=819.55 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 468764 0 0 0 81845 110 0 0 25 0 5 0 596263703 6479740928 1504774 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6327872 CPUtime=820.27 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 374112 0 0 0 81918 109 0 0 25 0 5 0 596263703 6479740928 1504774 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3279.26
Current children cumulated vsize (KiB) 6327872

[startup+882.301 s]
/proc/loadavg: 8.10 8.06 7.98 9/186 28504
/proc/meminfo: memFree=6565116/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6495368 CPUtime=3518.57 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1529841 0 0 0 351210 647 0 0 22 0 5 0 596263674 6651256832 1528475 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1623842 1528475 176 36 0 1621747 0
[pid=28434/tid=28436] ppid=28432 vsize=6495368 CPUtime=879.05 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 289913 0 0 0 87679 226 0 0 25 0 5 0 596263703 6651256832 1528475 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6495368 CPUtime=879.93 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 381555 0 0 0 87801 192 0 0 18 0 5 0 596263703 6651256832 1528475 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6495368 CPUtime=879.15 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 473235 0 0 0 87801 114 0 0 25 0 5 0 596263703 6651256832 1528475 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6495368 CPUtime=880.13 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 377495 0 0 0 87901 112 0 0 25 0 5 0 596263703 6651256832 1528475 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960489 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3518.57
Current children cumulated vsize (KiB) 6495368

[startup+942.307 s]
/proc/loadavg: 8.15 8.07 7.99 9/186 28505
/proc/meminfo: memFree=6423396/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6495764 CPUtime=3758.14 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1533633 0 0 0 375152 662 0 0 22 0 5 0 596263674 6651662336 1532267 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1623941 1532267 176 36 0 1621846 0
[pid=28434/tid=28436] ppid=28432 vsize=6495764 CPUtime=939.01 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 293375 0 0 0 93669 232 0 0 25 0 5 0 596263703 6651662336 1532267 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6495764 CPUtime=939.87 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 381678 0 0 0 93794 193 0 0 25 0 5 0 596263703 6651662336 1532267 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6495764 CPUtime=938.81 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 473349 0 0 0 93763 118 0 0 25 0 5 0 596263703 6651662336 1532267 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6495764 CPUtime=940.12 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 377588 0 0 0 93897 115 0 0 25 0 5 0 596263703 6651662336 1532267 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3758.14
Current children cumulated vsize (KiB) 6495764

[startup+1002.3 s]
/proc/loadavg: 8.09 8.07 7.99 9/186 28507
/proc/meminfo: memFree=6408376/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6496292 CPUtime=3997.36 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1534166 0 0 0 399061 675 0 0 22 0 5 0 596263674 6652203008 1532800 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1624073 1532800 176 36 0 1621978 0
[pid=28434/tid=28436] ppid=28432 vsize=6496292 CPUtime=998.88 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 293550 0 0 0 99650 238 0 0 25 0 5 0 596263703 6652203008 1532800 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6496292 CPUtime=999.76 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 381838 0 0 0 99781 195 0 0 25 0 5 0 596263703 6652203008 1532800 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6496292 CPUtime=998.46 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 473458 0 0 0 99725 121 0 0 25 0 5 0 596263703 6652203008 1532800 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6496292 CPUtime=999.94 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 377677 0 0 0 99876 118 0 0 25 0 5 0 596263703 6652203008 1532800 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3997.36
Current children cumulated vsize (KiB) 6496292

[startup+1062.3 s]
/proc/loadavg: 8.21 8.10 8.01 9/186 28509
/proc/meminfo: memFree=6343600/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6496820 CPUtime=4236.56 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1537282 0 0 0 422970 686 0 0 22 0 5 0 596263674 6652743680 1535916 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1624205 1535916 176 36 0 1622110 0
[pid=28434/tid=28436] ppid=28432 vsize=6496820 CPUtime=1058.77 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 295608 0 0 0 105633 244 0 0 25 0 5 0 596263703 6652743680 1535916 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6496820 CPUtime=1059.64 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 381979 0 0 0 105768 196 0 0 19 0 5 0 596263703 6652743680 1535916 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6496820 CPUtime=1058.07 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 474272 0 0 0 105684 123 0 0 25 0 5 0 596263703 6652743680 1535916 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960510 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6496820 CPUtime=1059.76 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 377780 0 0 0 105856 120 0 0 25 0 5 0 596263703 6652743680 1535916 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4236.56
Current children cumulated vsize (KiB) 6496820

[startup+1122.3 s]
/proc/loadavg: 8.15 8.10 8.01 9/186 28510
/proc/meminfo: memFree=6292084/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6497480 CPUtime=4475.83 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1538858 0 0 0 446885 698 0 0 22 0 5 0 596263674 6653419520 1537492 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1624370 1537492 176 36 0 1622275 0
[pid=28434/tid=28436] ppid=28432 vsize=6497480 CPUtime=1118.54 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 296836 0 0 0 111603 251 0 0 25 0 5 0 596263703 6653419520 1537492 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6497480 CPUtime=1119.53 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 382136 0 0 0 111756 197 0 0 18 0 5 0 596263703 6653419520 1537492 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6497480 CPUtime=1117.67 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 474369 0 0 0 111642 125 0 0 25 0 5 0 596263703 6653419520 1537492 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6497480 CPUtime=1119.76 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 377874 0 0 0 111854 122 0 0 25 0 5 0 596263703 6653419520 1537492 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4475.83
Current children cumulated vsize (KiB) 6497480

[startup+1182.3 s]
/proc/loadavg: 8.18 8.11 8.01 9/186 28512
/proc/meminfo: memFree=6226384/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6498008 CPUtime=4715.27 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1539307 0 0 0 470821 706 0 0 22 0 5 0 596263674 6653960192 1537941 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1624502 1537941 176 36 0 1622407 0
[pid=28434/tid=28436] ppid=28432 vsize=6498008 CPUtime=1178.43 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 296984 0 0 0 117587 256 0 0 25 0 5 0 596263703 6653960192 1537941 33554432000 4194304 4338100 140734133581952 18446744073709551615 218879960489 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6498008 CPUtime=1179.51 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 382272 0 0 0 117752 199 0 0 25 0 5 0 596263703 6653960192 1537941 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6498008 CPUtime=1177.31 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 474461 0 0 0 117605 126 0 0 25 0 5 0 596263703 6653960192 1537941 33554432000 4194304 4338100 140734133581952 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6498008 CPUtime=1179.72 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 377947 0 0 0 117849 123 0 0 25 0 5 0 596263703 6653960192 1537941 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4715.27
Current children cumulated vsize (KiB) 6498008



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.6 s]
/proc/loadavg: 8.12 8.10 8.01 9/186 28512
/proc/meminfo: memFree=6125432/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=6557704 CPUtime=4800.26 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) S 28432 28434 28330 0 -1 4202496 1554207 0 0 0 479312 714 0 0 22 0 5 0 596263674 6715088896 1552841 33554432000 4194304 4338100 140734133581952 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 1639426 1552841 176 36 0 1637331 0
[pid=28434/tid=28436] ppid=28432 vsize=6557704 CPUtime=1199.71 cores=0,2,4,6
/proc/28434/task/28436/stat : 28436 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 297044 0 0 0 119712 259 0 0 25 0 5 0 596263703 6715088896 1552841 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28434/tid=28437] ppid=28432 vsize=6557704 CPUtime=1200.77 cores=0,2,4,6
/proc/28434/task/28437/stat : 28437 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 385258 0 0 0 119876 201 0 0 18 0 5 0 596263703 6715088896 1552841 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=6557704 CPUtime=1198.48 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 480202 0 0 0 119721 127 0 0 25 0 5 0 596263703 6715088896 1552841 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28434/tid=28439] ppid=28432 vsize=6557704 CPUtime=1200.99 cores=0,2,4,6
/proc/28434/task/28439/stat : 28439 (ttsth-5-4) R 28432 28434 28330 0 -1 4202560 384060 0 0 0 119975 124 0 0 25 0 5 0 596263703 6715088896 1552841 33554432000 4194304 4338100 140734133581952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.26
Current children cumulated vsize (KiB) 6557704

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.71 s]
/proc/loadavg: 8.12 8.10 8.01 9/186 28512
/proc/meminfo: memFree=6125432/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=0 CPUtime=4800.39 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) Z 28432 28434 28330 0 -1 4203548 1554207 0 0 0 479315 724 0 0 18 0 2 0 596263674 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 0 0 0 0 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=0 CPUtime=1198.59 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4203588 480202 0 0 0 119722 137 0 0 25 0 2 0 596263703 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 0

[startup+1203.8 s]
/proc/loadavg: 8.12 8.10 8.01 9/186 28512
/proc/meminfo: memFree=6125432/32950928 swapFree=67111528/67111528
[pid=28434] ppid=28432 vsize=0 CPUtime=4800.48 cores=0,2,4,6
/proc/28434/stat : 28434 (ttsth-5-4) Z 28432 28434 28330 0 -1 4203548 1554207 0 0 0 479315 733 0 0 18 0 2 0 596263674 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28434/statm: 0 0 0 0 0 0 0
[pid=28434/tid=28438] ppid=28432 vsize=0 CPUtime=1198.68 cores=0,2,4,6
/proc/28434/task/28438/stat : 28438 (ttsth-5-4) R 28432 28434 28330 0 -1 4203588 480202 0 0 0 119722 146 0 0 25 0 2 0 596263703 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.48
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.88
CPU time (s): 4800.57
CPU user time (s): 4793.15
CPU system time (s): 7.41687
CPU usage (%): 398.757
Max. virtual memory (cumulated for all children) (KiB): 6557704

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4793.15
system time used= 7.41687
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1554207
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5921
involuntary context switches= 20911

runsolver used 1.9427 second user time and 4.45032 second system time

The end

Launcher Data 

Begin job on node112 at 2011-04-29 08:58:50
IDJOB=3249152
IDBENCH=83157
IDSOLVER=1654
FILE ID=node112/3249152-1304060330
RUNJOBID= node112-1304059146-28348
PBS_JOBID= 13170443
Free space on /tmp= 72564 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-6x34.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249152-1304060330/watcher-3249152-1304060330 -o /tmp/evaluation-result-3249152-1304060330/solver-3249152-1304060330 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249152-1304060330.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1633295f58c28a5b4150be8640bdf9df
RANDOM SEED=137941966

node112.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5261.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      12740760 kB
Buffers:       1995220 kB
Cached:       15797880 kB
SwapCached:          0 kB
Active:        8892252 kB
Inactive:     10759128 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      12740760 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             824 kB
Writeback:           0 kB
AnonPages:     1857796 kB
Mapped:          15312 kB
Slab:           492328 kB
PageTables:       7788 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  2058288 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72564 MiB
End job on node112 at 2011-04-29 09:18:56