Trace number 3265962

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.34 1202.4

General information on the benchmark

NameSAT05/crafted/sorge05/QG7/
dead-dnd003.shuffled-as.sat05-3109.cnf
MD5SUMd6a882056a7d77e09b3b073fbf028602
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark132.756
Satisfiable
(Un)Satisfiability was proved
Number of variables468
Number of clauses11753
Sum of the clauses size40843
Maximum clause length13
Minimum clause length1
Number of clauses of size 142
Number of clauses of size 21122
Number of clauses of size 38729
Number of clauses of size 4277
Number of clauses of size 5279
Number of clauses of size over 51304

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265962-1304103595.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 468
0.00/0.01	c Clauses = 11753
0.00/0.01	c Literals = 40843
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 468
0.00/0.01	c Clauses = 10283
0.00/0.01	c Literals = 31546
0.00/0.01	c a2
0.02/0.03	c a1
0.49/0.54	c Independent sub-problems = 1
0.49/0.54	c Sub-problem with 412 variables and 10283 clauses
0.49/0.54	c using permutation method 4
0.49/0.54	c count = 2
0.49/0.54	c Ordering finished 0.540 seconds
0.49/0.54	c mapped 0.540 seconds
0.49/0.54	c permuted 0.540 seconds
0.49/0.54	c about to sort
0.49/0.54	c sorted
0.49/0.56	c clauses now 10283
0.49/0.56	c init finished 0.540 seconds
0.49/0.56	c tree built 0.550 seconds
0.49/0.56	c 
0.49/0.56	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.56	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.56	c ................................ | ..........                    |                                  0.6         0.0             
0.49/0.56	c ................................ | ..........                  1 |                                  0.6         1.8             
0.49/0.56	c ................................ | ..........                  2 |                                  0.6         3.6             
0.49/0.56	c ................................ | ..........                  3 |                                  0.6         5.5             
0.49/0.56	c ................................ | ..........                  4 |                                  0.6         7.3             
0.49/0.56	c ................................ | ..........                  5 |                                  0.6         9.1             
0.49/0.57	c ................................ | ..........               1000 | 14           509                 0.6      1754.4        15632
96.17/24.50	c ................................ | ..........             906000 | 38328        413704             96.2      9418.9     17128129
192.02/48.55	c ................................ | ..........            1411000 | 59824        619923            192.2      7339.8     25017825
287.90/72.56	c ................................ | ..........            1766000 | 76496        770189            288.2      6128.5     31730152
384.16/96.60	c ................................ | ..........            2041000 | 89521        895176            384.2      5312.9     37681660
480.02/120.61	c ................................ | *.........            2350000 | 105327       1019626           480.1      4895.2     42948526
575.86/144.64	c ................................ | *.........            2687000 | 120980       1144503           576.0      4664.5     48974240
672.13/168.71	c ................................ | *.........            2991000 | 134818       1257838           672.2      4449.8     54023498
768.00/192.77	c ................................ | *.........            3279000 | 148499       1355161           768.3      4268.0     58118298
863.86/216.79	c ................................ | *.........            3528000 | 159431       1443117           864.2      4082.3     61727039
960.38/241.01	c ................................ | **........            3766000 | 169571       1527209           960.4      3921.1     65083611
1055.85/264.96	c ................................ | **........            4006000 | 181789       1609339          1056.1      3793.2     69216441
1152.12/289.04	c ................................ | **........            4243000 | 194382       1686140          1152.3      3682.2     73029720
1247.98/313.09	c ................................ | **........            4454000 | 207389       1758420          1248.3      3567.9     76097051
1343.79/337.05	c ................................ | **........            4651000 | 220033       1827712          1344.0      3460.5     78068489
1440.46/361.22	c ................................ | ***.......            4845000 | 230878       1892698          1440.6      3363.2     80162892
1535.94/385.18	c ................................ | ***.......            5018000 | 241329       1948779          1536.3      3266.3     82909004
1632.20/409.25	c ................................ | ***.......            5205000 | 250744       2008618          1632.4      3188.6     86009295
1728.09/433.29	c ................................ | ***.......            5381000 | 259342       2066374          1728.5      3113.2     88325173
1823.98/457.24	c ................................ | ***.......            5532000 | 267646       2120393          1824.1      3032.7     90997788
1919.84/481.25	c ................................ | ****......            5705000 | 275848       2182143          1920.0      2971.3     93114478
2015.71/505.27	c ................................ | ****......            5849000 | 284340       2238299          2016.0      2901.3     95505192
2112.00/529.33	c ................................ | ****......            5985000 | 292335       2288941          2112.1      2833.6     98116969
2208.28/553.46	c ................................ | ****......            6127000 | 299848       2343851          2208.5      2774.3    101077265
2303.75/577.37	c ................................ | ****......            6290000 | 307623       2399845          2304.0      2730.0    103360917
2399.66/601.39	c ................................ | *****.....            6452000 | 316211       2453301          2400.0      2688.3    104272933
2496.33/625.56	c ................................ | *****.....            6633000 | 326342       2509901          2496.6      2656.9    106948731
2592.18/649.52	c ................................ | *****.....            6795000 | 334959       2559527          2592.3      2621.3    107889762
2688.06/673.59	c ................................ | *****.....            6944000 | 343242       2608540          2688.4      2582.9    109566138
2783.95/697.52	c ................................ | *****.....            7100000 | 350579       2658509          2784.0      2550.3    112374156
2880.20/721.65	c ................................ | ******....            7253000 | 357829       2708986          2880.4      2518.1    114986671
2976.48/745.71	c ................................ | ******....            7389000 | 364503       2756595          2976.5      2482.4    117400618
3071.99/769.65	c ................................ | ******....            7543000 | 371587       2805366          3072.2      2455.3    119512703
3168.27/793.71	c ................................ | ******....            7704000 | 378628       2855317          3168.3      2431.6    121991295
3264.13/817.75	c ................................ | ******....            7842000 | 385538       2905992          3264.3      2402.3    123978821
3360.41/841.86	c ................................ | *******...            7987000 | 392726       2955910          3360.7      2376.6    126573978
3455.89/865.78	c ................................ | *******...            8121000 | 401421       2999436          3456.2      2349.7    128473074
3552.18/889.88	c ................................ | *******...            8259000 | 409131       3043863          3552.5      2324.8    130769864
3648.03/913.81	c ................................ | *******...            8401000 | 417677       3091832          3648.1      2302.9    132550992
3743.93/937.86	c ................................ | *******...            8528000 | 426878       3139952          3744.2      2277.7    134059973
3840.20/961.98	c ................................ | ********..            8655000 | 434493       3189485          3840.6      2253.6    136008665
3936.08/985.95	c ................................ | ********..            8789000 | 440957       3239114          3936.3      2232.8    138443514
4032.71/1010.10	c ................................ | ********..            8930000 | 447840       3289790          4032.7      2214.4    139901975
4128.20/1034.04	c ................................ | ********..            9066000 | 455317       3339041          4128.4      2196.0    141539985
4223.67/1057.99	c ................................ | ********..            9180000 | 461316       3382599          4224.1      2173.3    142096665
4319.98/1082.08	c ................................ | *********.            9312000 | 468503       3429729          4320.3      2155.4    143714890
4415.85/1106.07	c ................................ | *********.            9424000 | 475428       3473165          4416.1      2134.0    145565631
4512.49/1130.20	c ................................ | *********.            9560000 | 482438       3520068          4512.5      2118.6    146029066
4608.36/1154.23	c ................................ | *********.            9683000 | 489080       3564580          4608.5      2101.1    146029066
4704.24/1178.25	c ................................ | *********.            9812000 | 496328       3609687          4704.5      2085.7    146462767
4800.13/1202.20	c CPU limit (4800) reached
4800.13/1202.20	c Time = 4800.14 seconds
4800.13/1202.20	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265962-1304103595/watcher-3265962-1304103595 -o /tmp/evaluation-result-3265962-1304103595/solver-3265962-1304103595 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265962-1304103595.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.08 7.94 1/175 20476
/proc/meminfo: memFree=17904684/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=9424 CPUtime=0 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) R 20474 20476 19929 0 -1 4202496 407 0 0 0 0 0 0 0 25 0 1 0 521962137 9650176 349 33554432000 4194304 4338100 140734119801616 18446744073709551615 216634063407 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/20476/statm: 2356 349 147 36 0 261 0

[startup+0.0289109 s]
/proc/loadavg: 8.10 8.08 7.94 1/175 20476
/proc/meminfo: memFree=17904684/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=12456 CPUtime=0.02 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) R 20474 20476 19929 0 -1 4202496 1137 0 0 0 2 0 0 0 25 0 1 0 521962137 12754944 1072 33554432000 4194304 4338100 140734119801616 18446744073709551615 216633910648 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/20476/statm: 3114 1072 154 36 0 1019 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 12456

[startup+0.100899 s]
/proc/loadavg: 8.10 8.08 7.94 1/175 20476
/proc/meminfo: memFree=17904684/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=12616 CPUtime=0.09 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) R 20474 20476 19929 0 -1 4202496 1193 0 0 0 9 0 0 0 25 0 1 0 521962137 12918784 1126 33554432000 4194304 4338100 140734119801616 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/20476/statm: 3154 1126 155 36 0 1059 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12616

[startup+0.300864 s]
/proc/loadavg: 8.10 8.08 7.94 1/175 20476
/proc/meminfo: memFree=17904684/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=12616 CPUtime=0.29 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) R 20474 20476 19929 0 -1 4202496 1194 0 0 0 29 0 0 0 25 0 1 0 521962137 12918784 1127 33554432000 4194304 4338100 140734119801616 18446744073709551615 4295932 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/20476/statm: 3154 1127 155 36 0 1059 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 12616

[startup+1.46495 s]
/proc/loadavg: 8.10 8.08 7.94 1/175 20476
/proc/meminfo: memFree=17904684/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=260628 CPUtime=4.16 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 19015 0 0 0 411 5 0 0 25 0 5 0 521962137 266883072 18778 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 65157 18779 175 36 0 63062 0
Current children cumulated CPU time (s) 4.16
Current children cumulated vsize (KiB) 260628

[startup+1.50062 s]
/proc/loadavg: 8.10 8.08 7.94 1/175 20476
/proc/meminfo: memFree=17904684/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=266884 CPUtime=4.3 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 19424 0 0 0 425 5 0 0 25 0 5 0 521962137 273289216 19187 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 66721 19187 175 36 0 64626 0
Current children cumulated CPU time (s) 4.3
Current children cumulated vsize (KiB) 266884

[startup+3.10034 s]
/proc/loadavg: 8.17 8.10 7.94 9/186 20512
/proc/meminfo: memFree=17718968/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=324136 CPUtime=10.68 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 34811 0 0 0 1056 12 0 0 25 0 5 0 521962137 331915264 34574 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 81034 34574 175 36 0 78939 0
[pid=20476/tid=20478] ppid=20474 vsize=324136 CPUtime=2.53 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 12911 0 0 0 252 1 0 0 22 0 5 0 521962194 331915264 34574 33554432000 4194304 4338100 140734119801616 18446744073709551615 4288708 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=324136 CPUtime=2.52 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 4936 0 0 0 247 5 0 0 23 0 5 0 521962194 331915264 34574 33554432000 4194304 4338100 140734119801616 18446744073709551615 216633910337 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=324136 CPUtime=2.53 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 9135 0 0 0 252 1 0 0 22 0 5 0 521962194 331915264 34574 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=324136 CPUtime=2.53 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 5147 0 0 0 249 4 0 0 22 0 5 0 521962194 331915264 34574 33554432000 4194304 4338100 140734119801616 18446744073709551615 216633910369 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.68
Current children cumulated vsize (KiB) 324136

[startup+6.30078 s]
/proc/loadavg: 8.17 8.10 7.94 9/186 20513
/proc/meminfo: memFree=17583936/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=412756 CPUtime=23.47 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 56860 0 0 0 2315 32 0 0 25 0 5 0 521962137 422662144 56624 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 103189 56624 175 36 0 101094 0
[pid=20476/tid=20478] ppid=20474 vsize=412756 CPUtime=5.73 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 22442 0 0 0 570 3 0 0 25 0 5 0 521962194 422662144 56624 33554432000 4194304 4338100 140734119801616 18446744073709551615 216634513467 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=412756 CPUtime=5.7 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 8730 0 0 0 557 13 0 0 25 0 5 0 521962194 422662144 56625 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=412756 CPUtime=5.72 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 14039 0 0 0 570 2 0 0 25 0 5 0 521962194 422662144 56626 33554432000 4194304 4338100 140734119801616 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=412756 CPUtime=5.73 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 8970 0 0 0 561 12 0 0 25 0 5 0 521962194 422662144 56626 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.47
Current children cumulated vsize (KiB) 412756

[startup+12.7007 s]
/proc/loadavg: 7.90 8.05 7.93 6/182 20571
/proc/meminfo: memFree=17524620/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=555072 CPUtime=49.05 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 93385 0 0 0 4837 68 0 0 25 0 5 0 521962137 568393728 93148 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 138768 93148 175 36 0 136673 0
[pid=20476/tid=20478] ppid=20474 vsize=555072 CPUtime=12.12 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 38303 0 0 0 1206 6 0 0 25 0 5 0 521962194 568393728 93148 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=555072 CPUtime=12.08 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 14666 0 0 0 1179 29 0 0 25 0 5 0 521962194 568393728 93148 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=555072 CPUtime=12.13 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 22842 0 0 0 1210 3 0 0 25 0 5 0 521962194 568393728 93148 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=555072 CPUtime=12.14 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 14892 0 0 0 1186 28 0 0 25 0 5 0 521962194 568393728 93148 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.05
Current children cumulated vsize (KiB) 555072

[startup+25.5004 s]
/proc/loadavg: 7.46 7.95 7.89 6/182 20571
/proc/meminfo: memFree=17333944/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=754220 CPUtime=100.16 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 143587 0 0 0 9895 121 0 0 25 0 5 0 521962137 772321280 143350 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 188555 143350 175 36 0 186460 0
[pid=20476/tid=20478] ppid=20474 vsize=754220 CPUtime=24.84 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 57358 0 0 0 2474 10 0 0 25 0 5 0 521962194 772321280 143351 33554432000 4194304 4338100 140734119801616 18446744073709551615 4245241 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=754220 CPUtime=24.87 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 23972 0 0 0 2435 52 0 0 25 0 5 0 521962194 772321280 143352 33554432000 4194304 4338100 140734119801616 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=754220 CPUtime=24.92 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 35355 0 0 0 2486 6 0 0 25 0 5 0 521962194 772321280 143352 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=754220 CPUtime=24.93 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 24222 0 0 0 2442 51 0 0 25 0 5 0 521962194 772321280 143352 33554432000 4194304 4338100 140734119801616 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.16
Current children cumulated vsize (KiB) 754220

[startup+51.1009 s]
/proc/loadavg: 7.70 7.96 7.90 9/186 20576
/proc/meminfo: memFree=16654736/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=1028440 CPUtime=202.42 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 211531 0 0 0 20052 190 0 0 25 0 5 0 521962137 1053122560 211294 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 257110 211294 176 36 0 255015 0
[pid=20476/tid=20478] ppid=20474 vsize=1028440 CPUtime=50.31 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 89121 0 0 0 5013 18 0 0 25 0 5 0 521962194 1053122560 211294 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=1028440 CPUtime=50.47 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 34251 0 0 0 4968 79 0 0 25 0 5 0 521962194 1053122560 211294 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=1028440 CPUtime=50.52 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 50486 0 0 0 5042 10 0 0 25 0 5 0 521962194 1053122560 211294 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=1028440 CPUtime=50.52 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 34991 0 0 0 4971 81 0 0 25 0 5 0 521962194 1053122560 211294 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.42
Current children cumulated vsize (KiB) 1028440

[startup+102.301 s]
/proc/loadavg: 7.94 7.99 7.91 9/186 20586
/proc/meminfo: memFree=15817940/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=1453968 CPUtime=406.93 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 318336 0 0 0 40382 311 0 0 25 0 5 0 521962137 1488863232 318099 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 363492 318101 176 36 0 361397 0
[pid=20476/tid=20478] ppid=20474 vsize=1453968 CPUtime=101.43 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 135509 0 0 0 10113 30 0 0 25 0 5 0 521962194 1488863232 318101 33554432000 4194304 4338100 140734119801616 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=1453968 CPUtime=101.5 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 52392 0 0 0 10019 131 0 0 25 0 5 0 521962194 1488863232 318102 33554432000 4194304 4338100 140734119801616 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=1453968 CPUtime=101.72 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 74548 0 0 0 10155 17 0 0 25 0 5 0 521962194 1488863232 318102 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=1453968 CPUtime=101.69 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 53208 0 0 0 10038 131 0 0 25 0 5 0 521962194 1488863232 318103 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.93
Current children cumulated vsize (KiB) 1453968

[startup+162.3 s]
/proc/loadavg: 8.04 8.00 7.92 9/186 20588
/proc/meminfo: memFree=14997664/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=1892148 CPUtime=646.57 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 429109 0 0 0 64232 425 0 0 25 0 5 0 521962137 1937559552 428872 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 473037 428872 176 36 0 470942 0
[pid=20476/tid=20478] ppid=20474 vsize=1892148 CPUtime=161.43 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 189418 0 0 0 16099 44 0 0 25 0 5 0 521962194 1937559552 428872 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=1892148 CPUtime=161.48 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 69357 0 0 0 15968 180 0 0 25 0 5 0 521962194 1937559552 428872 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=1892148 CPUtime=161.72 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 96584 0 0 0 16150 22 0 0 25 0 5 0 521962194 1937559552 428872 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=1892148 CPUtime=161.35 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 71068 0 0 0 15958 177 0 0 25 0 5 0 521962194 1937559552 428872 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 646.57
Current children cumulated vsize (KiB) 1892148

[startup+222.301 s]
/proc/loadavg: 8.08 8.02 7.92 10/187 20592
/proc/meminfo: memFree=14436780/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=2201796 CPUtime=886.12 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 506813 0 0 0 88099 513 0 0 25 0 5 0 521962137 2254639104 506576 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 550449 506576 176 36 0 548354 0
[pid=20476/tid=20478] ppid=20474 vsize=2201796 CPUtime=221.42 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 230037 0 0 0 22086 56 0 0 25 0 5 0 521962194 2254639104 506576 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=11217920/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=3995472 CPUtime=3042.8 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 959497 0 0 0 303191 1089 0 0 25 0 5 0 521962137 4091363328 959260 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 998868 959260 176 36 0 996773 0
[pid=20476/tid=20478] ppid=20474 vsize=3995472 CPUtime=759.35 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 465520 0 0 0 75807 128 0 0 24 0 5 0 521962194 4091363328 959260 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=3995472 CPUtime=760.92 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 140011 0 0 0 75635 457 0 0 25 0 5 0 521962194 4091363328 959260 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=3995472 CPUtime=761.68 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 214493 0 0 0 76113 55 0 0 25 0 5 0 521962194 4091363328 959260 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=3995472 CPUtime=760.26 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 136791 0 0 0 75579 447 0 0 25 0 5 0 521962194 4091363328 959260 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.8
Current children cumulated vsize (KiB) 3995472

[startup+822.3 s]
/proc/loadavg: 8.07 8.07 7.99 9/186 20608
/proc/meminfo: memFree=10998668/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4172328 CPUtime=3282.52 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1003629 0 0 0 327103 1149 0 0 25 0 5 0 521962137 4272463872 1003392 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1043082 1003392 176 36 0 1040987 0
[pid=20476/tid=20478] ppid=20474 vsize=4172328 CPUtime=819.05 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 489257 0 0 0 81770 135 0 0 25 0 5 0 521962194 4272463872 1003392 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4172328 CPUtime=820.93 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 142912 0 0 0 81612 481 0 0 25 0 5 0 521962194 4272463872 1003392 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4172328 CPUtime=821.69 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 224121 0 0 0 82111 58 0 0 25 0 5 0 521962194 4272463872 1003392 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4172328 CPUtime=820.25 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 144657 0 0 0 81552 473 0 0 25 0 5 0 521962194 4272463872 1003392 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.52
Current children cumulated vsize (KiB) 4172328

[startup+882.301 s]
/proc/loadavg: 8.09 8.07 8.00 9/186 20610
/proc/meminfo: memFree=10690020/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4352064 CPUtime=3522.22 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1049087 0 0 0 351012 1210 0 0 25 0 5 0 521962137 4456513536 1048850 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1088016 1048850 176 36 0 1085921 0
[pid=20476/tid=20478] ppid=20474 vsize=4352064 CPUtime=878.76 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 510039 0 0 0 87734 142 0 0 25 0 5 0 521962194 4456513536 1048850 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4352064 CPUtime=880.92 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 150545 0 0 0 87584 508 0 0 25 0 5 0 521962194 4456513536 1048850 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4352064 CPUtime=881.69 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 234702 0 0 0 88107 62 0 0 25 0 5 0 521962194 4456513536 1048850 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4352064 CPUtime=880.25 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 151119 0 0 0 87529 496 0 0 25 0 5 0 521962194 4456513536 1048850 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3522.22
Current children cumulated vsize (KiB) 4352064

[startup+942.306 s]
/proc/loadavg: 8.16 8.08 8.01 9/186 20611
/proc/meminfo: memFree=10477224/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4481744 CPUtime=3761.91 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1081283 0 0 0 374925 1266 0 0 25 0 5 0 521962137 4589305856 1081046 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1120436 1081046 176 36 0 1118341 0
[pid=20476/tid=20478] ppid=20474 vsize=4481744 CPUtime=938.45 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 527326 0 0 0 93697 148 0 0 25 0 5 0 521962194 4589305856 1081046 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4481744 CPUtime=940.94 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 155730 0 0 0 93564 530 0 0 25 0 5 0 521962194 4589305856 1081046 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4481744 CPUtime=941.68 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 240326 0 0 0 94104 64 0 0 25 0 5 0 521962194 4589305856 1081046 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4481744 CPUtime=940.26 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 155219 0 0 0 93503 523 0 0 25 0 5 0 521962194 4589305856 1081046 33554432000 4194304 4338100 140734119801616 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3761.91
Current children cumulated vsize (KiB) 4481744

[startup+1002.3 s]
/proc/loadavg: 8.12 8.08 8.01 9/186 20613
/proc/meminfo: memFree=10225104/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4661464 CPUtime=4001.55 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1126999 0 0 0 398830 1325 0 0 25 0 5 0 521962137 4773339136 1126762 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1165366 1126762 176 36 0 1163271 0
[pid=20476/tid=20478] ppid=20474 vsize=4661464 CPUtime=998.14 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 547946 0 0 0 99659 155 0 0 25 0 5 0 521962194 4773339136 1126762 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4661464 CPUtime=1000.9 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 163330 0 0 0 99536 554 0 0 25 0 5 0 521962194 4773339136 1126762 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4661464 CPUtime=1001.66 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 251689 0 0 0 100100 66 0 0 25 0 5 0 521962194 4773339136 1126762 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4661464 CPUtime=1000.24 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 161352 0 0 0 99477 547 0 0 25 0 5 0 521962194 4773339136 1126762 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4001.55
Current children cumulated vsize (KiB) 4661464

[startup+1062.3 s]
/proc/loadavg: 8.04 8.07 8.00 9/186 20615
/proc/meminfo: memFree=10000644/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4738100 CPUtime=4241.26 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1146592 0 0 0 422758 1368 0 0 25 0 5 0 521962137 4851814400 1146355 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1184525 1146355 176 36 0 1182430 0
[pid=20476/tid=20478] ppid=20474 vsize=4738100 CPUtime=1057.86 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 560606 0 0 0 105626 160 0 0 25 0 5 0 521962194 4851814400 1146355 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4738100 CPUtime=1060.9 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 166397 0 0 0 105516 574 0 0 25 0 5 0 521962194 4851814400 1146355 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4738100 CPUtime=1061.67 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 254754 0 0 0 106100 67 0 0 25 0 5 0 521962194 4851814400 1146355 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4738100 CPUtime=1060.24 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 162153 0 0 0 105459 565 0 0 25 0 5 0 521962194 4851814400 1146355 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4241.26
Current children cumulated vsize (KiB) 4738100

[startup+1122.3 s]
/proc/loadavg: 8.09 8.07 8.00 9/186 20616
/proc/meminfo: memFree=9770604/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4893212 CPUtime=4480.92 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1184853 0 0 0 446672 1420 0 0 25 0 5 0 521962137 5010649088 1184616 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1223303 1184616 176 36 0 1221208 0
[pid=20476/tid=20478] ppid=20474 vsize=4893212 CPUtime=1117.55 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 579099 0 0 0 111588 167 0 0 25 0 5 0 521962194 5010649088 1184616 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4893212 CPUtime=1120.88 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 172549 0 0 0 111492 596 0 0 25 0 5 0 521962194 5010649088 1184616 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4893212 CPUtime=1121.66 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 261032 0 0 0 112097 69 0 0 25 0 5 0 521962194 5010649088 1184616 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4893212 CPUtime=1120.23 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 169491 0 0 0 111437 586 0 0 25 0 5 0 521962194 5010649088 1184616 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4480.92
Current children cumulated vsize (KiB) 4893212

[startup+1182.3 s]
/proc/loadavg: 8.18 8.10 8.01 9/186 20618
/proc/meminfo: memFree=9717768/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4913020 CPUtime=4720.63 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1190206 0 0 0 470595 1468 0 0 25 0 5 0 521962137 5030932480 1189969 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1228255 1189969 176 36 0 1226160 0
[pid=20476/tid=20478] ppid=20474 vsize=4913020 CPUtime=1177.25 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 583979 0 0 0 117556 169 0 0 25 0 5 0 521962194 5030932480 1189969 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4913020 CPUtime=1180.89 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 172708 0 0 0 117469 620 0 0 25 0 5 0 521962194 5030932480 1189969 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4913020 CPUtime=1181.67 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 261169 0 0 0 118097 70 0 0 25 0 5 0 521962194 5030932480 1189969 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4913020 CPUtime=1180.23 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 169668 0 0 0 117416 607 0 0 25 0 5 0 521962194 5030932480 1189969 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4720.63
Current children cumulated vsize (KiB) 4913020



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.13 8.09 8.01 9/186 20618
/proc/meminfo: memFree=9656880/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=4935440 CPUtime=4800.13 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) S 20474 20476 19929 0 -1 4202496 1195732 0 0 0 478534 1479 0 0 25 0 5 0 521962137 5053890560 1195495 33554432000 4194304 4338100 140734119801616 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 1233860 1195495 176 36 0 1231765 0
[pid=20476/tid=20478] ppid=20474 vsize=4935440 CPUtime=1197.05 cores=1,3,5,7
/proc/20476/task/20478/stat : 20478 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 589378 0 0 0 119534 171 0 0 25 0 5 0 521962194 5053890560 1195495 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=4935440 CPUtime=1200.79 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 172755 0 0 0 119455 624 0 0 25 0 5 0 521962194 5053890560 1195495 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=20476/tid=20480] ppid=20474 vsize=4935440 CPUtime=1201.57 cores=1,3,5,7
/proc/20476/task/20480/stat : 20480 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 261210 0 0 0 120087 70 0 0 25 0 5 0 521962194 5053890560 1195495 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=20476/tid=20481] ppid=20474 vsize=4935440 CPUtime=1200.14 cores=1,3,5,7
/proc/20476/task/20481/stat : 20481 (ttsth-5-4) R 20474 20476 19929 0 -1 4202560 169707 0 0 0 119402 612 0 0 25 0 5 0 521962194 5053890560 1195495 33554432000 4194304 4338100 140734119801616 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 4935440

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.31 s]
/proc/loadavg: 8.13 8.09 8.01 9/186 20618
/proc/meminfo: memFree=9656880/32951124 swapFree=67111528/67111528
[pid=20476] ppid=20474 vsize=0 CPUtime=4800.25 cores=1,3,5,7
/proc/20476/stat : 20476 (ttsth-5-4) Z 20474 20476 19929 0 -1 4203548 1195734 0 0 0 478535 1490 0 0 18 0 2 0 521962137 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/20476/statm: 0 0 0 0 0 0 0
[pid=20476/tid=20479] ppid=20474 vsize=0 CPUtime=1200.9 cores=1,3,5,7
/proc/20476/task/20479/stat : 20479 (ttsth-5-4) R 20474 20476 19929 0 -1 4203588 172755 0 0 0 119455 635 0 0 25 0 2 0 521962194 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.4
CPU time (s): 4800.34
CPU user time (s): 4785.36
CPU system time (s): 14.9837
CPU usage (%): 399.231
Max. virtual memory (cumulated for all children) (KiB): 4935440

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4785.36
system time used= 14.9837
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1195734
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3616
involuntary context switches= 30118

runsolver used 1.78173 second user time and 4.34734 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-29 20:59:55
IDJOB=3265962
IDBENCH=85841
IDSOLVER=1654
FILE ID=node103/3265962-1304103595
RUNJOBID= node103-1304102391-20388
PBS_JOBID= 13170621
Free space on /tmp= 71516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sorge05/QG7/dead-dnd003.shuffled-as.sat05-3109.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265962-1304103595/watcher-3265962-1304103595 -o /tmp/evaluation-result-3265962-1304103595/solver-3265962-1304103595 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265962-1304103595.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d6a882056a7d77e09b3b073fbf028602
RANDOM SEED=1133862941

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17904840 kB
Buffers:       2149024 kB
Cached:       12248004 kB
SwapCached:          0 kB
Active:        5587800 kB
Inactive:      8874900 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17904840 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:              44 kB
Writeback:           0 kB
AnonPages:       64360 kB
Mapped:          15068 kB
Slab:           519652 kB
PageTables:       4676 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   217512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71516 MiB
End job on node103 at 2011-04-29 21:19:59