Trace number 3250384

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1207.85

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-5-5_34309.cnf
MD5SUMe252fbf81407f3388187ab50f0786909
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables34307
Number of clauses708206
Sum of the clauses size3541030
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 5708206
Number of clauses of size over 50

Solver Data 

0.89/0.98	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250384-1304090009.cnf 
0.89/0.98	c Using SATTIMEOUT of 4800
0.89/0.98	c Using SATRAM of 15500
0.89/0.98	c Size of subset node = 32
0.89/0.98	c Variables = 34307
0.89/0.98	c Clauses = 708206
0.89/0.98	c Literals = 3541030
0.89/0.98	c Reduced size:
0.89/0.98	c Variables = 34307
0.89/0.98	c Clauses = 708206
0.89/0.98	c Literals = 3541030
0.89/0.98	c a2
3.39/3.40	c a1
4.09/4.11	c Independent sub-problems = 5
4.09/4.11	c Sub-problem with 4257 variables and 85800 clauses
4.09/4.11	c using permutation method 4
4.09/4.11	c count = 1
4.09/4.11	c Ordering finished 4.080 seconds
4.09/4.11	c mapped 4.110 seconds
4.09/4.11	c permuted 4.110 seconds
4.09/4.11	c about to sort
4.09/4.12	c sorted
4.49/4.51	c clauses now 85800
4.49/4.51	c init finished 4.160 seconds
4.49/4.51	c tree built 4.510 seconds
4.49/4.51	c 
4.49/4.51	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
4.49/4.51	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
4.49/4.51	c ................................ | ..........                    |                                  4.5         0.0             
4.49/4.51	c ................................ | ..........                  1 |                                  4.5         0.2             
4.49/4.51	c ................................ | ..........                  2 |                                  4.5         0.4             
4.49/4.51	c ................................ | ..........                  3 |                                  4.5         0.7             
4.49/4.51	c ................................ | ..........                  4 |                                  4.5         0.9             
4.49/4.51	c ................................ | ..........                  5 |                                  4.5         1.1             
4.49/4.53	c ................................ | ..........               1000 |              124                 4.6       219.3           13
95.94/27.62	c ................................ | ..........            1441000 | 36066        1061386            96.0     15005.7       729269
191.81/51.85	c ................................ | ..........            2854000 | 81121        1971625           192.0     14863.8      2191534
287.81/76.06	c ................................ | ..........            4126000 | 132234       2845156           288.1     14322.4      3741666
383.86/100.24	c ................................ | ..........            5352000 | 179153       3662359           384.0     13936.4      5069769
479.79/124.45	c ................................ | *.........            6472000 | 220640       4386374           480.0     13483.1      6290066
575.82/148.65	c ................................ | *.........            7505000 | 272206       4962579           576.0     13028.6      8218683
671.94/172.83	c ................................ | *.........            8542000 | 316930       5629258           672.1     12710.2      9528955
767.76/196.97	c ................................ | *.........            9361000 | 355429       6237505           768.0     12188.3     11201041
863.67/221.10	c ................................ | *.........           10014000 | 389451       6755405           864.1     11589.5     13420157
960.00/245.22	c ................................ | **........           10651000 | 423030       7190365           960.1     11093.8     15455926
1055.90/269.34	c ................................ | **........           11407000 | 461734       7650492          1056.1     10801.3     16976529
1152.05/293.50	c ................................ | **........           12137000 | 495559       8151542          1152.1     10535.0     18436187
1247.83/317.68	c ................................ | **........           12860000 | 525740       8645372          1248.2     10303.2     19886275
1343.96/341.88	c ................................ | **........           13723000 | 569381       9152137          1344.3     10208.4     21208925
1439.76/365.97	c ................................ | ***.......           14492000 | 612021       9578076          1440.0     10063.6     23539906
1535.68/390.09	c ................................ | ***.......           15027000 | 643007       9876836          1536.0      9782.9     26808288
1631.89/414.23	c ................................ | ***.......           15748000 | 674091       10291787         1632.0      9649.4     27839243
1727.75/438.37	c ................................ | ***.......           16391000 | 703577       10687649         1728.0      9485.4     29456534
1824.08/462.53	c ................................ | ***.......           16952000 | 728311       11049203         1824.2      9292.8     30736177
1919.97/486.62	c ................................ | ****......           17495000 | 755469       11381412         1920.1      9111.6     32272132
2015.86/510.75	c ................................ | ****......           18135000 | 783366       11698005         2016.1      8995.3     33717691
2111.72/534.79	c ................................ | ****......           18795000 | 812566       12013733         2112.1      8898.7     35985828
2208.01/558.89	c ................................ | ****......           19268000 | 839354       12309455         2208.4      8724.9     37996454
2304.27/582.94	c ................................ | ****......           19796000 | 863425       12615801         2304.4      8590.4     39602513
2400.13/606.97	c ................................ | *****.....           20230000 | 884389       12863355         2400.4      8427.7     41062872
2496.00/630.90	c ................................ | *****.....           20611000 | 905417       13094793         2496.0      8257.5     42789853
2591.87/654.94	c ................................ | *****.....           20982000 | 927923       13304289         2592.0      8094.8     44805662
2687.74/678.97	c ................................ | *****.....           21289000 | 945958       13486899         2688.1      7919.9     46513000
2784.01/703.00	c ................................ | *****.....           21551000 | 962594       13659966         2784.0      7741.0     48289791
2879.90/727.10	c ................................ | ******....           21789000 | 976582       13822127         2880.3      7564.9     49850981
2975.75/751.07	c ................................ | ******....           22001000 | 987811       13969744         2976.1      7392.7     51500030
3071.62/775.09	c ................................ | ******....           22195000 | 999296       14131080         3072.0      7224.9     52765680
3168.28/799.21	c ................................ | ******....           22382000 | 1011183      14278157         3168.3      7064.3     53965552
3263.75/823.18	c ................................ | ******....           22667000 | 1028519      14453361         3264.1      6944.4     55252746
3360.02/847.27	c ................................ | *******...           22977000 | 1049364      14641303         3360.3      6837.7     57058754
3455.92/871.27	c ................................ | *******...           23223000 | 1065314      14792254         3456.2      6719.3     58726914
3552.18/895.31	c ................................ | *******...           23494000 | 1082261      14952823         3552.2      6613.9     60411840
3648.44/919.42	c ................................ | *******...           23714000 | 1097744      15114652         3648.5      6499.6     62039751
3743.90/943.32	c ................................ | *******...           23950000 | 1112511      15281503         3744.0      6396.9     63199361
3840.18/967.42	c ................................ | ********..           24184000 | 1129629      15439167         3840.3      6297.5     64848251
3936.04/991.42	c ................................ | ********..           24374000 | 1142900      15581528         3936.1      6192.4     66095568
4031.91/1015.45	c ................................ | ********..           24512000 | 1154348      15706485         4032.1      6079.2     67092453
4128.20/1039.57	c ................................ | ********..           24639000 | 1164435      15821509         4128.5      5968.1     67946600
4223.66/1063.49	c ................................ | ********..           24894000 | 1183334      15962042         4224.0      5893.4     69511429
4320.32/1087.60	c ................................ | *********.           25137000 | 1198283      16106563         4320.3      5818.3     70676705
4415.79/1111.55	c ................................ | *********.           25593000 | 1220502      16333378         4416.0      5795.5     72152663
4512.07/1135.61	c ................................ | *********.           26045000 | 1239792      16574127         4512.1      5772.2     73391344
4607.94/1159.62	c ................................ | *********.           26515000 | 1263716      16819389         4608.1      5754.1     74325744
4703.82/1183.69	c ................................ | *********.           27046000 | 1292421      17064459         4704.2      5749.3     75632022

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250384-1304090009/watcher-3250384-1304090009 -o /tmp/evaluation-result-3250384-1304090009/solver-3250384-1304090009 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250384-1304090009.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 8.10 8.04 5/181 24521
/proc/meminfo: memFree=24846424/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=9244 CPUtime=0 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 366 0 0 0 0 0 0 0 23 0 1 0 520603767 9465856 308 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 2311 308 146 36 0 216 0

[startup+0.0710491 s]
/proc/loadavg: 8.11 8.10 8.04 5/181 24521
/proc/meminfo: memFree=24846424/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=16156 CPUtime=0.06 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 2061 0 0 0 6 0 0 0 23 0 1 0 520603767 16543744 2003 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 4039 2004 147 36 0 1944 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 16156

[startup+0.101041 s]
/proc/loadavg: 8.11 8.10 8.04 5/181 24521
/proc/meminfo: memFree=24846424/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=18928 CPUtime=0.09 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 2802 0 0 0 9 0 0 0 23 0 1 0 520603767 19382272 2744 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 4732 2744 147 36 0 2637 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18928

[startup+0.301008 s]
/proc/loadavg: 8.11 8.10 8.04 5/181 24521
/proc/meminfo: memFree=24846424/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=38984 CPUtime=0.29 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 7705 0 0 0 29 0 0 0 25 0 1 0 520603767 39919616 7647 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 9746 7648 147 36 0 7651 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 38984

[startup+0.700959 s]
/proc/loadavg: 8.11 8.10 8.04 5/181 24521
/proc/meminfo: memFree=24846424/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=78588 CPUtime=0.7 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 17507 0 0 0 68 2 0 0 25 0 1 0 520603767 80474112 17449 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370957056 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 19647 17450 148 36 0 17552 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 78588

[startup+1.50079 s]
/proc/loadavg: 7.86 8.05 8.02 6/182 24522
/proc/meminfo: memFree=24723556/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=317904 CPUtime=1.49 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 76794 0 0 0 140 9 0 0 25 0 1 0 520603767 325533696 76736 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370653963 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 79476 76738 151 36 0 77381 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 317904

[startup+3.1005 s]
/proc/loadavg: 7.86 8.05 8.02 6/182 24522
/proc/meminfo: memFree=24453112/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=350824 CPUtime=3.09 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) R 24519 24521 24182 0 -1 4202496 84943 0 0 0 299 10 0 0 25 0 1 0 520603767 359243776 84884 33554432000 4194304 4338100 140736681163264 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/24521/statm: 87706 84884 153 36 0 85611 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 350824

[startup+6.3009 s]
/proc/loadavg: 7.87 8.05 8.02 9/186 24526
/proc/meminfo: memFree=24366652/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=595236 CPUtime=11.52 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 89156 0 0 0 1136 16 0 0 25 0 5 0 520603767 609521664 86809 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 148809 86809 176 36 0 146714 0
[pid=24521/tid=24523] ppid=24519 vsize=595236 CPUtime=1.66 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 223 0 0 0 166 0 0 0 18 0 5 0 520604219 609521664 86809 33554432000 4194304 4338100 140736681163264 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=595236 CPUtime=1.77 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 160 0 0 0 177 0 0 0 20 0 5 0 520604219 609521664 86809 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382769870 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=595236 CPUtime=1.77 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 176 0 0 0 173 4 0 0 20 0 5 0 520604219 609521664 86809 33554432000 4194304 4338100 140736681163264 18446744073709551615 4287283 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=595236 CPUtime=1.77 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 165 0 0 0 177 0 0 0 25 0 5 0 520604219 609521664 86809 33554432000 4194304 4338100 140736681163264 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.52
Current children cumulated vsize (KiB) 595236

[startup+12.7008 s]
/proc/loadavg: 7.88 8.05 8.02 9/186 24527
/proc/meminfo: memFree=24218728/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=595236 CPUtime=36.84 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 90383 0 0 0 3653 31 0 0 25 0 5 0 520603767 609521664 88036 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 148809 88036 176 36 0 146714 0
[pid=24521/tid=24523] ppid=24519 vsize=595236 CPUtime=7.78 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 700 0 0 0 775 3 0 0 18 0 5 0 520604219 609521664 88036 33554432000 4194304 4338100 140736681163264 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=595236 CPUtime=8.17 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 287 0 0 0 817 0 0 0 25 0 5 0 520604219 609521664 88036 33554432000 4194304 4338100 140736681163264 18446744073709551615 4285491 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=595236 CPUtime=8.17 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 507 0 0 0 801 16 0 0 25 0 5 0 520604219 609521664 88036 33554432000 4194304 4338100 140736681163264 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=595236 CPUtime=8.17 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 457 0 0 0 817 0 0 0 25 0 5 0 520604219 609521664 88036 33554432000 4194304 4338100 140736681163264 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 36.84
Current children cumulated vsize (KiB) 595236

[startup+25.5005 s]
/proc/loadavg: 7.90 8.05 8.02 9/186 24527
/proc/meminfo: memFree=23956952/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=595236 CPUtime=87.61 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 94387 0 0 0 8699 62 0 0 25 0 5 0 520603767 609521664 92040 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 148809 92040 176 36 0 146714 0
[pid=24521/tid=24523] ppid=24519 vsize=595236 CPUtime=20.38 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 1752 0 0 0 2015 23 0 0 19 0 5 0 520604219 609521664 92040 33554432000 4194304 4338100 140736681163264 18446744073709551615 218370352524 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=595236 CPUtime=20.98 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 935 0 0 0 2097 1 0 0 25 0 5 0 520604219 609521664 92040 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=595236 CPUtime=20.74 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 1224 0 0 0 2049 25 0 0 18 0 5 0 520604219 609521664 92040 33554432000 4194304 4338100 140736681163264 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=595236 CPUtime=20.96 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 2044 0 0 0 2096 0 0 0 25 0 5 0 520604219 609521664 92040 33554432000 4194304 4338100 140736681163264 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 87.61
Current children cumulated vsize (KiB) 595236

[startup+51.1009 s]
/proc/loadavg: 8.01 8.06 8.02 9/186 24528
/proc/meminfo: memFree=23548848/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=595236 CPUtime=189.03 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 106900 0 0 0 18765 138 0 0 25 0 5 0 520603767 609521664 104553 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 148809 104553 176 36 0 146714 0
[pid=24521/tid=24523] ppid=24519 vsize=595236 CPUtime=45.41 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 3319 0 0 0 4477 64 0 0 18 0 5 0 520604219 609521664 104553 33554432000 4194304 4338100 140736681163264 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=595236 CPUtime=46.57 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 4442 0 0 0 4655 2 0 0 25 0 5 0 520604219 609521664 104553 33554432000 4194304 4338100 140736681163264 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=595236 CPUtime=45.92 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 3313 0 0 0 4537 55 0 0 18 0 5 0 520604219 609521664 104553 33554432000 4194304 4338100 140736681163264 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=595236 CPUtime=46.57 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 7394 0 0 0 4654 3 0 0 25 0 5 0 520604219 609521664 104553 33554432000 4194304 4338100 140736681163264 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 189.03
Current children cumulated vsize (KiB) 595236

[startup+102.307 s]
/proc/loadavg: 8.16 8.09 8.03 8/186 24529
/proc/meminfo: memFree=22901112/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=731320 CPUtime=392.2 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 136749 0 0 0 38934 286 0 0 25 0 5 0 520603767 748871680 134402 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 182830 134402 176 36 0 180735 0
[pid=24521/tid=24523] ppid=24519 vsize=731320 CPUtime=96.61 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 9121 0 0 0 9471 190 0 0 25 0 5 0 520604219 748871680 134402 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=731320 CPUtime=97.78 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 13678 0 0 0 9771 7 0 0 25 0 5 0 520604219 748871680 134402 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=731320 CPUtime=95.5 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 8638 0 0 0 9480 70 0 0 18 0 5 0 520604219 748871680 134402 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=731320 CPUtime=97.77 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 16880 0 0 0 9770 7 0 0 25 0 5 0 520604219 748871680 134402 33554432000 4194304 4338100 140736681163264 18446744073709551615 4289172 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 392.2
Current children cumulated vsize (KiB) 731320

[startup+162.301 s]
/proc/loadavg: 8.13 8.10 8.04 9/186 24531
/proc/meminfo: memFree=22235920/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=840272 CPUtime=630.24 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 179519 0 0 0 62587 437 0 0 25 0 5 0 520603767 860438528 177172 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 210068 177172 176 36 0 207973 0
[pid=24521/tid=24523] ppid=24519 vsize=840272 CPUtime=155.34 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 18400 0 0 0 15278 256 0 0 25 0 5 0 520604219 860438528 177172 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=840272 CPUtime=157.78 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 25743 0 0 0 15767 11 0 0 25 0 5 0 520604219 860438528 177172 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=840272 CPUtime=154.82 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 17496 0 0 0 15335 147 0 0 18 0 5 0 520604219 860438528 177172 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=840272 CPUtime=157.77 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 29448 0 0 0 15766 11 0 0 25 0 5 0 520604219 860438528 177172 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 630.24
Current children cumulated vsize (KiB) 840272

[startup+222.3 s]
/proc/loadavg: 8.13 8.10 8.04 9/186 24532
/proc/meminfo: memFree=21630412/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=1012596 CPUtime=868.82 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 225623 0 0 0 86324 558 0 0 25 0 5 0 520603767 1036898304 223276 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 253149 223276 176 36 0 251054 0
[pid=24521/tid=24523] ppid=24519 vsize=1012596 CPUtime=215.03 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 33077 0 0 0 21160 343 0 0 25 0 5 0 520604219 1036898304 223276 33554432000 4194304 4338100 140736681163264 18446744073709551615 4288215 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=1012596 CPUtime=217.78 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 36545 0 0 0 21763 15 0 0 25 0 5 0 520604219 1036898304 223276 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=1012596 CPUtime=213.7 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 30766 0 0 0 21198 172 0 0 18 0 5 0 520604219 1036898304 223276 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=1012596 CPUtime=217.77 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 36803 0 0 0 21761 16 0 0 25 0 5 0 520604219 1036898304 223276 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 868.82
Current children cumulated vsize (KiB) 1012596

################
# More data... #
################

/proc/meminfo: memFree=17778432/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=2517620 CPUtime=3020.91 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 583881 0 0 0 300103 1988 0 0 25 0 5 0 520603767 2578042880 581534 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 629405 581534 176 36 0 627310 0
[pid=24521/tid=24523] ppid=24519 vsize=2517620 CPUtime=752.91 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 111135 0 0 0 74063 1228 0 0 25 0 5 0 520604219 2578042880 581534 33554432000 4194304 4338100 140736681163264 18446744073709551615 4284433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=2517620 CPUtime=757.79 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 124212 0 0 0 75736 43 0 0 25 0 5 0 520604219 2578042880 581534 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=2517620 CPUtime=749 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 134155 0 0 0 74240 660 0 0 25 0 5 0 520604219 2578042880 581534 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=2517620 CPUtime=756.66 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 125947 0 0 0 75621 45 0 0 25 0 5 0 520604219 2578042880 581534 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3020.91
Current children cumulated vsize (KiB) 2517620

[startup+822.301 s]
/proc/loadavg: 8.17 8.09 8.04 9/186 24548
/proc/meminfo: memFree=17457380/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=2614568 CPUtime=3260.56 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 611483 0 0 0 323935 2121 0 0 25 0 5 0 520603767 2677317632 609136 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 653642 609136 176 36 0 651547 0
[pid=24521/tid=24523] ppid=24519 vsize=2614568 CPUtime=812.59 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 116312 0 0 0 79961 1298 0 0 25 0 5 0 520604219 2677317632 609136 33554432000 4194304 4338100 140736681163264 18446744073709551615 4279520 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=2614568 CPUtime=817.79 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 128276 0 0 0 81735 44 0 0 25 0 5 0 520604219 2677317632 609136 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=2614568 CPUtime=808.99 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 145315 0 0 0 80179 720 0 0 25 0 5 0 520604219 2677317632 609136 33554432000 4194304 4338100 140736681163264 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=2614568 CPUtime=816.65 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 133148 0 0 0 81618 47 0 0 25 0 5 0 520604219 2677317632 609136 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3260.56
Current children cumulated vsize (KiB) 2614568

[startup+882.301 s]
/proc/loadavg: 8.17 8.10 8.05 9/186 24550
/proc/meminfo: memFree=17110056/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=2787120 CPUtime=3500.24 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 647687 0 0 0 347710 2314 0 0 25 0 5 0 520603767 2854010880 645340 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 696780 645340 176 36 0 694685 0
[pid=24521/tid=24523] ppid=24519 vsize=2787120 CPUtime=872.26 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 124844 0 0 0 85826 1400 0 0 25 0 5 0 520604219 2854010880 645340 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382769958 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=2787120 CPUtime=877.79 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 134500 0 0 0 87733 46 0 0 25 0 5 0 520604219 2854010880 645340 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=2787120 CPUtime=868.98 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 158371 0 0 0 86093 805 0 0 25 0 5 0 520604219 2854010880 645340 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=2787120 CPUtime=876.65 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 141540 0 0 0 87615 50 0 0 25 0 5 0 520604219 2854010880 645340 33554432000 4194304 4338100 140736681163264 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3500.24
Current children cumulated vsize (KiB) 2787120

[startup+942.3 s]
/proc/loadavg: 8.10 8.09 8.05 9/186 24552
/proc/meminfo: memFree=16736552/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=2878916 CPUtime=3739.91 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 680105 0 0 0 371500 2491 0 0 25 0 5 0 520603767 2948009984 677758 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 719729 677758 176 36 0 717634 0
[pid=24521/tid=24523] ppid=24519 vsize=2878916 CPUtime=931.94 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 131842 0 0 0 91703 1491 0 0 25 0 5 0 520604219 2948009984 677758 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=2878916 CPUtime=937.8 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 140620 0 0 0 93731 49 0 0 25 0 5 0 520604219 2948009984 677758 33554432000 4194304 4338100 140736681163264 18446744073709551615 4256580 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=2878916 CPUtime=928.98 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 170152 0 0 0 92012 886 0 0 25 0 5 0 520604219 2948009984 677758 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=2878916 CPUtime=936.64 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 149059 0 0 0 93612 52 0 0 25 0 5 0 520604219 2948009984 677758 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3739.91
Current children cumulated vsize (KiB) 2878916

[startup+1002.3 s]
/proc/loadavg: 8.14 8.10 8.05 9/186 24553
/proc/meminfo: memFree=16351396/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=2966036 CPUtime=3979.59 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 709393 0 0 0 395351 2608 0 0 25 0 5 0 520603767 3037220864 707046 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 741509 707046 176 36 0 739414 0
[pid=24521/tid=24523] ppid=24519 vsize=2966036 CPUtime=991.62 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 137779 0 0 0 97610 1552 0 0 25 0 5 0 520604219 3037220864 707046 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=2966036 CPUtime=997.8 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 144830 0 0 0 99729 51 0 0 25 0 5 0 520604219 3037220864 707046 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=2966036 CPUtime=988.99 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 182048 0 0 0 97960 939 0 0 25 0 5 0 520604219 3037220864 707046 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=2966036 CPUtime=996.64 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 156304 0 0 0 99610 54 0 0 25 0 5 0 520604219 3037220864 707046 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3979.59
Current children cumulated vsize (KiB) 2966036

[startup+1062.3 s]
/proc/loadavg: 8.12 8.10 8.05 9/186 24555
/proc/meminfo: memFree=16035916/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=3121772 CPUtime=4219.26 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 735171 0 0 0 419215 2711 0 0 25 0 5 0 520603767 3196694528 732824 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 780443 732824 176 36 0 778348 0
[pid=24521/tid=24523] ppid=24519 vsize=3121772 CPUtime=1051.3 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 142134 0 0 0 103529 1601 0 0 25 0 5 0 520604219 3196694528 732824 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=3121772 CPUtime=1057.79 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 149182 0 0 0 105728 51 0 0 25 0 5 0 520604219 3196694528 732824 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=3121772 CPUtime=1048.99 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 191855 0 0 0 103909 990 0 0 25 0 5 0 520604219 3196694528 732824 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=3121772 CPUtime=1056.63 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 163568 0 0 0 105607 56 0 0 25 0 5 0 520604219 3196694528 732824 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4219.26
Current children cumulated vsize (KiB) 3121772

[startup+1122.3 s]
/proc/loadavg: 8.17 8.11 8.06 9/186 24556
/proc/meminfo: memFree=15667364/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=3221916 CPUtime=4458.93 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 763703 0 0 0 442933 2960 0 0 25 0 5 0 520603767 3299241984 761356 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 805479 761356 176 36 0 803384 0
[pid=24521/tid=24523] ppid=24519 vsize=3221916 CPUtime=1110.97 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 148321 0 0 0 109367 1730 0 0 25 0 5 0 520604219 3299241984 761356 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=3221916 CPUtime=1117.8 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 154111 0 0 0 111728 52 0 0 25 0 5 0 520604219 3299241984 761356 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=3221916 CPUtime=1108.99 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 202176 0 0 0 109792 1107 0 0 25 0 5 0 520604219 3299241984 761356 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=3221916 CPUtime=1116.63 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 170663 0 0 0 111604 59 0 0 25 0 5 0 520604219 3299241984 761356 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4458.93
Current children cumulated vsize (KiB) 3221916

[startup+1182.3 s]
/proc/loadavg: 8.18 8.12 8.06 9/186 24558
/proc/meminfo: memFree=15353616/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=3320852 CPUtime=4698.61 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 790586 0 0 0 466582 3279 0 0 25 0 5 0 520603767 3400552448 788239 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 830213 788239 176 36 0 828118 0
[pid=24521/tid=24523] ppid=24519 vsize=3320852 CPUtime=1170.65 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 154855 0 0 0 115168 1897 0 0 25 0 5 0 520604219 3400552448 788239 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=3320852 CPUtime=1177.8 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 159551 0 0 0 117725 55 0 0 25 0 5 0 520604219 3400552448 788239 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=3320852 CPUtime=1168.99 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 210375 0 0 0 115646 1253 0 0 25 0 5 0 520604219 3400552448 788239 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=3320852 CPUtime=1176.63 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 177373 0 0 0 117602 61 0 0 25 0 5 0 520604219 3400552448 788239 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4698.61
Current children cumulated vsize (KiB) 3320852



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1207.7 s]
/proc/loadavg: 7.23 7.91 8.00 5/176 24577
/proc/meminfo: memFree=22080356/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=3474532 CPUtime=4800.12 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) S 24519 24521 24182 0 -1 4202496 805398 0 0 0 476633 3379 0 0 25 0 5 0 520603767 3557920768 803051 33554432000 4194304 4338100 140736681163264 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 868633 803051 176 36 0 866538 0
[pid=24521/tid=24523] ppid=24519 vsize=3474532 CPUtime=1195.93 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 159421 0 0 0 117645 1948 0 0 25 0 5 0 520604219 3557920768 803051 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=24521/tid=24524] ppid=24519 vsize=3474532 CPUtime=1203.21 cores=0,2,4,6
/proc/24521/task/24524/stat : 24524 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 161378 0 0 0 120263 58 0 0 25 0 5 0 520604219 3557920768 803051 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=24521/tid=24525] ppid=24519 vsize=3474532 CPUtime=1194.4 cores=0,2,4,6
/proc/24521/task/24525/stat : 24525 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 215629 0 0 0 118141 1299 0 0 25 0 5 0 520604219 3557920768 803051 33554432000 4194304 4338100 140736681163264 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=24521/tid=24526] ppid=24519 vsize=3474532 CPUtime=1202.04 cores=0,2,4,6
/proc/24521/task/24526/stat : 24526 (ttsth-5-4) R 24519 24521 24182 0 -1 4202560 180538 0 0 0 120142 62 0 0 25 0 5 0 520604219 3557920768 803051 33554432000 4194304 4338100 140736681163264 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 3474532

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1207.82 s]
/proc/loadavg: 7.23 7.91 8.00 5/176 24577
/proc/meminfo: memFree=22080356/32951124 swapFree=67051468/67111528
[pid=24521] ppid=24519 vsize=0 CPUtime=4800.24 cores=0,2,4,6
/proc/24521/stat : 24521 (ttsth-5-4) Z 24519 24521 24182 0 -1 4203548 805400 0 0 0 476635 3389 0 0 18 0 2 0 520603767 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/24521/statm: 0 0 0 0 0 0 0
[pid=24521/tid=24523] ppid=24519 vsize=0 CPUtime=1196.03 cores=0,2,4,6
/proc/24521/task/24523/stat : 24523 (ttsth-5-4) R 24519 24521 24182 0 -1 4203588 159421 0 0 0 117645 1958 0 0 25 0 2 0 520604219 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1207.85
CPU time (s): 4800.28
CPU user time (s): 4766.36
CPU system time (s): 33.9198
CPU usage (%): 397.424
Max. virtual memory (cumulated for all children) (KiB): 3474532

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4766.36
system time used= 33.9198
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 805400
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 23672
involuntary context switches= 19793

runsolver used 1.9847 second user time and 4.48732 second system time

The end

Launcher Data 

Begin job on node101 at 2011-04-29 17:13:30
IDJOB=3250384
IDBENCH=83251
IDSOLVER=1654
FILE ID=node101/3250384-1304090009
RUNJOBID= node101-1304087588-24200
PBS_JOBID= 13170559
Free space on /tmp= 72000 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-5-5_34309.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250384-1304090009/watcher-3250384-1304090009 -o /tmp/evaluation-result-3250384-1304090009/solver-3250384-1304090009 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250384-1304090009.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e252fbf81407f3388187ab50f0786909
RANDOM SEED=315599574

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24846580 kB
Buffers:       2025644 kB
Cached:        5148648 kB
SwapCached:      30452 kB
Active:        3533760 kB
Inactive:      4126236 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24846580 kB
SwapTotal:    67111528 kB
SwapFree:     67051468 kB
Dirty:           23292 kB
Writeback:           0 kB
AnonPages:      476492 kB
Mapped:           9352 kB
Slab:           380944 kB
PageTables:       5616 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   708648 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72000 MiB
End job on node101 at 2011-04-29 17:33:40