Trace number 3249328

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.19 1206.62

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-21x33.cnf
MD5SUMf7c29bbc3bef00d1e7cc799553ba4813
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2772
Number of clauses448371
Sum of the clauses size1785168
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 24158
Number of clauses of size 30
Number of clauses of size 4444213
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/0.48	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249328-1304061568.cnf 
0.39/0.48	c Using SATTIMEOUT of 4800
0.39/0.48	c Using SATRAM of 15500
0.39/0.48	c Size of subset node = 32
0.39/0.48	c Variables = 2772
0.39/0.48	c Clauses = 448371
0.39/0.48	c Literals = 1785168
0.39/0.48	c Reduced size:
0.39/0.48	c Variables = 2772
0.39/0.48	c Clauses = 448371
0.39/0.48	c Literals = 1785168
0.39/0.48	c a2
1.39/1.42	c a1
4.19/4.29	c Independent sub-problems = 1
4.19/4.29	c Sub-problem with 2772 variables and 448371 clauses
4.19/4.29	c using permutation method 4
4.19/4.29	c count = 1
4.19/4.29	c Ordering finished 4.190 seconds
4.19/4.29	c mapped 4.280 seconds
4.19/4.29	c permuted 4.280 seconds
4.19/4.29	c about to sort
4.30/4.35	c sorted
5.19/5.26	c clauses now 448371
5.19/5.26	c init finished 4.400 seconds
5.19/5.26	c tree built 5.260 seconds
5.19/5.26	c 
5.19/5.26	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
5.19/5.26	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
5.19/5.26	c ................................ | ..........                    |                                  5.3         0.0             
5.19/5.26	c ................................ | ..........                  1 |                                  5.3         0.2             
5.19/5.26	c ................................ | ..........                  2 |              11                  5.3         0.4             
5.19/5.26	c ................................ | ..........                  3 |              12                  5.3         0.6             
5.19/5.26	c ................................ | ..........                  4 |              12                  5.3         0.8             
5.19/5.26	c ................................ | ..........                  5 |              12                  5.3         1.0             
5.19/5.27	c ................................ | ..........               1000 | 1            604                 5.3       188.7          103
96.01/28.00	c ................................ | ..........            1214000 | 70995        510438             96.0     12644.5      3186726
191.81/52.05	c ................................ | ..........            1984000 | 130481       913713            192.0     10332.3      5555320
288.00/76.12	c ................................ | ..........            2913000 | 199095       1256654           288.1     10111.8      8839551
383.82/100.17	c ................................ | ..........            3545000 | 251660       1586980           384.1      9229.8     10824290
479.98/124.21	c ................................ | *.........            4065000 | 294775       1837126           480.0      8468.4     12397535
575.75/148.28	c ................................ | *.........            4693000 | 349474       2213419           576.1      8146.4     14416490
671.93/172.31	c ................................ | *.........            5375000 | 410689       2672166           672.0      7998.5     17842986
767.73/196.37	c ................................ | *.........            6086000 | 448399       2959951           768.0      7924.4     20232935
863.92/220.47	c ................................ | *.........            6753000 | 476657       3139327           864.2      7813.9     22064571
959.73/244.49	c ................................ | **........            7270000 | 500544       3269326           960.1      7572.3     23457318
1055.93/268.52	c ................................ | **........            8034000 | 534885       3392552          1056.0      7607.7     25937006
1151.79/292.58	c ................................ | **........            8712000 | 568000       3504259          1152.1      7561.8     28320470
1247.97/316.63	c ................................ | **........            9332000 | 599547       3590814          1248.1      7477.0     30669208
1344.15/340.72	c ................................ | **........            9861000 | 628546       3706045          1344.2      7335.9     31388808
1439.94/364.72	c ................................ | ***.......           10324000 | 653305       3804402          1440.0      7169.3     32880974
1535.67/388.78	c ................................ | ***.......           10919000 | 681744       3925627          1536.0      7108.7     34295776
1631.86/412.86	c ................................ | ***.......           11411000 | 708755       4020056          1632.1      6991.6     34988924
1728.06/436.91	c ................................ | ***.......           11909000 | 736929       4139153          1728.1      6891.3     35943988
1823.89/460.93	c ................................ | ***.......           12276000 | 760531       4254830          1824.0      6730.2     37434167
1920.12/485.02	c ................................ | ****......           12592000 | 782597       4388724          1920.2      6557.7     38418774
2015.92/509.02	c ................................ | ****......           13165000 | 814078       4491200          2016.0      6530.2     40774738
2112.10/533.10	c ................................ | ****......           13586000 | 839814       4609214          2112.1      6432.3     42428307
2207.82/557.17	c ................................ | ****......           14007000 | 864950       4722728          2208.1      6343.4     43737411
2304.03/581.21	c ................................ | ****......           14441000 | 891890       4832261          2304.1      6267.6     44674021
2400.25/605.30	c ................................ | *****.....           14753000 | 914383       4939498          2400.3      6146.4     45912873
2496.47/629.40	c ................................ | *****.....           15020000 | 935856       5062049          2496.5      6016.4     47108546
2591.90/653.37	c ................................ | *****.....           15247000 | 953349       5154413          2592.2      5881.9     48396503
2687.67/677.39	c ................................ | *****.....           15564000 | 974533       5262156          2688.0      5790.1     49532753
2783.90/701.45	c ................................ | *****.....           16221000 | 1002869      5358319          2784.1      5826.3     51531721
2880.07/725.50	c ................................ | ******....           16843000 | 1033371      5451816          2880.1      5848.1     52264234
2975.85/749.56	c ................................ | ******....           17399000 | 1061287      5541294          2976.1      5846.2     52960857
3071.68/773.59	c ................................ | ******....           17929000 | 1088400      5610148          3072.1      5836.2     54226902
3167.87/797.69	c ................................ | ******....           18459000 | 1116740      5696044          3168.2      5826.2     55162733
3264.11/821.73	c ................................ | ******....           18890000 | 1141373      5782632          3264.2      5787.0     56167200
3359.84/845.77	c ................................ | *******...           19227000 | 1161470      5853089          3360.1      5722.1     57120670
3455.94/869.85	c ................................ | *******...           19865000 | 1196836      5948305          3456.2      5747.7     58102769
3551.75/893.87	c ................................ | *******...           20410000 | 1229897      6043362          3552.0      5746.0     58486158
3647.96/917.91	c ................................ | *******...           20900000 | 1257699      6124633          3648.0      5729.1     58945705
3743.76/941.99	c ................................ | *******...           21546000 | 1289769      6204904          3744.1      5754.6     60027823
3839.99/966.02	c ................................ | ********..           22180000 | 1323184      6284376          3840.1      5775.9     61934144
3935.85/990.07	c ................................ | ********..           22683000 | 1352028      6347958          3936.1      5762.8     63601447
4031.99/1014.16	c ................................ | ********..           23152000 | 1380359      6406436          4032.2      5741.8     64082060
4127.80/1038.16	c ................................ | ********..           23625000 | 1408008      6478623          4128.0      5723.1     64728996
4224.02/1062.20	c ................................ | ********..           24138000 | 1436744      6554553          4224.0      5714.4     65712656
4319.76/1086.29	c ................................ | *********.           24649000 | 1466571      6630584          4320.1      5705.6     67027278
4415.93/1110.32	c ................................ | *********.           25094000 | 1493696      6699289          4416.0      5682.5     68293192
4512.08/1134.41	c ................................ | *********.           25458000 | 1520203      6759668          4512.1      5642.1     69742652
4608.32/1158.50	c ................................ | *********.           25787000 | 1541295      6820718          4608.4      5595.7     70754438
4703.79/1182.47	c ................................ | *********.           26195000 | 1567026      6902497          4704.1      5568.6     72114990

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249328-1304061568/watcher-3249328-1304061568 -o /tmp/evaluation-result-3249328-1304061568/solver-3249328-1304061568 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249328-1304061568.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28572
/proc/meminfo: memFree=13706920/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 435 0 0 0 0 0 0 0 25 0 1 0 596387463 9633792 378 33554432000 4194304 4338100 140737107899136 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 2352 378 147 36 0 257 0

[startup+0.0526231 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28572
/proc/meminfo: memFree=13706920/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=15388 CPUtime=0.05 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 1914 0 0 0 5 0 0 0 25 0 1 0 596387463 15757312 1857 33554432000 4194304 4338100 140737107899136 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 3847 1858 148 36 0 1752 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 15388

[startup+0.100612 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28572
/proc/meminfo: memFree=13706920/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=21400 CPUtime=0.09 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 3354 0 0 0 9 0 0 0 25 0 1 0 596387463 21913600 3297 33554432000 4194304 4338100 140737107899136 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 5350 3298 148 36 0 3255 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 21400

[startup+0.300584 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28572
/proc/meminfo: memFree=13706920/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=46536 CPUtime=0.29 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 9394 0 0 0 29 0 0 0 25 0 1 0 596387463 47652864 9337 33554432000 4194304 4338100 140737107899136 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 11634 9338 148 36 0 9539 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 46536

[startup+0.700514 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28572
/proc/meminfo: memFree=13706920/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=152468 CPUtime=0.69 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 31516 0 0 0 66 3 0 0 25 0 1 0 596387463 156127232 30744 33554432000 4194304 4338100 140737107899136 18446744073709551615 218879960797 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 38117 30744 849 36 0 36022 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 152468

[startup+1.50038 s]
/proc/loadavg: 7.74 8.02 7.98 6/182 28573
/proc/meminfo: memFree=13573384/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=155972 CPUtime=1.49 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 32380 0 0 0 146 3 0 0 25 0 1 0 596387463 159715328 31608 33554432000 4194304 4338100 140737107899136 18446744073709551615 4228108 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 38993 31608 849 36 0 36898 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 155972

[startup+3.1011 s]
/proc/loadavg: 7.74 8.02 7.98 6/182 28573
/proc/meminfo: memFree=13606292/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=111284 CPUtime=3.09 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) R 28570 28572 28330 0 -1 4202496 44395 0 0 0 304 5 0 0 25 0 1 0 596387463 113954816 25130 33554432000 4194304 4338100 140737107899136 18446744073709551615 218880225989 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/28572/statm: 27821 25130 155 36 0 25726 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 111284

[startup+6.30046 s]
/proc/loadavg: 7.52 7.97 7.97 9/186 28577
/proc/meminfo: memFree=13492428/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=403752 CPUtime=9.39 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 58997 0 0 0 921 18 0 0 25 0 5 0 596387463 413442048 37979 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 100938 37979 176 36 0 98843 0
[pid=28572/tid=28574] ppid=28570 vsize=403752 CPUtime=1.02 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 362 0 0 0 100 2 0 0 25 0 5 0 596387989 413442048 37979 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=403752 CPUtime=1.01 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 652 0 0 0 99 2 0 0 19 0 5 0 596387989 413442048 37979 33554432000 4194304 4338100 140737107899136 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=403752 CPUtime=1.02 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 481 0 0 0 100 2 0 0 19 0 5 0 596387989 413442048 37979 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=403752 CPUtime=1.03 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 863 0 0 0 101 2 0 0 19 0 5 0 596387989 413442048 37979 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 9.39
Current children cumulated vsize (KiB) 403752

[startup+12.7003 s]
/proc/loadavg: 7.56 7.97 7.97 9/186 28578
/proc/meminfo: memFree=13358304/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=408632 CPUtime=34.94 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 68678 0 0 0 3399 95 0 0 25 0 5 0 596387463 418439168 47660 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 102158 47660 176 36 0 100063 0
[pid=28572/tid=28574] ppid=28570 vsize=408632 CPUtime=7.42 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 1850 0 0 0 720 22 0 0 25 0 5 0 596387989 418439168 47660 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=408632 CPUtime=7.38 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 3150 0 0 0 717 21 0 0 25 0 5 0 596387989 418439168 47660 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892390772 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=408632 CPUtime=7.42 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 2226 0 0 0 719 23 0 0 18 0 5 0 596387989 418439168 47660 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=408632 CPUtime=7.42 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 4813 0 0 0 722 20 0 0 25 0 5 0 596387989 418439168 47660 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34.94
Current children cumulated vsize (KiB) 408632

[startup+25.5011 s]
/proc/loadavg: 7.66 7.97 7.97 9/186 28578
/proc/meminfo: memFree=13099776/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=417244 CPUtime=86.04 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 83429 0 0 0 8315 289 0 0 25 0 5 0 596387463 427257856 62411 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 104311 62411 176 36 0 102216 0
[pid=28572/tid=28574] ppid=28570 vsize=417244 CPUtime=20.21 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 4626 0 0 0 1950 71 0 0 25 0 5 0 596387989 427257856 62411 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=417244 CPUtime=20.1 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 7089 0 0 0 1939 71 0 0 25 0 5 0 596387989 427257856 62411 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=417244 CPUtime=20.22 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 5559 0 0 0 1950 72 0 0 19 0 5 0 596387989 427257856 62411 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=417244 CPUtime=20.21 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 9516 0 0 0 1956 65 0 0 25 0 5 0 596387989 427257856 62411 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 86.04
Current children cumulated vsize (KiB) 417244

[startup+51.1007 s]
/proc/loadavg: 7.77 7.97 7.97 9/186 28579
/proc/meminfo: memFree=12688724/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=439268 CPUtime=188.21 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 105627 0 0 0 18065 756 0 0 25 0 5 0 596387463 449810432 84609 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 109817 84609 176 36 0 107722 0
[pid=28572/tid=28574] ppid=28570 vsize=439268 CPUtime=45.79 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 9150 0 0 0 4386 193 0 0 25 0 5 0 596387989 449810432 84609 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=439268 CPUtime=45.54 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 13143 0 0 0 4373 181 0 0 25 0 5 0 596387989 449810432 84609 33554432000 4194304 4338100 140737107899136 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=439268 CPUtime=45.8 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 11267 0 0 0 4389 191 0 0 18 0 5 0 596387989 449810432 84609 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=439268 CPUtime=45.79 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 15428 0 0 0 4396 183 0 0 21 0 5 0 596387989 449810432 84609 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 188.21
Current children cumulated vsize (KiB) 439268

[startup+102.307 s]
/proc/loadavg: 7.96 7.99 7.97 9/186 28580
/proc/meminfo: memFree=11894716/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=681520 CPUtime=392.6 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 155398 0 0 0 37586 1674 0 0 25 0 5 0 596387463 697876480 134380 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 170380 134380 176 36 0 168285 0
[pid=28572/tid=28574] ppid=28570 vsize=681520 CPUtime=96.95 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 19519 0 0 0 9266 429 0 0 25 0 5 0 596387989 697876480 134380 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=681520 CPUtime=96.45 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 26297 0 0 0 9241 404 0 0 18 0 5 0 596387989 697876480 134380 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=681520 CPUtime=96.96 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 23693 0 0 0 9272 424 0 0 21 0 5 0 596387989 697876480 134380 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=681520 CPUtime=96.95 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 29250 0 0 0 9286 409 0 0 25 0 5 0 596387989 697876480 134380 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 392.6
Current children cumulated vsize (KiB) 681520

[startup+162.3 s]
/proc/loadavg: 8.04 8.00 7.98 9/186 28582
/proc/meminfo: memFree=11209000/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=944784 CPUtime=632.01 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 205602 0 0 0 60471 2730 0 0 25 0 5 0 596387463 967458816 184584 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 236196 184584 176 36 0 234101 0
[pid=28572/tid=28574] ppid=28570 vsize=944784 CPUtime=156.56 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 30465 0 0 0 14970 686 0 0 24 0 5 0 596387989 967458816 184584 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892390772 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=944784 CPUtime=156.38 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 40723 0 0 0 14968 670 0 0 25 0 5 0 596387989 967458816 184584 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=944784 CPUtime=156.9 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 35528 0 0 0 14996 694 0 0 25 0 5 0 596387989 967458816 184584 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=944784 CPUtime=156.87 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 42247 0 0 0 15016 671 0 0 25 0 5 0 596387989 967458816 184584 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 632.01
Current children cumulated vsize (KiB) 944784

[startup+222.301 s]
/proc/loadavg: 8.10 8.03 7.99 9/186 28583
/proc/meminfo: memFree=10634740/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=1075076 CPUtime=871.51 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 253945 0 0 0 83559 3592 0 0 25 0 5 0 596387463 1100877824 232927 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 268769 232927 176 36 0 266674 0
[pid=28572/tid=28574] ppid=28570 vsize=1075076 CPUtime=216.41 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 39000 0 0 0 20741 900 0 0 20 0 5 0 596387989 1100877824 232927 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=1075076 CPUtime=216.34 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 54388 0 0 0 20747 887 0 0 25 0 5 0 596387989 1100877824 232927 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=1075076 CPUtime=216.64 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 46911 0 0 0 20751 913 0 0 25 0 5 0 596387989 1100877824 232927 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892379291 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=1075076 CPUtime=216.82 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 57007 0 0 0 20799 883 0 0 25 0 5 0 596387989 1100877824 232927 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 871.51
Current children cumulated vsize (KiB) 1075076

################
# More data... #
################

/proc/meminfo: memFree=5656568/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2190736 CPUtime=3027 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 526876 0 0 0 293190 9510 0 0 25 0 5 0 596387463 2243313664 505858 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 547684 505858 176 36 0 545589 0
[pid=28572/tid=28574] ppid=28570 vsize=2190736 CPUtime=755.42 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 95516 0 0 0 73149 2393 0 0 25 0 5 0 596387989 2243313664 505858 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2190736 CPUtime=755.95 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 128366 0 0 0 73213 2382 0 0 25 0 5 0 596387989 2243313664 505858 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2190736 CPUtime=755.52 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 117590 0 0 0 73174 2378 0 0 25 0 5 0 596387989 2243313664 505858 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2190736 CPUtime=754.81 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 128765 0 0 0 73133 2348 0 0 19 0 5 0 596387989 2243313664 505858 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3027
Current children cumulated vsize (KiB) 2190736

[startup+822.301 s]
/proc/loadavg: 8.13 8.09 8.01 9/186 28599
/proc/meminfo: memFree=5173216/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2347140 CPUtime=3266.48 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 548161 0 0 0 316573 10075 0 0 25 0 5 0 596387463 2403471360 527143 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 586785 527143 176 36 0 584690 0
[pid=28572/tid=28574] ppid=28570 vsize=2347140 CPUtime=815.37 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 95911 0 0 0 79002 2535 0 0 25 0 5 0 596387989 2403471360 527143 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2347140 CPUtime=815.93 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 135542 0 0 0 79061 2532 0 0 25 0 5 0 596387989 2403471360 527143 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2347140 CPUtime=815.47 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 124054 0 0 0 79032 2515 0 0 25 0 5 0 596387989 2403471360 527143 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2347140 CPUtime=814.43 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 136015 0 0 0 78958 2485 0 0 25 0 5 0 596387989 2403471360 527143 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3266.48
Current children cumulated vsize (KiB) 2347140

[startup+882.301 s]
/proc/loadavg: 8.11 8.08 8.01 9/186 28601
/proc/meminfo: memFree=4715460/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2377384 CPUtime=3505.84 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 570275 0 0 0 339921 10663 0 0 25 0 5 0 596387463 2434441216 549257 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 594346 549257 176 36 0 592251 0
[pid=28572/tid=28574] ppid=28570 vsize=2377384 CPUtime=875.29 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 96329 0 0 0 84843 2686 0 0 25 0 5 0 596387989 2434441216 549257 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2377384 CPUtime=875.84 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 142846 0 0 0 84906 2678 0 0 25 0 5 0 596387989 2434441216 549257 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2377384 CPUtime=875.38 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 132410 0 0 0 84874 2664 0 0 25 0 5 0 596387989 2434441216 549257 33554432000 4194304 4338100 140737107899136 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2377384 CPUtime=874.04 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 142051 0 0 0 84778 2626 0 0 25 0 5 0 596387989 2434441216 549257 33554432000 4194304 4338100 140737107899136 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3505.84
Current children cumulated vsize (KiB) 2377384

[startup+942.301 s]
/proc/loadavg: 8.14 8.10 8.01 9/186 28602
/proc/meminfo: memFree=4445336/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2449260 CPUtime=3745.35 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 585237 0 0 0 363273 11262 0 0 25 0 5 0 596387463 2508042240 564219 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 612315 564219 176 36 0 610220 0
[pid=28572/tid=28574] ppid=28570 vsize=2449260 CPUtime=935.24 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 96407 0 0 0 90688 2836 0 0 25 0 5 0 596387989 2508042240 564219 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2449260 CPUtime=935.53 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 146932 0 0 0 90727 2826 0 0 25 0 5 0 596387989 2508042240 564219 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2449260 CPUtime=935.34 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 142807 0 0 0 90718 2816 0 0 25 0 5 0 596387989 2508042240 564219 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2449260 CPUtime=933.95 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 142452 0 0 0 90619 2776 0 0 25 0 5 0 596387989 2508042240 564219 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3745.35
Current children cumulated vsize (KiB) 2449260

[startup+1002.3 s]
/proc/loadavg: 8.11 8.09 8.01 9/186 28604
/proc/meminfo: memFree=3979912/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2580332 CPUtime=3984.9 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 616748 0 0 0 386684 11806 0 0 25 0 5 0 596387463 2642259968 595730 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 645083 595730 176 36 0 642988 0
[pid=28572/tid=28574] ppid=28570 vsize=2580332 CPUtime=995.19 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 96407 0 0 0 96548 2971 0 0 25 0 5 0 596387989 2642259968 595730 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2580332 CPUtime=995.18 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 158279 0 0 0 96557 2961 0 0 25 0 5 0 596387989 2642259968 595730 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2580332 CPUtime=995.32 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 153808 0 0 0 96581 2951 0 0 25 0 5 0 596387989 2642259968 595730 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2580332 CPUtime=993.9 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 151615 0 0 0 96477 2913 0 0 21 0 5 0 596387989 2642259968 595730 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3984.9
Current children cumulated vsize (KiB) 2580332

[startup+1062.3 s]
/proc/loadavg: 8.18 8.10 8.02 9/186 28606
/proc/meminfo: memFree=3596064/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2665896 CPUtime=4224.41 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 632446 0 0 0 410088 12353 0 0 25 0 5 0 596387463 2729877504 611428 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 666474 611428 176 36 0 664379 0
[pid=28572/tid=28574] ppid=28570 vsize=2665896 CPUtime=1055.1 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 96591 0 0 0 102401 3109 0 0 25 0 5 0 596387989 2729877504 611428 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2665896 CPUtime=1054.91 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 159025 0 0 0 102397 3094 0 0 20 0 5 0 596387989 2729877504 611428 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2665896 CPUtime=1055.23 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 160712 0 0 0 102434 3089 0 0 25 0 5 0 596387989 2729877504 611428 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2665896 CPUtime=1053.87 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 159479 0 0 0 102335 3052 0 0 25 0 5 0 596387989 2729877504 611428 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4224.41
Current children cumulated vsize (KiB) 2665896

[startup+1122.3 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 28607
/proc/meminfo: memFree=3087980/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2760992 CPUtime=4463.85 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 660204 0 0 0 433492 12893 0 0 25 0 5 0 596387463 2827255808 639186 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 690248 639186 176 36 0 688153 0
[pid=28572/tid=28574] ppid=28570 vsize=2760992 CPUtime=1114.73 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 102919 0 0 0 108226 3247 0 0 25 0 5 0 596387989 2827255808 639186 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2760992 CPUtime=1114.83 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 169643 0 0 0 108252 3231 0 0 25 0 5 0 596387989 2827255808 639186 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2760992 CPUtime=1115.17 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 167135 0 0 0 108295 3222 0 0 18 0 5 0 596387989 2827255808 639186 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2760992 CPUtime=1113.83 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 163868 0 0 0 108198 3185 0 0 18 0 5 0 596387989 2827255808 639186 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4463.85
Current children cumulated vsize (KiB) 2760992

[startup+1182.3 s]
/proc/loadavg: 7.54 7.98 7.98 5/176 28630
/proc/meminfo: memFree=11862844/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2918244 CPUtime=4703.38 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 685086 0 0 0 456859 13479 0 0 25 0 5 0 596387463 2988281856 664068 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 729561 664068 176 36 0 727466 0
[pid=28572/tid=28574] ppid=28570 vsize=2918244 CPUtime=1174.35 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 109239 0 0 0 114045 3390 0 0 25 0 5 0 596387989 2988281856 664068 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2918244 CPUtime=1174.79 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 177070 0 0 0 114104 3375 0 0 25 0 5 0 596387989 2988281856 664068 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2918244 CPUtime=1175.15 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 172365 0 0 0 114148 3367 0 0 25 0 5 0 596387989 2988281856 664068 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2918244 CPUtime=1173.78 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 169773 0 0 0 114041 3337 0 0 25 0 5 0 596387989 2988281856 664068 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4703.38
Current children cumulated vsize (KiB) 2918244



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1206.5 s]
/proc/loadavg: 6.33 7.66 7.87 5/176 28630
/proc/meminfo: memFree=11840884/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=2918772 CPUtime=4800.06 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) S 28570 28572 28330 0 -1 4202496 690889 0 0 0 466269 13737 0 0 25 0 5 0 596387463 2988822528 669871 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892367845 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 729693 669871 176 36 0 727598 0
[pid=28572/tid=28574] ppid=28570 vsize=2918772 CPUtime=1198.44 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 111488 0 0 0 116390 3454 0 0 20 0 5 0 596387989 2988822528 669871 33554432000 4194304 4338100 140737107899136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28572/tid=28575] ppid=28570 vsize=2918772 CPUtime=1198.98 cores=1,3,5,7
/proc/28572/task/28575/stat : 28575 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 177197 0 0 0 116458 3440 0 0 25 0 5 0 596387989 2988822528 669871 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28572/tid=28576] ppid=28570 vsize=2918772 CPUtime=1199.35 cores=1,3,5,7
/proc/28572/task/28576/stat : 28576 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 174475 0 0 0 116502 3433 0 0 25 0 5 0 596387989 2988822528 669871 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28572/tid=28577] ppid=28570 vsize=2918772 CPUtime=1197.98 cores=1,3,5,7
/proc/28572/task/28577/stat : 28577 (ttsth-5-4) R 28570 28572 28330 0 -1 4202560 171090 0 0 0 116397 3401 0 0 25 0 5 0 596387989 2988822528 669871 33554432000 4194304 4338100 140737107899136 18446744073709551615 218892391093 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.06
Current children cumulated vsize (KiB) 2918772

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1206.61 s]
/proc/loadavg: 6.33 7.66 7.87 2/174 28631
/proc/meminfo: memFree=14217420/32950928 swapFree=67111528/67111528
[pid=28572] ppid=28570 vsize=0 CPUtime=4800.18 cores=1,3,5,7
/proc/28572/stat : 28572 (ttsth-5-4) Z 28570 28572 28330 0 -1 4203548 690889 0 0 0 466271 13747 0 0 18 0 2 0 596387463 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/28572/statm: 0 0 0 0 0 0 0
[pid=28572/tid=28574] ppid=28570 vsize=0 CPUtime=1198.55 cores=1,3,5,7
/proc/28572/task/28574/stat : 28574 (ttsth-5-4) R 28570 28572 28330 0 -1 4203588 111488 0 0 0 116391 3464 0 0 21 0 2 0 596387989 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1206.62
CPU time (s): 4800.19
CPU user time (s): 4662.71
CPU system time (s): 137.484
CPU usage (%): 397.82
Max. virtual memory (cumulated for all children) (KiB): 2918772

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4662.71
system time used= 137.484
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 690889
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 850541
involuntary context switches= 31048

runsolver used 1.93571 second user time and 4.63929 second system time

The end

Launcher Data 

Begin job on node112 at 2011-04-29 09:19:28
IDJOB=3249328
IDBENCH=83165
IDSOLVER=1654
FILE ID=node112/3249328-1304061568
RUNJOBID= node112-1304059146-28349
PBS_JOBID= 13170443
Free space on /tmp= 72556 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-21x33.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249328-1304061568/watcher-3249328-1304061568 -o /tmp/evaluation-result-3249328-1304061568/solver-3249328-1304061568 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249328-1304061568.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f7c29bbc3bef00d1e7cc799553ba4813
RANDOM SEED=1262489792

node112.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5261.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.93
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13707200 kB
Buffers:       1995312 kB
Cached:       15809700 kB
SwapCached:          0 kB
Active:        7926692 kB
Inactive:     10760072 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13707200 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           11456 kB
Writeback:          32 kB
AnonPages:      881156 kB
Mapped:          15216 kB
Slab:           492980 kB
PageTables:       6348 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  1069152 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72556 MiB
End job on node112 at 2011-04-29 09:39:37