Trace number 3251528

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.45 1209.04

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m42-q585-n40-p15-s54275047.cnf
MD5SUM67948a6aed9ea09317abe3c8a9a27a03
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark2.77358
Satisfiable
(Un)Satisfiability was proved
Number of variables1590
Number of clauses272291
Sum of the clauses size546092
Maximum clause length42
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2272251
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.09/0.20	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251528-1304108446.cnf 
0.09/0.20	c Using SATTIMEOUT of 4800
0.09/0.20	c Using SATRAM of 15500
0.09/0.20	c Size of subset node = 32
0.09/0.20	c Variables = 1590
0.09/0.20	c Clauses = 272291
0.09/0.20	c Literals = 546092
0.09/0.20	c Reduced size:
0.09/0.20	c Variables = 1590
0.09/0.20	c Clauses = 272291
0.09/0.20	c Literals = 546092
0.09/0.20	c a2
0.29/0.35	c a1
0.79/0.82	c Independent sub-problems = 1
0.79/0.82	c Sub-problem with 1590 variables and 272291 clauses
0.79/0.82	c using permutation method 4
0.79/0.82	c count = 1
0.79/0.82	c Ordering finished 0.780 seconds
0.79/0.82	c mapped 0.810 seconds
0.79/0.82	c permuted 0.810 seconds
0.79/0.82	c about to sort
0.79/0.84	c sorted
1.69/1.71	c clauses now 272291
1.69/1.71	c init finished 0.900 seconds
1.69/1.71	c tree built 1.700 seconds
1.69/1.71	c 
1.69/1.71	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.69/1.71	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.69/1.71	c ................................ | ..........                    |                                  1.7         0.0             
1.69/1.71	c ................................ | ..........                  1 |                                  1.7         0.6             
1.69/1.71	c ................................ | ..........                  2 |                                  1.7         1.2             
1.69/1.71	c ................................ | ..........                  3 |                                  1.7         1.8             
1.69/1.71	c ................................ | ..........                  4 |                                  1.7         2.4             
1.69/1.71	c ................................ | ..........                  5 |                                  1.7         2.9             
1.69/1.72	c ................................ | ..........               1000 |              87                  1.7       581.4         5412
95.90/25.62	c ................................ | ..........            5200000 | 4116         550088             96.0     54166.7     22058567
191.66/49.69	c ................................ | ..........           10072000 | 7783         1068233           192.0     52455.6     38064697
288.07/73.90	c ................................ | ..........           14780000 | 12431        1564643           288.1     51307.0     49073972
383.99/98.10	c ................................ | ..........           19283000 | 16190        2030331           384.0     50216.1     55568905
479.74/122.36	c ................................ | *.........           23626000 | 19954        2498583           480.0     49220.8     65388171
575.62/146.59	c ................................ | *.........           28062000 | 22193        2969969           576.0     48717.9     74654564
671.90/170.82	c ................................ | *.........           32670000 | 24776        3435344           672.0     48615.3     92016083
767.94/194.91	c ................................ | *.........           37031000 | 27593        3912714           768.0     48216.8    103500899
863.76/219.25	c ................................ | *.........           41462000 | 30288        4368955           864.0     47988.4    112268091
959.94/243.31	c ................................ | **........           45649000 | 32799        4810184           960.0     47551.0    119846740
1055.92/267.41	c ................................ | **........           49693000 | 35208        5251075          1056.0     47057.8    131183698
1151.84/291.64	c ................................ | **........           53467000 | 37722        5680772          1152.0     46412.3    136077428
1247.78/315.85	c ................................ | **........           57394000 | 41677        6107925          1248.0     45988.8    137189688
1343.67/340.08	c ................................ | **........           61517000 | 46518        6540746          1344.0     45771.2    147522175
1440.01/364.30	c ................................ | ***.......           65676000 | 50631        6975568          1440.0     45608.0    164044834
1536.30/388.69	c ................................ | ***.......           69480000 | 54892        7377169          1536.7     45213.8    174726785
1631.77/412.76	c ................................ | ***.......           73339000 | 57779        7767400          1632.0     44937.8    180637745
1727.92/436.92	c ................................ | ***.......           77201000 | 59848        8170635          1728.0     44676.5    182180960
1823.83/461.14	c ................................ | ***.......           81047000 | 62641        8566218          1824.0     44433.4    188040819
1919.94/485.43	c ................................ | ****......           84990000 | 65400        8970597          1920.1     44263.8    197615124
2015.68/509.47	c ................................ | ****......           88601000 | 68799        9347457          2016.0     43948.9    200450787
2111.84/533.54	c ................................ | ****......           92043000 | 70696        9721914          2112.0     43580.6    202806066
2207.63/557.59	c ................................ | ****......           95917000 | 73124        10099038         2208.0     43440.7    207769448
2303.81/581.65	c ................................ | ****......           99754000 | 75119        10479233         2304.0     43295.8    214054187
2399.91/605.72	c ................................ | *****.....          103486000 | 77453        10853781         2400.0     43119.2    221153653
2495.64/629.79	c ................................ | *****.....          107307000 | 79958        11238498         2496.0     42991.4    232043119
2591.63/653.89	c ................................ | *****.....          110781000 | 81730        11616700         2592.0     42739.3    240428264
2687.79/677.95	c ................................ | *****.....          114603000 | 84394        12000715         2688.0     42635.0    242099093
2783.82/702.04	c ................................ | *****.....          118386000 | 87411        12371313         2784.0     42523.7    248283640
2879.96/726.11	c ................................ | ******....          121888000 | 90387        12719271         2880.0     42321.9    252220976
2975.66/750.18	c ................................ | ******....          125445000 | 93185        13075679         2976.0     42152.1    253667985
3071.85/774.24	c ................................ | ******....          128983000 | 95953        13435598         3072.0     41986.4    263013207
3167.97/798.31	c ................................ | ******....          132386000 | 97513        13788503         3168.0     41788.4    264198737
3264.02/822.50	c ................................ | ******....          136029000 | 99259        14164861         3264.0     41675.3    268561475
3359.92/846.73	c ................................ | *******...          139466000 | 100954       14530312         3360.0     41507.5    269463506
3455.87/870.94	c ................................ | *******...          142824000 | 102400       14889044         3456.0     41326.3    271744400
3551.94/895.12	c ................................ | *******...          146419000 | 104410       15308891         3552.0     41221.4    277492104
3647.98/919.21	c ................................ | *******...          149910000 | 106586       15690693         3648.0     41093.5    280878381
3743.78/943.36	c ................................ | *******...          153315000 | 108994       16076839         3744.0     40949.3    284876213
3839.71/967.57	c ................................ | ********..          156533000 | 110744       16453756         3840.0     40763.7    285626879
3935.72/991.77	c ................................ | ********..          160146000 | 112886       16848923         3936.0     40687.5    295404604
4031.69/1015.88	c ................................ | ********..          163566000 | 115035       17220365         4032.0     40566.9    303437866
4127.73/1039.96	c ................................ | ********..          166855000 | 117672       17592401         4128.0     40420.3    306177407
4223.71/1064.08	c ................................ | ********..          170386000 | 119296       17968221         4224.0     40337.6    314104233
4319.99/1088.20	c ................................ | *********.          173614000 | 120819       18322949         4320.0     40188.3    318440043
4415.73/1112.27	c ................................ | *********.          177050000 | 122947       18707007         4416.0     40092.8    319310905
4512.07/1136.42	c ................................ | *********.          180230000 | 126156       19062366         4512.2     39943.0    322693040
4607.81/1160.44	c ................................ | *********.          183439000 | 128558       19434594         4608.0     39808.8    322693040
4703.83/1184.55	c ................................ | *********.          186426000 | 130728       19775601         4704.0     39631.2    323093221
4799.93/1208.63	c CPU limit (4800) reached
4799.93/1208.63	c Time = 4800.03 seconds
4799.93/1208.63	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251528-1304108446/watcher-3251528-1304108446 -o /tmp/evaluation-result-3251528-1304108446/solver-3251528-1304108446 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251528-1304108446.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 8.08 8.03 2/175 26749
/proc/meminfo: memFree=30831704/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=9548 CPUtime=0 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) R 26747 26749 26305 0 -1 4202496 471 0 0 0 0 0 0 0 25 0 1 0 522444208 9777152 413 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/26749/statm: 2387 414 147 36 0 292 0

[startup+0.034601 s]
/proc/loadavg: 8.04 8.08 8.03 2/175 26749
/proc/meminfo: memFree=30831704/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=15000 CPUtime=0.03 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) R 26747 26749 26305 0 -1 4202496 1804 0 0 0 3 0 0 0 25 0 1 0 522444208 15360000 1746 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/26749/statm: 3750 1748 148 36 0 1655 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 15000

[startup+0.100623 s]
/proc/loadavg: 8.04 8.08 8.03 2/175 26749
/proc/meminfo: memFree=30831704/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=27360 CPUtime=0.09 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) R 26747 26749 26305 0 -1 4202496 4796 0 0 0 9 0 0 0 25 0 1 0 522444208 28016640 4738 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/26749/statm: 6840 4741 148 36 0 4745 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27360

[startup+0.300525 s]
/proc/loadavg: 8.04 8.08 8.03 2/175 26749
/proc/meminfo: memFree=30831704/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=63324 CPUtime=0.29 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) R 26747 26749 26305 0 -1 4202496 13131 0 0 0 28 1 0 0 25 0 1 0 522444208 64843776 13072 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664203458 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/26749/statm: 15831 13072 153 36 0 13736 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 63324

[startup+0.70044 s]
/proc/loadavg: 8.04 8.08 8.03 2/175 26749
/proc/meminfo: memFree=30831704/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=69576 CPUtime=0.69 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) R 26747 26749 26305 0 -1 4202496 20706 0 0 0 67 2 0 0 25 0 1 0 522444208 71245824 15266 33554432000 4194304 4338100 140735709593776 18446744073709551615 4297518 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/26749/statm: 17394 15266 156 36 0 15299 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 69576

[startup+1.50032 s]
/proc/loadavg: 8.04 8.08 8.03 7/180 26754
/proc/meminfo: memFree=30698248/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=133860 CPUtime=1.49 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) R 26747 26749 26305 0 -1 4202496 37375 0 0 0 146 3 0 0 25 0 1 0 522444208 137072640 31403 33554432000 4194304 4338100 140735709593776 18446744073709551615 4278319 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/26749/statm: 33465 31403 164 36 0 31370 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 133860

[startup+3.10099 s]
/proc/loadavg: 7.80 8.02 8.01 10/184 26758
/proc/meminfo: memFree=30558160/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=318644 CPUtime=6.75 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 57014 0 0 0 659 16 0 0 25 0 5 0 522444208 326291456 50510 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 79661 50510 176 36 0 77566 0
[pid=26749/tid=26755] ppid=26747 vsize=318644 CPUtime=1.31 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 4011 0 0 0 131 0 0 0 19 0 5 0 522444380 326291456 50510 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676621006 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=318644 CPUtime=0.95 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1680 0 0 0 90 5 0 0 25 0 5 0 522444380 326291456 50510 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664913390 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=318644 CPUtime=1.37 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 4403 0 0 0 137 0 0 0 19 0 5 0 522444380 326291456 50510 33554432000 4194304 4338100 140735709593776 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=318644 CPUtime=1.37 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 3309 0 0 0 132 5 0 0 19 0 5 0 522444380 326291456 50510 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676621094 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 6.75
Current children cumulated vsize (KiB) 318644

[startup+6.30044 s]
/proc/loadavg: 7.80 8.02 8.01 9/184 26758
/proc/meminfo: memFree=30296612/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=504140 CPUtime=18.84 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 86007 0 0 0 1829 55 0 0 25 0 5 0 522444208 516239360 79503 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 126035 79503 176 36 0 123940 0
[pid=26749/tid=26755] ppid=26747 vsize=504140 CPUtime=4.43 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 12939 0 0 0 440 3 0 0 25 0 5 0 522444380 516239360 79504 33554432000 4194304 4338100 140735709593776 18446744073709551615 4229520 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=504140 CPUtime=3.63 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 6190 0 0 0 342 21 0 0 25 0 5 0 522444380 516239360 79504 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=504140 CPUtime=4.57 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 13736 0 0 0 455 2 0 0 25 0 5 0 522444380 516239360 79505 33554432000 4194304 4338100 140735709593776 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=504140 CPUtime=4.47 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 9533 0 0 0 424 23 0 0 25 0 5 0 522444380 516239360 79505 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 18.84
Current children cumulated vsize (KiB) 504140

[startup+12.7003 s]
/proc/loadavg: 7.83 8.02 8.01 9/184 26758
/proc/meminfo: memFree=29875828/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=670568 CPUtime=44.4 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 127177 0 0 0 4290 150 0 0 25 0 5 0 522444208 686661632 120673 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 167642 120673 176 36 0 165547 0
[pid=26749/tid=26755] ppid=26747 vsize=670568 CPUtime=10.79 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 27400 0 0 0 1071 8 0 0 25 0 5 0 522444380 686661632 120673 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664203396 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=670568 CPUtime=10.02 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 12712 0 0 0 936 66 0 0 25 0 5 0 522444380 686661632 120673 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=670568 CPUtime=10.97 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 27930 0 0 0 1092 5 0 0 25 0 5 0 522444380 686661632 120673 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664504800 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=670568 CPUtime=10.87 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 15524 0 0 0 1021 66 0 0 25 0 5 0 522444380 686661632 120673 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 44.4
Current children cumulated vsize (KiB) 670568

[startup+25.5011 s]
/proc/loadavg: 7.86 8.02 8.01 9/184 26758
/proc/meminfo: memFree=29076904/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=1042232 CPUtime=95.5 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 220697 0 0 0 9210 340 0 0 25 0 5 0 522444208 1067245568 214193 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 260558 214194 176 36 0 258463 0
[pid=26749/tid=26755] ppid=26747 vsize=1042232 CPUtime=23.52 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 59719 0 0 0 2336 16 0 0 25 0 5 0 522444380 1067245568 214194 33554432000 4194304 4338100 140735709593776 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=1042232 CPUtime=22.82 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 28375 0 0 0 2127 155 0 0 25 0 5 0 522444380 1067245568 214194 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=1042232 CPUtime=23.74 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 57363 0 0 0 2364 10 0 0 25 0 5 0 522444380 1067245568 214194 33554432000 4194304 4338100 140735709593776 18446744073709551615 4286331 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=1042232 CPUtime=23.66 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 31630 0 0 0 2213 153 0 0 25 0 5 0 522444380 1067245568 214194 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 95.5
Current children cumulated vsize (KiB) 1042232

[startup+51.1007 s]
/proc/loadavg: 7.96 8.03 8.01 9/184 26759
/proc/meminfo: memFree=27791176/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=1627400 CPUtime=197.65 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 362190 0 0 0 19054 711 0 0 25 0 5 0 522444208 1666457600 353777 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 406850 353777 176 36 0 404755 0
[pid=26749/tid=26755] ppid=26747 vsize=1627400 CPUtime=48.99 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 113869 0 0 0 4868 31 0 0 25 0 5 0 522444380 1666457600 353777 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664504761 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=1627400 CPUtime=48.4 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 52657 0 0 0 4507 333 0 0 18 0 5 0 522444380 1666457600 353777 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=1627400 CPUtime=49.29 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 96873 0 0 0 4912 17 0 0 25 0 5 0 522444380 1666457600 353777 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664504817 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=1627400 CPUtime=49.23 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 55180 0 0 0 4598 325 0 0 18 0 5 0 522444380 1666457600 353777 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 197.65
Current children cumulated vsize (KiB) 1627400

[startup+102.307 s]
/proc/loadavg: 8.04 8.04 8.01 9/184 26760
/proc/meminfo: memFree=26319516/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=2278024 CPUtime=400.62 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 525570 0 0 0 39130 932 0 0 25 0 5 0 522444208 2332696576 517157 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 569506 517157 176 36 0 567411 0
[pid=26749/tid=26755] ppid=26747 vsize=2278024 CPUtime=100.12 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 193460 0 0 0 9955 57 0 0 25 0 5 0 522444380 2332696576 517157 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=2278024 CPUtime=98.93 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 68796 0 0 0 9463 430 0 0 25 0 5 0 522444380 2332696576 517157 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=2278024 CPUtime=100.19 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 148375 0 0 0 9990 29 0 0 25 0 5 0 522444380 2332696576 517157 33554432000 4194304 4338100 140735709593776 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=2278024 CPUtime=99.63 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 71328 0 0 0 9552 411 0 0 18 0 5 0 522444380 2332696576 517157 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 400.62
Current children cumulated vsize (KiB) 2278024

[startup+162.3 s]
/proc/loadavg: 8.09 8.05 8.01 9/184 26762
/proc/meminfo: memFree=24404424/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=3147608 CPUtime=638.24 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 744455 0 0 0 62670 1154 0 0 25 0 5 0 522444208 3223150592 736042 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 786902 736042 176 36 0 784807 0
[pid=26749/tid=26755] ppid=26747 vsize=3147608 CPUtime=160.04 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 301428 0 0 0 15917 87 0 0 25 0 5 0 522444380 3223150592 736042 33554432000 4194304 4338100 140735709593776 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=3147608 CPUtime=158.87 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 87673 0 0 0 15309 578 0 0 25 0 5 0 522444380 3223150592 736042 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=3147608 CPUtime=159.78 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 209678 0 0 0 15939 39 0 0 25 0 5 0 522444380 3223150592 736042 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=3147608 CPUtime=157.79 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 102065 0 0 0 15335 444 0 0 18 0 5 0 522444380 3223150592 736042 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 638.24
Current children cumulated vsize (KiB) 3147608

[startup+222.301 s]
/proc/loadavg: 8.14 8.07 8.02 9/184 26795
/proc/meminfo: memFree=26150344/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=3954632 CPUtime=876.15 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 944801 0 0 0 85796 1819 0 0 25 0 5 0 522444208 4049543168 936388 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 988658 936388 176 36 0 986563 0
[pid=26749/tid=26755] ppid=26747 vsize=3954632 CPUtime=219.77 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 402283 0 0 0 21859 118 0 0 25 0 5 0 522444380 4049543168 936388 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1084247 0 0 0 75431 329 0 0 25 0 5 0 522444380 8887767040 2110517 33554432000 4194304 4338100 140735709593776 18446744073709551615 4257952 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=8679460 CPUtime=756.72 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 183606 0 0 0 72448 3224 0 0 25 0 5 0 522444380 8887767040 2110517 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=8679460 CPUtime=758 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 570433 0 0 0 75694 106 0 0 25 0 5 0 522444380 8887767040 2110517 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=8679460 CPUtime=750.26 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 237033 0 0 0 72322 2704 0 0 25 0 5 0 522444380 8887767040 2110518 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3024.34
Current children cumulated vsize (KiB) 8679460

[startup+822.3 s]
/proc/loadavg: 8.12 8.06 8.01 9/184 26811
/proc/meminfo: memFree=14465060/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=8976620 CPUtime=3263.21 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2193758 0 0 0 319424 6897 0 0 25 0 5 0 522444208 9192058880 2185345 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2244155 2185345 176 36 0 2242060 0
[pid=26749/tid=26755] ppid=26747 vsize=8976620 CPUtime=817.19 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1138694 0 0 0 81371 348 0 0 25 0 5 0 522444380 9192058880 2185345 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=8976620 CPUtime=816.61 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 183606 0 0 0 78171 3490 0 0 25 0 5 0 522444380 9192058880 2185345 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=8976620 CPUtime=817.92 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 590814 0 0 0 81683 109 0 0 25 0 5 0 522444380 9192058880 2185345 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=8976620 CPUtime=809.75 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 237033 0 0 0 78030 2945 0 0 18 0 5 0 522444380 9192058880 2185345 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3263.21
Current children cumulated vsize (KiB) 8976620

[startup+882.301 s]
/proc/loadavg: 7.97 8.03 8.00 9/184 26812
/proc/meminfo: memFree=13645268/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=9267952 CPUtime=3500.99 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2262627 0 0 0 343023 7076 0 0 25 0 5 0 522444208 9490382848 2254214 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2316988 2254214 176 36 0 2314893 0
[pid=26749/tid=26755] ppid=26747 vsize=9267952 CPUtime=876.69 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1185609 0 0 0 87302 367 0 0 25 0 5 0 522444380 9490382848 2254214 33554432000 4194304 4338100 140735709593776 18446744073709551615 4286319 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=9267952 CPUtime=876.49 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 186450 0 0 0 84029 3620 0 0 25 0 5 0 522444380 9490382848 2254214 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=9267952 CPUtime=877.87 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 606125 0 0 0 87674 113 0 0 25 0 5 0 522444380 9490382848 2254214 33554432000 4194304 4338100 140735709593776 18446744073709551615 4242487 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=9267952 CPUtime=868.19 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 240832 0 0 0 83849 2970 0 0 18 0 5 0 522444380 9490382848 2254214 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3500.99
Current children cumulated vsize (KiB) 9267952

[startup+942.3 s]
/proc/loadavg: 7.99 8.02 8.00 9/184 26814
/proc/meminfo: memFree=12812192/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=9527708 CPUtime=3739.81 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2328131 0 0 0 366262 7719 0 0 25 0 5 0 522444208 9756372992 2319718 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2381927 2319718 176 36 0 2379832 0
[pid=26749/tid=26755] ppid=26747 vsize=9527708 CPUtime=936.45 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1242076 0 0 0 93257 388 0 0 25 0 5 0 522444380 9756372992 2319718 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=9527708 CPUtime=936.13 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 186469 0 0 0 89680 3933 0 0 18 0 5 0 522444380 9756372992 2319718 33554432000 4194304 4338100 140735709593776 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=9527708 CPUtime=937.77 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 615135 0 0 0 93661 116 0 0 25 0 5 0 522444380 9756372992 2319718 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=9527708 CPUtime=927.71 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 240840 0 0 0 89494 3277 0 0 25 0 5 0 522444380 9756372992 2319718 33554432000 4194304 4338100 140735709593776 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3739.81
Current children cumulated vsize (KiB) 9527708

[startup+1002.3 s]
/proc/loadavg: 7.96 8.00 7.99 9/184 26815
/proc/meminfo: memFree=11932744/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=10012548 CPUtime=3977.98 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2452923 0 0 0 389758 8040 0 0 25 0 5 0 522444208 10252849152 2444510 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2503137 2444510 176 36 0 2501042 0
[pid=26749/tid=26755] ppid=26747 vsize=10012548 CPUtime=996.1 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1321936 0 0 0 99197 413 0 0 25 0 5 0 522444380 10252849152 2444510 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254413 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=10012548 CPUtime=994.92 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 188423 0 0 0 95456 4036 0 0 25 0 5 0 522444380 10252849152 2444510 33554432000 4194304 4338100 140735709593776 18446744073709551615 4256000 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=10012548 CPUtime=997.63 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 656100 0 0 0 99639 124 0 0 25 0 5 0 522444380 10252849152 2444511 33554432000 4194304 4338100 140735709593776 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=10012548 CPUtime=987.58 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 242854 0 0 0 95298 3460 0 0 25 0 5 0 522444380 10252849152 2444511 33554432000 4194304 4338100 140735709593776 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3977.98
Current children cumulated vsize (KiB) 10012548

[startup+1062.3 s]
/proc/loadavg: 8.03 8.01 8.00 9/184 26817
/proc/meminfo: memFree=10837904/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=10437956 CPUtime=4216.95 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2559187 0 0 0 413002 8693 0 0 25 0 5 0 522444208 10688466944 2550774 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2609489 2550774 176 36 0 2607394 0
[pid=26749/tid=26755] ppid=26747 vsize=10437956 CPUtime=1055.76 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1400553 0 0 0 105136 440 0 0 25 0 5 0 522444380 10688466944 2550774 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664466176 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=10437956 CPUtime=1054.77 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 188425 0 0 0 101138 4339 0 0 25 0 5 0 522444380 10688466944 2550774 33554432000 4194304 4338100 140735709593776 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=10437956 CPUtime=1057.51 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 683744 0 0 0 105622 129 0 0 25 0 5 0 522444380 10688466944 2550774 33554432000 4194304 4338100 140735709593776 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=10437956 CPUtime=1047.16 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 242854 0 0 0 100936 3780 0 0 25 0 5 0 522444380 10688466944 2550774 33554432000 4194304 4338100 140735709593776 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4216.95
Current children cumulated vsize (KiB) 10437956

[startup+1122.3 s]
/proc/loadavg: 8.06 8.02 8.00 9/184 26819
/proc/meminfo: memFree=10156104/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=10647664 CPUtime=4455.96 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2612894 0 0 0 436282 9314 0 0 25 0 5 0 522444208 10903207936 2604481 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2661916 2604481 176 36 0 2659821 0
[pid=26749/tid=26755] ppid=26747 vsize=10647664 CPUtime=1115.47 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1442731 0 0 0 111089 458 0 0 25 0 5 0 522444380 10903207936 2604481 33554432000 4194304 4338100 140735709593776 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=10647664 CPUtime=1114.53 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 188433 0 0 0 106809 4644 0 0 25 0 5 0 522444380 10903207936 2604481 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=10647664 CPUtime=1117.39 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 695253 0 0 0 111607 132 0 0 25 0 5 0 522444380 10903207936 2604481 33554432000 4194304 4338100 140735709593776 18446744073709551615 274664203280 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=10647664 CPUtime=1106.82 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 242866 0 0 0 106608 4074 0 0 25 0 5 0 522444380 10903207936 2604481 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4455.96
Current children cumulated vsize (KiB) 10647664

[startup+1182.31 s]
/proc/loadavg: 8.08 8.03 8.00 9/184 26820
/proc/meminfo: memFree=9729120/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=10889836 CPUtime=4695.13 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) S 26747 26749 26305 0 -1 4202496 2647884 0 0 0 459516 9997 0 0 25 0 5 0 522444208 11151192064 2639471 33554432000 4194304 4338100 140735709593776 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 2722459 2639471 176 36 0 2720364 0
[pid=26749/tid=26755] ppid=26747 vsize=10889836 CPUtime=1175.24 cores=0,2,4,6
/proc/26749/task/26755/stat : 26755 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 1463802 0 0 0 117053 471 0 0 25 0 5 0 522444380 11151192064 2639471 33554432000 4194304 4338100 140735709593776 18446744073709551615 4235906 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26749/tid=26756] ppid=26747 vsize=10889836 CPUtime=1174.26 cores=0,2,4,6
/proc/26749/task/26756/stat : 26756 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 194881 0 0 0 112445 4981 0 0 25 0 5 0 522444380 11151192064 2639471 33554432000 4194304 4338100 140735709593776 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=10889836 CPUtime=1177.32 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 699321 0 0 0 117598 134 0 0 25 0 5 0 522444380 11151192064 2639471 33554432000 4194304 4338100 140735709593776 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26749/tid=26758] ppid=26747 vsize=10889836 CPUtime=1166.57 cores=0,2,4,6
/proc/26749/task/26758/stat : 26758 (ttsth-5-4) R 26747 26749 26305 0 -1 4202560 246269 0 0 0 112250 4407 0 0 25 0 5 0 522444380 11151192064 2639471 33554432000 4194304 4338100 140735709593776 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4695.13
Current children cumulated vsize (KiB) 10889836



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1208.71 s]
/proc/loadavg: 8.05 8.03 8.00 6/181 26821
/proc/meminfo: memFree=11167316/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) Z 26747 26749 26305 0 -1 4203548 2661049 0 0 0 469707 10304 0 0 18 0 2 0 522444208 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 0 0 0 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=0 CPUtime=1203.69 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4203588 701261 0 0 0 120226 143 0 0 25 0 2 0 522444380 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1208.9 s]
/proc/loadavg: 8.05 8.03 8.00 6/181 26821
/proc/meminfo: memFree=11167316/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=0 CPUtime=4800.3 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) Z 26747 26749 26305 0 -1 4203548 2661049 0 0 0 469707 10323 0 0 18 0 2 0 522444208 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 0 0 0 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=0 CPUtime=1203.88 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4203588 701261 0 0 0 120226 162 0 0 25 0 2 0 522444380 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

[startup+1209 s]
/proc/loadavg: 8.05 8.03 8.00 6/181 26821
/proc/meminfo: memFree=11167316/32951132 swapFree=67111356/67111528
[pid=26749] ppid=26747 vsize=0 CPUtime=4800.4 cores=0,2,4,6
/proc/26749/stat : 26749 (ttsth-5-4) Z 26747 26749 26305 0 -1 4203548 2661049 0 0 0 469707 10333 0 0 18 0 2 0 522444208 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/26749/statm: 0 0 0 0 0 0 0
[pid=26749/tid=26757] ppid=26747 vsize=0 CPUtime=1203.98 cores=0,2,4,6
/proc/26749/task/26757/stat : 26757 (ttsth-5-4) R 26747 26749 26305 0 -1 4203588 701261 0 0 0 120226 172 0 0 25 0 2 0 522444380 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.4
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1209.04
CPU time (s): 4800.45
CPU user time (s): 4697.07
CPU system time (s): 103.372
CPU usage (%): 397.045
Max. virtual memory (cumulated for all children) (KiB): 10974884

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4697.07
system time used= 103.372
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2661049
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 33796
involuntary context switches= 22659

runsolver used 1.9597 second user time and 4.68329 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 22:20:46
IDJOB=3251528
IDBENCH=88225
IDSOLVER=1654
FILE ID=node131/3251528-1304108446
RUNJOBID= node131-1304107233-26621
PBS_JOBID= 13170641
Free space on /tmp= 71380 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m42-q585-n40-p15-s54275047.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251528-1304108446/watcher-3251528-1304108446 -o /tmp/evaluation-result-3251528-1304108446/solver-3251528-1304108446 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251528-1304108446.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 67948a6aed9ea09317abe3c8a9a27a03
RANDOM SEED=1368944089

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30832488 kB
Buffers:        205528 kB
Cached:        1518292 kB
SwapCached:        168 kB
Active:         173188 kB
Inactive:      1639064 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30832488 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            4304 kB
Writeback:           8 kB
AnonPages:       87788 kB
Mapped:          15388 kB
Slab:           242800 kB
PageTables:       4524 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   233488 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71380 MiB
End job on node131 at 2011-04-29 22:40:56