Trace number 3259626

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.19 1203.59

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v2640c305320gyes3.cnf
MD5SUM3b033385d50481e5bd2fe507f2941f4f
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2640
Number of clauses305320
Sum of the clauses size613120
Maximum clause length33
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2305240
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 580

Solver Data 

0.19/0.20	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259626-1304056163.cnf 
0.19/0.20	c Using SATTIMEOUT of 4800
0.19/0.20	c Using SATRAM of 15500
0.19/0.20	c Size of subset node = 32
0.19/0.20	c Variables = 2640
0.19/0.20	c Clauses = 305320
0.19/0.20	c Literals = 613120
0.19/0.20	c Reduced size:
0.19/0.20	c Variables = 2640
0.19/0.20	c Clauses = 305320
0.19/0.20	c Literals = 613120
0.19/0.20	c a2
0.29/0.38	c a1
0.79/0.81	c Independent sub-problems = 1
0.79/0.81	c Sub-problem with 2640 variables and 305320 clauses
0.79/0.81	c using permutation method 4
0.79/0.81	c count = 1
0.79/0.81	c Ordering finished 0.780 seconds
0.79/0.81	c mapped 0.810 seconds
0.79/0.81	c permuted 0.810 seconds
0.79/0.81	c about to sort
0.79/0.84	c sorted
1.59/1.64	c clauses now 305320
1.59/1.64	c init finished 0.900 seconds
1.59/1.64	c tree built 1.630 seconds
1.59/1.64	c 
1.59/1.64	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.59/1.64	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.59/1.64	c ................................ | ..........                    |                                  1.6         0.0             
1.59/1.64	c ................................ | ..........                  1 |                                  1.6         0.6             
1.59/1.64	c ................................ | ..........                  2 |                                  1.6         1.2             
1.59/1.64	c ................................ | ..........                  3 |                                  1.6         1.8             
1.59/1.64	c ................................ | ..........                  4 |                                  1.6         2.5             
1.59/1.64	c ................................ | ..........                  5 |                                  1.6         3.1             
1.59/1.64	c ................................ | ..........               1000 | 2            69                  1.6       609.8         1555
95.75/25.27	c ................................ | ..........            3387000 | 72764        259959             96.0     35273.9     10346273
192.01/49.30	c ................................ | ..........            6419000 | 133887       469984            192.0     33430.6     18841996
287.87/73.33	c ................................ | ..........            9130000 | 188812       669163            288.0     31701.4     24481140
383.74/97.37	c ................................ | ..........           12060000 | 248394       864020            384.0     31405.4     28725457
480.01/121.40	c ................................ | *.........           14626000 | 317481       1026805           480.0     30469.6     35001810
575.85/145.44	c ................................ | *.........           17025000 | 373169       1190209           576.0     29555.8     38727280
671.74/169.48	c ................................ | *.........           19273000 | 435888       1330059           672.0     28677.9     41221831
767.96/193.52	c ................................ | *.........           21833000 | 478718       1477500           768.0     28427.6     42208254
863.80/217.55	c ................................ | *.........           23977000 | 515655       1611696           864.0     27750.5     42208254
959.65/241.58	c ................................ | **........           26121000 | 560663       1753748           960.0     27209.1     42208254
1055.92/265.62	c ................................ | **........           28145000 | 599311       1882030          1056.0     26652.2     43834830
1151.76/289.66	c ................................ | **........           30026000 | 634665       1999687          1152.0     26064.0     45859454
1247.63/313.70	c ................................ | **........           32135000 | 675604       2111086          1248.0     25748.8     46766229
1343.88/337.75	c ................................ | **........           34238000 | 712065       2222367          1344.1     25473.6     47720485
1439.71/361.78	c ................................ | ***.......           36336000 | 751294       2332813          1440.0     25232.8     49153383
1535.96/385.81	c ................................ | ***.......           38374000 | 802129       2435061          1536.0     24982.7     51972692
1631.80/409.85	c ................................ | ***.......           40449000 | 849588       2540510          1632.0     24784.6     54826912
1727.65/433.90	c ................................ | ***.......           42393000 | 890962       2645983          1728.0     24532.4     56884115
1823.89/457.93	c ................................ | ***.......           44179000 | 925647       2746119          1824.0     24220.8     60692956
1919.71/481.97	c ................................ | ****......           45865000 | 956761       2842895          1920.0     23887.9     64133602
2015.98/506.02	c ................................ | ****......           47728000 | 994145       2934397          2016.0     23674.0     66215566
2111.76/530.07	c ................................ | ****......           49486000 | 1024581      3027427          2112.0     23430.4     67516855
2207.92/554.12	c ................................ | ****......           51165000 | 1059460      3116661          2208.0     23172.6     69577925
2303.64/578.19	c ................................ | ****......           52847000 | 1090537      3204778          2304.0     22936.9     70243979
2399.75/602.27	c ................................ | *****.....           54521000 | 1122472      3298099          2400.0     22716.8     70476226
2495.85/626.34	c ................................ | *****.....           56266000 | 1154761      3386930          2496.0     22542.3     72026537
2591.96/650.42	c ................................ | *****.....           57821000 | 1180463      3477535          2592.1     22306.9     73107423
2687.82/674.45	c ................................ | *****.....           59245000 | 1207189      3557116          2688.0     22040.2     73417994
2783.68/698.48	c ................................ | *****.....           60681000 | 1234640      3640084          2784.0     21796.3     74843060
2879.92/722.52	c ................................ | ******....           62022000 | 1259173      3716518          2880.0     21535.3     76708420
2975.76/746.56	c ................................ | ******....           63511000 | 1295778      3791261          2976.0     21341.0     78397252
3071.92/770.62	c ................................ | ******....           64897000 | 1324597      3870386          3072.0     21125.3     81294280
3167.76/794.66	c ................................ | ******....           66197000 | 1350356      3944138          3168.0     20895.3     82313351
3264.02/818.70	c ................................ | ******....           67656000 | 1381242      4032858          3264.0     20727.8     82799724
3359.89/842.73	c ................................ | *******...           68989000 | 1411705      4112827          3360.0     20532.4     85046401
3455.73/866.77	c ................................ | *******...           70347000 | 1437521      4194005          3456.0     20354.9     85426466
3551.79/890.97	c ................................ | *******...           71541000 | 1464512      4264403          3552.1     20140.6     86032471
3647.63/914.99	c ................................ | *******...           72782000 | 1492612      4340927          3648.0     19951.1     86741748
3743.90/939.02	c ................................ | *******...           73900000 | 1516452      4419981          3744.0     19738.2     87147024
3839.75/963.06	c ................................ | ********..           75176000 | 1540275      4492250          3840.0     19577.0     87147024
3935.60/987.10	c ................................ | ********..           76394000 | 1563966      4571102          3936.0     19409.0     88035363
4031.86/1011.14	c ................................ | ********..           77547000 | 1588902      4645559          4032.0     19232.7     89995744
4127.76/1035.17	c ................................ | ********..           78731000 | 1614529      4721085          4128.0     19072.3     92482575
4224.02/1059.21	c ................................ | ********..           79900000 | 1637279      4790926          4224.0     18915.5     94408279
4319.89/1083.23	c ................................ | *********.           81089000 | 1663497      4858820          4320.0     18770.4     96996247
4415.76/1107.26	c ................................ | *********.           82854000 | 1702813      4939962          4416.0     18762.1     98571620
4512.00/1131.30	c ................................ | *********.           84501000 | 1730853      5021276          4512.0     18728.0     99339878
4607.84/1155.34	c ................................ | *********.           86143000 | 1757588      5107412          4608.0     18694.2    100297663
4703.72/1179.38	c ................................ | *********.           87800000 | 1791740      5186754          4704.0     18664.8    102186553
4799.94/1203.42	c CPU limit (4800) reached
4799.94/1203.42	c Time = 4800.02 seconds
4799.94/1203.42	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259626-1304056163/watcher-3259626-1304056163 -o /tmp/evaluation-result-3259626-1304056163/solver-3259626-1304056163 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259626-1304056163.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.80 7.79 7.91 2/177 2663
/proc/meminfo: memFree=18285712/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=9244 CPUtime=0 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) R 2661 2663 1994 0 -1 4202496 391 0 0 0 0 0 0 0 21 0 1 0 517216764 9465856 333 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2663/statm: 2311 334 147 36 0 216 0

[startup+0.0985449 s]
/proc/loadavg: 6.80 7.79 7.91 2/177 2663
/proc/meminfo: memFree=18285712/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=28280 CPUtime=0.09 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) R 2661 2663 1994 0 -1 4202496 5074 0 0 0 8 1 0 0 21 0 1 0 517216764 28958720 5016 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2663/statm: 7070 5017 148 36 0 4975 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28280

[startup+0.100544 s]
/proc/loadavg: 6.80 7.79 7.91 2/177 2663
/proc/meminfo: memFree=18285712/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=28676 CPUtime=0.09 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) R 2661 2663 1994 0 -1 4202496 5171 0 0 0 8 1 0 0 21 0 1 0 517216764 29364224 5113 33554432000 4194304 4338100 140734325463872 18446744073709551615 230809165568 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2663/statm: 7169 5115 148 36 0 5074 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28676

[startup+0.300507 s]
/proc/loadavg: 6.80 7.79 7.91 2/177 2663
/proc/meminfo: memFree=18285712/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=67716 CPUtime=0.29 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) R 2661 2663 1994 0 -1 4202496 14979 0 0 0 27 2 0 0 22 0 1 0 517216764 69341184 14920 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808560883 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2663/statm: 16929 14920 153 36 0 14834 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 67716

[startup+0.700413 s]
/proc/loadavg: 6.80 7.79 7.91 2/177 2663
/proc/meminfo: memFree=18285712/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=76912 CPUtime=0.69 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) R 2661 2663 1994 0 -1 4202496 18497 0 0 0 66 3 0 0 24 0 1 0 517216764 78757888 17244 33554432000 4194304 4338100 140734325463872 18446744073709551615 4295813 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2663/statm: 19228 17244 156 36 0 17133 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 76912

[startup+1.50023 s]
/proc/loadavg: 6.80 7.79 7.91 7/182 2668
/proc/meminfo: memFree=18100300/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=178748 CPUtime=1.49 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) R 2661 2663 1994 0 -1 4202496 44552 0 0 0 143 6 0 0 25 0 1 0 517216764 183037952 42702 33554432000 4194304 4338100 140734325463872 18446744073709551615 4278080 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2663/statm: 44687 42702 164 36 0 42592 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 178748

[startup+3.1009 s]
/proc/loadavg: 6.80 7.79 7.91 11/186 2672
/proc/meminfo: memFree=17952592/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=414356 CPUtime=7.46 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 59262 0 0 0 730 16 0 0 25 0 5 0 517216764 424300544 56815 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 103589 56815 176 36 0 101494 0
[pid=2663/tid=2669] ppid=2661 vsize=414356 CPUtime=1.45 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 2785 0 0 0 144 1 0 0 20 0 5 0 517216928 424300544 56815 33554432000 4194304 4338100 140734325463872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=414356 CPUtime=1.45 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 1893 0 0 0 142 3 0 0 19 0 5 0 517216928 424300544 56815 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=414356 CPUtime=1.45 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 2497 0 0 0 145 0 0 0 20 0 5 0 517216928 424300544 56815 33554432000 4194304 4338100 140734325463872 18446744073709551615 4256284 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=414356 CPUtime=1.45 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 1843 0 0 0 141 4 0 0 19 0 5 0 517216928 424300544 56815 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 7.46
Current children cumulated vsize (KiB) 414356

[startup+6.30032 s]
/proc/loadavg: 6.90 7.79 7.91 9/186 2672
/proc/meminfo: memFree=17812064/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=464800 CPUtime=20.23 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 72294 0 0 0 1983 40 0 0 25 0 5 0 517216764 475955200 69847 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 116200 69847 176 36 0 114105 0
[pid=2663/tid=2669] ppid=2661 vsize=464800 CPUtime=4.65 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 6850 0 0 0 462 3 0 0 25 0 5 0 517216928 475955200 69847 33554432000 4194304 4338100 140734325463872 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=464800 CPUtime=4.65 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 4391 0 0 0 451 14 0 0 25 0 5 0 517216928 475955200 69847 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808862128 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=464800 CPUtime=4.62 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 6422 0 0 0 462 0 0 0 25 0 5 0 517216928 475955200 69847 33554432000 4194304 4338100 140734325463872 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=464800 CPUtime=4.64 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 4387 0 0 0 449 15 0 0 25 0 5 0 517216928 475955200 69847 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808561077 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 20.23
Current children cumulated vsize (KiB) 464800

[startup+12.7011 s]
/proc/loadavg: 6.98 7.79 7.91 9/186 2672
/proc/meminfo: memFree=17594404/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=559956 CPUtime=45.81 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 96218 0 0 0 4497 84 0 0 25 0 5 0 517216764 573394944 93771 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 139989 93771 176 36 0 137894 0
[pid=2663/tid=2669] ppid=2661 vsize=559956 CPUtime=11.05 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 16026 0 0 0 1100 5 0 0 25 0 5 0 517216928 573394944 93771 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808561378 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=559956 CPUtime=11.04 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 8798 0 0 0 1070 34 0 0 18 0 5 0 517216928 573394944 93771 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=559956 CPUtime=10.99 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 12719 0 0 0 1098 1 0 0 25 0 5 0 517216928 573394944 93771 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=559956 CPUtime=11.05 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 8431 0 0 0 1070 35 0 0 25 0 5 0 517216928 573394944 93771 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 45.81
Current children cumulated vsize (KiB) 559956

[startup+25.5008 s]
/proc/loadavg: 7.21 7.80 7.91 10/186 2672
/proc/meminfo: memFree=17209128/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=714908 CPUtime=96.94 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 134411 0 0 0 9520 174 0 0 25 0 5 0 517216764 732065792 131964 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 178727 131964 176 36 0 176632 0
[pid=2663/tid=2669] ppid=2661 vsize=714908 CPUtime=23.85 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 30856 0 0 0 2375 10 0 0 25 0 5 0 517216928 732065792 131964 33554432000 4194304 4338100 140734325463872 18446744073709551615 230809163835 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=714908 CPUtime=23.84 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 14151 0 0 0 2309 75 0 0 25 0 5 0 517216928 732065792 131964 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=714908 CPUtime=23.72 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 25080 0 0 0 2369 3 0 0 25 0 5 0 517216928 732065792 131964 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=714908 CPUtime=23.85 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 14080 0 0 0 2308 77 0 0 25 0 5 0 517216928 732065792 131964 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 96.94
Current children cumulated vsize (KiB) 714908

[startup+51.1011 s]
/proc/loadavg: 7.48 7.82 7.91 9/186 2673
/proc/meminfo: memFree=16652876/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=1065092 CPUtime=199.2 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 205822 0 0 0 19571 349 0 0 25 0 5 0 517216764 1090654208 203375 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 266273 203375 176 36 0 264178 0
[pid=2663/tid=2669] ppid=2661 vsize=1065092 CPUtime=49.44 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 60727 0 0 0 4925 19 0 0 25 0 5 0 517216928 1090654208 203375 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=1065092 CPUtime=49.45 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 24979 0 0 0 4787 158 0 0 25 0 5 0 517216928 1090654208 203375 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=1065092 CPUtime=49.2 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 44869 0 0 0 4913 7 0 0 25 0 5 0 517216928 1090654208 203375 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=1065092 CPUtime=49.44 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 25003 0 0 0 4788 156 0 0 25 0 5 0 517216928 1090654208 203375 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 199.2
Current children cumulated vsize (KiB) 1065092

[startup+102.307 s]
/proc/loadavg: 7.82 7.86 7.92 10/186 2674
/proc/meminfo: memFree=15831612/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=1407912 CPUtime=403.73 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 289955 0 0 0 39720 653 0 0 25 0 5 0 517216764 1441701888 287508 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 351978 287508 176 36 0 349883 0
[pid=2663/tid=2669] ppid=2661 vsize=1407912 CPUtime=100.65 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 109219 0 0 0 10027 38 0 0 25 0 5 0 517216928 1441701888 287508 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=1407912 CPUtime=100.65 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 33488 0 0 0 9763 302 0 0 25 0 5 0 517216928 1441701888 287508 33554432000 4194304 4338100 140734325463872 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=1407912 CPUtime=100.13 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 66720 0 0 0 10001 12 0 0 25 0 5 0 517216928 1441701888 287508 33554432000 4194304 4338100 140734325463872 18446744073709551615 4256655 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=1407912 CPUtime=100.64 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 30284 0 0 0 9771 293 0 0 25 0 5 0 517216928 1441701888 287508 33554432000 4194304 4338100 140734325463872 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 403.73
Current children cumulated vsize (KiB) 1407912

[startup+162.301 s]
/proc/loadavg: 8.02 7.91 7.93 9/186 2676
/proc/meminfo: memFree=15178728/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=1760240 CPUtime=643.36 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 378754 0 0 0 63337 999 0 0 25 0 5 0 517216764 1802485760 376307 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 440060 376307 176 36 0 437965 0
[pid=2663/tid=2669] ppid=2661 vsize=1760240 CPUtime=160.62 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 165216 0 0 0 16006 56 0 0 25 0 5 0 517216928 1802485760 376307 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808560777 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=1760240 CPUtime=160.55 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 40653 0 0 0 15593 462 0 0 25 0 5 0 517216928 1802485760 376307 33554432000 4194304 4338100 140734325463872 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=1760240 CPUtime=159.89 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 87196 0 0 0 15972 17 0 0 25 0 5 0 517216928 1802485760 376307 33554432000 4194304 4338100 140734325463872 18446744073709551615 4253935 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=1760240 CPUtime=160.63 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 35445 0 0 0 15608 455 0 0 25 0 5 0 517216928 1802485760 376307 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254849 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 643.36
Current children cumulated vsize (KiB) 1760240

[startup+222.301 s]
/proc/loadavg: 8.01 7.92 7.93 10/186 2678
/proc/meminfo: memFree=14830768/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=1954508 CPUtime=882.97 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 416700 0 0 0 86989 1308 0 0 25 0 5 0 517216764 2001416192 413005 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 488627 413005 176 36 0 486532 0
[pid=2663/tid=2669] ppid=2661 vsize=1954508 CPUtime=220.62 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 178772 0 0 0 21995 67 0 0 25 0 5 0 517216928 2001416192 413005 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 47764 0 0 0 68591 1418 0 0 25 0 5 0 517216928 3090763776 675056 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2799.24
Current children cumulated vsize (KiB) 3018324

[startup+762.307 s]
/proc/loadavg: 8.16 8.08 8.00 9/186 2701
/proc/meminfo: memFree=12004836/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3196620 CPUtime=3038.82 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 723231 0 0 0 300691 3191 0 0 25 0 5 0 517216764 3273338880 719536 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 799155 719536 176 36 0 797060 0
[pid=2663/tid=2669] ppid=2661 vsize=3196620 CPUtime=759.89 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 393608 0 0 0 75825 164 0 0 25 0 5 0 517216928 3273338880 719536 33554432000 4194304 4338100 140734325463872 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3196620 CPUtime=757.79 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 52269 0 0 0 74311 1468 0 0 25 0 5 0 517216928 3273338880 719536 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3196620 CPUtime=759.55 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 179346 0 0 0 75912 43 0 0 25 0 5 0 517216928 3273338880 719536 33554432000 4194304 4338100 140734325463872 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3196620 CPUtime=759.91 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 47764 0 0 0 74483 1508 0 0 25 0 5 0 517216928 3273338880 719536 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3038.82
Current children cumulated vsize (KiB) 3196620

[startup+822.301 s]
/proc/loadavg: 8.16 8.10 8.01 9/186 2703
/proc/meminfo: memFree=11901260/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3274820 CPUtime=3278.4 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 745125 0 0 0 324438 3402 0 0 25 0 5 0 517216764 3353415680 741430 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 818705 741430 176 36 0 816610 0
[pid=2663/tid=2669] ppid=2661 vsize=3274820 CPUtime=819.86 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 410829 0 0 0 81813 173 0 0 25 0 5 0 517216928 3353415680 741430 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808560666 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3274820 CPUtime=817.77 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 53089 0 0 0 80211 1566 0 0 25 0 5 0 517216928 3353415680 741430 33554432000 4194304 4338100 140734325463872 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3274820 CPUtime=819.52 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 181422 0 0 0 81909 43 0 0 25 0 5 0 517216928 3353415680 741430 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3274820 CPUtime=819.58 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 49541 0 0 0 80346 1612 0 0 25 0 5 0 517216928 3353415680 741430 33554432000 4194304 4338100 140734325463872 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3278.4
Current children cumulated vsize (KiB) 3274820

[startup+882.301 s]
/proc/loadavg: 8.24 8.12 8.02 11/187 2707
/proc/meminfo: memFree=11519520/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3365532 CPUtime=3518.01 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 771402 0 0 0 348181 3620 0 0 25 0 5 0 517216764 3446304768 767707 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 841383 767707 176 36 0 839288 0
[pid=2663/tid=2669] ppid=2661 vsize=3365532 CPUtime=879.84 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 426179 0 0 0 87802 182 0 0 25 0 5 0 517216928 3446304768 767707 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3365532 CPUtime=877.76 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 54572 0 0 0 86108 1668 0 0 25 0 5 0 517216928 3446304768 767707 33554432000 4194304 4338100 140734325463872 18446744073709551615 4256161 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3365532 CPUtime=879.5 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 189722 0 0 0 87905 45 0 0 25 0 5 0 517216928 3446304768 767707 33554432000 4194304 4338100 140734325463872 18446744073709551615 4286433 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3365532 CPUtime=879.23 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 50685 0 0 0 86207 1716 0 0 25 0 5 0 517216928 3446304768 767707 33554432000 4194304 4338100 140734325463872 18446744073709551615 4256655 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3518.01
Current children cumulated vsize (KiB) 3365532

[startup+942.301 s]
/proc/loadavg: 8.29 8.17 8.04 10/186 2709
/proc/meminfo: memFree=11165912/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3399940 CPUtime=3757.08 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 786824 0 0 0 371863 3845 0 0 25 0 5 0 517216764 3481538560 783129 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 849985 783129 176 36 0 847890 0
[pid=2663/tid=2669] ppid=2661 vsize=3399940 CPUtime=939.84 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 434894 0 0 0 93795 189 0 0 25 0 5 0 517216928 3481538560 783129 33554432000 4194304 4338100 140734325463872 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3399940 CPUtime=937.03 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 58004 0 0 0 91927 1776 0 0 25 0 5 0 517216928 3481538560 783129 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808560687 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3399940 CPUtime=939.49 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 189722 0 0 0 93904 45 0 0 25 0 5 0 517216928 3481538560 783129 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3399940 CPUtime=939.05 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 53960 0 0 0 92078 1827 0 0 25 0 5 0 517216928 3481538560 783129 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808560865 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3757.08
Current children cumulated vsize (KiB) 3399940

[startup+1002.3 s]
/proc/loadavg: 8.15 8.15 8.04 9/186 2710
/proc/meminfo: memFree=10917116/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3465628 CPUtime=3996.71 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 802744 0 0 0 395610 4061 0 0 25 0 5 0 517216764 3548803072 799049 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 866407 799049 176 36 0 864312 0
[pid=2663/tid=2669] ppid=2661 vsize=3465628 CPUtime=999.82 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 448223 0 0 0 99786 196 0 0 25 0 5 0 517216928 3548803072 799049 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3465628 CPUtime=996.7 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 58004 0 0 0 97791 1879 0 0 25 0 5 0 517216928 3548803072 799049 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3465628 CPUtime=999.48 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 192313 0 0 0 99902 46 0 0 25 0 5 0 517216928 3548803072 799049 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3465628 CPUtime=999.04 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 53960 0 0 0 97973 1931 0 0 25 0 5 0 517216928 3548803072 799049 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808560772 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3996.71
Current children cumulated vsize (KiB) 3465628

[startup+1062.3 s]
/proc/loadavg: 7.98 8.10 8.03 10/186 2712
/proc/meminfo: memFree=10483456/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3643924 CPUtime=4236.4 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 846664 0 0 0 419376 4264 0 0 25 0 5 0 517216764 3731378176 842969 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 910981 842969 176 36 0 908886 0
[pid=2663/tid=2669] ppid=2661 vsize=3643924 CPUtime=1059.83 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 475062 0 0 0 105776 207 0 0 25 0 5 0 517216928 3731378176 842969 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3643924 CPUtime=1056.63 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 58004 0 0 0 103690 1973 0 0 25 0 5 0 517216928 3731378176 842969 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808561402 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3643924 CPUtime=1059.23 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 209394 0 0 0 105872 51 0 0 25 0 5 0 517216928 3731378176 842969 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3643924 CPUtime=1059.04 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 53960 0 0 0 103880 2024 0 0 25 0 5 0 517216928 3731378176 842969 33554432000 4194304 4338100 140734325463872 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4236.4
Current children cumulated vsize (KiB) 3643924

[startup+1122.3 s]
/proc/loadavg: 8.04 8.10 8.03 10/186 2714
/proc/meminfo: memFree=10308076/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3762788 CPUtime=4476.06 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 877260 0 0 0 443140 4466 0 0 25 0 5 0 517216764 3853094912 873565 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 940697 873565 176 36 0 938602 0
[pid=2663/tid=2669] ppid=2661 vsize=3762788 CPUtime=1119.8 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 497432 0 0 0 111762 218 0 0 25 0 5 0 517216928 3853094912 873565 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3762788 CPUtime=1116.62 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 58004 0 0 0 109594 2068 0 0 25 0 5 0 517216928 3853094912 873565 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991348 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3762788 CPUtime=1118.94 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 217620 0 0 0 111840 54 0 0 25 0 5 0 517216928 3853094912 873565 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3762788 CPUtime=1119.04 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 53960 0 0 0 109786 2118 0 0 25 0 5 0 517216928 3853094912 873565 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4476.06
Current children cumulated vsize (KiB) 3762788

[startup+1182.3 s]
/proc/loadavg: 8.01 8.08 8.02 9/186 2715
/proc/meminfo: memFree=9982792/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=3881652 CPUtime=4715.69 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) S 2661 2663 1994 0 -1 4202496 906589 0 0 0 466894 4675 0 0 25 0 5 0 517216764 3974811648 902894 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 970413 902894 176 36 0 968318 0
[pid=2663/tid=2669] ppid=2661 vsize=3881652 CPUtime=1179.79 cores=0,2,4,6
/proc/2663/task/2669/stat : 2669 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 526761 0 0 0 117748 231 0 0 25 0 5 0 517216928 3974811648 902894 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2663/tid=2670] ppid=2661 vsize=3881652 CPUtime=1176.59 cores=0,2,4,6
/proc/2663/task/2670/stat : 2670 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 58004 0 0 0 115492 2167 0 0 25 0 5 0 517216928 3974811648 902894 33554432000 4194304 4338100 140734325463872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2663/tid=2671] ppid=2661 vsize=3881652 CPUtime=1178.61 cores=0,2,4,6
/proc/2663/task/2671/stat : 2671 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 217620 0 0 0 117807 54 0 0 25 0 5 0 517216928 3974811648 902894 33554432000 4194304 4338100 140734325463872 18446744073709551615 230808561031 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=3881652 CPUtime=1179.03 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4202560 53960 0 0 0 115688 2215 0 0 25 0 5 0 517216928 3974811648 902894 33554432000 4194304 4338100 140734325463872 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4715.69
Current children cumulated vsize (KiB) 3881652



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.5 s]
/proc/loadavg: 7.45 7.95 7.98 2/177 2717
/proc/meminfo: memFree=16170668/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=0 CPUtime=4800.1 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) Z 2661 2663 1994 0 -1 4203548 934856 0 0 0 475256 4754 0 0 18 0 2 0 517216764 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 0 0 0 0 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=0 CPUtime=1200.2 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4203588 58559 0 0 0 117766 2254 0 0 25 0 2 0 517216928 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.5 s]
/proc/loadavg: 7.45 7.95 7.98 2/177 2717
/proc/meminfo: memFree=16170668/32951124 swapFree=67111528/67111528
[pid=2663] ppid=2661 vsize=0 CPUtime=4800.1 cores=0,2,4,6
/proc/2663/stat : 2663 (ttsth-5-4) Z 2661 2663 1994 0 -1 4203548 934856 0 0 0 475256 4754 0 0 18 0 2 0 517216764 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2663/statm: 0 0 0 0 0 0 0
[pid=2663/tid=2672] ppid=2661 vsize=0 CPUtime=1200.2 cores=0,2,4,6
/proc/2663/task/2672/stat : 2672 (ttsth-5-4) R 2661 2663 1994 0 -1 4203588 58559 0 0 0 117766 2254 0 0 25 0 2 0 517216928 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.59
CPU time (s): 4800.19
CPU user time (s): 4752.57
CPU system time (s): 47.6278
CPU usage (%): 398.824
Max. virtual memory (cumulated for all children) (KiB): 4137916

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4752.57
system time used= 47.6278
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 934856
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6119
involuntary context switches= 24875

runsolver used 1.9867 second user time and 4.28635 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 07:49:23
IDJOB=3259626
IDBENCH=71603
IDSOLVER=1654
FILE ID=node104/3259626-1304056163
RUNJOBID= node104-1304056163-2643
PBS_JOBID= 13170409
Free space on /tmp= 71544 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v2640c305320gyes3.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259626-1304056163/watcher-3259626-1304056163 -o /tmp/evaluation-result-3259626-1304056163/solver-3259626-1304056163 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259626-1304056163.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3b033385d50481e5bd2fe507f2941f4f
RANDOM SEED=1155975116

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18286240 kB
Buffers:       2196056 kB
Cached:       11802856 kB
SwapCached:          0 kB
Active:        2362184 kB
Inactive:     11716908 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18286240 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            5632 kB
Writeback:           0 kB
AnonPages:       79188 kB
Mapped:          14872 kB
Slab:           522996 kB
PageTables:       4844 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   223824 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71544 MiB
End job on node104 at 2011-04-29 08:09:29