Trace number 3263366

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.44 1212.11

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_60_28_40_7-log.shuffled-as.sat03-127.cnf
MD5SUMf1f35d90237c4ca07d41a6a9b08e6279
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables140
Number of clauses58376
Sum of the clauses size583480
Maximum clause length10
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 556
Number of clauses of size over 558320

Solver Data 

0.09/0.14	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263366-1304096259.cnf 
0.09/0.14	c Using SATTIMEOUT of 4800
0.09/0.14	c Using SATRAM of 15500
0.09/0.14	c Size of subset node = 32
0.09/0.14	c Variables = 140
0.09/0.14	c Clauses = 58376
0.09/0.14	c Literals = 583480
0.09/0.14	c Reduced size:
0.09/0.14	c Variables = 140
0.09/0.14	c Clauses = 58376
0.09/0.14	c Literals = 583480
0.09/0.14	c a2
0.99/1.03	c a1
2.89/2.99	c Independent sub-problems = 1
2.89/2.99	c Sub-problem with 140 variables and 58376 clauses
2.89/2.99	c using permutation method 4
2.89/2.99	c count = 1
2.89/2.99	c Ordering finished 2.950 seconds
2.89/2.99	c mapped 2.980 seconds
2.89/2.99	c permuted 2.980 seconds
2.89/2.99	c about to sort
2.99/3.01	c sorted
3.09/3.15	c clauses now 58376
3.09/3.15	c init finished 3.020 seconds
3.09/3.15	c tree built 3.140 seconds
3.09/3.15	c 
3.09/3.15	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
3.09/3.15	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
3.09/3.15	c ................................ | ..........                    |                                  3.1         0.0             
3.09/3.15	c ................................ | ..........                  1 |                                  3.1         0.3             
3.09/3.15	c ................................ | ..........                  2 |                                  3.1         0.6             
3.09/3.15	c ................................ | ..........                  3 |                                  3.1         1.0             
3.09/3.15	c ................................ | ..........                  4 |              1                   3.1         1.3             
3.09/3.15	c ................................ | ..........                  5 |              5                   3.1         1.6            1
3.09/3.15	c ................................ | ..........               1000 | 140          1990                3.1       317.5        12283
59.08/17.28	c *............................... | ..........            1281947 | 328252       1642104            59.4     21578.0     15821771
95.75/26.47	c *............................... | ..........            1866000 | 478050       2409136            96.0     19437.5     21194008
191.72/50.57	c *............................... | ..........            3065000 | 792557       4098131           192.0     15962.7     34655652
288.05/74.71	c *............................... | ..........            4143000 | 1074410      5521233           288.1     14382.4     43193060
383.76/98.86	c *............................... | ..........            5148000 | 1338837      6861453           384.0     13405.2     48301164
479.72/122.97	c *............................... | *.........            6063000 | 1578697      8106010           480.0     12630.2     52919119
575.99/147.10	c *............................... | *.........            6934000 | 1808073      9289861           576.0     12038.2     59403336
671.79/171.25	c *............................... | *.........            7749000 | 2025108      10422082          672.0     11531.1     64578839
767.77/195.48	c *............................... | *.........            8558000 | 2237158      11491766          768.1     11141.8     66811230
863.77/219.56	c *............................... | *.........            9343000 | 2442299      12540429          864.0     10813.3     70612440
960.00/243.70	c *............................... | **........           10105000 | 2643212      13573889          960.0     10525.9     75014291
1055.77/267.86	c *............................... | **........           10865000 | 2840232      14584132         1056.0     10288.5     76994038
1151.98/292.03	c *............................... | **........           11597000 | 3030077      15612020         1152.1     10066.0     77446725
1248.00/316.21	c *............................... | **........           12373000 | 3233709      16626651         1248.1      9913.8     81215516
1344.03/340.32	c *............................... | **........           13108000 | 3424495      17629784         1344.1      9752.0     84474510
1439.86/364.56	c *............................... | ***.......           13789000 | 3601816      18564693         1440.1      9575.0     87791007
1535.86/388.76	c *............................... | ***.......           14455000 | 3776080      19472076         1536.1      9410.1     92175958
1631.94/412.94	c *............................... | ***.......           15102000 | 3946063      20363025         1632.1      9253.1     93834592
1727.81/437.06	c *............................... | ***.......           15810000 | 4129931      21289331         1728.1      9148.9     96028940
1823.97/461.22	c *............................... | ***.......           16468000 | 4302132      22191142         1824.0      9028.3     97787767
1920.02/485.43	c *............................... | ****......           17142000 | 4478844      23090940         1920.1      8927.5     99996160
2015.79/509.58	c *............................... | ****......           17765000 | 4643689      23971325         2016.1      8811.5    104265885
2111.99/533.83	c *............................... | ****......           18395000 | 4810043      24822106         2112.1      8709.3    104853538
2207.76/558.08	c *............................... | ****......           19052000 | 4982370      25714042         2208.1      8628.3    105732802
2304.00/582.21	c *............................... | ****......           19715000 | 5154144      26602205         2304.1      8556.7    107356293
2399.88/606.46	c *............................... | *****.....           20334000 | 5316428      27465407         2400.1      8472.0    107459966
2495.83/630.64	c *............................... | *****.....           20984000 | 5486661      28343878         2496.0      8407.0    107600781
2592.04/654.81	c *............................... | *****.....           21608000 | 5650444      29206187         2592.1      8336.1    108104672
2688.01/679.00	c *............................... | *****.....           22225000 | 5813130      30044734         2688.0      8268.1    108866287
2784.01/703.20	c *............................... | *****.....           22888000 | 5984791      30911626         2784.1      8221.1    109601314
2879.78/727.36	c *............................... | ******....           23564000 | 6161878      31770318         2880.1      8181.8    111015896
2976.08/751.70	c *............................... | ******....           24208000 | 6331131      32625355         2976.1      8134.1    112302259
3071.79/775.98	c *............................... | ******....           24851000 | 6501160      33478285         3072.1      8089.1    113417048
3167.96/800.21	c *............................... | ******....           25475000 | 6665991      34304288         3168.0      8041.4    113705997
3263.79/824.48	c *............................... | ******....           26107000 | 6828954      35146008         3264.1      7998.2    114786544
3359.95/848.62	c *............................... | *******...           26710000 | 6985459      35951006         3360.1      7949.3    115813805
3456.21/872.94	c *............................... | *******...           27310000 | 7141679      36761462         3456.4      7901.3    116646271
3551.77/897.05	c *............................... | *******...           27943000 | 7308237      37586134         3552.0      7866.8    117376285
3647.90/921.26	c *............................... | *******...           28600000 | 7480425      38427058         3648.2      7839.6    118107384
3743.87/945.43	c *............................... | *******...           29221000 | 7645231      39255970         3744.0      7804.7    120560982
3839.95/969.74	c *............................... | ********..           29825000 | 7805166      40065908         3840.1      7766.7    121513340
3936.02/993.90	c *............................... | ********..           30438000 | 7967022      40848567         3936.0      7733.2    122444786
4031.69/1018.09	c *............................... | ********..           31058000 | 8131657      41647669         4032.1      7702.8    123000749
4127.79/1042.25	c *............................... | ********..           31660000 | 8292463      42429677         4128.0      7669.6    123327121
4224.08/1066.50	c *............................... | ********..           32234000 | 8441590      43192748         4224.1      7631.0    123687542
4319.99/1090.70	c *............................... | *********.           32817000 | 8593897      43978350         4320.0      7596.5    123687542
4415.84/1114.88	c *............................... | *********.           33390000 | 8744793      44771697         4416.2      7560.8    123928915
4511.93/1139.04	c *............................... | *********.           33970000 | 8897756      45575264         4512.1      7528.6    124705470
4607.74/1163.27	c *............................... | *********.           34518000 | 9042208      46343898         4608.0      7490.8    126509358
4703.94/1187.55	c *............................... | *********.           35103000 | 9196718      47126374         4704.2      7462.1    126577778
4799.99/1211.84	c CPU limit (4800) reached
4799.99/1211.84	c Time = 4800.16 seconds
4799.99/1211.84	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263366-1304096259/watcher-3263366-1304096259 -o /tmp/evaluation-result-3263366-1304096259/solver-3263366-1304096259 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263366-1304096259.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.43 4.96 6.57 3/176 21597
/proc/meminfo: memFree=30877360/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=9104 CPUtime=0 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 341 0 0 0 0 0 0 0 25 0 1 0 599856489 9322496 283 33554432000 4194304 4338100 140733238159056 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21597/statm: 2276 284 147 36 0 181 0

[startup+0.028967 s]
/proc/loadavg: 3.43 4.96 6.57 3/176 21597
/proc/meminfo: memFree=30877360/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=10700 CPUtime=0.02 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 724 0 0 0 2 0 0 0 25 0 1 0 599856489 10956800 666 33554432000 4194304 4338100 140733238159056 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21597/statm: 2675 667 147 36 0 580 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 10700

[startup+0.100984 s]
/proc/loadavg: 3.43 4.96 6.57 3/176 21597
/proc/meminfo: memFree=30877360/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=15792 CPUtime=0.09 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 1986 0 0 0 9 0 0 0 25 0 1 0 599856489 16171008 1928 33554432000 4194304 4338100 140733238159056 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21597/statm: 3948 1929 148 36 0 1853 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 15792

[startup+0.300944 s]
/proc/loadavg: 3.43 4.96 6.57 3/176 21597
/proc/meminfo: memFree=30877360/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=88752 CPUtime=0.29 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 11290 0 0 0 28 1 0 0 25 0 1 0 599856489 90882048 10656 33554432000 4194304 4338100 140733238159056 18446744073709551615 260019791013 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21597/statm: 22188 10656 314 36 0 20093 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 88752

[startup+0.700866 s]
/proc/loadavg: 3.43 4.96 6.57 3/176 21597
/proc/meminfo: memFree=30877360/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=88752 CPUtime=0.69 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 11295 0 0 0 68 1 0 0 25 0 1 0 599856489 90882048 10656 33554432000 4194304 4338100 140733238159056 18446744073709551615 260019791042 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21597/statm: 22188 10656 309 36 0 20093 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 88752

[startup+1.50073 s]
/proc/loadavg: 3.31 4.91 6.54 4/178 21599
/proc/meminfo: memFree=30822924/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=23084 CPUtime=1.49 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 11745 0 0 0 147 2 0 0 25 0 1 0 599856489 23638016 3760 33554432000 4194304 4338100 140733238159056 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21597/statm: 5771 3760 166 36 0 3676 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 23084

[startup+3.10038 s]
/proc/loadavg: 3.31 4.91 6.54 7/182 21603
/proc/meminfo: memFree=30833060/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=29744 CPUtime=3.09 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) R 21593 21597 21560 0 -1 4202496 13532 0 0 0 307 2 0 0 25 0 1 0 599856489 30457856 5421 33554432000 4194304 4338100 140733238159056 18446744073709551615 4278048 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21597/statm: 7436 5421 164 36 0 5341 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 29744

[startup+6.30075 s]
/proc/loadavg: 3.31 4.91 6.54 10/186 21607
/proc/meminfo: memFree=30464152/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=393920 CPUtime=15.68 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 62648 0 0 0 1554 14 0 0 25 0 5 0 599856489 403374080 54537 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 98480 54538 176 36 0 96385 0
[pid=21597/tid=21604] ppid=21593 vsize=393920 CPUtime=3.13 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 12530 0 0 0 310 3 0 0 25 0 5 0 599856805 403374080 54538 33554432000 4194304 4338100 140733238159056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=393920 CPUtime=3.13 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 12049 0 0 0 311 2 0 0 25 0 5 0 599856805 403374080 54538 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=393920 CPUtime=3.13 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 11837 0 0 0 310 3 0 0 18 0 5 0 599856805 403374080 54539 33554432000 4194304 4338100 140733238159056 18446744073709551615 260020057121 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=393920 CPUtime=3.11 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 12693 0 0 0 309 2 0 0 25 0 5 0 599856805 403374080 54539 33554432000 4194304 4338100 140733238159056 18446744073709551615 4284198 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 15.68
Current children cumulated vsize (KiB) 393920

[startup+12.7006 s]
/proc/loadavg: 4.03 5.01 6.56 10/186 21607
/proc/meminfo: memFree=29932376/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=602608 CPUtime=41.17 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 118377 0 0 0 4090 27 0 0 25 0 5 0 599856489 617070592 110266 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 150652 110267 176 36 0 148557 0
[pid=21597/tid=21604] ppid=21593 vsize=602608 CPUtime=9.51 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 26480 0 0 0 944 7 0 0 25 0 5 0 599856805 617070592 110267 33554432000 4194304 4338100 140733238159056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=602608 CPUtime=9.51 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 25479 0 0 0 947 4 0 0 25 0 5 0 599856805 617070592 110267 33554432000 4194304 4338100 140733238159056 18446744073709551615 4263517 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=602608 CPUtime=9.51 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 26034 0 0 0 945 6 0 0 25 0 5 0 599856805 617070592 110267 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=602608 CPUtime=9.45 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 26844 0 0 0 940 5 0 0 25 0 5 0 599856805 617070592 110267 33554432000 4194304 4338100 140733238159056 18446744073709551615 260019791598 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 41.17
Current children cumulated vsize (KiB) 602608

[startup+25.5012 s]
/proc/loadavg: 4.64 5.11 6.57 10/186 21608
/proc/meminfo: memFree=29269300/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=888016 CPUtime=92.14 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 194526 0 0 0 9167 47 0 0 25 0 5 0 599856489 909328384 186415 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 222004 186415 176 36 0 219909 0
[pid=21597/tid=21604] ppid=21593 vsize=888016 CPUtime=22.19 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 50432 0 0 0 2206 13 0 0 25 0 5 0 599856805 909328384 186416 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=888016 CPUtime=22.29 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 44605 0 0 0 2220 9 0 0 25 0 5 0 599856805 909328384 186416 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=888016 CPUtime=22.27 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 42010 0 0 0 2217 10 0 0 25 0 5 0 599856805 909328384 186416 33554432000 4194304 4338100 140733238159056 18446744073709551615 4239504 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=888016 CPUtime=22.19 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 43939 0 0 0 2210 9 0 0 25 0 5 0 599856805 909328384 186416 33554432000 4194304 4338100 140733238159056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 92.14
Current children cumulated vsize (KiB) 888016

[startup+51.1005 s]
/proc/loadavg: 5.85 5.36 6.61 10/186 21608
/proc/meminfo: memFree=28316460/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=1417844 CPUtime=194.1 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 320132 0 0 0 19333 77 0 0 25 0 5 0 599856489 1451872256 312021 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 354461 312022 176 36 0 352366 0
[pid=21597/tid=21604] ppid=21593 vsize=1417844 CPUtime=47.57 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 81170 0 0 0 4737 20 0 0 25 0 5 0 599856805 1451872256 312022 33554432000 4194304 4338100 140733238159056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=1417844 CPUtime=47.83 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 75649 0 0 0 4766 17 0 0 25 0 5 0 599856805 1451872256 312022 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=1417844 CPUtime=47.79 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 69825 0 0 0 4764 15 0 0 25 0 5 0 599856805 1451872256 312023 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=1417844 CPUtime=47.72 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 79948 0 0 0 4752 20 0 0 25 0 5 0 599856805 1451872256 312023 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 194.1
Current children cumulated vsize (KiB) 1417844

[startup+102.307 s]
/proc/loadavg: 7.26 5.83 6.70 10/186 21610
/proc/meminfo: memFree=27162496/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=1890912 CPUtime=397.74 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 454953 0 0 0 39649 125 0 0 25 0 5 0 599856489 1936293888 446842 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 472728 446842 176 36 0 470633 0
[pid=21597/tid=21604] ppid=21593 vsize=1890912 CPUtime=98.3 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 125107 0 0 0 9791 39 0 0 25 0 5 0 599856805 1936293888 446842 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=1890912 CPUtime=98.86 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 104383 0 0 0 9859 27 0 0 25 0 5 0 599856805 1936293888 446842 33554432000 4194304 4338100 140733238159056 18446744073709551615 260019791290 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=1890912 CPUtime=98.79 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 101254 0 0 0 9854 25 0 0 25 0 5 0 599856805 1936293888 446843 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=1890912 CPUtime=98.6 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 110668 0 0 0 9831 29 0 0 25 0 5 0 599856805 1936293888 446843 33554432000 4194304 4338100 140733238159056 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 397.74
Current children cumulated vsize (KiB) 1890912

[startup+162.301 s]
/proc/loadavg: 7.80 6.24 6.79 10/186 21611
/proc/meminfo: memFree=26295808/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=2466268 CPUtime=636.35 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 582999 0 0 0 63461 174 0 0 25 0 5 0 599856489 2525458432 574888 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 616567 574888 176 36 0 614472 0
[pid=21597/tid=21604] ppid=21593 vsize=2466268 CPUtime=157.83 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 166970 0 0 0 15725 58 0 0 25 0 5 0 599856805 2525458432 574888 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=2466268 CPUtime=158.57 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 134404 0 0 0 15818 39 0 0 25 0 5 0 599856805 2525458432 574888 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=2466268 CPUtime=158.55 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 123295 0 0 0 15822 33 0 0 25 0 5 0 599856805 2525458432 574888 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=2466268 CPUtime=158.21 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 144789 0 0 0 15782 39 0 0 25 0 5 0 599856805 2525458432 574888 33554432000 4194304 4338100 140733238159056 18446744073709551615 260019790869 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 636.35
Current children cumulated vsize (KiB) 2466268

[startup+222.301 s]
/proc/loadavg: 7.92 6.56 6.86 10/186 21622
/proc/meminfo: memFree=25460980/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=2829980 CPUtime=874.89 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 671169 0 0 0 87276 213 0 0 25 0 5 0 599856489 2897899520 663058 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 707495 663058 176 36 0 705400 0
[pid=21597/tid=21604] ppid=21593 vsize=2829980 CPUtime=217.37 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 200585 0 0 0 21664 73 0 0 25 0 5 0 599856805 2897899520 663058 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=2829980 CPUtime=218.35 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 149047 0 0 0 21789 46 0 0 25 0 5 0 599856805 2897899520 663058 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=2829980 CPUtime=218.26 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 141814 0 0 0 21786 40 0 0 25 0 5 0 599856805 2897899520 663058 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=2829980 CPUtime=217.71 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 166182 0 0 0 21722 49 0 0 25 0 5 0 599856805 2897899520 663058 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 874.89
Current children cumulated vsize (KiB) 2829980

################
# More data... #
################

/proc/21597/statm: 1182469 1155519 176 36 0 1180374 0
[pid=21597/tid=21604] ppid=21593 vsize=4729876 CPUtime=751.2 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 424304 0 0 0 74934 186 0 0 25 0 5 0 599856805 4843393024 1155519 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254332 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=4729876 CPUtime=754.97 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 236104 0 0 0 75401 96 0 0 25 0 5 0 599856805 4843393024 1155519 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=4729876 CPUtime=755.2 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 217496 0 0 0 75428 92 0 0 25 0 5 0 599856805 4843393024 1155519 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=4729876 CPUtime=753.48 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 272185 0 0 0 75246 102 0 0 25 0 5 0 599856805 4843393024 1155519 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3018.05
Current children cumulated vsize (KiB) 4729876

[startup+822.3 s]
/proc/loadavg: 8.06 7.84 7.41 10/186 21638
/proc/meminfo: memFree=21287900/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=4939480 CPUtime=3255.44 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1195409 0 0 0 325037 507 0 0 25 0 5 0 599856489 5058027520 1187298 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1234870 1187298 176 36 0 1232775 0
[pid=21597/tid=21604] ppid=21593 vsize=4939480 CPUtime=810.42 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 442731 0 0 0 80845 197 0 0 25 0 5 0 599856805 5058027520 1187298 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=4939480 CPUtime=814.29 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 240166 0 0 0 81327 102 0 0 25 0 5 0 599856805 5058027520 1187298 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=4939480 CPUtime=814.65 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 221288 0 0 0 81368 97 0 0 25 0 5 0 599856805 5058027520 1187298 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=4939480 CPUtime=812.91 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 277683 0 0 0 81184 107 0 0 25 0 5 0 599856805 5058027520 1187298 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3255.44
Current children cumulated vsize (KiB) 4939480

[startup+882.3 s]
/proc/loadavg: 7.95 7.85 7.44 10/186 21640
/proc/meminfo: memFree=20952928/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=5123188 CPUtime=3493.54 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1229628 0 0 0 348818 536 0 0 25 0 5 0 599856489 5246144512 1221517 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1280797 1221517 176 36 0 1278702 0
[pid=21597/tid=21604] ppid=21593 vsize=5123188 CPUtime=869.84 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 456839 0 0 0 86778 206 0 0 25 0 5 0 599856805 5246144512 1221517 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=5123188 CPUtime=873.79 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 244977 0 0 0 87270 109 0 0 25 0 5 0 599856805 5246144512 1221517 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=5123188 CPUtime=874.16 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 227037 0 0 0 87312 104 0 0 25 0 5 0 599856805 5246144512 1221517 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=5123188 CPUtime=872.57 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 287234 0 0 0 87145 112 0 0 25 0 5 0 599856805 5246144512 1221517 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3493.54
Current children cumulated vsize (KiB) 5123188

[startup+942.3 s]
/proc/loadavg: 7.98 7.88 7.47 10/186 21641
/proc/meminfo: memFree=20679464/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=5250844 CPUtime=3731.7 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1272842 0 0 0 372606 564 0 0 25 0 5 0 599856489 5376864256 1264731 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1312711 1264731 176 36 0 1310616 0
[pid=21597/tid=21604] ppid=21593 vsize=5250844 CPUtime=929.35 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 484261 0 0 0 92714 221 0 0 25 0 5 0 599856805 5376864256 1264731 33554432000 4194304 4338100 140733238159056 18446744073709551615 4258054 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=5250844 CPUtime=933.07 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 248992 0 0 0 93193 114 0 0 25 0 5 0 599856805 5376864256 1264731 33554432000 4194304 4338100 140733238159056 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=5250844 CPUtime=933.93 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 230906 0 0 0 93284 109 0 0 25 0 5 0 599856805 5376864256 1264731 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=5250844 CPUtime=932.17 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 295142 0 0 0 93102 115 0 0 25 0 5 0 599856805 5376864256 1264731 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3731.7
Current children cumulated vsize (KiB) 5250844

[startup+1002.3 s]
/proc/loadavg: 8.15 7.94 7.51 10/186 21643
/proc/meminfo: memFree=20414184/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=5328452 CPUtime=3969.25 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1304784 0 0 0 396336 589 0 0 25 0 5 0 599856489 5456334848 1296673 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1332113 1296673 176 36 0 1330018 0
[pid=21597/tid=21604] ppid=21593 vsize=5328452 CPUtime=988.46 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 503456 0 0 0 98611 235 0 0 25 0 5 0 599856805 5456334848 1296673 33554432000 4194304 4338100 140733238159056 18446744073709551615 260020053809 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=5328452 CPUtime=992.42 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) S 21593 21597 21560 0 -1 4202560 252777 0 0 0 99125 117 0 0 25 0 5 0 599856805 5456334848 1296673 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032208646 0 0 4096 0 18446604452551891904 0 0 -1 0 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=5328452 CPUtime=993.48 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 234701 0 0 0 99234 114 0 0 25 0 5 0 599856805 5456334848 1296673 33554432000 4194304 4338100 140733238159056 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=5328452 CPUtime=991.69 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) S 21593 21597 21560 0 -1 4202560 300309 0 0 0 99051 118 0 0 19 0 5 0 599856805 5456334848 1296673 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032208646 0 0 4096 0 18446604452551891904 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3969.25
Current children cumulated vsize (KiB) 5328452

[startup+1062.3 s]
/proc/loadavg: 8.17 7.98 7.55 10/186 21644
/proc/meminfo: memFree=20150528/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=5515236 CPUtime=4207.41 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1329668 0 0 0 420131 610 0 0 25 0 5 0 599856489 5647601664 1321557 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1378809 1321557 176 36 0 1376714 0
[pid=21597/tid=21604] ppid=21593 vsize=5515236 CPUtime=1047.81 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 517025 0 0 0 104536 245 0 0 25 0 5 0 599856805 5647601664 1321557 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=5515236 CPUtime=1052.22 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 256521 0 0 0 105102 120 0 0 25 0 5 0 599856805 5647601664 1321557 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=5515236 CPUtime=1053.07 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 238615 0 0 0 105188 119 0 0 25 0 5 0 599856805 5647601664 1321557 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=5515236 CPUtime=1051.14 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 303966 0 0 0 104991 123 0 0 25 0 5 0 599856805 5647601664 1321557 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4207.41
Current children cumulated vsize (KiB) 5515236

[startup+1122.3 s]
/proc/loadavg: 8.23 8.03 7.59 10/186 21646
/proc/meminfo: memFree=19901492/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=5536408 CPUtime=4445.62 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1349170 0 0 0 443930 632 0 0 25 0 5 0 599856489 5669281792 1341059 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1384102 1341059 176 36 0 1382007 0
[pid=21597/tid=21604] ppid=21593 vsize=5536408 CPUtime=1107.18 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 523142 0 0 0 110465 253 0 0 25 0 5 0 599856805 5669281792 1341059 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=5536408 CPUtime=1111.89 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 260293 0 0 0 111064 125 0 0 25 0 5 0 599856805 5669281792 1341059 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=5536408 CPUtime=1112.6 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 244491 0 0 0 111136 124 0 0 25 0 5 0 599856805 5669281792 1341059 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=5536408 CPUtime=1110.76 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 307703 0 0 0 110951 125 0 0 25 0 5 0 599856805 5669281792 1341059 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4445.62
Current children cumulated vsize (KiB) 5536408

[startup+1182.3 s]
/proc/loadavg: 8.11 8.03 7.62 10/186 21648
/proc/meminfo: memFree=19623684/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=5701316 CPUtime=4683.36 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) S 21593 21597 21560 0 -1 4202496 1384601 0 0 0 467680 656 0 0 25 0 5 0 599856489 5838147584 1376490 33554432000 4194304 4338100 140733238159056 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 1425329 1376490 176 36 0 1423234 0
[pid=21597/tid=21604] ppid=21593 vsize=5701316 CPUtime=1166.45 cores=0,2,4,6
/proc/21597/task/21604/stat : 21604 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 541437 0 0 0 116381 264 0 0 25 0 5 0 599856805 5838147584 1376490 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21597/tid=21605] ppid=21593 vsize=5701316 CPUtime=1171.1 cores=0,2,4,6
/proc/21597/task/21605/stat : 21605 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 263973 0 0 0 116980 130 0 0 25 0 5 0 599856805 5838147584 1376490 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21597/tid=21606] ppid=21593 vsize=5701316 CPUtime=1172.32 cores=0,2,4,6
/proc/21597/task/21606/stat : 21606 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 248600 0 0 0 117103 129 0 0 25 0 5 0 599856805 5838147584 1376490 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=5701316 CPUtime=1170.31 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4202560 317050 0 0 0 116902 129 0 0 25 0 5 0 599856805 5838147584 1376490 33554432000 4194304 4338100 140733238159056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4683.36
Current children cumulated vsize (KiB) 5701316



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1211.9 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21649
/proc/meminfo: memFree=26434168/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=0 CPUtime=4800.23 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) Z 21593 21597 21560 0 -1 4203548 1393336 0 0 0 479351 672 0 0 18 0 2 0 599856489 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 0 0 0 0 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=0 CPUtime=1199.74 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4203588 318811 0 0 0 119838 136 0 0 25 0 2 0 599856805 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1212.01 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21649
/proc/meminfo: memFree=26434168/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=0 CPUtime=4800.33 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) Z 21593 21597 21560 0 -1 4203548 1393336 0 0 0 479351 682 0 0 18 0 2 0 599856489 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 0 0 0 0 0 0 0
[pid=21597/tid=21607] ppid=21593 vsize=0 CPUtime=1199.85 cores=0,2,4,6
/proc/21597/task/21607/stat : 21607 (ttsth-5-4) R 21593 21597 21560 0 -1 4203588 318811 0 0 0 119838 147 0 0 25 0 2 0 599856805 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.33
Current children cumulated vsize (KiB) 0

[startup+1212.1 s]
/proc/loadavg: 8.06 8.03 7.63 2/178 21650
/proc/meminfo: memFree=30811740/32951124 swapFree=67111464/67111528
[pid=21597] ppid=21593 vsize=0 CPUtime=4800.42 cores=0,2,4,6
/proc/21597/stat : 21597 (ttsth-5-4) Z 21593 21597 21560 0 -1 4203548 1393336 0 0 0 479351 691 0 0 18 0 1 0 599856489 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21597/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.42
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1212.11
CPU time (s): 4800.44
CPU user time (s): 4793.52
CPU system time (s): 6.91995
CPU usage (%): 396.041
Max. virtual memory (cumulated for all children) (KiB): 5714700

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4793.52
system time used= 6.91995
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1393336
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 30742
involuntary context switches= 29529

runsolver used 1.78773 second user time and 4.83127 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 18:57:39
IDJOB=3263366
IDBENCH=83965
IDSOLVER=1654
FILE ID=node110/3263366-1304096259
RUNJOBID= node110-1304096259-21578
PBS_JOBID= 13170606
Free space on /tmp= 72836 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_60_28_40_7-log.shuffled-as.sat03-127.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263366-1304096259/watcher-3263366-1304096259 -o /tmp/evaluation-result-3263366-1304096259/solver-3263366-1304096259 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263366-1304096259.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f1f35d90237c4ca07d41a6a9b08e6279
RANDOM SEED=1350559274

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30877928 kB
Buffers:        902960 kB
Cached:         698164 kB
SwapCached:         64 kB
Active:         122176 kB
Inactive:      1543076 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30877928 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:            4028 kB
Writeback:           0 kB
AnonPages:       63696 kB
Mapped:          14584 kB
Slab:           341824 kB
PageTables:       4488 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   186868 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72836 MiB
End job on node110 at 2011-04-29 19:17:53