Trace number 3249812

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1203.09

General information on the benchmark

Namecrafted/skvortsov/
automata-synchronization/crn_20_360_u.cnf
MD5SUM4b70d66c676eba3660e2a5eff473f8e7
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark178.103
Satisfiable
(Un)Satisfiability was proved
Number of variables7920
Number of clauses14950
Sum of the clauses size44260
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2590
Number of clauses of size 314360
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249812-1304072801.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 7920
0.00/0.01	c Clauses = 14950
0.00/0.01	c Literals = 44260
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 7920
0.00/0.01	c Clauses = 14950
0.00/0.01	c Literals = 44260
0.00/0.01	c a2
0.00/0.02	c a1
0.09/0.12	c Independent sub-problems = 1
0.09/0.12	c Sub-problem with 7920 variables and 14950 clauses
0.09/0.12	c using permutation method 4
0.09/0.12	c count = 2
0.09/0.12	c Ordering finished 0.110 seconds
0.09/0.12	c mapped 0.110 seconds
0.09/0.12	c permuted 0.110 seconds
0.09/0.12	c about to sort
0.09/0.12	c sorted
0.19/0.22	c clauses now 14950
0.19/0.22	c init finished 0.180 seconds
0.19/0.22	c tree built 0.220 seconds
0.19/0.22	c 
0.19/0.22	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.22	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.22	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.22	c ................................ | ..........                  1 |                                  0.2         4.5             
0.19/0.22	c ................................ | ..........                  2 |                                  0.2         9.1             
0.19/0.22	c ................................ | ..........                  3 |                                  0.2        13.6             
0.19/0.22	c ................................ | ..........                  4 |                                  0.2        18.2             
0.19/0.22	c ................................ | ..........                  5 |                                  0.2        22.7             
0.19/0.22	c ................................ | ..........               1000 |              42                  0.2      4545.5             
95.94/24.22	c ................................ | ..........            2205000 | 69608        158641             96.0     22964.0     14199606
191.77/48.28	c ................................ | ..........            3888000 | 121833       267168            192.1     20240.5     20499385
288.00/72.30	c ................................ | ..........            5491000 | 172793       368313            288.0     19064.0     26914794
383.80/96.35	c ................................ | ..........            7174000 | 226254       472621            384.0     18682.3     32312759
479.99/120.41	c ................................ | *.........            9000000 | 284890       581587            480.0     18748.4     37400450
575.76/144.47	c ................................ | *.........           10871000 | 344250       691339            576.0     18871.6     42372541
671.98/168.50	c ................................ | *.........           12771000 | 404548       799530            672.0     19004.5     46874379
767.74/192.58	c ................................ | *.........           14646000 | 464432       911419            768.1     19068.6     52765473
863.95/216.63	c ................................ | *.........           16093000 | 512767       1008861           864.1     18624.2     59349109
959.81/240.64	c ................................ | **........           16646000 | 529934       1060404           960.0     17339.6     63777399
1055.68/264.69	c ................................ | **........           17270000 | 548935       1132986          1056.0     16353.7     67176504
1151.89/288.75	c ................................ | **........           18019000 | 571734       1206308          1152.1     15640.4     70543210
1247.69/312.78	c ................................ | **........           18888000 | 598251       1254591          1248.0     15134.4     72953180
1343.95/336.82	c ................................ | **........           19703000 | 623448       1323847          1344.1     14659.3     77224968
1439.71/360.88	c ................................ | ***.......           20644000 | 652600       1405975          1440.0     14335.9     82020970
1535.92/384.92	c ................................ | ***.......           21597000 | 682091       1475037          1536.0     14060.4     85819802
1632.12/409.00	c ................................ | ***.......           22573000 | 712695       1523736          1632.1     13830.4     87999251
1727.89/433.04	c ................................ | ***.......           23561000 | 743617       1591911          1728.1     13634.4     91439978
1823.71/457.08	c ................................ | ***.......           24472000 | 772428       1667677          1824.0     13416.5     93496744
1919.93/481.13	c ................................ | ****......           25345000 | 799914       1745991          1920.0     13200.2     96676961
2016.11/505.20	c ................................ | ****......           26300000 | 830251       1789341          2016.1     13044.8     99577557
2111.90/529.23	c ................................ | ****......           27242000 | 860081       1846997          2112.1     12898.4    102045609
2207.70/553.28	c ................................ | ****......           28124000 | 887915       1917469          2208.1     12737.0    104945807
2303.92/577.33	c ................................ | ****......           28993000 | 915016       1983827          2304.1     12583.4    107961747
2399.74/601.37	c ................................ | *****.....           30034000 | 945922       2052818          2400.0     12514.0    110977290
2495.98/625.42	c ................................ | *****.....           30888000 | 971829       2106076          2496.0     12374.8    113730275
2591.68/649.48	c ................................ | *****.....           31946000 | 1004701      2175360          2592.1     12324.6    117297355
2687.89/673.52	c ................................ | *****.....           33083000 | 1040695      2244630          2688.0     12307.7    119411524
2783.68/697.59	c ................................ | *****.....           33988000 | 1069064      2316395          2784.1     12208.1    123170542
2879.85/721.64	c ................................ | ******....           35139000 | 1105030      2368572          2880.0     12201.0    125265636
2976.00/745.70	c ................................ | ******....           36316000 | 1141634      2436478          2976.0     12202.8    127968959
3071.74/769.77	c ................................ | ******....           37262000 | 1171214      2503586          3072.0     12129.4    131166516
3167.87/793.83	c ................................ | ******....           38446000 | 1208495      2576525          3168.0     12135.7    131947337
3263.68/817.89	c ................................ | ******....           39596000 | 1244652      2641368          3264.0     12131.1    132291331
3359.85/841.94	c ................................ | *******...           40609000 | 1276727      2704029          3360.0     12086.0    133339016
3455.95/866.02	c ................................ | *******...           41856000 | 1316429      2774264          3456.0     12111.0    134521081
3551.75/890.08	c ................................ | *******...           43200000 | 1359282      2855518          3552.1     12161.9    134544320
3647.95/914.16	c ................................ | *******...           44234000 | 1392587      2912375          3648.2     12124.9    135399875
3744.05/938.20	c ................................ | *******...           45437000 | 1430429      2977984          3744.1     12135.8    135977515
3839.84/962.25	c ................................ | ********..           46778000 | 1473542      3052006          3840.0     12181.7    135977515
3936.01/986.33	c ................................ | ********..           48123000 | 1516686      3127525          3936.1     12226.0    135977515
4031.70/1010.38	c ................................ | ********..           49144000 | 1549509      3185026          4032.0     12188.4    137118235
4127.78/1034.46	c ................................ | ********..           50455000 | 1591059      3257741          4128.0     12222.5    138178535
4223.89/1058.54	c ................................ | ********..           51923000 | 1638282      3335324          4224.1     12292.2    140085131
4319.68/1082.59	c ................................ | *********.           52997000 | 1673282      3401105          4320.1     12267.7    141304662
4415.89/1106.63	c ................................ | *********.           54197000 | 1711174      3467148          4416.0     12272.9    142694727
4511.65/1130.69	c ................................ | *********.           55654000 | 1758046      3541395          4512.0     12334.6    143431133
4607.68/1154.79	c ................................ | *********.           57192000 | 1807117      3623974          4608.0     12411.4    146138649
4703.89/1178.82	c ................................ | *********.           58266000 | 1841541      3681322          4704.0     12386.5    147867340
4799.63/1202.90	c CPU limit (4800) reached
4799.63/1202.90	c Time = 4800.03 seconds
4799.63/1202.90	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249812-1304072801/watcher-3249812-1304072801 -o /tmp/evaluation-result-3249812-1304072801/solver-3249812-1304072801 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249812-1304072801.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.06 8.07 7.90 1/175 23795
/proc/meminfo: memFree=14410508/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) R 23793 23795 23403 0 -1 4202496 419 0 0 0 0 0 0 0 25 0 1 0 518881081 9633792 361 33554432000 4194304 4338100 140736969010224 18446744073709551615 270612771525 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/23795/statm: 2352 361 147 36 0 257 0

[startup+0.0423591 s]
/proc/loadavg: 8.06 8.07 7.90 1/175 23795
/proc/meminfo: memFree=14410508/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=14244 CPUtime=0.04 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) R 23793 23795 23403 0 -1 4202496 1662 0 0 0 4 0 0 0 25 0 1 0 518881081 14585856 1574 33554432000 4194304 4338100 140736969010224 18446744073709551615 4245168 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/23795/statm: 3561 1574 153 36 0 1466 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 14244

[startup+0.100348 s]
/proc/loadavg: 8.06 8.07 7.90 1/175 23795
/proc/meminfo: memFree=14410508/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=14816 CPUtime=0.09 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) R 23793 23795 23403 0 -1 4202496 1795 0 0 0 9 0 0 0 25 0 1 0 518881081 15171584 1707 33554432000 4194304 4338100 140736969010224 18446744073709551615 4298020 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/23795/statm: 3704 1707 153 36 0 1609 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 14816

[startup+0.313867 s]
/proc/loadavg: 8.06 8.07 7.90 1/175 23795
/proc/meminfo: memFree=14410508/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=388296 CPUtime=3.17 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 36620 0 0 0 307 10 0 0 25 0 5 0 518881081 397615104 36501 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 97074 36501 174 36 0 94979 0
Current children cumulated CPU time (s) 3.17
Current children cumulated vsize (KiB) 388296

[startup+1.06612 s]
/proc/loadavg: 8.06 8.07 7.90 1/175 23795
/proc/meminfo: memFree=14410508/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=394684 CPUtime=3.57 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 37472 0 0 0 347 10 0 0 25 0 5 0 518881081 404156416 37353 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 98671 37354 174 36 0 96576 0
Current children cumulated CPU time (s) 3.57
Current children cumulated vsize (KiB) 394684

[startup+1.50105 s]
/proc/loadavg: 8.06 8.07 7.90 1/175 23795
/proc/meminfo: memFree=14410508/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=407196 CPUtime=5.31 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 41138 0 0 0 516 15 0 0 25 0 5 0 518881081 416968704 41019 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 101799 41020 174 36 0 99704 0
Current children cumulated CPU time (s) 5.31
Current children cumulated vsize (KiB) 407196

[startup+3.10076 s]
/proc/loadavg: 7.49 7.95 7.86 6/182 23827
/proc/meminfo: memFree=14211300/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=454248 CPUtime=11.69 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 52711 0 0 0 1137 32 0 0 25 0 5 0 518881081 465149952 52592 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 113562 52592 174 36 0 111467 0
[pid=23795/tid=23797] ppid=23793 vsize=454248 CPUtime=2.85 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 7158 0 0 0 277 8 0 0 24 0 5 0 518881104 465149952 52592 33554432000 4194304 4338100 140736969010224 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=454248 CPUtime=2.86 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 6299 0 0 0 280 6 0 0 24 0 5 0 518881104 465149952 52592 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=454248 CPUtime=2.86 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 7694 0 0 0 279 7 0 0 24 0 5 0 518881104 465149952 52592 33554432000 4194304 4338100 140736969010224 18446744073709551615 270612505759 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=454248 CPUtime=2.86 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 6875 0 0 0 280 6 0 0 23 0 5 0 518881104 465149952 52592 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.69
Current children cumulated vsize (KiB) 454248

[startup+6.3012 s]
/proc/loadavg: 7.29 7.90 7.84 9/186 23831
/proc/meminfo: memFree=14117632/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=523376 CPUtime=24.47 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 69983 0 0 0 2379 68 0 0 25 0 5 0 518881081 535937024 69864 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 130844 69864 174 36 0 128749 0
[pid=23795/tid=23797] ppid=23793 vsize=523376 CPUtime=6.03 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 11303 0 0 0 586 17 0 0 25 0 5 0 518881104 535937024 69864 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=523376 CPUtime=6.06 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 10808 0 0 0 590 16 0 0 25 0 5 0 518881104 535937024 69864 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=523376 CPUtime=6.06 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 12093 0 0 0 590 16 0 0 25 0 5 0 518881104 535937024 69864 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=523376 CPUtime=6.07 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 11094 0 0 0 592 15 0 0 25 0 5 0 518881104 535937024 69864 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.47
Current children cumulated vsize (KiB) 523376

[startup+12.7071 s]
/proc/loadavg: 7.11 7.85 7.83 6/186 23862
/proc/meminfo: memFree=13123660/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=636116 CPUtime=50.05 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 98652 0 0 0 4879 126 0 0 25 0 5 0 518881081 651382784 98533 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 159029 98533 174 36 0 156934 0
[pid=23795/tid=23797] ppid=23793 vsize=636116 CPUtime=12.4 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 18555 0 0 0 1207 33 0 0 25 0 5 0 518881104 651382784 98533 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=636116 CPUtime=12.47 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 17692 0 0 0 1217 30 0 0 25 0 5 0 518881104 651382784 98533 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=636116 CPUtime=12.46 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 19402 0 0 0 1218 28 0 0 25 0 5 0 518881104 651382784 98533 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=636116 CPUtime=12.46 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 18318 0 0 0 1216 30 0 0 25 0 5 0 518881104 651382784 98533 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.05
Current children cumulated vsize (KiB) 636116

[startup+25.5008 s]
/proc/loadavg: 7.10 7.82 7.82 9/186 23894
/proc/meminfo: memFree=13805372/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=824344 CPUtime=101.13 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 143812 0 0 0 9869 244 0 0 25 0 5 0 518881081 844128256 143693 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 206086 143693 174 36 0 203991 0
[pid=23795/tid=23797] ppid=23793 vsize=824344 CPUtime=25.11 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 29713 0 0 0 2446 65 0 0 25 0 5 0 518881104 844128256 143693 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=824344 CPUtime=25.26 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 28955 0 0 0 2470 56 0 0 25 0 5 0 518881104 844128256 143693 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=824344 CPUtime=25.26 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 30583 0 0 0 2467 59 0 0 25 0 5 0 518881104 844128256 143693 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=824344 CPUtime=25.26 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 29876 0 0 0 2466 60 0 0 25 0 5 0 518881104 844128256 143693 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.13
Current children cumulated vsize (KiB) 824344

[startup+51.1003 s]
/proc/loadavg: 7.53 7.85 7.83 9/186 23894
/proc/meminfo: memFree=13516396/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=1055092 CPUtime=203.36 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 200142 0 0 0 19894 442 0 0 25 0 5 0 518881081 1080414208 199975 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 263773 199975 175 36 0 261678 0
[pid=23795/tid=23797] ppid=23793 vsize=1055092 CPUtime=50.66 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 44662 0 0 0 4947 119 0 0 25 0 5 0 518881104 1080414208 199975 33554432000 4194304 4338100 140736969010224 18446744073709551615 4285314 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=1055092 CPUtime=50.86 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 44585 0 0 0 4984 102 0 0 25 0 5 0 518881104 1080414208 199975 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=1055092 CPUtime=50.76 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 43052 0 0 0 4967 109 0 0 25 0 5 0 518881104 1080414208 199975 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=1055092 CPUtime=50.85 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 43158 0 0 0 4977 108 0 0 25 0 5 0 518881104 1080414208 199975 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 203.36
Current children cumulated vsize (KiB) 1055092

[startup+102.301 s]
/proc/loadavg: 7.72 7.85 7.82 9/186 23896
/proc/meminfo: memFree=13067144/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=1424196 CPUtime=407.74 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 292896 0 0 0 39891 883 0 0 25 0 5 0 518881081 1458376704 292729 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 356049 292729 175 36 0 353954 0
[pid=23795/tid=23797] ppid=23793 vsize=1424196 CPUtime=101.79 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 72592 0 0 0 9939 240 0 0 25 0 5 0 518881104 1458376704 292729 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=1424196 CPUtime=102.03 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 66762 0 0 0 9998 205 0 0 25 0 5 0 518881104 1458376704 292729 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=1424196 CPUtime=101.91 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 64601 0 0 0 9968 223 0 0 25 0 5 0 518881104 1458376704 292729 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=1424196 CPUtime=101.75 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 64256 0 0 0 9964 211 0 0 25 0 5 0 518881104 1458376704 292729 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.74
Current children cumulated vsize (KiB) 1424196

[startup+162.307 s]
/proc/loadavg: 8.08 7.93 7.85 9/186 23897
/proc/meminfo: memFree=12621596/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=1797284 CPUtime=647.25 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 386348 0 0 0 63372 1353 0 0 25 0 5 0 518881081 1840418816 386181 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 449321 386181 175 36 0 447226 0
[pid=23795/tid=23797] ppid=23793 vsize=1797284 CPUtime=161.73 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 106330 0 0 0 15792 381 0 0 25 0 5 0 518881104 1840418816 386182 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=1797284 CPUtime=161.83 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 82768 0 0 0 15880 303 0 0 25 0 5 0 518881104 1840418816 386182 33554432000 4194304 4338100 140736969010224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=1797284 CPUtime=161.89 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 92397 0 0 0 15838 351 0 0 25 0 5 0 518881104 1840418816 386182 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=1797284 CPUtime=161.54 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 80169 0 0 0 15841 313 0 0 25 0 5 0 518881104 1840418816 386182 33554432000 4194304 4338100 140736969010224 18446744073709551615 270612506350 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 647.25
Current children cumulated vsize (KiB) 1797284

[startup+222.301 s]
/proc/loadavg: 8.16 7.97 7.87 9/186 23899
/proc/meminfo: memFree=12081820/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=2282388 CPUtime=886.71 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 506304 0 0 0 86879 1792 0 0 25 0 5 0 518881081 2337165312 506137 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 570597 506137 175 36 0 568502 0
[pid=23795/tid=23797] ppid=23793 vsize=2282388 CPUtime=221.63 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 141187 0 0 0 21662 501 0 0 25 0 5 0 518881104 2337165312 506137 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0

################
# More data... #
################

/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 233354 0 0 0 69005 1013 0 0 25 0 5 0 518881104 4374589440 1004799 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254231 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2802.87
Current children cumulated vsize (KiB) 4272060

[startup+762.3 s]
/proc/loadavg: 8.05 8.06 7.94 9/186 23913
/proc/meminfo: memFree=9437932/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4484764 CPUtime=3042.27 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1057221 0 0 0 299958 4269 0 0 25 0 5 0 518881081 4592398336 1057054 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1121191 1057054 175 36 0 1119096 0
[pid=23795/tid=23797] ppid=23793 vsize=4484764 CPUtime=760.77 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 298581 0 0 0 74982 1095 0 0 25 0 5 0 518881104 4592398336 1057054 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4484764 CPUtime=760.24 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 253393 0 0 0 74959 1065 0 0 25 0 5 0 518881104 4592398336 1057054 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4484764 CPUtime=760.89 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 242297 0 0 0 75056 1033 0 0 25 0 5 0 518881104 4592398336 1057054 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4484764 CPUtime=760.13 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 238265 0 0 0 74941 1072 0 0 25 0 5 0 518881104 4592398336 1057054 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3042.27
Current children cumulated vsize (KiB) 4484764

[startup+822.307 s]
/proc/loadavg: 8.09 8.06 7.95 9/186 23915
/proc/meminfo: memFree=9275728/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4599512 CPUtime=3281.67 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1081230 0 0 0 323618 4549 0 0 25 0 5 0 518881081 4709900288 1081063 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1149878 1081063 175 36 0 1147783 0
[pid=23795/tid=23797] ppid=23793 vsize=4599512 CPUtime=820.37 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 306124 0 0 0 80863 1174 0 0 25 0 5 0 518881104 4709900288 1081063 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4599512 CPUtime=820.18 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 259828 0 0 0 80887 1131 0 0 25 0 5 0 518881104 4709900288 1081063 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4599512 CPUtime=820.77 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 245756 0 0 0 80974 1103 0 0 25 0 5 0 518881104 4709900288 1081063 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4599512 CPUtime=820.09 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 244837 0 0 0 80873 1136 0 0 25 0 5 0 518881104 4709900288 1081063 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3281.67
Current children cumulated vsize (KiB) 4599512

[startup+882.3 s]
/proc/loadavg: 8.10 8.06 7.95 9/186 23917
/proc/meminfo: memFree=9141176/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4688808 CPUtime=3520.99 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1101185 0 0 0 347247 4852 0 0 25 0 5 0 518881081 4801339392 1101018 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1172202 1101018 175 36 0 1170107 0
[pid=23795/tid=23797] ppid=23793 vsize=4688808 CPUtime=880 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 319786 0 0 0 86741 1259 0 0 25 0 5 0 518881104 4801339392 1101018 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4688808 CPUtime=880.13 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 260235 0 0 0 86812 1201 0 0 25 0 5 0 518881104 4801339392 1101018 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4688808 CPUtime=880.59 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 251318 0 0 0 86879 1180 0 0 25 0 5 0 518881104 4801339392 1101018 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4688808 CPUtime=880.03 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 245161 0 0 0 86795 1208 0 0 25 0 5 0 518881104 4801339392 1101018 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3520.99
Current children cumulated vsize (KiB) 4688808

[startup+942.301 s]
/proc/loadavg: 8.10 8.06 7.96 9/186 23918
/proc/meminfo: memFree=9035892/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4776688 CPUtime=3760.41 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1123170 0 0 0 370917 5124 0 0 25 0 5 0 518881081 4891328512 1123003 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1194172 1123003 175 36 0 1192077 0
[pid=23795/tid=23797] ppid=23793 vsize=4776688 CPUtime=939.56 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 332687 0 0 0 92614 1342 0 0 25 0 5 0 518881104 4891328512 1123003 33554432000 4194304 4338100 140736969010224 18446744073709551615 270612807221 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4776688 CPUtime=940.1 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 263367 0 0 0 92751 1259 0 0 25 0 5 0 518881104 4891328512 1123003 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4776688 CPUtime=940.54 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 254404 0 0 0 92804 1250 0 0 25 0 5 0 518881104 4891328512 1123003 33554432000 4194304 4338100 140736969010224 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4776688 CPUtime=939.97 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 248027 0 0 0 92728 1269 0 0 25 0 5 0 518881104 4891328512 1123003 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3760.41
Current children cumulated vsize (KiB) 4776688

[startup+1002.3 s]
/proc/loadavg: 8.07 8.06 7.96 9/186 23920
/proc/meminfo: memFree=9004016/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4798584 CPUtime=3999.84 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1128771 0 0 0 394563 5421 0 0 25 0 5 0 518881081 4913750016 1128604 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1199646 1128604 175 36 0 1197551 0
[pid=23795/tid=23797] ppid=23793 vsize=4798584 CPUtime=999.41 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 338280 0 0 0 98514 1427 0 0 25 0 5 0 518881104 4913750016 1128604 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4798584 CPUtime=1000.04 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 263370 0 0 0 98682 1322 0 0 25 0 5 0 518881104 4913750016 1128604 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4798584 CPUtime=1000.49 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 254406 0 0 0 98716 1333 0 0 25 0 5 0 518881104 4913750016 1128604 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4798584 CPUtime=999.64 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 248030 0 0 0 98630 1334 0 0 25 0 5 0 518881104 4913750016 1128604 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3999.84
Current children cumulated vsize (KiB) 4798584

[startup+1062.3 s]
/proc/loadavg: 8.08 8.06 7.97 9/186 23921
/proc/meminfo: memFree=8838592/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4923704 CPUtime=4239.05 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1159553 0 0 0 418165 5740 0 0 25 0 5 0 518881081 5041872896 1159386 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1230926 1159386 175 36 0 1228831 0
[pid=23795/tid=23797] ppid=23793 vsize=4923704 CPUtime=1059.11 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 356741 0 0 0 104383 1528 0 0 25 0 5 0 518881104 5041872896 1159386 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4923704 CPUtime=1059.94 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 263370 0 0 0 104605 1389 0 0 25 0 5 0 518881104 5041872896 1159386 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4923704 CPUtime=1060.37 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 266727 0 0 0 104613 1424 0 0 25 0 5 0 518881104 5041872896 1159386 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4923704 CPUtime=1059.38 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 248030 0 0 0 104544 1394 0 0 25 0 5 0 518881104 5041872896 1159386 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4239.05
Current children cumulated vsize (KiB) 4923704

[startup+1122.3 s]
/proc/loadavg: 8.15 8.08 7.98 9/186 23923
/proc/meminfo: memFree=8708136/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=4992652 CPUtime=4478.53 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1177328 0 0 0 441726 6127 0 0 25 0 5 0 518881081 5112475648 1177161 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1248163 1177161 175 36 0 1246068 0
[pid=23795/tid=23797] ppid=23793 vsize=4992652 CPUtime=1118.72 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 364192 0 0 0 110236 1636 0 0 25 0 5 0 518881104 5112475648 1177161 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=4992652 CPUtime=1119.89 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 263373 0 0 0 110514 1475 0 0 25 0 5 0 518881104 5112475648 1177161 33554432000 4194304 4338100 140736969010224 18446744073709551615 270612505849 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=4992652 CPUtime=1120.33 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 277046 0 0 0 110499 1534 0 0 25 0 5 0 518881104 5112475648 1177161 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=4992652 CPUtime=1119.34 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 248032 0 0 0 110456 1478 0 0 25 0 5 0 518881104 5112475648 1177161 33554432000 4194304 4338100 140736969010224 18446744073709551615 270612505849 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4478.53
Current children cumulated vsize (KiB) 4992652

[startup+1182.3 s]
/proc/loadavg: 8.12 8.08 7.98 9/186 23925
/proc/meminfo: memFree=8508236/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=5161564 CPUtime=4717.88 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) S 23793 23795 23403 0 -1 4202496 1219443 0 0 0 465343 6445 0 0 25 0 5 0 518881081 5285441536 1219276 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 1290391 1219276 175 36 0 1288296 0
[pid=23795/tid=23797] ppid=23793 vsize=5161564 CPUtime=1178.42 cores=1,3,5,7
/proc/23795/task/23797/stat : 23797 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 389665 0 0 0 116110 1732 0 0 25 0 5 0 518881104 5285441536 1219276 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=5161564 CPUtime=1179.86 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 263377 0 0 0 116444 1542 0 0 25 0 5 0 518881104 5285441536 1219276 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=23795/tid=23799] ppid=23793 vsize=5161564 CPUtime=1180.05 cores=1,3,5,7
/proc/23795/task/23799/stat : 23799 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 293683 0 0 0 116384 1621 0 0 25 0 5 0 518881104 5285441536 1219276 33554432000 4194304 4338100 140736969010224 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=23795/tid=23800] ppid=23793 vsize=5161564 CPUtime=1179.29 cores=1,3,5,7
/proc/23795/task/23800/stat : 23800 (ttsth-5-4) R 23793 23795 23403 0 -1 4202560 248033 0 0 0 116384 1545 0 0 25 0 5 0 518881104 5285441536 1219276 33554432000 4194304 4338100 140736969010224 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4717.88
Current children cumulated vsize (KiB) 5161564



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.91 s]
/proc/loadavg: 8.08 8.07 7.98 6/183 23925
/proc/meminfo: memFree=8517536/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=0 CPUtime=4800.03 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) Z 23793 23795 23403 0 -1 4203548 1223984 0 0 0 473448 6555 0 0 18 0 2 0 518881081 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 0 0 0 0 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=0 CPUtime=1200.45 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4203588 263378 0 0 0 118481 1564 0 0 25 0 2 0 518881104 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.03
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.01 s]
/proc/loadavg: 8.08 8.07 7.98 6/183 23925
/proc/meminfo: memFree=8517536/32951124 swapFree=67111528/67111528
[pid=23795] ppid=23793 vsize=0 CPUtime=4800.13 cores=1,3,5,7
/proc/23795/stat : 23795 (ttsth-5-4) Z 23793 23795 23403 0 -1 4203548 1223984 0 0 0 473448 6565 0 0 18 0 2 0 518881081 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/23795/statm: 0 0 0 0 0 0 0
[pid=23795/tid=23798] ppid=23793 vsize=0 CPUtime=1200.55 cores=1,3,5,7
/proc/23795/task/23798/stat : 23798 (ttsth-5-4) R 23793 23795 23403 0 -1 4203588 263378 0 0 0 118481 1574 0 0 25 0 2 0 518881104 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.09
CPU time (s): 4800.22
CPU user time (s): 4734.48
CPU system time (s): 65.732
CPU usage (%): 398.992
Max. virtual memory (cumulated for all children) (KiB): 5180332

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4734.48
system time used= 65.732
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1223984
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 36810
involuntary context switches= 32523

runsolver used 1.9417 second user time and 4.6273 second system time

The end

Launcher Data 

Begin job on node108 at 2011-04-29 12:26:41
IDJOB=3249812
IDBENCH=83189
IDSOLVER=1654
FILE ID=node108/3249812-1304072801
RUNJOBID= node108-1304071593-23707
PBS_JOBID= 13170476
Free space on /tmp= 72120 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/skvortsov/automata-synchronization/crn_20_360_u.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249812-1304072801/watcher-3249812-1304072801 -o /tmp/evaluation-result-3249812-1304072801/solver-3249812-1304072801 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249812-1304072801.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 4b70d66c676eba3660e2a5eff473f8e7
RANDOM SEED=1322246585

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14410664 kB
Buffers:       2272548 kB
Cached:       15621288 kB
SwapCached:          0 kB
Active:        9013440 kB
Inactive:      8945812 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14410664 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             412 kB
Writeback:           0 kB
AnonPages:       65176 kB
Mapped:          15080 kB
Slab:           518564 kB
PageTables:       4292 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   216000 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72120 MiB
End job on node108 at 2011-04-29 12:46:46