Trace number 3250912

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.45 1205.82

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q918.75-n50-p30-s52376212.cnf
MD5SUMf34a5dd1cdce70a322dd395bcb5f1303
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2548
Number of clauses582679
Sum of the clauses size1167806
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2582629
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data 

0.29/0.39	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250912-1304105724.cnf 
0.29/0.39	c Using SATTIMEOUT of 4800
0.29/0.39	c Using SATRAM of 15500
0.29/0.39	c Size of subset node = 32
0.29/0.39	c Variables = 2548
0.29/0.39	c Clauses = 582679
0.29/0.39	c Literals = 1167806
0.29/0.39	c Reduced size:
0.29/0.39	c Variables = 2548
0.29/0.39	c Clauses = 582679
0.29/0.39	c Literals = 1167806
0.29/0.39	c a2
0.69/0.71	c a1
1.79/1.86	c Independent sub-problems = 1
1.79/1.86	c Sub-problem with 2548 variables and 582679 clauses
1.79/1.86	c using permutation method 4
1.79/1.86	c count = 1
1.79/1.86	c Ordering finished 1.790 seconds
1.79/1.86	c mapped 1.860 seconds
1.79/1.86	c permuted 1.860 seconds
1.79/1.86	c about to sort
1.89/1.91	c sorted
4.09/4.16	c clauses now 582679
4.09/4.16	c init finished 2.050 seconds
4.09/4.16	c tree built 4.150 seconds
4.09/4.16	c 
4.09/4.16	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
4.09/4.16	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
4.09/4.16	c ................................ | ..........                    |                                  4.2         0.0             
4.09/4.16	c ................................ | ..........                  1 |                                  4.2         0.2             
4.09/4.16	c ................................ | ..........                  2 |                                  4.2         0.5             
4.09/4.16	c ................................ | ..........                  3 |                                  4.2         0.7             
4.09/4.16	c ................................ | ..........                  4 |                                  4.2         1.0             
4.09/4.16	c ................................ | ..........                  5 |                                  4.2         1.2             
4.09/4.16	c ................................ | ..........               1000 |              62                  4.2       240.4         4424
95.76/27.16	c ................................ | ..........            4456000 | 720          379085             96.0     46416.7     13527405
191.63/51.19	c ................................ | ..........            8794000 | 1403         747528            192.0     45802.1     30225419
287.88/75.23	c ................................ | ..........           12782000 | 2129         1099519           288.0     44380.4     41599030
383.74/99.26	c ................................ | ..........           16981000 | 2547         1445768           384.0     44221.4     53429855
479.62/123.30	c ................................ | *.........           20948000 | 3420         1783619           480.0     43641.7     55653017
575.88/147.33	c ................................ | *.........           24931000 | 4171         2123805           576.0     43283.0     67905320
671.61/171.50	c ................................ | *.........           28651000 | 4979         2447153           672.0     42634.8     75467153
767.61/195.70	c ................................ | *.........           32575000 | 5722         2775152           768.0     42414.8     82042436
863.98/219.80	c ................................ | *.........           36409000 | 7245         3104141           864.0     42140.0     96556738
959.84/243.84	c ................................ | **........           40355000 | 8034         3434433           960.0     42036.5    111085984
1055.71/267.87	c ................................ | **........           44062000 | 8814         3748092          1056.0     41725.4    117532081
1151.96/291.91	c ................................ | **........           47764000 | 9380         4063582          1152.0     41461.4    128878445
1247.80/315.95	c ................................ | **........           51362000 | 9951         4364587          1248.0     41155.1    135412861
1343.64/339.99	c ................................ | **........           54904000 | 10415        4665767          1344.0     40851.2    144744419
1439.89/364.03	c ................................ | ***.......           58655000 | 12523        4970812          1440.0     40732.6    158527276
1535.74/388.06	c ................................ | ***.......           62032000 | 13538        5256380          1536.0     40385.2    159270536
1632.00/412.10	c ................................ | ***.......           65567000 | 14338        5552358          1632.0     40175.9    167327916
1727.86/436.13	c ................................ | ***.......           69004000 | 15226        5840612          1728.0     39932.6    174145369
1823.71/460.17	c ................................ | ***.......           72470000 | 15991        6136479          1824.0     39730.9    186372000
1919.97/484.20	c ................................ | ****......           75810000 | 17331        6417966          1920.0     39484.4    193905408
2015.83/508.24	c ................................ | ****......           78930000 | 17882        6695606          2016.0     39151.6    193905408
2111.65/532.28	c ................................ | ****......           82242000 | 18657        6982027          2112.0     38940.3    198756496
2207.89/556.33	c ................................ | ****......           85461000 | 19635        7279221          2208.0     38704.6    202945856
2303.71/580.37	c ................................ | ****......           88591000 | 20415        7582643          2304.0     38450.8    208268736
2399.96/604.41	c ................................ | *****.....           91790000 | 21448        7880521          2400.0     38245.5    214424849
2495.81/628.44	c ................................ | *****.....           94872000 | 22038        8171329          2496.0     38009.6    218375460
2591.67/652.48	c ................................ | *****.....           98700000 | 22879        8475330          2592.0     38078.4    224773269
2687.94/676.52	c ................................ | *****.....          102437000 | 24124        8768155          2688.0     38108.7    226051938
2783.77/700.56	c ................................ | *****.....          106264000 | 25117        9070233          2784.0     38169.5    232116330
2880.01/724.60	c ................................ | ******....          109944000 | 26024        9365684          2880.0     38174.9    237646070
2975.87/748.64	c ................................ | ******....          113345000 | 27127        9651130          2976.0     38086.1    241271787
3071.70/772.67	c ................................ | ******....          117287000 | 28140        9936881          3072.0     38179.2    247418045
3167.92/796.72	c ................................ | ******....          120721000 | 29071        10203464         3168.0     38106.3    250190350
3263.77/820.75	c ................................ | ******....          124449000 | 30480        10490837         3264.0     38127.8    253263074
3360.01/844.80	c ................................ | *******...          127933000 | 31500        10763649         3360.0     38075.1    256512373
3455.86/868.83	c ................................ | *******...          131409000 | 32979        11040128         3456.0     38023.4    262244888
3551.69/892.87	c ................................ | *******...          134738000 | 33937        11311016         3552.0     37933.0    267274320
3647.94/916.91	c ................................ | *******...          138423000 | 35099        11589933         3648.0     37944.9    276296861
3743.79/940.95	c ................................ | *******...          141827000 | 36028        11858966         3744.0     37881.0    280170487
3839.65/964.99	c ................................ | ********..          145276000 | 36849        12130469         3840.0     37832.2    285728389
3935.88/989.03	c ................................ | ********..          148821000 | 37377        12401887         3936.0     37810.1    292930578
4031.77/1013.06	c ................................ | ********..          152252000 | 37973        12675613         4032.0     37760.8    297016833
4127.95/1037.11	c ................................ | ********..          155276000 | 38661        12943540         4128.0     37615.2    301856240
4223.80/1061.15	c ................................ | ********..          158563000 | 39901        13227723         4224.0     37538.5    301856240
4319.66/1085.18	c ................................ | *********.          161775000 | 40788        13504998         4320.0     37447.7    301890349
4415.90/1109.22	c ................................ | *********.          165048000 | 41426        13780290         4416.0     37374.9    304650132
4511.75/1133.26	c ................................ | *********.          168239000 | 42884        14047662         4512.0     37287.0    306699364
4608.01/1157.30	c ................................ | *********.          171451000 | 44333        14313553         4608.0     37207.1    310300086
4703.90/1181.33	c ................................ | *********.          174518000 | 45415        14585999         4704.0     37099.9    313316951
4799.68/1205.38	c CPU limit (4800) reached
4799.68/1205.38	c Time = 4800.00 seconds
4799.68/1205.38	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250912-1304105724/watcher-3250912-1304105724 -o /tmp/evaluation-result-3250912-1304105724/solver-3250912-1304105724 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250912-1304105724.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.88 7.89 7.95 5/181 3142
/proc/meminfo: memFree=27996344/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=9424 CPUtime=0 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 432 0 0 0 0 0 0 0 25 0 1 0 502988328 9650176 375 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 2356 377 147 36 0 261 0

[startup+0.0634309 s]
/proc/loadavg: 7.88 7.89 7.95 5/181 3142
/proc/meminfo: memFree=27996344/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=21416 CPUtime=0.05 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 3350 0 0 0 5 0 0 0 25 0 1 0 502988328 21929984 3293 33554432000 4194304 4338100 140735388734960 18446744073709551615 220816236288 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 5354 3295 148 36 0 3259 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 21416

[startup+0.100427 s]
/proc/loadavg: 7.88 7.89 7.95 5/181 3142
/proc/meminfo: memFree=27996344/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=28820 CPUtime=0.09 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 5194 0 0 0 9 0 0 0 25 0 1 0 502988328 29511680 5137 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 7205 5139 148 36 0 5110 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28820

[startup+0.300388 s]
/proc/loadavg: 7.88 7.89 7.95 5/181 3142
/proc/meminfo: memFree=27996344/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=69364 CPUtime=0.29 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 15047 0 0 0 28 1 0 0 25 0 1 0 502988328 71028736 14990 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 17374 14992 148 36 0 15279 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 69364

[startup+0.700307 s]
/proc/loadavg: 7.88 7.89 7.95 5/181 3142
/proc/meminfo: memFree=27996344/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=121460 CPUtime=0.69 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 26587 0 0 0 67 2 0 0 25 0 1 0 502988328 124375040 26513 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631573 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 30365 26513 239 36 0 28270 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 121460

[startup+1.50116 s]
/proc/loadavg: 7.88 7.89 7.95 6/182 3143
/proc/meminfo: memFree=27882512/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=135552 CPUtime=1.49 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 41517 0 0 0 145 4 0 0 25 0 1 0 502988328 138805248 31275 33554432000 4194304 4338100 140735388734960 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 33888 31276 155 36 0 31793 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 135552

[startup+3.10084 s]
/proc/loadavg: 7.88 7.89 7.95 6/182 3143
/proc/meminfo: memFree=27824108/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=226496 CPUtime=3.09 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) R 3140 3142 2468 0 -1 4202496 66049 0 0 0 302 7 0 0 25 0 1 0 502988328 231931904 54668 33554432000 4194304 4338100 140735388734960 18446744073709551615 4278345 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3142/statm: 56624 54668 164 36 0 54529 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 226496

[startup+6.30121 s]
/proc/loadavg: 7.89 7.89 7.95 9/186 3147
/proc/meminfo: memFree=27543964/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=502336 CPUtime=12.68 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 107444 0 0 0 1235 33 0 0 25 0 5 0 502988328 514392064 96062 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 125584 96063 176 36 0 123489 0
[pid=3142/tid=3144] ppid=3140 vsize=502336 CPUtime=2.13 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 5087 0 0 0 212 1 0 0 21 0 5 0 502988744 514392064 96063 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828050587 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=502336 CPUtime=2.12 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 3300 0 0 0 202 10 0 0 21 0 5 0 502988744 514392064 96063 33554432000 4194304 4338100 140735388734960 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=502336 CPUtime=2.13 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 6198 0 0 0 212 1 0 0 21 0 5 0 502988744 514392064 96063 33554432000 4194304 4338100 140735388734960 18446744073709551615 4245205 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=502336 CPUtime=2.11 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 3210 0 0 0 202 9 0 0 21 0 5 0 502988744 514392064 96063 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631806 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 502336

[startup+12.701 s]
/proc/loadavg: 7.90 7.89 7.95 9/186 3147
/proc/meminfo: memFree=27226812/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=696316 CPUtime=38.25 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 155199 0 0 0 3711 114 0 0 25 0 5 0 502988328 713027584 143817 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 174079 143817 176 36 0 171984 0
[pid=3142/tid=3144] ppid=3140 vsize=696316 CPUtime=8.52 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 20648 0 0 0 847 5 0 0 25 0 5 0 502988744 713027584 143817 33554432000 4194304 4338100 140735388734960 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=696316 CPUtime=8.52 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 11561 0 0 0 805 47 0 0 25 0 5 0 502988744 713027584 143817 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631376 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=696316 CPUtime=8.52 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 22604 0 0 0 848 4 0 0 25 0 5 0 502988744 713027584 143817 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631376 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=696316 CPUtime=8.47 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 10736 0 0 0 802 45 0 0 25 0 5 0 502988744 713027584 143817 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631585 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38.25
Current children cumulated vsize (KiB) 696316

[startup+25.5007 s]
/proc/loadavg: 7.92 7.90 7.95 9/186 3147
/proc/meminfo: memFree=26870048/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=973672 CPUtime=89.36 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 205261 0 0 0 8654 282 0 0 25 0 5 0 502988328 997040128 193879 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 243418 193879 176 36 0 241323 0
[pid=3142/tid=3144] ppid=3140 vsize=973672 CPUtime=21.32 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 42322 0 0 0 2121 11 0 0 25 0 5 0 502988744 997040128 193879 33554432000 4194304 4338100 140735388734960 18446744073709551615 4286174 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=973672 CPUtime=21.32 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 19060 0 0 0 2004 128 0 0 25 0 5 0 502988744 997040128 193879 33554432000 4194304 4338100 140735388734960 18446744073709551615 4286331 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=973672 CPUtime=21.33 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 36814 0 0 0 2126 7 0 0 25 0 5 0 502988744 997040128 193879 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631785 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=973672 CPUtime=21.2 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 17415 0 0 0 1995 125 0 0 25 0 5 0 502988744 997040128 193879 33554432000 4194304 4338100 140735388734960 18446744073709551615 4286319 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 89.36
Current children cumulated vsize (KiB) 973672

[startup+51.101 s]
/proc/loadavg: 7.95 7.90 7.95 9/186 3148
/proc/meminfo: memFree=25955768/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=1478352 CPUtime=191.63 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 338216 0 0 0 18539 624 0 0 25 0 5 0 502988328 1513832448 326834 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 369588 326834 176 36 0 367493 0
[pid=3142/tid=3144] ppid=3140 vsize=1478352 CPUtime=46.93 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 94046 0 0 0 4666 27 0 0 25 0 5 0 502988744 1513832448 326834 33554432000 4194304 4338100 140735388734960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=1478352 CPUtime=46.78 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 36826 0 0 0 4385 293 0 0 25 0 5 0 502988744 1513832448 326834 33554432000 4194304 4338100 140735388734960 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=1478352 CPUtime=46.92 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 81771 0 0 0 4678 14 0 0 25 0 5 0 502988744 1513832448 326834 33554432000 4194304 4338100 140735388734960 18446744073709551615 4241466 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=1478352 CPUtime=46.8 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 35923 0 0 0 4401 279 0 0 25 0 5 0 502988744 1513832448 326834 33554432000 4194304 4338100 140735388734960 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 191.63
Current children cumulated vsize (KiB) 1478352

[startup+102.307 s]
/proc/loadavg: 7.98 7.92 7.95 9/186 3149
/proc/meminfo: memFree=24462292/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=2317868 CPUtime=396.15 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 527915 0 0 0 38285 1330 0 0 25 0 5 0 502988328 2373496832 516533 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 579467 516533 176 36 0 577372 0
[pid=3142/tid=3144] ppid=3140 vsize=2317868 CPUtime=98.13 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 177616 0 0 0 9758 55 0 0 25 0 5 0 502988744 2373496832 516533 33554432000 4194304 4338100 140735388734960 18446744073709551615 4286311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=2317868 CPUtime=97.91 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 63561 0 0 0 9166 625 0 0 25 0 5 0 502988744 2373496832 516533 33554432000 4194304 4338100 140735388734960 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=2317868 CPUtime=98.13 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 136491 0 0 0 9789 24 0 0 25 0 5 0 502988744 2373496832 516533 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631443 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=2317868 CPUtime=97.79 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 60597 0 0 0 9164 615 0 0 25 0 5 0 502988744 2373496832 516533 33554432000 4194304 4338100 140735388734960 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 396.15
Current children cumulated vsize (KiB) 2317868

[startup+162.3 s]
/proc/loadavg: 7.99 7.93 7.95 9/186 3151
/proc/meminfo: memFree=23531120/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=2999260 CPUtime=635.49 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 693298 0 0 0 61491 2058 0 0 25 0 5 0 502988328 3071242240 681916 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 749815 681916 176 36 0 747720 0
[pid=3142/tid=3144] ppid=3140 vsize=2999260 CPUtime=158.12 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 266170 0 0 0 15731 81 0 0 25 0 5 0 502988744 3071242240 681916 33554432000 4194304 4338100 140735388734960 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=2999260 CPUtime=157.83 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 78562 0 0 0 14804 979 0 0 25 0 5 0 502988744 3071242240 681917 33554432000 4194304 4338100 140735388734960 18446744073709551615 4241517 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=2999260 CPUtime=158.06 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 185171 0 0 0 15773 33 0 0 25 0 5 0 502988744 3071242240 681917 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631797 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=2999260 CPUtime=157.28 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 73745 0 0 0 14775 953 0 0 18 0 5 0 502988744 3071242240 681917 33554432000 4194304 4338100 140735388734960 18446744073709551615 4254849 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 635.49
Current children cumulated vsize (KiB) 2999260

[startup+222.3 s]
/proc/loadavg: 8.03 7.95 7.95 9/186 3154
/proc/meminfo: memFree=22317124/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=3781116 CPUtime=873.96 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 885888 0 0 0 84932 2464 0 0 25 0 5 0 502988328 3871862784 874506 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 945279 874506 176 36 0 943184 0
[pid=3142/tid=3144] ppid=3140 vsize=3781116 CPUtime=218.1 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 361620 0 0 0 21702 108 0 0 25 0 5 0 502988744 3871862784 874506 33554432000 4194304 4338100 140735388734960 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=3781116 CPUtime=217.74 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 99413 0 0 0 20561 1213 0 0 25 0 5 0 502988744 3871862784 874506 33554432000 4194304 4338100 140735388734960 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=3781116 CPUtime=217.82 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 236527 0 0 0 21738 44 0 0 25 0 5 0 502988744 3871862784 874506 33554432000 4194304 4338100 140735388734960 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=3781116 CPUtime=216.11 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 98678 0 0 0 20523 1088 0 0 25 0 5 0 502988744 3871862784 874506 33554432000 4194304 4338100 140735388734960 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 873.96
Current children cumulated vsize (KiB) 3781116

################
# More data... #
################

/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 188181 0 0 0 65259 4207 0 0 25 0 5 0 502988744 8464863232 1948620 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631794 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2790.96
Current children cumulated vsize (KiB) 8266468

[startup+762.307 s]
/proc/loadavg: 8.18 8.08 8.01 9/186 3168
/proc/meminfo: memFree=13164688/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=8631556 CPUtime=3030.61 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2057531 0 0 0 293320 9741 0 0 25 0 5 0 502988328 8838713344 2046149 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2157889 2046149 176 36 0 2155794 0
[pid=3142/tid=3144] ppid=3140 vsize=8631556 CPUtime=757.12 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1062606 0 0 0 75381 331 0 0 25 0 5 0 502988744 8838713344 2046149 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631609 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=8631556 CPUtime=756.87 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 190746 0 0 0 70974 4713 0 0 25 0 5 0 502988744 8838713344 2046149 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=8631556 CPUtime=757.79 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 526219 0 0 0 75672 107 0 0 25 0 5 0 502988744 8838713344 2046149 33554432000 4194304 4338100 140735388734960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=8631556 CPUtime=754.64 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 188310 0 0 0 70885 4579 0 0 25 0 5 0 502988744 8838713344 2046149 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3030.61
Current children cumulated vsize (KiB) 8631556

[startup+822.301 s]
/proc/loadavg: 8.11 8.08 8.01 9/186 3170
/proc/meminfo: memFree=12764892/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=9027940 CPUtime=3270.17 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2133724 0 0 0 316545 10472 0 0 25 0 5 0 502988328 9244610560 2122342 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2256985 2122342 176 36 0 2254890 0
[pid=3142/tid=3144] ppid=3140 vsize=9027940 CPUtime=816.84 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1122848 0 0 0 81331 353 0 0 25 0 5 0 502988744 9244610560 2122342 33554432000 4194304 4338100 140735388734960 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=9027940 CPUtime=816.84 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193187 0 0 0 76615 5069 0 0 25 0 5 0 502988744 9244610560 2122342 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=9027940 CPUtime=817.69 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 534166 0 0 0 81661 108 0 0 25 0 5 0 502988744 9244610560 2122342 33554432000 4194304 4338100 140735388734960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=9027940 CPUtime=814.61 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193873 0 0 0 76530 4931 0 0 25 0 5 0 502988744 9244610560 2122343 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3270.17
Current children cumulated vsize (KiB) 9027940

[startup+882.301 s]
/proc/loadavg: 8.14 8.09 8.01 9/186 3171
/proc/meminfo: memFree=12304292/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=9428428 CPUtime=3509.76 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2229840 0 0 0 339750 11226 0 0 25 0 5 0 502988328 9654710272 2218458 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2357107 2218458 176 36 0 2355012 0
[pid=3142/tid=3144] ppid=3140 vsize=9428428 CPUtime=876.83 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1191649 0 0 0 87303 380 0 0 25 0 5 0 502988744 9654710272 2218458 33554432000 4194304 4338100 140735388734960 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=9428428 CPUtime=876.82 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193187 0 0 0 82245 5437 0 0 25 0 5 0 502988744 9654710272 2218459 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=9428428 CPUtime=877.35 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 561480 0 0 0 87623 112 0 0 25 0 5 0 502988744 9654710272 2218459 33554432000 4194304 4338100 140735388734960 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=9428428 CPUtime=874.58 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193874 0 0 0 82171 5287 0 0 25 0 5 0 502988744 9654710272 2218459 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3509.76
Current children cumulated vsize (KiB) 9428428

[startup+942.301 s]
/proc/loadavg: 8.14 8.10 8.02 9/186 3176
/proc/meminfo: memFree=11599192/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=9907060 CPUtime=3749.39 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2344358 0 0 0 362950 11989 0 0 25 0 5 0 502988328 10144829440 2332976 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2476765 2332976 176 36 0 2474670 0
[pid=3142/tid=3144] ppid=3140 vsize=9907060 CPUtime=936.81 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1275799 0 0 0 93275 406 0 0 25 0 5 0 502988744 10144829440 2332976 33554432000 4194304 4338100 140735388734960 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=9907060 CPUtime=936.8 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193187 0 0 0 87874 5806 0 0 25 0 5 0 502988744 10144829440 2332976 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=9907060 CPUtime=937.01 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 591848 0 0 0 93584 117 0 0 25 0 5 0 502988744 10144829440 2332976 33554432000 4194304 4338100 140735388734960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=9907060 CPUtime=934.57 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193874 0 0 0 87809 5648 0 0 25 0 5 0 502988744 10144829440 2332976 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3749.39
Current children cumulated vsize (KiB) 9907060

[startup+1002.3 s]
/proc/loadavg: 8.10 8.10 8.02 9/186 3177
/proc/meminfo: memFree=10725708/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=10350708 CPUtime=3989 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2445721 0 0 0 386150 12750 0 0 25 0 5 0 502988328 10599124992 2434339 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2587677 2434339 176 36 0 2585582 0
[pid=3142/tid=3144] ppid=3140 vsize=10350708 CPUtime=996.78 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1350247 0 0 0 99248 430 0 0 25 0 5 0 502988744 10599124992 2434339 33554432000 4194304 4338100 140735388734960 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=10350708 CPUtime=996.78 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 201827 0 0 0 93498 6180 0 0 25 0 5 0 502988744 10599124992 2434339 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=10350708 CPUtime=996.67 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 610119 0 0 0 99546 121 0 0 25 0 5 0 502988744 10599124992 2434339 33554432000 4194304 4338100 140735388734960 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=10350708 CPUtime=994.57 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193878 0 0 0 93449 6008 0 0 25 0 5 0 502988744 10599124992 2434339 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3989
Current children cumulated vsize (KiB) 10350708

[startup+1062.3 s]
/proc/loadavg: 8.04 8.08 8.01 9/186 3180
/proc/meminfo: memFree=10170400/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=10665172 CPUtime=4228.59 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2526498 0 0 0 409342 13517 0 0 25 0 5 0 502988328 10921136128 2515116 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2666293 2515116 176 36 0 2664198 0
[pid=3142/tid=3144] ppid=3140 vsize=10665172 CPUtime=1056.69 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1403549 0 0 0 105218 451 0 0 25 0 5 0 502988744 10921136128 2515116 33554432000 4194304 4338100 140735388734960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=10665172 CPUtime=1056.75 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 209333 0 0 0 99114 6561 0 0 25 0 5 0 502988744 10921136128 2515116 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=10665172 CPUtime=1056.43 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 630081 0 0 0 105518 125 0 0 25 0 5 0 502988744 10921136128 2515116 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631797 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=10665172 CPUtime=1054.53 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 193885 0 0 0 99084 6369 0 0 25 0 5 0 502988744 10921136128 2515116 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4228.59
Current children cumulated vsize (KiB) 10665172

[startup+1122.3 s]
/proc/loadavg: 5.66 7.40 7.78 5/176 3200
/proc/meminfo: memFree=20665232/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=10818204 CPUtime=4468.22 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2566412 0 0 0 432995 13827 0 0 25 0 5 0 502988328 11077840896 2555030 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2704551 2555030 176 36 0 2702456 0
[pid=3142/tid=3144] ppid=3140 vsize=10818204 CPUtime=1116.52 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1440761 0 0 0 111183 469 0 0 25 0 5 0 502988744 11077840896 2555030 33554432000 4194304 4338100 140735388734960 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=10818204 CPUtime=1116.68 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 211203 0 0 0 104960 6708 0 0 25 0 5 0 502988744 11077840896 2555030 33554432000 4194304 4338100 140735388734960 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=10818204 CPUtime=1116.43 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 630082 0 0 0 111518 125 0 0 25 0 5 0 502988744 11077840896 2555030 33554432000 4194304 4338100 140735388734960 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=10818204 CPUtime=1114.4 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 194716 0 0 0 104926 6514 0 0 25 0 5 0 502988744 11077840896 2555030 33554432000 4194304 4338100 140735388734960 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4468.22
Current children cumulated vsize (KiB) 10818204

[startup+1182.3 s]
/proc/loadavg: 4.61 6.78 7.54 5/176 3201
/proc/meminfo: memFree=20429568/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=11039612 CPUtime=4707.88 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) S 3140 3142 2468 0 -1 4202496 2627133 0 0 0 456684 14104 0 0 25 0 5 0 502988328 11304562688 2615751 33554432000 4194304 4338100 140735388734960 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 2759903 2615752 176 36 0 2757808 0
[pid=3142/tid=3144] ppid=3140 vsize=11039612 CPUtime=1176.51 cores=0,2,4,6
/proc/3142/task/3144/stat : 3144 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 1495178 0 0 0 117161 490 0 0 25 0 5 0 502988744 11304562688 2615752 33554432000 4194304 4338100 140735388734960 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3142/tid=3145] ppid=3140 vsize=11039612 CPUtime=1176.49 cores=0,2,4,6
/proc/3142/task/3145/stat : 3145 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 213110 0 0 0 110812 6837 0 0 25 0 5 0 502988744 11304562688 2615752 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631797 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=11039612 CPUtime=1176.43 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 630088 0 0 0 117518 125 0 0 25 0 5 0 502988744 11304562688 2615752 33554432000 4194304 4338100 140735388734960 18446744073709551615 4279576 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3142/tid=3147] ppid=3140 vsize=11039612 CPUtime=1174.25 cores=0,2,4,6
/proc/3142/task/3147/stat : 3147 (ttsth-5-4) R 3140 3142 2468 0 -1 4202560 199108 0 0 0 110785 6640 0 0 25 0 5 0 502988744 11304562688 2615752 33554432000 4194304 4338100 140735388734960 18446744073709551615 220815631381 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4707.88
Current children cumulated vsize (KiB) 11039612



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.4 s]
/proc/loadavg: 4.40 6.56 7.45 2/173 3202
/proc/meminfo: memFree=20745248/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=0 CPUtime=4800.03 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) Z 3140 3142 2468 0 -1 4203548 2631087 0 0 0 465792 14211 0 0 18 0 2 0 502988328 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 0 0 0 0 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=0 CPUtime=1199.54 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4203588 630090 0 0 0 119826 128 0 0 25 0 2 0 502988744 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.03
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.8 s]
/proc/loadavg: 4.40 6.56 7.45 2/174 3203
/proc/meminfo: memFree=23023268/32951124 swapFree=67105204/67111528
[pid=3142] ppid=3140 vsize=0 CPUtime=4800.42 cores=0,2,4,6
/proc/3142/stat : 3142 (ttsth-5-4) Z 3140 3142 2468 0 -1 4203548 2631087 0 0 0 465792 14250 0 0 18 0 2 0 502988328 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3142/statm: 0 0 0 0 0 0 0
[pid=3142/tid=3146] ppid=3140 vsize=0 CPUtime=1199.93 cores=0,2,4,6
/proc/3142/task/3146/stat : 3146 (ttsth-5-4) R 3140 3142 2468 0 -1 4203588 630090 0 0 0 119826 167 0 0 25 0 2 0 502988744 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.42
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.82
CPU time (s): 4800.45
CPU user time (s): 4657.93
CPU system time (s): 142.526
CPU usage (%): 398.107
Max. virtual memory (cumulated for all children) (KiB): 11124788

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4657.93
system time used= 142.526
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2631087
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 11889
involuntary context switches= 22973

runsolver used 1.9487 second user time and 4.52231 second system time

The end

Launcher Data 

Begin job on node147 at 2011-04-29 21:35:24
IDJOB=3250912
IDBENCH=88204
IDSOLVER=1654
FILE ID=node147/3250912-1304105724
RUNJOBID= node147-1304105580-3011
PBS_JOBID= 13170622
Free space on /tmp= 72636 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q918.75-n50-p30-s52376212.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250912-1304105724/watcher-3250912-1304105724 -o /tmp/evaluation-result-3250912-1304105724/solver-3250912-1304105724 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250912-1304105724.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f34a5dd1cdce70a322dd395bcb5f1303
RANDOM SEED=1597456754

node147.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27996624 kB
Buffers:        283040 kB
Cached:        1391312 kB
SwapCached:       3064 kB
Active:        3049492 kB
Inactive:      1575680 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27996624 kB
SwapTotal:    67111528 kB
SwapFree:     67105204 kB
Dirty:            7516 kB
Writeback:           0 kB
AnonPages:     2948576 kB
Mapped:          15184 kB
Slab:           260972 kB
PageTables:      10572 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  3219140 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72652 MiB
End job on node147 at 2011-04-29 21:55:32