Trace number 3251396

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.35 1204.25

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m47-q841.5-n45-p15-s16393788.cnf
MD5SUM683c9f75b69575e6451b8563d12f176b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.85272
Satisfiable
(Un)Satisfiability was proved
Number of variables1981
Number of clauses333838
Sum of the clauses size669567
Maximum clause length47
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2333793
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.19/0.23	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251396-1304107772.cnf 
0.19/0.23	c Using SATTIMEOUT of 4800
0.19/0.23	c Using SATRAM of 15500
0.19/0.23	c Size of subset node = 32
0.19/0.23	c Variables = 1981
0.19/0.23	c Clauses = 333838
0.19/0.23	c Literals = 669567
0.19/0.23	c Reduced size:
0.19/0.23	c Variables = 1981
0.19/0.23	c Clauses = 333838
0.19/0.23	c Literals = 669567
0.19/0.23	c a2
0.39/0.41	c a1
0.89/0.95	c Independent sub-problems = 1
0.89/0.95	c Sub-problem with 1981 variables and 333838 clauses
0.89/0.95	c using permutation method 4
0.89/0.95	c count = 1
0.89/0.95	c Ordering finished 0.900 seconds
0.89/0.95	c mapped 0.940 seconds
0.89/0.95	c permuted 0.940 seconds
0.89/0.95	c about to sort
0.89/0.98	c sorted
1.99/2.01	c clauses now 333838
1.99/2.01	c init finished 1.060 seconds
1.99/2.01	c tree built 2.010 seconds
1.99/2.01	c 
1.99/2.01	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.99/2.01	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.99/2.01	c ................................ | ..........                    |                                  2.0         0.0             
1.99/2.01	c ................................ | ..........                  1 |                                  2.0         0.5             
1.99/2.01	c ................................ | ..........                  2 |                                  2.0         1.0             
1.99/2.01	c ................................ | ..........                  3 |                                  2.0         1.5             
1.99/2.01	c ................................ | ..........                  4 |                                  2.0         2.0             
1.99/2.01	c ................................ | ..........                  5 |                                  2.0         2.5             
1.99/2.01	c ................................ | ..........               1000 |              90                  2.0       495.0         3899
95.80/25.55	c ................................ | ..........            5038000 | 9124         469731             96.0     52479.2     19026337
191.69/49.58	c ................................ | ..........            9925000 | 17266        921516            192.0     51692.7     33631635
287.94/73.62	c ................................ | ..........           14464000 | 20597        1351368           288.0     50222.2     42524893
383.76/97.66	c ................................ | ..........           18818000 | 28725        1768355           384.0     49003.9     47497159
480.00/121.70	c ................................ | *.........           23016000 | 36710        2174390           480.0     47950.0     53347318
575.86/145.73	c ................................ | *.........           27248000 | 46090        2579443           576.0     47305.6     62665810
671.69/169.78	c ................................ | *.........           31296000 | 55420        2971990           672.0     46570.0     72649011
767.90/193.82	c ................................ | *.........           35327000 | 63897        3363255           768.0     45998.1     82082867
863.76/217.86	c ................................ | *.........           39591000 | 71908        3755489           864.0     45822.9     91260166
959.99/241.90	c ................................ | **........           43837000 | 79652        4122694           960.0     45663.5     94210091
1056.64/266.11	c ................................ | **........           48050000 | 86623        4496299          1056.7     45473.9    101472366
1151.99/290.00	c ................................ | **........           52000000 | 96019        4857453          1152.0     45138.5    106945745
1248.24/314.11	c ................................ | **........           55820000 | 105889       5207841          1248.3     44717.2    111889595
1343.64/338.09	c ................................ | **........           59975000 | 115335       5596368          1344.0     44624.3    116577195
1439.82/362.14	c ................................ | ***.......           64210000 | 124365       5974032          1440.0     44590.0    119220516
1535.65/386.19	c ................................ | ***.......           68222000 | 130079       6347965          1536.0     44415.1    121819164
1631.89/410.23	c ................................ | ***.......           72226000 | 139866       6720145          1632.0     44256.1    127692234
1727.70/434.27	c ................................ | ***.......           76160000 | 150977       7092345          1728.0     44073.8    135920573
1823.94/458.31	c ................................ | ***.......           80300000 | 157701       7452871          1824.0     44024.1    139384076
1919.76/482.36	c ................................ | ****......           84375000 | 162756       7810923          1920.0     43945.1    143146886
2016.00/506.40	c ................................ | ****......           88371000 | 167340       8164854          2016.0     43834.6    149667856
2111.82/530.45	c ................................ | ****......           92350000 | 174567       8522022          2112.1     43725.3    159292711
2208.00/554.50	c ................................ | ****......           95965000 | 181753       8858831          2208.0     43462.2    159292711
2303.82/578.55	c ................................ | ****......           99764000 | 188459       9232210          2304.0     43300.2    166035444
2399.63/602.59	c ................................ | *****.....          103880000 | 195438       9597740          2400.0     43283.2    171238544
2495.80/626.65	c ................................ | *****.....          107983000 | 199827       9962563          2496.0     43262.2    175076462
2591.61/650.69	c ................................ | *****.....          111950000 | 203265       10320737         2592.0     43190.6    180359345
2687.81/674.74	c ................................ | *****.....          115725000 | 209185       10680048         2688.0     43052.5    184122110
2783.66/698.79	c ................................ | *****.....          119480000 | 214785       11044073         2784.0     42916.4    190686726
2879.87/722.83	c ................................ | ******....          123559000 | 221346       11404557         2880.0     42902.4    197045435
2975.69/746.88	c ................................ | ******....          127747000 | 229392       11758890         2976.0     42925.7    206792293
3071.87/770.93	c ................................ | ******....          131696000 | 236268       12094456         3072.0     42869.7    211306465
3167.97/795.01	c ................................ | ******....          135669000 | 241677       12432132         3168.0     42824.7    216213642
3263.72/819.07	c ................................ | ******....          139559000 | 247802       12781112         3264.0     42756.8    223495281
3359.94/843.11	c ................................ | *******...          143183000 | 254653       13133862         3360.0     42614.0    225363471
3455.74/867.16	c ................................ | *******...          146691000 | 262209       13479246         3456.0     42445.3    226882358
3551.92/891.22	c ................................ | *******...          150123000 | 268592       13823263         3552.0     42264.4    226882358
3647.75/915.27	c ................................ | *******...          153524000 | 276856       14163977         3648.0     42084.3    227410188
3743.94/939.32	c ................................ | *******...          157689000 | 284228       14538345         3744.0     42117.7    236143479
3839.62/963.39	c ................................ | ********..          161438000 | 290469       14881846         3840.0     42040.9    236419561
3935.82/987.45	c ................................ | ********..          165351000 | 294904       15231382         3936.0     42009.7    240592390
4031.62/1011.49	c ................................ | ********..          169279000 | 300633       15583718         4032.0     41983.9    249156473
4127.84/1035.54	c ................................ | ********..          173027000 | 307434       15922668         4128.0     41915.5    255323087
4223.64/1059.59	c ................................ | ********..          176543000 | 313652       16254237         4224.0     41795.1    258315204
4319.86/1083.63	c ................................ | *********.          180083000 | 320468       16576270         4320.0     41685.8    263293827
4416.01/1107.70	c ................................ | *********.          183780000 | 328391       16921654         4416.0     41616.8    266144163
4511.80/1131.75	c ................................ | *********.          187564000 | 334660       17270190         4512.0     41569.9    271091056
4607.96/1155.81	c ................................ | *********.          191124000 | 341545       17609280         4608.0     41476.6    274265692
4705.31/1180.25	c ................................ | *********.          194510000 | 348763       17940475         4705.5     41336.7    275492460
4799.89/1203.93	c CPU limit (4800) reached
4799.89/1203.93	c Time = 4800.03 seconds
4799.89/1203.93	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251396-1304107772/watcher-3251396-1304107772 -o /tmp/evaluation-result-3251396-1304107772/solver-3251396-1304107772 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251396-1304107772.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.20 8.15 8.10 5/185 11628
/proc/meminfo: memFree=23097596/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=9768 CPUtime=0 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) R 11626 11628 11448 0 -1 4202496 491 0 0 0 0 0 0 0 25 0 1 0 522375412 10002432 433 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516834048 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11628/statm: 2442 434 147 36 0 347 0

[startup+0.0194519 s]
/proc/loadavg: 8.20 8.15 8.10 5/185 11628
/proc/meminfo: memFree=23097596/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=12468 CPUtime=0.01 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) R 11626 11628 11448 0 -1 4202496 1165 0 0 0 1 0 0 0 25 0 1 0 522375412 12767232 1107 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11628/statm: 3117 1109 148 36 0 1022 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 12468

[startup+0.100441 s]
/proc/loadavg: 8.20 8.15 8.10 5/185 11628
/proc/meminfo: memFree=23097596/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=28816 CPUtime=0.09 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) R 11626 11628 11448 0 -1 4202496 5193 0 0 0 9 0 0 0 25 0 1 0 522375412 29507584 5135 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11628/statm: 7204 5137 148 36 0 5109 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28816

[startup+0.300402 s]
/proc/loadavg: 8.20 8.15 8.10 5/185 11628
/proc/meminfo: memFree=23097596/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=76548 CPUtime=0.29 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) R 11626 11628 11448 0 -1 4202496 16065 0 0 0 28 1 0 0 25 0 1 0 522375412 78385152 16006 33554432000 4194304 4338100 140734358368048 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11628/statm: 19137 16006 153 36 0 17042 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 76548

[startup+0.70033 s]
/proc/loadavg: 8.20 8.15 8.10 5/185 11628
/proc/meminfo: memFree=23097596/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=77660 CPUtime=0.69 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) R 11626 11628 11448 0 -1 4202496 20784 0 0 0 67 2 0 0 25 0 1 0 522375412 79523840 17426 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516495045 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11628/statm: 19415 17426 155 36 0 17320 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 77660

[startup+1.50019 s]
/proc/loadavg: 7.94 8.10 8.08 6/186 11629
/proc/meminfo: memFree=22997152/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=131828 CPUtime=1.49 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) R 11626 11628 11448 0 -1 4202496 34385 0 0 0 145 4 0 0 25 0 1 0 522375412 134991872 30374 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516492091 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/11628/statm: 32957 30374 164 36 0 30862 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 131828

[startup+3.10086 s]
/proc/loadavg: 7.94 8.10 8.08 9/190 11633
/proc/meminfo: memFree=22865928/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=320296 CPUtime=6.34 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 56985 0 0 0 617 17 0 0 25 0 5 0 522375412 327983104 52321 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 80074 52321 176 36 0 77979 0
[pid=11628/tid=11630] ppid=11626 vsize=320296 CPUtime=1.07 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 3004 0 0 0 107 0 0 0 25 0 5 0 522375613 327983104 52322 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=320296 CPUtime=1.08 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1510 0 0 0 103 5 0 0 19 0 5 0 522375613 327983104 52322 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229172 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=320296 CPUtime=1.07 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 3027 0 0 0 107 0 0 0 25 0 5 0 522375613 327983104 52323 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229252 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=320296 CPUtime=1.07 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1760 0 0 0 102 5 0 0 19 0 5 0 522375613 327983104 52323 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.34
Current children cumulated vsize (KiB) 320296

[startup+6.30027 s]
/proc/loadavg: 7.94 8.10 8.08 9/190 11633
/proc/meminfo: memFree=22696048/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=403120 CPUtime=19.12 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 78903 0 0 0 1863 49 0 0 25 0 5 0 522375412 412794880 74239 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 100780 74242 176 36 0 98685 0
[pid=11628/tid=11630] ppid=11626 vsize=403120 CPUtime=4.27 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 10429 0 0 0 425 2 0 0 25 0 5 0 522375613 412794880 74242 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=403120 CPUtime=4.27 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 4803 0 0 0 407 20 0 0 25 0 5 0 522375613 412794880 74242 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=403120 CPUtime=4.27 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 10803 0 0 0 427 0 0 0 25 0 5 0 522375613 412794880 74242 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=403120 CPUtime=4.25 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 5186 0 0 0 406 19 0 0 25 0 5 0 522375613 412794880 74242 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254041 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 19.12
Current children cumulated vsize (KiB) 403120

[startup+12.7011 s]
/proc/loadavg: 7.95 8.10 8.08 9/190 11633
/proc/meminfo: memFree=22468272/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=631156 CPUtime=44.68 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 124799 0 0 0 4355 113 0 0 25 0 5 0 522375412 646303744 120135 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 157789 120135 176 36 0 155694 0
[pid=11628/tid=11630] ppid=11626 vsize=631156 CPUtime=10.67 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 26753 0 0 0 1061 6 0 0 25 0 5 0 522375613 646303744 120135 33554432000 4194304 4338100 140734358368048 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=631156 CPUtime=10.67 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 11372 0 0 0 1016 51 0 0 25 0 5 0 522375613 646303744 120136 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=631156 CPUtime=10.67 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 26621 0 0 0 1064 3 0 0 25 0 5 0 522375613 646303744 120136 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516530912 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=631156 CPUtime=10.61 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 12369 0 0 0 1015 46 0 0 25 0 5 0 522375613 646303744 120136 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229159 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 44.68
Current children cumulated vsize (KiB) 631156

[startup+25.5008 s]
/proc/loadavg: 7.96 8.09 8.08 9/190 11633
/proc/meminfo: memFree=22031324/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=951248 CPUtime=95.8 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 200229 0 0 0 9319 261 0 0 25 0 5 0 522375412 974077952 195565 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 237812 195566 176 36 0 235717 0
[pid=11628/tid=11630] ppid=11626 vsize=951248 CPUtime=23.47 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 54345 0 0 0 2333 14 0 0 25 0 5 0 522375613 974077952 195566 33554432000 4194304 4338100 140734358368048 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=951248 CPUtime=23.47 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 19928 0 0 0 2226 121 0 0 25 0 5 0 522375613 974077952 195566 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=951248 CPUtime=23.43 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 55199 0 0 0 2335 8 0 0 25 0 5 0 522375613 974077952 195566 33554432000 4194304 4338100 140734358368048 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=951248 CPUtime=23.38 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 23073 0 0 0 2227 111 0 0 25 0 5 0 522375613 974077952 195566 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 95.8
Current children cumulated vsize (KiB) 951248

[startup+51.1011 s]
/proc/loadavg: 8.09 8.11 8.09 9/190 11634
/proc/meminfo: memFree=21344092/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=1440812 CPUtime=198.06 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 322261 0 0 0 19239 567 0 0 25 0 5 0 522375412 1475391488 317597 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 360203 317597 176 36 0 358108 0
[pid=11628/tid=11630] ppid=11626 vsize=1440812 CPUtime=49.07 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 102916 0 0 0 4878 29 0 0 25 0 5 0 522375613 1475391488 317597 33554432000 4194304 4338100 140734358368048 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=1440812 CPUtime=49.07 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 29791 0 0 0 4643 264 0 0 25 0 5 0 522375613 1475391488 317597 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=1440812 CPUtime=48.89 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 102897 0 0 0 4872 17 0 0 25 0 5 0 522375613 1475391488 317597 33554432000 4194304 4338100 140734358368048 18446744073709551615 4229565 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=1440812 CPUtime=48.98 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 38972 0 0 0 4648 250 0 0 25 0 5 0 522375613 1475391488 317597 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 198.06
Current children cumulated vsize (KiB) 1440812

[startup+102.307 s]
/proc/loadavg: 8.16 8.12 8.09 9/190 11635
/proc/meminfo: memFree=20379340/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=2018960 CPUtime=402.55 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 454781 0 0 0 39095 1160 0 0 25 0 5 0 522375412 2067415040 450117 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 504740 450117 176 36 0 502645 0
[pid=11628/tid=11630] ppid=11626 vsize=2018960 CPUtime=100.27 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 158477 0 0 0 9975 52 0 0 25 0 5 0 522375613 2067415040 450117 33554432000 4194304 4338100 140734358368048 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=2018960 CPUtime=100.1 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 40563 0 0 0 9460 550 0 0 25 0 5 0 522375613 2067415040 450117 33554432000 4194304 4338100 140734358368048 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=2018960 CPUtime=99.96 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 157372 0 0 0 9968 28 0 0 25 0 5 0 522375613 2067415040 450118 33554432000 4194304 4338100 140734358368048 18446744073709551615 4253920 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=2018960 CPUtime=100.18 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 50684 0 0 0 9495 523 0 0 25 0 5 0 522375613 2067415040 450118 33554432000 4194304 4338100 140734358368048 18446744073709551615 4263456 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 402.55
Current children cumulated vsize (KiB) 2018960

[startup+162.301 s]
/proc/loadavg: 8.24 8.16 8.10 9/190 11637
/proc/meminfo: memFree=19132480/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=2648200 CPUtime=642.14 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 606415 0 0 0 62335 1879 0 0 25 0 5 0 522375412 2711756800 601751 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 662050 601751 176 36 0 659955 0
[pid=11628/tid=11630] ppid=11626 vsize=2648200 CPUtime=160.23 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 236794 0 0 0 15941 82 0 0 25 0 5 0 522375613 2711756800 601751 33554432000 4194304 4338100 140734358368048 18446744073709551615 4263461 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=2648200 CPUtime=159.76 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 40570 0 0 0 15080 896 0 0 25 0 5 0 522375613 2711756800 601752 33554432000 4194304 4338100 140734358368048 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=2648200 CPUtime=159.94 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 230682 0 0 0 15952 42 0 0 25 0 5 0 522375613 2711756800 601752 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=2648200 CPUtime=160.17 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 50685 0 0 0 15164 853 0 0 25 0 5 0 522375613 2711756800 601752 33554432000 4194304 4338100 140734358368048 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 642.14
Current children cumulated vsize (KiB) 2648200

[startup+222.301 s]
/proc/loadavg: 8.16 8.15 8.10 9/190 11639
/proc/meminfo: memFree=17666256/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=3375120 CPUtime=881.72 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 799954 0 0 0 85598 2574 0 0 25 0 5 0 522375412 3456122880 795290 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 843780 795290 176 36 0 841685 0
[pid=11628/tid=11630] ppid=11626 vsize=3375120 CPUtime=220.21 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 332412 0 0 0 21906 115 0 0 25 0 5 0 522375613 3456122880 795291 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 83689 0 0 0 66164 3804 0 0 25 0 5 0 522375613 7132459008 1614743 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2798.05
Current children cumulated vsize (KiB) 6965292

[startup+762.307 s]
/proc/loadavg: 8.11 8.15 8.10 10/190 11653
/proc/meminfo: memFree=8488520/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=7559132 CPUtime=3037.56 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 1762145 0 0 0 294880 8876 0 0 25 0 5 0 522375412 7740551168 1757481 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 1889783 1757481 176 36 0 1887688 0
[pid=11628/tid=11630] ppid=11626 vsize=7559132 CPUtime=758.08 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 871141 0 0 0 75474 334 0 0 25 0 5 0 522375613 7740551168 1757481 33554432000 4194304 4338100 140734358368048 18446744073709551615 4257231 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=7559132 CPUtime=759.25 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 72517 0 0 0 71672 4253 0 0 25 0 5 0 522375613 7740551168 1757481 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=7559132 CPUtime=758.53 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 687110 0 0 0 75715 138 0 0 25 0 5 0 522375613 7740551168 1757481 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=7559132 CPUtime=759.65 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 83692 0 0 0 71821 4144 0 0 25 0 5 0 522375613 7740551168 1757481 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3037.56
Current children cumulated vsize (KiB) 7559132

[startup+822.301 s]
/proc/loadavg: 7.90 8.10 8.08 9/190 11687
/proc/meminfo: memFree=22250348/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=8003612 CPUtime=3276.91 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 1871586 0 0 0 318126 9565 0 0 25 0 5 0 522375412 8195698688 1866922 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2000903 1866922 176 36 0 1998808 0
[pid=11628/tid=11630] ppid=11626 vsize=8003612 CPUtime=817.97 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 940565 0 0 0 81434 363 0 0 25 0 5 0 522375613 8195698688 1866922 33554432000 4194304 4338100 140734358368048 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=8003612 CPUtime=819.18 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 73248 0 0 0 77338 4580 0 0 25 0 5 0 522375613 8195698688 1866922 33554432000 4194304 4338100 140734358368048 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=8003612 CPUtime=818.36 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 725087 0 0 0 81689 147 0 0 25 0 5 0 522375613 8195698688 1866922 33554432000 4194304 4338100 140734358368048 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=8003612 CPUtime=819.36 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 85001 0 0 0 77467 4469 0 0 25 0 5 0 522375613 8195698688 1866922 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229257 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3276.91
Current children cumulated vsize (KiB) 8003612

[startup+882.301 s]
/proc/loadavg: 7.92 8.08 8.07 9/190 11688
/proc/meminfo: memFree=20594696/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=8131956 CPUtime=3516.39 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 1918666 0 0 0 341342 10297 0 0 25 0 5 0 522375412 8327122944 1914002 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2032989 1914002 176 36 0 2030894 0
[pid=11628/tid=11630] ppid=11626 vsize=8131956 CPUtime=877.8 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 964724 0 0 0 87402 378 0 0 25 0 5 0 522375613 8327122944 1914002 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=8131956 CPUtime=879.06 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 84348 0 0 0 82960 4946 0 0 25 0 5 0 522375613 8327122944 1914002 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528648448 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=8131956 CPUtime=878.25 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 726726 0 0 0 87678 147 0 0 25 0 5 0 522375613 8327122944 1914002 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229172 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=8131956 CPUtime=879.24 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 95183 0 0 0 83105 4819 0 0 25 0 5 0 522375613 8327122944 1914002 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528660149 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3516.39
Current children cumulated vsize (KiB) 8131956

[startup+942.301 s]
/proc/loadavg: 8.13 8.12 8.09 9/190 11690
/proc/meminfo: memFree=19352068/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=8408716 CPUtime=3755.9 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 1997054 0 0 0 364547 11043 0 0 25 0 5 0 522375412 8610525184 1992390 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2102179 1992390 176 36 0 2100084 0
[pid=11628/tid=11630] ppid=11626 vsize=8408716 CPUtime=937.74 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1005822 0 0 0 93373 401 0 0 25 0 5 0 522375613 8610525184 1992390 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516530684 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=8408716 CPUtime=939.02 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 86573 0 0 0 88594 5308 0 0 25 0 5 0 522375613 8610525184 1992390 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=8408716 CPUtime=938.21 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 760403 0 0 0 93666 155 0 0 25 0 5 0 522375613 8610525184 1992390 33554432000 4194304 4338100 140734358368048 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=8408716 CPUtime=938.88 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 96571 0 0 0 88716 5172 0 0 25 0 5 0 522375613 8610525184 1992390 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3755.9
Current children cumulated vsize (KiB) 8408716

[startup+1002.3 s]
/proc/loadavg: 8.01 8.08 8.08 9/190 11691
/proc/meminfo: memFree=18312036/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=9073012 CPUtime=3995.28 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 2086360 0 0 0 387757 11771 0 0 25 0 5 0 522375412 9290764288 2081696 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2268253 2081696 176 36 0 2266158 0
[pid=11628/tid=11630] ppid=11626 vsize=9073012 CPUtime=997.68 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1054438 0 0 0 99344 424 0 0 25 0 5 0 522375613 9290764288 2081697 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229601 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=9073012 CPUtime=998.75 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 92796 0 0 0 94219 5656 0 0 25 0 5 0 522375613 9290764288 2081697 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=9073012 CPUtime=998.13 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 789148 0 0 0 99652 161 0 0 25 0 5 0 522375613 9290764288 2081697 33554432000 4194304 4338100 140734358368048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=9073012 CPUtime=998.68 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 102294 0 0 0 94345 5523 0 0 25 0 5 0 522375613 9290764288 2081697 33554432000 4194304 4338100 140734358368048 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3995.28
Current children cumulated vsize (KiB) 9073012

[startup+1062.3 s]
/proc/loadavg: 8.11 8.10 8.09 9/190 11693
/proc/meminfo: memFree=17161756/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=9493868 CPUtime=4234.81 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 2187851 0 0 0 410974 12507 0 0 25 0 5 0 522375412 9721720832 2183187 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2373467 2183187 176 36 0 2371372 0
[pid=11628/tid=11630] ppid=11626 vsize=9493868 CPUtime=1057.63 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1108359 0 0 0 105316 447 0 0 25 0 5 0 522375613 9721720832 2183187 33554432000 4194304 4338100 140734358368048 18446744073709551615 4283481 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=9493868 CPUtime=1058.39 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 92861 0 0 0 99831 6008 0 0 25 0 5 0 522375613 9721720832 2183187 33554432000 4194304 4338100 140734358368048 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=9493868 CPUtime=1058.12 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 836652 0 0 0 105637 175 0 0 25 0 5 0 522375613 9721720832 2183187 33554432000 4194304 4338100 140734358368048 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=9493868 CPUtime=1058.64 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 102294 0 0 0 99992 5872 0 0 25 0 5 0 522375613 9721720832 2183187 33554432000 4194304 4338100 140734358368048 18446744073709551615 4285738 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4234.81
Current children cumulated vsize (KiB) 9493868

[startup+1122.3 s]
/proc/loadavg: 8.04 8.08 8.08 9/190 11695
/proc/meminfo: memFree=16087600/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=9791412 CPUtime=4474.27 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 2267121 0 0 0 434145 13282 0 0 25 0 5 0 522375412 10026405888 2262457 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2447853 2262457 176 36 0 2445758 0
[pid=11628/tid=11630] ppid=11626 vsize=9791412 CPUtime=1117.55 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1152106 0 0 0 111287 468 0 0 25 0 5 0 522375613 10026405888 2262457 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229506 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=9791412 CPUtime=1118.01 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 99337 0 0 0 105420 6381 0 0 25 0 5 0 522375613 10026405888 2262457 33554432000 4194304 4338100 140734358368048 18446744073709551615 4279576 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=9791412 CPUtime=1118.07 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 862234 0 0 0 111627 180 0 0 25 0 5 0 522375613 10026405888 2262457 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=9791412 CPUtime=1118.61 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 105759 0 0 0 105613 6248 0 0 25 0 5 0 522375613 10026405888 2262457 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229262 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4474.27
Current children cumulated vsize (KiB) 9791412

[startup+1182.3 s]
/proc/loadavg: 8.34 8.17 8.11 9/190 11696
/proc/meminfo: memFree=15105848/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=10013236 CPUtime=4713.66 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) S 11626 11628 11448 0 -1 4202496 2344649 0 0 0 457324 14042 0 0 25 0 5 0 522375412 10253553664 2339985 33554432000 4194304 4338100 140734358368048 18446744073709551615 246528636901 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 2503309 2339985 176 36 0 2501214 0
[pid=11628/tid=11630] ppid=11626 vsize=10013236 CPUtime=1177.44 cores=1,3,5,7
/proc/11628/task/11630/stat : 11630 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 1187109 0 0 0 117258 486 0 0 25 0 5 0 522375613 10253553664 2339985 33554432000 4194304 4338100 140734358368048 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=10013236 CPUtime=1177.72 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 108429 0 0 0 111024 6748 0 0 25 0 5 0 522375613 10253553664 2339985 33554432000 4194304 4338100 140734358368048 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=11628/tid=11632] ppid=11626 vsize=10013236 CPUtime=1177.91 cores=1,3,5,7
/proc/11628/task/11632/stat : 11632 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 885815 0 0 0 117606 185 0 0 25 0 5 0 522375613 10253553664 2339985 33554432000 4194304 4338100 140734358368048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=11628/tid=11633] ppid=11626 vsize=10013236 CPUtime=1178.54 cores=1,3,5,7
/proc/11628/task/11633/stat : 11633 (ttsth-5-4) R 11626 11628 11448 0 -1 4202560 115611 0 0 0 111238 6616 0 0 25 0 5 0 522375613 10253553664 2339985 33554432000 4194304 4338100 140734358368048 18446744073709551615 246516229882 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4713.66
Current children cumulated vsize (KiB) 10013236



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204 s]
/proc/loadavg: 8.32 8.17 8.11 6/187 11697
/proc/meminfo: memFree=16787752/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) Z 11626 11628 11448 0 -1 4203548 2344971 0 0 0 465702 14308 0 0 18 0 2 0 522375412 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 0 0 0 0 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=0 CPUtime=1199.41 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4203588 108568 0 0 0 113055 6886 0 0 25 0 2 0 522375613 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.2 s]
/proc/loadavg: 8.32 8.17 8.11 6/187 11697
/proc/meminfo: memFree=16787752/32951124 swapFree=67102692/67111528
[pid=11628] ppid=11626 vsize=0 CPUtime=4800.29 cores=1,3,5,7
/proc/11628/stat : 11628 (ttsth-5-4) Z 11626 11628 11448 0 -1 4203548 2344971 0 0 0 465702 14327 0 0 18 0 2 0 522375412 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/11628/statm: 0 0 0 0 0 0 0
[pid=11628/tid=11631] ppid=11626 vsize=0 CPUtime=1199.61 cores=1,3,5,7
/proc/11628/task/11631/stat : 11631 (ttsth-5-4) R 11626 11628 11448 0 -1 4203588 108568 0 0 0 113055 6906 0 0 25 0 2 0 522375613 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.25
CPU time (s): 4800.35
CPU user time (s): 4657.03
CPU system time (s): 143.326
CPU usage (%): 398.617
Max. virtual memory (cumulated for all children) (KiB): 10013236

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4657.03
system time used= 143.326
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2344971
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12011
involuntary context switches= 35662

runsolver used 1.83072 second user time and 4.74628 second system time

The end

Launcher Data 

Begin job on node134 at 2011-04-29 22:09:32
IDJOB=3251396
IDBENCH=88221
IDSOLVER=1654
FILE ID=node134/3251396-1304107772
RUNJOBID= node134-1304106566-11467
PBS_JOBID= 13170663
Free space on /tmp= 71580 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m47-q841.5-n45-p15-s16393788.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251396-1304107772/watcher-3251396-1304107772 -o /tmp/evaluation-result-3251396-1304107772/solver-3251396-1304107772 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251396-1304107772.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 683c9f75b69575e6451b8563d12f176b
RANDOM SEED=1318163959

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23097752 kB
Buffers:        989880 kB
Cached:        1364192 kB
SwapCached:       4400 kB
Active:        7141244 kB
Inactive:      2300440 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23097752 kB
SwapTotal:    67111528 kB
SwapFree:     67102692 kB
Dirty:            4316 kB
Writeback:           0 kB
AnonPages:     7084820 kB
Mapped:          15252 kB
Slab:           334532 kB
PageTables:      18552 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  7410312 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71572 MiB
End job on node134 at 2011-04-29 22:29:38