Trace number 3259714

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1205.68

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v2400c266431gyes2.cnf
MD5SUMaac2bd287b8cd6b96cbf7ecc463acf19
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data 

0.09/0.18	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259714-1304056189.cnf 
0.09/0.18	c Using SATTIMEOUT of 4800
0.09/0.18	c Using SATRAM of 15500
0.09/0.18	c Size of subset node = 32
0.09/0.18	c Variables = 2400
0.09/0.18	c Clauses = 266431
0.09/0.18	c Literals = 535112
0.09/0.18	c Reduced size:
0.09/0.18	c Variables = 2400
0.09/0.18	c Clauses = 266431
0.09/0.18	c Literals = 535112
0.09/0.18	c a2
0.29/0.33	c a1
0.69/0.70	c Independent sub-problems = 1
0.69/0.70	c Sub-problem with 2400 variables and 266431 clauses
0.69/0.70	c using permutation method 4
0.69/0.70	c count = 1
0.69/0.70	c Ordering finished 0.660 seconds
0.69/0.70	c mapped 0.690 seconds
0.69/0.70	c permuted 0.690 seconds
0.69/0.70	c about to sort
0.69/0.72	c sorted
1.38/1.40	c clauses now 266431
1.38/1.40	c init finished 0.770 seconds
1.38/1.40	c tree built 1.390 seconds
1.38/1.40	c 
1.38/1.40	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.38/1.40	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.38/1.40	c ................................ | ..........                    |                                  1.4         0.0             
1.38/1.40	c ................................ | ..........                  1 |              14                  1.4         0.7             
1.38/1.40	c ................................ | ..........                  2 |              17                  1.4         1.4             
1.38/1.40	c ................................ | ..........                  3 |              18                  1.4         2.2             
1.38/1.40	c ................................ | ..........                  4 |              18                  1.4         2.9             
1.38/1.40	c ................................ | ..........                  5 |              18                  1.4         3.6             
1.38/1.40	c ................................ | ..........               1000 |              73                  1.4       709.2         1434
95.69/25.18	c ................................ | ..........            3898000 | 54749        264810             96.0     40604.2     12413996
191.63/49.29	c ................................ | ..........            7488000 | 101377       508135            192.0     38998.0     21018414
287.97/73.40	c ................................ | ..........           10629000 | 138516       731048            288.0     36906.2     29366395
383.93/97.52	c ................................ | ..........           13404000 | 179676       908486            384.0     34904.4     36256643
479.92/121.62	c ................................ | *.........           16000000 | 216124       1081198           480.0     33331.9     41285294
575.91/145.72	c ................................ | *.........           18362000 | 248329       1240178           576.0     31877.4     47034084
671.91/169.82	c ................................ | *.........           20648000 | 286327       1373986           672.0     30725.3     51899186
767.96/193.91	c ................................ | *.........           23077000 | 320795       1513293           768.0     30047.4     58374815
863.95/218.01	c ................................ | *.........           25272000 | 356533       1657490           864.0     29249.7     60102700
959.96/242.11	c ................................ | **........           27198000 | 391225       1789524           960.0     28331.0     63523338
1056.00/266.21	c ................................ | **........           29240000 | 422952       1948195          1056.0     27688.6     69527301
1152.03/290.30	c ................................ | **........           31392000 | 455774       2098342          1152.0     27249.1     77568850
1247.66/314.39	c ................................ | **........           33599000 | 488508       2242126          1248.0     26922.1     83689478
1343.70/338.48	c ................................ | **........           35470000 | 526315       2365978          1344.0     26390.6     87902345
1439.73/362.57	c ................................ | ***.......           37715000 | 558473       2513155          1440.0     26190.6     90720060
1535.74/386.67	c ................................ | ***.......           39771000 | 595546       2637131          1536.0     25892.2     90823755
1631.78/410.75	c ................................ | ***.......           41612000 | 628508       2758000          1632.0     25497.5     93189233
1727.77/434.86	c ................................ | ***.......           43380000 | 659590       2880798          1728.0     25103.6     96659357
1823.78/458.96	c ................................ | ***.......           45155000 | 691417       3000299          1824.0     24755.6    100528125
1919.81/483.05	c ................................ | ****......           47143000 | 719761       3131202          1920.0     24553.1    104689066
2015.84/507.14	c ................................ | ****......           48810000 | 748154       3244748          2016.0     24211.3    107377417
2111.85/531.24	c ................................ | ****......           50590000 | 773767       3362104          2112.0     23953.4    107712372
2207.89/555.32	c ................................ | ****......           52283000 | 802194       3475840          2208.0     23678.9    108690065
2303.96/579.42	c ................................ | ****......           54345000 | 832861       3596220          2304.0     23586.8    110494315
2400.00/603.50	c ................................ | *****.....           56360000 | 862390       3705456          2400.0     23483.1    112639003
2496.01/627.60	c ................................ | *****.....           58292000 | 892554       3814704          2496.0     23354.1    116403427
2592.04/651.70	c ................................ | *****.....           60011000 | 922287       3908342          2592.1     23151.9    118328529
2687.68/675.79	c ................................ | *****.....           61448000 | 946995       3992798          2688.0     22859.9    121105040
2783.73/699.88	c ................................ | *****.....           62872000 | 973319       4081321          2784.1     22582.9    124262168
2879.83/723.94	c ................................ | ******....           64589000 | 1000493      4179275          2880.0     22426.6    125701230
2975.66/747.99	c ................................ | ******....           66400000 | 1029806      4274625          2976.0     22311.8    129291159
3071.83/772.04	c ................................ | ******....           68482000 | 1058758      4386265          3072.0     22292.2    133756622
3167.68/796.08	c ................................ | ******....           70317000 | 1086751      4491846          3168.0     22196.0    140472167
3263.91/820.12	c ................................ | ******....           72191000 | 1108439      4600988          3264.0     22117.3    147607645
3359.74/844.16	c ................................ | *******...           73869000 | 1135513      4701952          3360.0     21984.8    148665466
3455.98/868.21	c ................................ | *******...           75386000 | 1160099      4798962          3456.1     21812.8    149577716
3551.79/892.26	c ................................ | *******...           77037000 | 1186213      4900089          3552.1     21687.9    152631338
3648.03/916.31	c ................................ | *******...           78572000 | 1207254      5008529          3648.1     21538.1    155487201
3743.70/940.37	c ................................ | *******...           80140000 | 1227317      5121465          3744.0     21404.9    160315486
3839.80/964.46	c ................................ | ********..           81611000 | 1248368      5229134          3840.1     21252.6    165391471
3935.82/988.55	c ................................ | ********..           83595000 | 1276239      5351742          3936.0     21238.4    168859086
4031.87/1012.64	c ................................ | ********..           85465000 | 1305283      5457042          4032.0     21196.6    171814726
4127.85/1036.74	c ................................ | ********..           87325000 | 1332643      5572037          4128.0     21154.2    175167534
4223.91/1060.82	c ................................ | ********..           89019000 | 1364913      5673417          4224.0     21074.5    179906657
4319.98/1084.91	c ................................ | *********.           90588000 | 1398804      5764168          4320.0     20969.3    184356495
4416.04/1109.00	c ................................ | *********.           91876000 | 1427560      5860805          4416.1     20805.0    186633668
4511.74/1133.08	c ................................ | *********.           93156000 | 1456372      5964898          4512.0     20646.1    186633668
4607.74/1157.18	c ................................ | *********.           94548000 | 1482191      6067305          4608.1     20518.0    186633668
4703.68/1181.29	c ................................ | *********.           96100000 | 1508139      6178315          4704.0     20429.2    186633668
4799.96/1205.41	c CPU limit (4800) reached
4799.96/1205.41	c Time = 4800.00 seconds
4799.96/1205.41	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259714-1304056189/watcher-3259714-1304056189 -o /tmp/evaluation-result-3259714-1304056189/solver-3259714-1304056189 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259714-1304056189.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.05 8.09 8.04 3/177 30793
/proc/meminfo: memFree=20447124/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=9760 CPUtime=0 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) R 30791 30793 30360 0 -1 4202496 488 0 0 0 0 0 0 0 25 0 1 0 595849620 9994240 430 33554432000 4194304 4338100 140734785060896 18446744073709551615 231895490304 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30793/statm: 2440 430 147 36 0 345 0

[startup+0.0415991 s]
/proc/loadavg: 8.05 8.09 8.04 3/177 30793
/proc/meminfo: memFree=20447124/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=16820 CPUtime=0.03 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) R 30791 30793 30360 0 -1 4202496 2241 0 0 0 3 0 0 0 25 0 1 0 595849620 17223680 2183 33554432000 4194304 4338100 140734785060896 18446744073709551615 231895151301 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30793/statm: 4205 2185 148 36 0 2110 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 16820

[startup+0.100589 s]
/proc/loadavg: 8.05 8.09 8.04 3/177 30793
/proc/meminfo: memFree=20447124/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=28676 CPUtime=0.09 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) R 30791 30793 30360 0 -1 4202496 5161 0 0 0 9 0 0 0 25 0 1 0 595849620 29364224 5103 33554432000 4194304 4338100 140734785060896 18446744073709551615 231895151301 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30793/statm: 7169 5105 148 36 0 5074 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28676

[startup+0.300552 s]
/proc/loadavg: 8.05 8.09 8.04 3/177 30793
/proc/meminfo: memFree=20447124/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=61324 CPUtime=0.29 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) R 30791 30793 30360 0 -1 4202496 13327 0 0 0 28 1 0 0 25 0 1 0 595849620 62795776 13268 33554432000 4194304 4338100 140734785060896 18446744073709551615 231894885485 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30793/statm: 15331 13268 153 36 0 13236 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 61324

[startup+0.700482 s]
/proc/loadavg: 8.05 8.09 8.04 3/177 30793
/proc/meminfo: memFree=20447124/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=68104 CPUtime=0.69 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) R 30791 30793 30360 0 -1 4202496 16162 0 0 0 68 1 0 0 25 0 1 0 595849620 69738496 15061 33554432000 4194304 4338100 140734785060896 18446744073709551615 4228256 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30793/statm: 17026 15061 161 36 0 14931 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 68104

[startup+1.50022 s]
/proc/loadavg: 7.57 7.99 8.01 3/178 30794
/proc/meminfo: memFree=20237492/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=363160 CPUtime=1.77 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 46061 0 0 0 171 6 0 0 25 0 5 0 595849620 371875840 43917 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 90790 43918 176 36 0 88695 0
Current children cumulated CPU time (s) 1.77
Current children cumulated vsize (KiB) 363160

[startup+3.10091 s]
/proc/loadavg: 7.57 7.99 8.01 10/186 30802
/proc/meminfo: memFree=20085776/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=395364 CPUtime=8.14 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 55111 0 0 0 804 10 0 0 25 0 5 0 595849620 404852736 52967 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 98841 52967 176 36 0 96746 0
[pid=30793/tid=30799] ppid=30791 vsize=395364 CPUtime=1.68 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 3214 0 0 0 167 1 0 0 20 0 5 0 595849761 404852736 52967 33554432000 4194304 4338100 140734785060896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=395364 CPUtime=1.67 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 2141 0 0 0 166 1 0 0 18 0 5 0 595849761 404852736 52967 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=395364 CPUtime=1.68 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 3356 0 0 0 168 0 0 0 20 0 5 0 595849761 404852736 52967 33554432000 4194304 4338100 140734785060896 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=395364 CPUtime=1.68 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 2090 0 0 0 166 2 0 0 20 0 5 0 595849761 404852736 52967 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 8.14
Current children cumulated vsize (KiB) 395364

[startup+6.30032 s]
/proc/loadavg: 7.57 7.99 8.01 9/186 30802
/proc/meminfo: memFree=19912072/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=451520 CPUtime=20.87 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 68629 0 0 0 2068 19 0 0 25 0 5 0 595849620 462356480 66485 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 112880 66485 176 36 0 110785 0
[pid=30793/tid=30799] ppid=30791 vsize=451520 CPUtime=4.86 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 8178 0 0 0 484 2 0 0 25 0 5 0 595849761 462356480 66485 33554432000 4194304 4338100 140734785060896 18446744073709551615 4241600 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=451520 CPUtime=4.8 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 4637 0 0 0 478 2 0 0 18 0 5 0 595849761 462356480 66485 33554432000 4194304 4338100 140734785060896 18446744073709551615 4253946 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=451520 CPUtime=4.88 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 7078 0 0 0 487 1 0 0 25 0 5 0 595849761 462356480 66485 33554432000 4194304 4338100 140734785060896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=451520 CPUtime=4.88 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 4426 0 0 0 481 7 0 0 18 0 5 0 595849761 462356480 66485 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 20.87
Current children cumulated vsize (KiB) 451520

[startup+12.7011 s]
/proc/loadavg: 7.79 8.02 8.02 9/186 30802
/proc/meminfo: memFree=19634864/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=558584 CPUtime=46.35 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 95425 0 0 0 4601 34 0 0 25 0 5 0 595849620 571990016 93281 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 139646 93281 176 36 0 137551 0
[pid=30793/tid=30799] ppid=30791 vsize=558584 CPUtime=11.22 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 17473 0 0 0 1118 4 0 0 25 0 5 0 595849761 571990016 93281 33554432000 4194304 4338100 140734785060896 18446744073709551615 231895186912 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=558584 CPUtime=11.12 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 8964 0 0 0 1108 4 0 0 18 0 5 0 595849761 571990016 93281 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=558584 CPUtime=11.29 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 16609 0 0 0 1126 3 0 0 25 0 5 0 595849761 571990016 93281 33554432000 4194304 4338100 140734785060896 18446744073709551615 4256025 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=558584 CPUtime=11.29 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 8069 0 0 0 1112 17 0 0 25 0 5 0 595849761 571990016 93281 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 46.35
Current children cumulated vsize (KiB) 558584

[startup+25.5007 s]
/proc/loadavg: 7.89 8.04 8.02 9/186 30803
/proc/meminfo: memFree=19127208/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=756928 CPUtime=97.27 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 145529 0 0 0 9663 64 0 0 25 0 5 0 595849620 775094272 143385 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 189232 143385 176 36 0 187137 0
[pid=30793/tid=30799] ppid=30791 vsize=756928 CPUtime=23.95 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 31171 0 0 0 2387 8 0 0 25 0 5 0 595849761 775094272 143385 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254816 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=756928 CPUtime=23.72 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 16262 0 0 0 2365 7 0 0 18 0 5 0 595849761 775094272 143385 33554432000 4194304 4338100 140734785060896 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=756928 CPUtime=24.08 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 38998 0 0 0 2402 6 0 0 25 0 5 0 595849761 775094272 143385 33554432000 4194304 4338100 140734785060896 18446744073709551615 231894885606 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=756928 CPUtime=24.08 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 14788 0 0 0 2373 35 0 0 25 0 5 0 595849761 775094272 143386 33554432000 4194304 4338100 140734785060896 18446744073709551615 231895186864 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 97.27
Current children cumulated vsize (KiB) 756928

[startup+51.1009 s]
/proc/loadavg: 8.06 8.07 8.03 9/186 30803
/proc/meminfo: memFree=18372460/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=1049824 CPUtime=199.19 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 217862 0 0 0 19798 121 0 0 25 0 5 0 595849620 1075019776 215718 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 262456 215718 176 36 0 260361 0
[pid=30793/tid=30799] ppid=30791 vsize=1049824 CPUtime=49.41 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 59009 0 0 0 4925 16 0 0 25 0 5 0 595849761 1075019776 215718 33554432000 4194304 4338100 140734785060896 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=1049824 CPUtime=48.98 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 25708 0 0 0 4884 14 0 0 18 0 5 0 595849761 1075019776 215718 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=1049824 CPUtime=49.69 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 65978 0 0 0 4957 12 0 0 25 0 5 0 595849761 1075019776 215718 33554432000 4194304 4338100 140734785060896 18446744073709551615 231895186987 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=1049824 CPUtime=49.69 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 22857 0 0 0 4896 73 0 0 25 0 5 0 595849761 1075019776 215718 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 199.19
Current children cumulated vsize (KiB) 1049824

[startup+102.306 s]
/proc/loadavg: 8.08 8.07 8.03 9/186 30805
/proc/meminfo: memFree=17324016/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=1548604 CPUtime=403.06 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 342363 0 0 0 40073 233 0 0 25 0 5 0 595849620 1585770496 340219 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 387151 340219 176 36 0 385056 0
[pid=30793/tid=30799] ppid=30791 vsize=1548604 CPUtime=100.34 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 115870 0 0 0 9999 35 0 0 25 0 5 0 595849761 1585770496 340219 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=1548604 CPUtime=99.51 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 40667 0 0 0 9924 27 0 0 18 0 5 0 595849761 1585770496 340219 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=1548604 CPUtime=100.89 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 103000 0 0 0 10070 19 0 0 25 0 5 0 595849761 1585770496 340219 33554432000 4194304 4338100 140734785060896 18446744073709551615 231894885810 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=1548604 CPUtime=100.89 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 38516 0 0 0 9943 146 0 0 25 0 5 0 595849761 1585770496 340219 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 403.06
Current children cumulated vsize (KiB) 1548604

[startup+162.301 s]
/proc/loadavg: 8.09 8.07 8.03 9/186 30806
/proc/meminfo: memFree=16370692/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=1964196 CPUtime=642.04 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 445748 0 0 0 63861 343 0 0 25 0 5 0 595849620 2011336704 442101 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 491049 442101 176 36 0 488954 0
[pid=30793/tid=30799] ppid=30791 vsize=1964196 CPUtime=160.01 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 166810 0 0 0 15949 52 0 0 25 0 5 0 595849761 2011336704 442101 33554432000 4194304 4338100 140734785060896 18446744073709551615 231894886126 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=1964196 CPUtime=158.82 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 50616 0 0 0 15843 39 0 0 18 0 5 0 595849761 2011336704 442101 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=1964196 CPUtime=160.88 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 135816 0 0 0 16063 25 0 0 25 0 5 0 595849761 2011336704 442101 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=1964196 CPUtime=160.88 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 48196 0 0 0 15868 220 0 0 25 0 5 0 595849761 2011336704 442101 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 642.04
Current children cumulated vsize (KiB) 1964196

[startup+222.301 s]
/proc/loadavg: 8.08 8.07 8.03 9/186 30808
/proc/meminfo: memFree=15679980/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=2311124 CPUtime=881.07 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 531439 0 0 0 87664 443 0 0 25 0 5 0 595849620 2366590976 527792 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 577781 527792 176 36 0 575686 0
[pid=30793/tid=30799] ppid=30791 vsize=2311124 CPUtime=219.66 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 204254 0 0 0 21899 67 0 0 25 0 5 0 595849761 2366590976 527792 33554432000 4194304 4338100 140734785060896 18446744073709551615 4288444 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 82723 0 0 0 69284 538 0 0 25 0 5 0 595849761 4483551232 1037762 33554432000 4194304 4338100 140734785060896 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2793.7
Current children cumulated vsize (KiB) 4378468

[startup+762.307 s]
/proc/loadavg: 8.14 8.11 8.05 9/186 30831
/proc/meminfo: memFree=10579504/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=4625580 CPUtime=3033.12 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1102845 0 0 0 301975 1337 0 0 25 0 5 0 595849620 4736593920 1099198 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1156395 1099198 176 36 0 1154300 0
[pid=30793/tid=30799] ppid=30791 vsize=4625580 CPUtime=756.85 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 513405 0 0 0 75504 181 0 0 25 0 5 0 595849761 4736593920 1099198 33554432000 4194304 4338100 140734785060896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=4625580 CPUtime=756.03 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 86437 0 0 0 75168 435 0 0 25 0 5 0 595849761 4736593920 1099198 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=4625580 CPUtime=760.8 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 372269 0 0 0 76003 77 0 0 25 0 5 0 595849761 4736593920 1099198 33554432000 4194304 4338100 140734785060896 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=4625580 CPUtime=758.01 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 86424 0 0 0 75164 637 0 0 25 0 5 0 595849761 4736593920 1099198 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3033.12
Current children cumulated vsize (KiB) 4625580

[startup+822.301 s]
/proc/loadavg: 8.12 8.10 8.05 9/186 30833
/proc/meminfo: memFree=9640808/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=5132316 CPUtime=3272.7 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1229851 0 0 0 325645 1625 0 0 25 0 5 0 595849620 5255491584 1226204 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1283079 1226204 176 36 0 1280984 0
[pid=30793/tid=30799] ppid=30791 vsize=5132316 CPUtime=816.83 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 582557 0 0 0 81484 199 0 0 25 0 5 0 595849761 5255491584 1226204 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907303118 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=5132316 CPUtime=815.83 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 94843 0 0 0 81020 563 0 0 25 0 5 0 595849761 5255491584 1226204 33554432000 4194304 4338100 140734785060896 18446744073709551615 4240519 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=5132316 CPUtime=820.78 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 411567 0 0 0 81992 86 0 0 25 0 5 0 595849761 5255491584 1226204 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=5132316 CPUtime=817.83 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 96574 0 0 0 81013 770 0 0 25 0 5 0 595849761 5255491584 1226204 33554432000 4194304 4338100 140734785060896 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3272.7
Current children cumulated vsize (KiB) 5132316

[startup+882.3 s]
/proc/loadavg: 8.14 8.11 8.05 9/186 30835
/proc/meminfo: memFree=9540344/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=5229284 CPUtime=3512.26 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1253437 0 0 0 349333 1893 0 0 25 0 5 0 595849620 5354786816 1249790 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1307321 1249790 176 36 0 1305226 0
[pid=30793/tid=30799] ppid=30791 vsize=5229284 CPUtime=876.83 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 600328 0 0 0 87474 209 0 0 25 0 5 0 595849761 5354786816 1249790 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=5229284 CPUtime=875.46 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 94867 0 0 0 86855 691 0 0 25 0 5 0 595849761 5354786816 1249790 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=5229284 CPUtime=880.76 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 416750 0 0 0 87989 87 0 0 25 0 5 0 595849761 5354786816 1249790 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=5229284 CPUtime=877.78 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 97182 0 0 0 86878 900 0 0 25 0 5 0 595849761 5354786816 1249790 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3512.26
Current children cumulated vsize (KiB) 5229284

[startup+942.301 s]
/proc/loadavg: 7.93 8.06 8.03 9/186 30836
/proc/meminfo: memFree=9177388/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=5517060 CPUtime=3751.67 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1325725 0 0 0 373087 2080 0 0 25 0 5 0 595849620 5649469440 1322078 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1379265 1322078 176 36 0 1377170 0
[pid=30793/tid=30799] ppid=30791 vsize=5517060 CPUtime=936.65 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 630669 0 0 0 93446 219 0 0 25 0 5 0 595849761 5649469440 1322078 33554432000 4194304 4338100 140734785060896 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=5517060 CPUtime=935.3 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 99235 0 0 0 92742 788 0 0 25 0 5 0 595849761 5649469440 1322078 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=5517060 CPUtime=940.76 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 450495 0 0 0 93980 96 0 0 25 0 5 0 595849761 5649469440 1322078 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=5517060 CPUtime=937.53 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 101016 0 0 0 92782 971 0 0 18 0 5 0 595849761 5649469440 1322078 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3751.67
Current children cumulated vsize (KiB) 5517060

[startup+1002.3 s]
/proc/loadavg: 8.04 8.06 8.03 9/186 30838
/proc/meminfo: memFree=8854332/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=5839244 CPUtime=3990.84 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1405705 0 0 0 396917 2167 0 0 25 0 5 0 595849620 5979385856 1402058 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1459811 1402058 176 36 0 1457716 0
[pid=30793/tid=30799] ppid=30791 vsize=5839244 CPUtime=996.33 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 674044 0 0 0 99397 236 0 0 25 0 5 0 595849761 5979385856 1402058 33554432000 4194304 4338100 140734785060896 18446744073709551615 231894885822 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=5839244 CPUtime=995.3 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 105081 0 0 0 98684 846 0 0 25 0 5 0 595849761 5979385856 1402058 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=5839244 CPUtime=1000.73 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 474627 0 0 0 99973 100 0 0 25 0 5 0 595849761 5979385856 1402058 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=5839244 CPUtime=997.05 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 107643 0 0 0 98726 979 0 0 18 0 5 0 595849761 5979385856 1402058 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3990.84
Current children cumulated vsize (KiB) 5839244

[startup+1062.3 s]
/proc/loadavg: 7.84 8.00 8.01 9/186 30839
/proc/meminfo: memFree=8508716/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=6123892 CPUtime=4229.9 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1477514 0 0 0 420743 2247 0 0 25 0 5 0 595849620 6270865408 1473867 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1530973 1473867 176 36 0 1528878 0
[pid=30793/tid=30799] ppid=30791 vsize=6123892 CPUtime=1056.2 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 717694 0 0 0 105369 251 0 0 25 0 5 0 595849761 6270865408 1473867 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=6123892 CPUtime=1055.27 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 109945 0 0 0 104628 899 0 0 25 0 5 0 595849761 6270865408 1473867 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=6123892 CPUtime=1060.48 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 494069 0 0 0 105944 104 0 0 25 0 5 0 595849761 6270865408 1473867 33554432000 4194304 4338100 140734785060896 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=6123892 CPUtime=1056.52 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 111496 0 0 0 104665 987 0 0 18 0 5 0 595849761 6270865408 1473867 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4229.9
Current children cumulated vsize (KiB) 6123892

[startup+1122.3 s]
/proc/loadavg: 8.00 8.01 8.01 9/186 30841
/proc/meminfo: memFree=8104716/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=6371300 CPUtime=4469.09 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1539475 0 0 0 444588 2321 0 0 25 0 5 0 595849620 6524211200 1535828 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1592825 1535828 176 36 0 1590730 0
[pid=30793/tid=30799] ppid=30791 vsize=6371300 CPUtime=1116.2 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 747337 0 0 0 111359 261 0 0 25 0 5 0 595849761 6524211200 1535828 33554432000 4194304 4338100 140734785060896 18446744073709551615 231894885772 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=6371300 CPUtime=1115.25 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 114463 0 0 0 110578 947 0 0 25 0 5 0 595849761 6524211200 1535828 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=6371300 CPUtime=1120.17 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 515383 0 0 0 111907 110 0 0 25 0 5 0 595849761 6524211200 1535828 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=6371300 CPUtime=1116.03 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 117982 0 0 0 110607 996 0 0 18 0 5 0 595849761 6524211200 1535828 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4469.09
Current children cumulated vsize (KiB) 6371300

[startup+1182.3 s]
/proc/loadavg: 8.10 8.04 8.02 9/186 30843
/proc/meminfo: memFree=7928496/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=6371300 CPUtime=4708 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) S 30791 30793 30360 0 -1 4202496 1544955 0 0 0 468400 2400 0 0 25 0 5 0 595849620 6524211200 1541308 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 1592825 1541308 176 36 0 1590730 0
[pid=30793/tid=30799] ppid=30791 vsize=6371300 CPUtime=1176.19 cores=0,2,4,6
/proc/30793/task/30799/stat : 30799 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 747337 0 0 0 117354 265 0 0 25 0 5 0 595849761 6524211200 1541308 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=6371300 CPUtime=1175.23 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 117330 0 0 0 116508 1015 0 0 25 0 5 0 595849761 6524211200 1541308 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30793/tid=30801] ppid=30791 vsize=6371300 CPUtime=1179.79 cores=0,2,4,6
/proc/30793/task/30801/stat : 30801 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 515383 0 0 0 117869 110 0 0 25 0 5 0 595849761 6524211200 1541308 33554432000 4194304 4338100 140734785060896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30793/tid=30802] ppid=30791 vsize=6371300 CPUtime=1175.36 cores=0,2,4,6
/proc/30793/task/30802/stat : 30802 (ttsth-5-4) R 30791 30793 30360 0 -1 4202560 120595 0 0 0 116532 1004 0 0 18 0 5 0 595849761 6524211200 1541308 33554432000 4194304 4338100 140734785060896 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4708
Current children cumulated vsize (KiB) 6371300



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.5 s]
/proc/loadavg: 8.07 8.04 8.01 2/174 30861
/proc/meminfo: memFree=16224112/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=0 CPUtime=4800.1 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) Z 30791 30793 30360 0 -1 4203548 1548790 0 0 0 477572 2438 0 0 18 0 2 0 595849620 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 0 0 0 0 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=0 CPUtime=1198.41 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4203588 117330 0 0 0 118793 1048 0 0 25 0 2 0 595849761 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.62 s]
/proc/loadavg: 8.07 8.04 8.01 2/174 30861
/proc/meminfo: memFree=16224112/32950928 swapFree=67111524/67111528
[pid=30793] ppid=30791 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/30793/stat : 30793 (ttsth-5-4) Z 30791 30793 30360 0 -1 4203548 1548790 0 0 0 477572 2448 0 0 18 0 2 0 595849620 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30793/statm: 0 0 0 0 0 0 0
[pid=30793/tid=30800] ppid=30791 vsize=0 CPUtime=1198.52 cores=0,2,4,6
/proc/30793/task/30800/stat : 30800 (ttsth-5-4) R 30791 30793 30360 0 -1 4203588 117330 0 0 0 118793 1059 0 0 25 0 2 0 595849761 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.68
CPU time (s): 4800.28
CPU user time (s): 4775.73
CPU system time (s): 24.5523
CPU usage (%): 398.138
Max. virtual memory (cumulated for all children) (KiB): 6383812

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4775.73
system time used= 24.5523
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1548790
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 14430
involuntary context switches= 23147

runsolver used 1.9877 second user time and 4.49432 second system time

The end

Launcher Data 

Begin job on node114 at 2011-04-29 07:49:49
IDJOB=3259714
IDBENCH=71612
IDSOLVER=1654
FILE ID=node114/3259714-1304056189
RUNJOBID= node114-1304054983-30674
PBS_JOBID= 13170413
Free space on /tmp= 71796 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v2400c266431gyes2.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259714-1304056189/watcher-3259714-1304056189 -o /tmp/evaluation-result-3259714-1304056189/solver-3259714-1304056189 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259714-1304056189.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= aac2bd287b8cd6b96cbf7ecc463acf19
RANDOM SEED=929096166

node114.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      20447280 kB
Buffers:       2255972 kB
Cached:        9543312 kB
SwapCached:          4 kB
Active:        3464748 kB
Inactive:      8458692 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      20447280 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            7396 kB
Writeback:           0 kB
AnonPages:      123812 kB
Mapped:          15076 kB
Slab:           516056 kB
PageTables:       5060 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   268824 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71800 MiB
End job on node114 at 2011-04-29 08:09:57