Trace number 3250032

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.21 1203.11

General information on the benchmark

Namecrafted/skvortsov/battleship/
battleship-16-16-unsat.cnf
MD5SUMc07743bac800f91e403382f26717b1d0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables256
Number of clauses2176
Sum of the clauses size7936
Maximum clause length16
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21920
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 5256

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250032-1304088233.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 256
0.00/0.00	c Clauses = 2176
0.00/0.00	c Literals = 7936
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 256
0.00/0.00	c Clauses = 2176
0.00/0.00	c Literals = 7936
0.00/0.00	c a2
0.00/0.01	c a1
0.19/0.22	c Independent sub-problems = 1
0.19/0.22	c Sub-problem with 256 variables and 2176 clauses
0.19/0.22	c using permutation method 4
0.19/0.22	c count = 2
0.19/0.22	c Ordering finished 0.210 seconds
0.19/0.22	c mapped 0.210 seconds
0.19/0.22	c permuted 0.210 seconds
0.19/0.22	c about to sort
0.19/0.22	c sorted
0.19/0.22	c clauses now 2176
0.19/0.22	c init finished 0.220 seconds
0.19/0.22	c tree built 0.220 seconds
0.19/0.22	c 
0.19/0.22	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.22	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.22	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.22	c ................................ | ..........                  1 |                                  0.2         4.5             
0.19/0.22	c ................................ | ..........                  2 |                                  0.2         9.1             
0.19/0.22	c ................................ | ..........                  3 |                                  0.2        13.6             
0.19/0.22	c ................................ | ..........                  4 |                                  0.2        18.2             
0.19/0.22	c ................................ | ..........                  5 |                                  0.2        22.7             
0.19/0.23	c ................................ | ..........               1000 | 207          263                 0.2      4347.8         4737
95.80/24.29	c ................................ | ..........            1026000 | 319003       2926               96.2     10668.6      7098866
192.33/48.43	c ................................ | ..........            1691000 | 523284       3700              192.5      8785.8     13450496
287.70/72.39	c ................................ | ..........            2124000 | 659027       4094              288.1      7373.7     19984877
383.82/96.49	c ................................ | ..........            2446000 | 758164       4939              384.1      6368.8     23591478
480.46/120.61	c ................................ | *.........            2699000 | 835520       5053              480.2      5620.7     25951522
576.03/144.63	c ................................ | *.........            2886000 | 895148       5055              576.1      5009.5     28807984
671.79/168.69	c ................................ | *.........            3181000 | 989503       5068              672.0      4733.6     30061279
767.92/192.77	c ................................ | *.........            3482000 | 1086212      5076              768.2      4532.6     30927163
864.11/216.82	c ................................ | *.........            3606000 | 1125623      5417              864.2      4172.6     34093252
960.29/240.96	c ................................ | **........            3768000 | 1173547      5578              960.5      3922.8     37709859
1056.24/265.09	c ................................ | **........            4002000 | 1246987      5735             1056.6      3787.5     39898498
1153.00/289.34	c ................................ | **........            4281000 | 1334286      5811             1153.1      3712.7     40777490
1248.81/313.36	c ................................ | **........            4511000 | 1405205      5850             1249.1      3611.5     41479824
1344.55/337.31	c ................................ | **........            4669000 | 1454380      5890             1344.3      3473.2     42951898
1439.97/361.21	c ................................ | ***.......            4858000 | 1513792      5960             1440.0      3373.5     44897191
1536.05/385.39	c ................................ | ***.......            5097000 | 1587853      5960             1536.1      3318.2     46167580
1631.88/409.34	c ................................ | ***.......            5217000 | 1624356      6069             1632.1      3196.6     46167580
1727.77/433.37	c ................................ | ***.......            5376000 | 1673829      6088             1728.1      3111.0     46167580
1824.42/457.58	c ................................ | ***.......            5483000 | 1707255      6091             1824.8      3004.8     46167580
1920.24/481.58	c ................................ | ****......            5611000 | 1746804      6205             1920.6      2921.5     46167580
2016.07/505.51	c ................................ | ****......            5714000 | 1778257      6337             2016.1      2834.2     46167580
2111.89/529.53	c ................................ | ****......            5832000 | 1815509      6359             2112.0      2761.3     46167580
2207.76/553.57	c ................................ | ****......            5943000 | 1849163      6368             2208.1      2691.5     46842620
2305.23/577.98	c ................................ | ****......            6023000 | 1874693      6434             2305.6      2612.4     48773967
2399.91/601.65	c ................................ | *****.....            6098000 | 1898167      6441             2400.1      2540.7     49183236
2496.17/625.72	c ................................ | *****.....            6223000 | 1938493      6441             2496.2      2492.9     50402687
2591.85/649.74	c ................................ | *****.....            6369000 | 1985878      6441             2592.0      2457.1     52427264
2688.12/673.86	c ................................ | *****.....            6499000 | 2028022      6444             2688.3      2417.5     52610432
2784.22/697.92	c ................................ | *****.....            6711000 | 2095734      6533             2784.3      2410.3     54100757
2880.86/722.12	c ................................ | ******....            6778000 | 2118085      6623             2880.9      2352.7     55897876
2976.71/746.18	c ................................ | ******....            6830000 | 2133571      6623             2977.1      2294.2     57170698
3071.72/769.99	c ................................ | ******....            6905000 | 2157716      6623             3072.1      2247.7     57676630
3168.80/794.23	c ................................ | ******....            7011000 | 2191784      6639             3168.9      2212.4     58187605
3263.73/818.08	c ................................ | ******....            7126000 | 2228239      6639             3264.1      2183.2     59536305
3360.21/842.24	c ................................ | *******...            7225000 | 2258688      6700             3360.4      2150.0     63649516
3456.22/866.39	c ................................ | *******...            7347000 | 2295101      6840             3456.6      2125.5     65129579
3553.66/890.70	c ................................ | *******...            7389000 | 2308581      6840             3553.7      2079.3     65627216
3648.14/914.43	c ................................ | *******...            7472000 | 2333999      6843             3648.2      2048.1     67130993
3744.30/938.51	c ................................ | *******...            7557000 | 2358876      6929             3744.4      2018.2     67130993
3840.15/962.59	c ................................ | ********..            7647000 | 2386835      7120             3840.2      1991.3     68497846
3936.25/986.67	c ................................ | ********..            7753000 | 2420000      7263             3936.4      1969.5     69515022
4031.91/1010.62	c ................................ | ********..            7884000 | 2461699      7263             4032.0      1955.4     69678723
4128.56/1034.83	c ................................ | ********..            8019000 | 2503618      7263             4128.7      1942.3     69678723
4223.99/1058.75	c ................................ | ********..            8085000 | 2523313      7263             4224.2      1914.0     69678723
4321.02/1083.03	c ................................ | *********.            8159000 | 2546507      7266             4321.1      1888.2     70493915
4416.05/1106.84	c ................................ | *********.            8256000 | 2577055      7266             4416.2      1869.5     70961815
4512.31/1130.95	c ................................ | *********.            8356000 | 2608280      7266             4512.5      1851.7     71846415
4607.95/1154.98	c ................................ | *********.            8453000 | 2638015      7266             4608.3      1834.3     71868285
4704.22/1179.00	c ................................ | *********.            8559000 | 2670845      7266             4704.3      1819.4     72481398

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250032-1304088233/watcher-3250032-1304088233 -o /tmp/evaluation-result-3250032-1304088233/solver-3250032-1304088233 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250032-1304088233.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.00 7.99 7.52 5/181 25353
/proc/meminfo: memFree=15583152/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=10036 CPUtime=0 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) R 25351 25353 25048 0 -1 4202496 598 0 0 0 0 0 0 0 25 0 1 0 520424200 10276864 540 33554432000 4194304 4338100 140735293878864 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/25353/statm: 2509 540 152 36 0 414 0

[startup+0.0335289 s]
/proc/loadavg: 8.00 7.99 7.52 5/181 25353
/proc/meminfo: memFree=15583152/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=10180 CPUtime=0.03 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) R 25351 25353 25048 0 -1 4202496 609 0 0 0 3 0 0 0 25 0 1 0 520424200 10424320 551 33554432000 4194304 4338100 140735293878864 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/25353/statm: 2545 551 155 36 0 450 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 10180

[startup+0.100533 s]
/proc/loadavg: 8.00 7.99 7.52 5/181 25353
/proc/meminfo: memFree=15583152/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=10180 CPUtime=0.09 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) R 25351 25353 25048 0 -1 4202496 609 0 0 0 9 0 0 0 25 0 1 0 520424200 10424320 551 33554432000 4194304 4338100 140735293878864 18446744073709551615 4295890 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/25353/statm: 2545 551 155 36 0 450 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 10180

[startup+0.345878 s]
/proc/loadavg: 8.00 7.99 7.52 5/181 25353
/proc/meminfo: memFree=15583152/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=263316 CPUtime=1.36 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 3287 0 0 0 135 1 0 0 25 0 5 0 520424200 269635584 3130 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 65829 3130 175 36 0 63734 0
Current children cumulated CPU time (s) 1.36
Current children cumulated vsize (KiB) 263316

[startup+1.00011 s]
/proc/loadavg: 8.00 7.99 7.52 5/181 25353
/proc/meminfo: memFree=15583152/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=275828 CPUtime=3.58 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 6514 0 0 0 354 4 0 0 25 0 5 0 520424200 282447872 6358 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 68957 6358 175 36 0 66862 0
Current children cumulated CPU time (s) 3.58
Current children cumulated vsize (KiB) 275828

[startup+1.51884 s]
/proc/loadavg: 8.00 7.99 7.52 5/181 25353
/proc/meminfo: memFree=15583152/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=292988 CPUtime=5.36 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 10264 0 0 0 531 5 0 0 25 0 5 0 520424200 300019712 10089 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 73247 10089 176 36 0 71152 0
Current children cumulated CPU time (s) 5.36
Current children cumulated vsize (KiB) 292988

[startup+3.10095 s]
/proc/loadavg: 8.00 7.99 7.53 8/186 25358
/proc/meminfo: memFree=15521492/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=318172 CPUtime=11.65 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 16605 0 0 0 1155 10 0 0 25 0 5 0 520424200 325808128 16430 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 79543 16430 176 36 0 77448 0
[pid=25353/tid=25355] ppid=25351 vsize=318172 CPUtime=2.85 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 4965 0 0 0 284 1 0 0 20 0 5 0 520424223 325808128 16431 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=318172 CPUtime=2.85 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 3776 0 0 0 282 3 0 0 19 0 5 0 520424223 325808128 16431 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=318172 CPUtime=2.86 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 3079 0 0 0 284 2 0 0 20 0 5 0 520424223 325808128 16431 33554432000 4194304 4338100 140735293878864 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=318172 CPUtime=2.83 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 3404 0 0 0 281 2 0 0 19 0 5 0 520424223 325808128 16431 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.65
Current children cumulated vsize (KiB) 318172

[startup+6.30038 s]
/proc/loadavg: 8.00 7.99 7.53 9/186 25358
/proc/meminfo: memFree=15436312/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=390520 CPUtime=24.41 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 34531 0 0 0 2421 20 0 0 25 0 5 0 520424200 399892480 34348 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 97630 34348 176 36 0 95535 0
[pid=25353/tid=25355] ppid=25351 vsize=390520 CPUtime=6.02 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 9456 0 0 0 600 2 0 0 25 0 5 0 520424223 399892480 34349 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=390520 CPUtime=6.05 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 7336 0 0 0 598 7 0 0 25 0 5 0 520424223 399892480 34349 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=390520 CPUtime=6.05 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 9851 0 0 0 602 3 0 0 25 0 5 0 520424223 399892480 34349 33554432000 4194304 4338100 140735293878864 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=390520 CPUtime=6.04 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 6507 0 0 0 598 6 0 0 21 0 5 0 520424223 399892480 34349 33554432000 4194304 4338100 140735293878864 18446744073709551615 4286825 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.41
Current children cumulated vsize (KiB) 390520

[startup+12.7062 s]
/proc/loadavg: 8.00 7.99 7.53 9/186 25358
/proc/meminfo: memFree=15374296/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=432144 CPUtime=49.93 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 45694 0 0 0 4961 32 0 0 25 0 5 0 520424200 442515456 45511 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 108036 45511 176 36 0 105941 0
[pid=25353/tid=25355] ppid=25351 vsize=432144 CPUtime=12.38 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 14041 0 0 0 1235 3 0 0 25 0 5 0 520424223 442515456 45511 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=432144 CPUtime=12.43 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 9960 0 0 0 1232 11 0 0 18 0 5 0 520424223 442515456 45511 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=432144 CPUtime=12.45 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 12392 0 0 0 1239 6 0 0 25 0 5 0 520424223 442515456 45511 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=432144 CPUtime=12.42 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 7919 0 0 0 1233 9 0 0 21 0 5 0 520424223 442515456 45511 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.93
Current children cumulated vsize (KiB) 432144

[startup+25.501 s]
/proc/loadavg: 8.00 7.99 7.54 9/186 25358
/proc/meminfo: memFree=15294680/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=501992 CPUtime=100.98 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 63064 0 0 0 10038 60 0 0 25 0 5 0 520424200 514039808 62881 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 125498 62881 176 36 0 123403 0
[pid=25353/tid=25355] ppid=25351 vsize=501992 CPUtime=25.1 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 18536 0 0 0 2504 6 0 0 25 0 5 0 520424223 514039808 62881 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=501992 CPUtime=25.2 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 14111 0 0 0 2499 21 0 0 25 0 5 0 520424223 514039808 62881 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=501992 CPUtime=25.23 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 18216 0 0 0 2512 11 0 0 25 0 5 0 520424223 514039808 62881 33554432000 4194304 4338100 140735293878864 18446744073709551615 4286820 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=501992 CPUtime=25.19 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 10819 0 0 0 2499 20 0 0 24 0 5 0 520424223 514039808 62881 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.98
Current children cumulated vsize (KiB) 501992

[startup+51.1005 s]
/proc/loadavg: 8.06 8.01 7.56 9/186 25359
/proc/meminfo: memFree=15010952/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=711904 CPUtime=203.14 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 115347 0 0 0 20201 113 0 0 25 0 5 0 520424200 728989696 115164 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 177976 115164 176 36 0 175881 0
[pid=25353/tid=25355] ppid=25351 vsize=711904 CPUtime=50.58 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 34444 0 0 0 5048 10 0 0 25 0 5 0 520424223 728989696 115164 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=711904 CPUtime=50.78 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 23837 0 0 0 5036 42 0 0 25 0 5 0 520424223 728989696 115164 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=711904 CPUtime=50.75 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 37231 0 0 0 5056 19 0 0 25 0 5 0 520424223 728989696 115164 33554432000 4194304 4338100 140735293878864 18446744073709551615 270612505665 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=711904 CPUtime=50.78 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 18453 0 0 0 5038 40 0 0 25 0 5 0 520424223 728989696 115164 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 203.14
Current children cumulated vsize (KiB) 711904

[startup+102.3 s]
/proc/loadavg: 8.14 8.04 7.59 9/186 25360
/proc/meminfo: memFree=14555212/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=1027832 CPUtime=407.38 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 193789 0 0 0 40550 188 0 0 25 0 5 0 520424200 1052499968 193606 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 256958 193606 176 36 0 254863 0
[pid=25353/tid=25355] ppid=25351 vsize=1027832 CPUtime=101.58 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 59793 0 0 0 10140 18 0 0 25 0 5 0 520424223 1052499968 193606 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=1027832 CPUtime=101.89 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 38431 0 0 0 10119 70 0 0 25 0 5 0 520424223 1052499968 193606 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=1027832 CPUtime=101.78 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 62253 0 0 0 10145 33 0 0 19 0 5 0 520424223 1052499968 193606 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=1027832 CPUtime=101.87 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 31930 0 0 0 10123 64 0 0 22 0 5 0 520424223 1052499968 193606 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 407.38
Current children cumulated vsize (KiB) 1027832

[startup+162.307 s]
/proc/loadavg: 8.19 8.07 7.63 9/186 25362
/proc/meminfo: memFree=14230056/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=1247220 CPUtime=646.7 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 245569 0 0 0 64438 232 0 0 25 0 5 0 520424200 1277153280 245386 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 311805 245386 176 36 0 309710 0
[pid=25353/tid=25355] ppid=25351 vsize=1247220 CPUtime=161.26 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 69346 0 0 0 16104 22 0 0 25 0 5 0 520424223 1277153280 245386 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=1247220 CPUtime=161.8 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 47714 0 0 0 16093 87 0 0 25 0 5 0 520424223 1277153280 245386 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=1247220 CPUtime=161.7 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 84611 0 0 0 16125 45 0 0 19 0 5 0 520424223 1277153280 245386 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=1247220 CPUtime=161.69 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 42516 0 0 0 16092 77 0 0 19 0 5 0 520424223 1277153280 245386 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 646.7
Current children cumulated vsize (KiB) 1247220

[startup+222.301 s]
/proc/loadavg: 8.13 8.07 7.65 9/186 25363
/proc/meminfo: memFree=14024796/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=1403620 CPUtime=886.03 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 285751 0 0 0 88308 295 0 0 25 0 5 0 520424200 1437306880 285568 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 350905 285568 176 36 0 348810 0
[pid=25353/tid=25355] ppid=25351 vsize=1403620 CPUtime=221.09 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 74952 0 0 0 22085 24 0 0 25 0 5 0 520424223 1437306880 285568 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=12415920/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2182228 CPUtime=3041.37 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 471283 0 0 0 303395 742 0 0 25 0 5 0 520424200 2234601472 471100 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 545557 471100 176 36 0 543462 0
[pid=25353/tid=25355] ppid=25351 vsize=2182228 CPUtime=759.68 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 119617 0 0 0 75927 41 0 0 25 0 5 0 520424223 2234601472 471100 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2182228 CPUtime=760.81 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 97918 0 0 0 75794 287 0 0 25 0 5 0 520424223 2234601472 471100 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2182228 CPUtime=761.11 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 184360 0 0 0 75972 139 0 0 25 0 5 0 520424223 2234601472 471100 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2182228 CPUtime=759.51 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 68006 0 0 0 75678 273 0 0 18 0 5 0 520424223 2234601472 471100 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3041.37
Current children cumulated vsize (KiB) 2182228

[startup+822.306 s]
/proc/loadavg: 7.22 7.91 7.79 5/182 25408
/proc/meminfo: memFree=15408544/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2266816 CPUtime=3280.84 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 492904 0 0 0 327310 774 0 0 25 0 5 0 520424200 2321219584 492721 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 566704 492721 176 36 0 564609 0
[pid=25353/tid=25355] ppid=25351 vsize=2266816 CPUtime=819.38 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 128524 0 0 0 81893 45 0 0 19 0 5 0 520424223 2321219584 492721 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254413 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2266816 CPUtime=820.78 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 101413 0 0 0 81779 299 0 0 25 0 5 0 520424223 2321219584 492721 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2266816 CPUtime=821.03 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 190535 0 0 0 81957 146 0 0 22 0 5 0 520424223 2321219584 492721 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2266816 CPUtime=819.39 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 71050 0 0 0 81657 282 0 0 18 0 5 0 520424223 2321219584 492721 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3280.84
Current children cumulated vsize (KiB) 2266816

[startup+882.3 s]
/proc/loadavg: 7.80 7.95 7.81 9/186 25413
/proc/meminfo: memFree=14641512/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2426344 CPUtime=3520.1 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 531298 0 0 0 351199 811 0 0 25 0 5 0 520424200 2484576256 531115 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 606586 531115 176 36 0 604491 0
[pid=25353/tid=25355] ppid=25351 vsize=2426344 CPUtime=879.29 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 132933 0 0 0 87884 45 0 0 25 0 5 0 520424223 2484576256 531115 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2426344 CPUtime=880.32 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 109476 0 0 0 87722 310 0 0 25 0 5 0 520424223 2484576256 531115 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2426344 CPUtime=880.92 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 208825 0 0 0 87934 158 0 0 24 0 5 0 520424223 2484576256 531115 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2426344 CPUtime=879.33 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78682 0 0 0 87636 297 0 0 25 0 5 0 520424223 2484576256 531115 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3520.1
Current children cumulated vsize (KiB) 2426344

[startup+942.301 s]
/proc/loadavg: 8.21 8.03 7.85 9/186 25415
/proc/meminfo: memFree=14281728/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2485776 CPUtime=3759.47 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 546628 0 0 0 375107 840 0 0 25 0 5 0 520424200 2545434624 546445 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 621444 546445 176 36 0 619349 0
[pid=25353/tid=25355] ppid=25351 vsize=2485776 CPUtime=939.23 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 132933 0 0 0 93878 45 0 0 25 0 5 0 520424223 2545434624 546445 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2485776 CPUtime=939.91 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 112267 0 0 0 93673 318 0 0 18 0 5 0 520424223 2545434624 546445 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2485776 CPUtime=940.77 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 221363 0 0 0 93911 166 0 0 25 0 5 0 520424223 2545434624 546445 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2485776 CPUtime=939.3 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78683 0 0 0 93621 309 0 0 25 0 5 0 520424223 2545434624 546445 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3759.47
Current children cumulated vsize (KiB) 2485776

[startup+1002.3 s]
/proc/loadavg: 8.13 8.04 7.86 9/186 25416
/proc/meminfo: memFree=14011844/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2563976 CPUtime=3998.75 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 566194 0 0 0 399007 868 0 0 25 0 5 0 520424200 2625511424 566011 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 640994 566011 176 36 0 638899 0
[pid=25353/tid=25355] ppid=25351 vsize=2563976 CPUtime=999.16 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 132934 0 0 0 99870 46 0 0 25 0 5 0 520424223 2625511424 566011 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2563976 CPUtime=999.68 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 116744 0 0 0 99643 325 0 0 21 0 5 0 520424223 2625511424 566011 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2563976 CPUtime=1000.43 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 236447 0 0 0 99868 175 0 0 23 0 5 0 520424223 2625511424 566011 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2563976 CPUtime=999.22 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78687 0 0 0 99602 320 0 0 25 0 5 0 520424223 2625511424 566011 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3998.75
Current children cumulated vsize (KiB) 2563976

[startup+1062.3 s]
/proc/loadavg: 8.20 8.08 7.88 9/186 25427
/proc/meminfo: memFree=13888004/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2567104 CPUtime=4238.35 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 566999 0 0 0 422948 887 0 0 25 0 5 0 520424200 2628714496 566816 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 641776 566816 176 36 0 639681 0
[pid=25353/tid=25355] ppid=25351 vsize=2567104 CPUtime=1059.12 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 132934 0 0 0 105866 46 0 0 25 0 5 0 520424223 2628714496 566816 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2567104 CPUtime=1059.35 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 116745 0 0 0 105605 330 0 0 25 0 5 0 520424223 2628714496 566816 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2567104 CPUtime=1060.4 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 237248 0 0 0 105860 180 0 0 25 0 5 0 520424223 2628714496 566816 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2567104 CPUtime=1059.22 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78690 0 0 0 105593 329 0 0 25 0 5 0 520424223 2628714496 566816 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4238.35
Current children cumulated vsize (KiB) 2567104

[startup+1122.3 s]
/proc/loadavg: 8.19 8.09 7.90 9/186 25428
/proc/meminfo: memFree=13697364/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2617152 CPUtime=4477.96 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 580644 0 0 0 446889 907 0 0 25 0 5 0 520424200 2679963648 580461 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 654288 580461 176 36 0 652193 0
[pid=25353/tid=25355] ppid=25351 vsize=2617152 CPUtime=1119.13 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 135184 0 0 0 111866 47 0 0 25 0 5 0 520424223 2679963648 580461 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2617152 CPUtime=1118.98 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 117030 0 0 0 111564 334 0 0 25 0 5 0 520424223 2679963648 580461 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2617152 CPUtime=1120.39 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 248358 0 0 0 111852 187 0 0 25 0 5 0 520424223 2679963648 580461 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2617152 CPUtime=1119.21 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78690 0 0 0 111583 338 0 0 25 0 5 0 520424223 2679963648 580461 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4477.96
Current children cumulated vsize (KiB) 2617152

[startup+1182.3 s]
/proc/loadavg: 8.14 8.09 7.91 9/186 25430
/proc/meminfo: memFree=13418304/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2672040 CPUtime=4717.38 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 589015 0 0 0 470814 924 0 0 25 0 5 0 520424200 2736168960 588832 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 668010 588832 176 36 0 665915 0
[pid=25353/tid=25355] ppid=25351 vsize=2672040 CPUtime=1178.91 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 137117 0 0 0 117843 48 0 0 25 0 5 0 520424223 2736168960 588832 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2672040 CPUtime=1178.76 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 117535 0 0 0 117539 337 0 0 25 0 5 0 520424223 2736168960 588832 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2672040 CPUtime=1180.31 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 254291 0 0 0 117842 189 0 0 25 0 5 0 520424223 2736168960 588832 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2672040 CPUtime=1179.14 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78690 0 0 0 117567 347 0 0 25 0 5 0 520424223 2736168960 588832 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4717.38
Current children cumulated vsize (KiB) 2672040



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203 s]
/proc/loadavg: 8.18 8.10 7.91 11/187 25434
/proc/meminfo: memFree=13325652/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=2687680 CPUtime=4800.08 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) S 25351 25353 25048 0 -1 4202496 593005 0 0 0 479077 931 0 0 25 0 5 0 520424200 2752184320 592822 33554432000 4194304 4338100 140735293878864 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 671920 592822 176 36 0 669825 0
[pid=25353/tid=25355] ppid=25351 vsize=2687680 CPUtime=1199.51 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 137117 0 0 0 119903 48 0 0 25 0 5 0 520424223 2752184320 592822 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25353/tid=25356] ppid=25351 vsize=2687680 CPUtime=1199.45 cores=0,2,4,6
/proc/25353/task/25356/stat : 25356 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 117535 0 0 0 119606 339 0 0 18 0 5 0 520424223 2752184320 592822 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25353/tid=25357] ppid=25351 vsize=2687680 CPUtime=1201.02 cores=0,2,4,6
/proc/25353/task/25357/stat : 25357 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 258281 0 0 0 119910 192 0 0 25 0 5 0 520424223 2752184320 592822 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25353/tid=25358] ppid=25351 vsize=2687680 CPUtime=1199.84 cores=0,2,4,6
/proc/25353/task/25358/stat : 25358 (ttsth-5-4) R 25351 25353 25048 0 -1 4202560 78690 0 0 0 119634 350 0 0 25 0 5 0 520424223 2752184320 592822 33554432000 4194304 4338100 140735293878864 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 2687680

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.11 s]
/proc/loadavg: 8.18 8.10 7.91 11/187 25434
/proc/meminfo: memFree=13325652/32951124 swapFree=67111528/67111528
[pid=25353] ppid=25351 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/25353/stat : 25353 (ttsth-5-4) Z 25351 25353 25048 0 -1 4203548 593006 0 0 0 479079 941 0 0 18 0 2 0 520424200 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/25353/statm: 0 0 0 0 0 0 0
[pid=25353/tid=25355] ppid=25351 vsize=0 CPUtime=1199.61 cores=0,2,4,6
/proc/25353/task/25355/stat : 25355 (ttsth-5-4) R 25351 25353 25048 0 -1 4203588 137117 0 0 0 119903 58 0 0 25 0 2 0 520424223 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.11
CPU time (s): 4800.21
CPU user time (s): 4790.79
CPU system time (s): 9.41957
CPU usage (%): 398.982
Max. virtual memory (cumulated for all children) (KiB): 2687680

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4790.79
system time used= 9.41957
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 593006
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10448
involuntary context switches= 34222

runsolver used 1.80972 second user time and 4.21236 second system time

The end

Launcher Data 

Begin job on node108 at 2011-04-29 16:43:53
IDJOB=3250032
IDBENCH=83223
IDSOLVER=1654
FILE ID=node108/3250032-1304088233
RUNJOBID= node108-1304086608-25066
PBS_JOBID= 13170551
Free space on /tmp= 72120 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/skvortsov/battleship/battleship-16-16-unsat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250032-1304088233/watcher-3250032-1304088233 -o /tmp/evaluation-result-3250032-1304088233/solver-3250032-1304088233 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250032-1304088233.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c07743bac800f91e403382f26717b1d0
RANDOM SEED=1304341668

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15583308 kB
Buffers:       2164596 kB
Cached:       12756140 kB
SwapCached:          0 kB
Active:        6002336 kB
Inactive:     10781504 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15583308 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:              16 kB
Writeback:           0 kB
AnonPages:     1862692 kB
Mapped:          15336 kB
Slab:           517816 kB
PageTables:       8472 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2063512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72120 MiB
End job on node108 at 2011-04-29 17:03:58