Trace number 3250252

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.85 1202.83

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-4-5_4231.cnf
MD5SUMa52d60bf6e72caa628ad712a5fe0fb7a
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables4229
Number of clauses100397
Sum of the clauses size411595
Maximum clause length5
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 490390
Number of clauses of size 510007
Number of clauses of size over 50

Solver Data 

0.09/0.11	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250252-1304089438.cnf 
0.09/0.11	c Using SATTIMEOUT of 4800
0.09/0.11	c Using SATRAM of 15500
0.09/0.11	c Size of subset node = 32
0.09/0.11	c Variables = 4229
0.09/0.11	c Clauses = 100397
0.09/0.11	c Literals = 411595
0.09/0.11	c Reduced size:
0.09/0.11	c Variables = 4229
0.09/0.11	c Clauses = 100321
0.09/0.11	c Literals = 411291
0.09/0.11	c a2
0.29/0.34	c a1
0.49/0.55	c Independent sub-problems = 2
0.49/0.55	c Sub-problem with 2103 variables and 49022 clauses
0.49/0.55	c using permutation method 4
0.49/0.55	c count = 1
0.49/0.55	c Ordering finished 0.540 seconds
0.49/0.55	c mapped 0.550 seconds
0.49/0.55	c permuted 0.550 seconds
0.49/0.55	c about to sort
0.49/0.56	c sorted
0.69/0.74	c clauses now 49022
0.69/0.74	c init finished 0.570 seconds
0.69/0.74	c tree built 0.730 seconds
0.69/0.74	c 
0.69/0.74	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.74	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.74	c ................................ | ..........                    |                                  0.7         0.0             
0.69/0.74	c ................................ | ..........                  1 |                                  0.7         1.4             
0.69/0.74	c ................................ | ..........                  2 |                                  0.7         2.7             
0.69/0.74	c ................................ | ..........                  3 |                                  0.7         4.1             
0.69/0.74	c ................................ | ..........                  4 |                                  0.7         5.5             
0.69/0.74	c ................................ | ..........                  5 |                                  0.7         6.8             
0.69/0.75	c ................................ | ..........               1000 | 2            266                 0.8      1315.8          565
96.01/24.61	c ................................ | ..........             910000 | 74794        740051             96.1      9473.2     18779093
191.88/48.67	c ................................ | ..........            1522000 | 130108       1194206           192.2      7920.1     38920839
287.77/72.66	c ................................ | ..........            2067000 | 174631       1590118           288.0      7177.1     57480167
384.03/96.76	c ................................ | ..........            2455000 | 211612       1866611           384.3      6388.7     73827503
479.88/120.76	c ................................ | *.........            2735000 | 243379       2112429           480.1      5696.3     88310564
575.75/144.78	c ................................ | *.........            3028000 | 278098       2346772           576.1      5256.1    104668985
672.02/168.81	c ................................ | *.........            3481000 | 315433       2609171           672.1      5179.4    116436523
767.89/192.85	c ................................ | *.........            3884000 | 347916       2847701           768.1      5056.5    127451509
863.74/216.89	c ................................ | *.........            4196000 | 380648       3055718           864.1      4855.7    138786003
960.42/241.02	c ................................ | **........            4467000 | 413874       3251482           960.5      4650.6    151245341
1055.88/264.98	c ................................ | **........            4720000 | 447358       3430062          1056.2      4468.7    162429846
1152.15/289.07	c ................................ | **........            4905000 | 470583       3586467          1152.5      4256.1    171813314
1248.41/313.13	c ................................ | **........            5065000 | 493043       3721020          1248.6      4056.7    182051098
1344.28/337.14	c ................................ | **........            5264000 | 520339       3850976          1344.5      3915.3    192599143
1440.15/361.16	c ................................ | ***.......            5473000 | 539683       4030348          1440.4      3799.7    200686333
1536.41/385.27	c ................................ | ***.......            5634000 | 557874       4192617          1536.7      3666.3    208885206
1632.31/409.27	c ................................ | ***.......            5768000 | 575351       4351203          1632.6      3533.0    217306286
1728.57/433.38	c ................................ | ***.......            5897000 | 591328       4495422          1728.9      3410.8    225833172
1824.02/457.26	c ................................ | ***.......            6006000 | 607581       4607198          1824.3      3292.3    234774688
1920.29/481.34	c ................................ | ****......            6149000 | 624312       4722019          1920.5      3201.8    243741294
2016.15/505.33	c ................................ | ****......            6278000 | 641554       4837949          2016.3      3113.6    251598702
2112.02/529.31	c ................................ | ****......            6424000 | 663839       4931921          2112.1      3041.5    259711183
2208.29/553.44	c ................................ | ****......            6555000 | 681217       5024102          2208.4      2968.2    267572054
2304.58/577.53	c ................................ | ****......            6667000 | 695902       5114227          2304.7      2892.8    274443670
2400.43/601.52	c ................................ | *****.....            6789000 | 711160       5201563          2400.5      2828.1    281187723
2495.90/625.45	c ................................ | *****.....            6876000 | 723543       5292665          2496.1      2754.7    287472190
2592.56/649.64	c ................................ | *****.....            6967000 | 736345       5390063          2592.8      2687.1    293754802
2690.04/674.02	c ................................ | *****.....            7029000 | 746563       5474286          2690.1      2612.9    299648088
2784.70/697.76	c ................................ | *****.....            7089000 | 757358       5543212          2785.0      2545.5    304723766
2880.18/721.64	c ................................ | ******....            7150000 | 769195       5615832          2880.3      2482.3    310303909
2977.63/746.00	c ................................ | ******....            7201000 | 779558       5689686          2977.6      2418.4    315715593
3073.53/770.01	c ................................ | ******....            7266000 | 790342       5777691          3073.6      2364.0    320571995
3168.96/793.91	c ................................ | ******....            7322000 | 799689       5863735          3169.0      2310.5    325513909
3265.23/818.06	c ................................ | ******....            7390000 | 810028       5957642          3265.5      2263.1    331727425
3360.30/841.87	c ................................ | *******...            7447000 | 819541       6034617          3360.6      2216.0    336991044
3456.17/865.85	c ................................ | *******...            7502000 | 828195       6110512          3456.4      2170.5    341932217
3554.05/890.34	c ................................ | *******...            7542000 | 835990       6183283          3554.2      2122.0    347068442
3648.33/913.91	c ................................ | *******...            7588000 | 844905       6246310          3648.4      2079.8    351560967
3744.21/937.93	c ................................ | *******...            7628000 | 853207       6311509          3744.3      2037.2    356380953
3841.30/962.29	c ................................ | ********..            7704000 | 864273       6374310          3841.7      2005.4    360784283
3936.78/986.15	c ................................ | ********..            7798000 | 876209       6440750          3937.0      1980.7    366677355
4032.26/1010.04	c ................................ | ********..            7876000 | 886643       6503182          4032.4      1953.2    372090044
4128.56/1034.14	c ................................ | ********..            7942000 | 895277       6566949          4128.7      1923.6    376849978
4224.86/1058.24	c ................................ | ********..            8010000 | 904728       6634378          4225.0      1895.9    381533976
4319.94/1082.03	c ................................ | *********.            8063000 | 913575       6693164          4320.1      1866.4    386127316
4416.22/1106.15	c ................................ | *********.            8112000 | 921351       6750375          4416.4      1836.8    390696161
4512.90/1130.31	c ................................ | *********.            8156000 | 928781       6818776          4513.0      1807.2    392752811
4609.19/1154.49	c ................................ | *********.            8192000 | 935282       6900096          4609.6      1777.2    398087330
4705.08/1178.40	c ................................ | *********.            8228000 | 942360       6980132          4705.1      1748.7    403562845

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250252-1304089438/watcher-3250252-1304089438 -o /tmp/evaluation-result-3250252-1304089438/solver-3250252-1304089438 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250252-1304089438.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.18 8.10 7.91 5/181 25461
/proc/meminfo: memFree=15706040/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) R 25459 25461 25048 0 -1 4202496 401 0 0 0 0 0 0 0 25 0 1 0 520544723 9633792 344 33554432000 4194304 4338100 140735993333328 18446744073709551615 270613110528 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/25461/statm: 2352 344 146 36 0 257 0

[startup+0.0612411 s]
/proc/loadavg: 8.18 8.10 7.91 5/181 25461
/proc/meminfo: memFree=15706040/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=16420 CPUtime=0.05 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) R 25459 25461 25048 0 -1 4202496 2127 0 0 0 5 0 0 0 25 0 1 0 520544723 16814080 2070 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612771525 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/25461/statm: 4105 2071 147 36 0 2010 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 16420

[startup+0.10024 s]
/proc/loadavg: 8.18 8.10 7.91 5/181 25461
/proc/meminfo: memFree=15706040/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=20876 CPUtime=0.09 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) R 25459 25461 25048 0 -1 4202496 3199 0 0 0 9 0 0 0 25 0 1 0 520544723 21377024 3142 33554432000 4194304 4338100 140735993333328 18446744073709551615 4209648 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/25461/statm: 5219 3142 150 36 0 3124 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20876

[startup+0.300231 s]
/proc/loadavg: 8.18 8.10 7.91 5/181 25461
/proc/meminfo: memFree=15706040/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=48856 CPUtime=0.29 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) R 25459 25461 25048 0 -1 4202496 10216 0 0 0 28 1 0 0 25 0 1 0 520544723 50028544 10158 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612506310 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/25461/statm: 12214 10158 153 36 0 10119 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 48856

[startup+0.701155 s]
/proc/loadavg: 8.18 8.10 7.91 5/181 25461
/proc/meminfo: memFree=15706040/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=85088 CPUtime=0.69 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) R 25459 25461 25048 0 -1 4202496 19663 0 0 0 66 3 0 0 25 0 1 0 520544723 87130112 19307 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612771525 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/25461/statm: 21272 19309 164 36 0 19177 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 85088

[startup+1.50095 s]
/proc/loadavg: 7.92 8.05 7.90 9/186 25466
/proc/meminfo: memFree=15607924/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=291800 CPUtime=3.75 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 27217 0 0 0 371 4 0 0 25 0 5 0 520544723 298803200 26860 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 72950 26861 176 36 0 70855 0
[pid=25461/tid=25463] ppid=25459 vsize=291800 CPUtime=0.75 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1756 0 0 0 75 0 0 0 18 0 5 0 520544797 298803200 26861 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=291800 CPUtime=0.75 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 737 0 0 0 75 0 0 0 18 0 5 0 520544797 298803200 26861 33554432000 4194304 4338100 140735993333328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=291800 CPUtime=0.74 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1302 0 0 0 74 0 0 0 25 0 5 0 520544797 298803200 26862 33554432000 4194304 4338100 140735993333328 18446744073709551615 4289168 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=291800 CPUtime=0.75 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1526 0 0 0 75 0 0 0 18 0 5 0 520544797 298803200 26862 33554432000 4194304 4338100 140735993333328 18446744073709551615 4287287 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3.75
Current children cumulated vsize (KiB) 291800

[startup+3.10067 s]
/proc/loadavg: 7.92 8.05 7.90 9/186 25466
/proc/meminfo: memFree=15578660/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=335368 CPUtime=10.14 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 37881 0 0 0 1006 8 0 0 25 0 5 0 520544723 343416832 37524 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 83842 37524 176 36 0 81747 0
[pid=25461/tid=25463] ppid=25459 vsize=335368 CPUtime=2.35 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 4844 0 0 0 234 1 0 0 22 0 5 0 520544797 343416832 37524 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=335368 CPUtime=2.34 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 3160 0 0 0 234 0 0 0 22 0 5 0 520544797 343416832 37525 33554432000 4194304 4338100 140735993333328 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=335368 CPUtime=2.33 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 3744 0 0 0 233 0 0 0 25 0 5 0 520544797 343416832 37525 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612506068 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=335368 CPUtime=2.34 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 4237 0 0 0 233 1 0 0 21 0 5 0 520544797 343416832 37525 33554432000 4194304 4338100 140735993333328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 10.14
Current children cumulated vsize (KiB) 335368

[startup+6.3011 s]
/proc/loadavg: 7.92 8.05 7.90 9/186 25466
/proc/meminfo: memFree=15478904/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=416604 CPUtime=22.93 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 59208 0 0 0 2281 12 0 0 25 0 5 0 520544723 426602496 58851 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 104151 58851 176 36 0 102056 0
[pid=25461/tid=25463] ppid=25459 vsize=416604 CPUtime=5.55 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 9855 0 0 0 553 2 0 0 25 0 5 0 520544797 426602496 58851 33554432000 4194304 4338100 140735993333328 18446744073709551615 4256025 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=416604 CPUtime=5.55 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 6582 0 0 0 554 1 0 0 25 0 5 0 520544797 426602496 58851 33554432000 4194304 4338100 140735993333328 18446744073709551615 4253920 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=416604 CPUtime=5.51 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 11528 0 0 0 549 2 0 0 25 0 5 0 520544797 426602496 58851 33554432000 4194304 4338100 140735993333328 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=416604 CPUtime=5.54 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 9346 0 0 0 552 2 0 0 25 0 5 0 520544797 426602496 58851 33554432000 4194304 4338100 140735993333328 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 22.93
Current children cumulated vsize (KiB) 416604

[startup+12.701 s]
/proc/loadavg: 7.93 8.04 7.90 9/186 25466
/proc/meminfo: memFree=15300376/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=579300 CPUtime=48.49 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 100127 0 0 0 4829 20 0 0 25 0 5 0 520544723 593203200 99770 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 144825 99770 176 36 0 142730 0
[pid=25461/tid=25463] ppid=25459 vsize=579300 CPUtime=11.95 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 19407 0 0 0 1191 4 0 0 25 0 5 0 520544797 593203200 99770 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=579300 CPUtime=11.95 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 13840 0 0 0 1192 3 0 0 25 0 5 0 520544797 593203200 99770 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612505843 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=579300 CPUtime=11.88 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 23979 0 0 0 1183 5 0 0 25 0 5 0 520544797 593203200 99770 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=579300 CPUtime=11.95 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 21004 0 0 0 1191 4 0 0 25 0 5 0 520544797 593203200 99770 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624923342 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 48.49
Current children cumulated vsize (KiB) 579300

[startup+25.5007 s]
/proc/loadavg: 7.94 8.04 7.90 9/186 25466
/proc/meminfo: memFree=14954124/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=903240 CPUtime=99.61 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 182389 0 0 0 9922 39 0 0 25 0 5 0 520544723 924917760 182032 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 225810 182032 176 36 0 223715 0
[pid=25461/tid=25463] ppid=25459 vsize=903240 CPUtime=24.7 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 35254 0 0 0 2463 7 0 0 25 0 5 0 520544797 924917760 182032 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612506016 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=903240 CPUtime=24.75 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 33411 0 0 0 2468 7 0 0 25 0 5 0 520544797 924917760 182032 33554432000 4194304 4338100 140735993333328 18446744073709551615 270612506362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=903240 CPUtime=24.64 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 43489 0 0 0 2454 10 0 0 25 0 5 0 520544797 924917760 182032 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=903240 CPUtime=24.75 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 48338 0 0 0 2465 10 0 0 25 0 5 0 520544797 924917760 182032 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 99.61
Current children cumulated vsize (KiB) 903240

[startup+51.1012 s]
/proc/loadavg: 8.04 8.05 7.91 9/186 25467
/proc/meminfo: memFree=14248860/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=1572992 CPUtime=201.87 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 352554 0 0 0 20114 73 0 0 25 0 5 0 520544723 1610743808 352197 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 393248 352197 176 36 0 391153 0
[pid=25461/tid=25463] ppid=25459 vsize=1572992 CPUtime=50.17 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 72293 0 0 0 5002 15 0 0 25 0 5 0 520544797 1610743808 352197 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=1572992 CPUtime=50.34 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 75701 0 0 0 5019 15 0 0 25 0 5 0 520544797 1610743808 352197 33554432000 4194304 4338100 140735993333328 18446744073709551615 4285817 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=1572992 CPUtime=50.24 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 79439 0 0 0 5007 17 0 0 25 0 5 0 520544797 1610743808 352197 33554432000 4194304 4338100 140735993333328 18446744073709551615 4284198 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=1572992 CPUtime=50.35 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 103224 0 0 0 5014 21 0 0 25 0 5 0 520544797 1610743808 352197 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 201.87
Current children cumulated vsize (KiB) 1572992

[startup+102.306 s]
/proc/loadavg: 8.14 8.07 7.92 9/186 25468
/proc/meminfo: memFree=12954764/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=2725368 CPUtime=406.4 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 643302 0 0 0 40515 125 0 0 25 0 5 0 520544723 2790776832 642945 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 681342 642945 176 36 0 679247 0
[pid=25461/tid=25463] ppid=25459 vsize=2725368 CPUtime=101.1 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 113050 0 0 0 10087 23 0 0 25 0 5 0 520544797 2790776832 642945 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=2725368 CPUtime=101.55 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 163078 0 0 0 10125 30 0 0 25 0 5 0 520544797 2790776832 642945 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=2725368 CPUtime=101.44 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 143764 0 0 0 10114 30 0 0 25 0 5 0 520544797 2790776832 642945 33554432000 4194304 4338100 140735993333328 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=2725368 CPUtime=101.55 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 201513 0 0 0 10117 38 0 0 25 0 5 0 520544797 2790776832 642946 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 406.4
Current children cumulated vsize (KiB) 2725368

[startup+162.301 s]
/proc/loadavg: 8.17 8.09 7.93 9/186 25470
/proc/meminfo: memFree=11700488/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=3878136 CPUtime=646.05 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 933820 0 0 0 64427 178 0 0 25 0 5 0 520544723 3971211264 933463 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 969534 933463 176 36 0 967439 0
[pid=25461/tid=25463] ppid=25459 vsize=3878136 CPUtime=160.76 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 170396 0 0 0 16040 36 0 0 25 0 5 0 520544797 3971211264 933463 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=3878136 CPUtime=161.55 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 246632 0 0 0 16111 44 0 0 25 0 5 0 520544797 3971211264 933463 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=3878136 CPUtime=161.43 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 201847 0 0 0 16102 41 0 0 25 0 5 0 520544797 3971211264 933463 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=3878136 CPUtime=161.55 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 293048 0 0 0 16102 53 0 0 25 0 5 0 520544797 3971211264 933463 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 646.05
Current children cumulated vsize (KiB) 3878136


################
# More data... #
################

/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 865076 0 0 0 75981 175 0 0 25 0 5 0 520544797 10629611520 2571945 33554432000 4194304 4338100 140735993333328 18446744073709551615 4241600 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3042.77
Current children cumulated vsize (KiB) 10380480

[startup+822.301 s]
/proc/loadavg: 7.91 8.08 8.00 5/177 25506
/proc/meminfo: memFree=6651812/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=10798880 CPUtime=3282.41 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 2679227 0 0 0 327665 576 0 0 25 0 5 0 520544723 11058053120 2678870 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 2699720 2678870 176 36 0 2697625 0
[pid=25461/tid=25463] ppid=25459 vsize=10798880 CPUtime=820.56 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 505398 0 0 0 81932 124 0 0 25 0 5 0 520544797 11058053120 2678870 33554432000 4194304 4338100 140735993333328 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=10798880 CPUtime=821.1 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 713031 0 0 0 81969 141 0 0 25 0 5 0 520544797 11058053120 2678870 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=10798880 CPUtime=818.42 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 536795 0 0 0 81720 122 0 0 25 0 5 0 520544797 11058053120 2678870 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=10798880 CPUtime=821.56 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 902106 0 0 0 81972 184 0 0 25 0 5 0 520544797 11058053120 2678870 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3282.41
Current children cumulated vsize (KiB) 10798880

[startup+882.301 s]
/proc/loadavg: 5.51 7.35 7.75 5/176 25507
/proc/meminfo: memFree=6242652/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=11207500 CPUtime=3522.09 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 2780822 0 0 0 351602 607 0 0 25 0 5 0 520544723 11476480000 2780465 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 2801875 2780465 176 36 0 2799780 0
[pid=25461/tid=25463] ppid=25459 vsize=11207500 CPUtime=880.56 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 522548 0 0 0 87927 129 0 0 25 0 5 0 520544797 11476480000 2780465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=11207500 CPUtime=881.09 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 735729 0 0 0 87961 148 0 0 25 0 5 0 520544797 11476480000 2780465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=11207500 CPUtime=878.11 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 561732 0 0 0 87682 129 0 0 25 0 5 0 520544797 11476480000 2780465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=11207500 CPUtime=881.56 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 938916 0 0 0 87960 196 0 0 25 0 5 0 520544797 11476480000 2780465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3522.09
Current children cumulated vsize (KiB) 11207500

[startup+942.301 s]
/proc/loadavg: 4.62 6.76 7.52 5/176 25509
/proc/meminfo: memFree=5863044/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=11587004 CPUtime=3761.8 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 2876486 0 0 0 375546 634 0 0 25 0 5 0 520544723 11865092096 2876129 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 2896751 2876129 176 36 0 2894656 0
[pid=25461/tid=25463] ppid=25459 vsize=11587004 CPUtime=940.56 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 536091 0 0 0 93923 133 0 0 25 0 5 0 520544797 11865092096 2876129 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=11587004 CPUtime=941.08 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 756837 0 0 0 93955 153 0 0 25 0 5 0 520544797 11865092096 2876129 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=11587004 CPUtime=937.82 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 583305 0 0 0 93648 134 0 0 25 0 5 0 520544797 11865092096 2876129 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=11587004 CPUtime=941.56 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 978356 0 0 0 93948 208 0 0 25 0 5 0 520544797 11865092096 2876129 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3761.8
Current children cumulated vsize (KiB) 11587004

[startup+1002.3 s]
/proc/loadavg: 4.23 6.25 7.30 5/176 25510
/proc/meminfo: memFree=5449200/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=11988972 CPUtime=4001.5 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 2978645 0 0 0 399485 665 0 0 25 0 5 0 520544723 12276707328 2978288 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 2997243 2978288 176 36 0 2995148 0
[pid=25461/tid=25463] ppid=25459 vsize=11988972 CPUtime=1000.57 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 557127 0 0 0 99917 140 0 0 25 0 5 0 520544797 12276707328 2978288 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=11988972 CPUtime=1001.07 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 781263 0 0 0 99947 160 0 0 25 0 5 0 520544797 12276707328 2978288 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=11988972 CPUtime=997.53 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 607126 0 0 0 99610 143 0 0 25 0 5 0 520544797 12276707328 2978288 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=11988972 CPUtime=1001.56 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1011232 0 0 0 99939 217 0 0 25 0 5 0 520544797 12276707328 2978288 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4001.5
Current children cumulated vsize (KiB) 11988972

[startup+1062.3 s]
/proc/loadavg: 4.08 5.84 7.09 5/176 25512
/proc/meminfo: memFree=5069748/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=12372304 CPUtime=4241.22 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 3074372 0 0 0 423430 692 0 0 25 0 5 0 520544723 12669239296 3074015 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 3093076 3074015 176 36 0 3090981 0
[pid=25461/tid=25463] ppid=25459 vsize=12372304 CPUtime=1060.57 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 574221 0 0 0 105912 145 0 0 25 0 5 0 520544797 12669239296 3074015 33554432000 4194304 4338100 140735993333328 18446744073709551615 4288662 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=12372304 CPUtime=1061.08 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 803581 0 0 0 105941 167 0 0 25 0 5 0 520544797 12669239296 3074015 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=12372304 CPUtime=1057.26 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 629115 0 0 0 105577 149 0 0 25 0 5 0 520544797 12669239296 3074015 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=12372304 CPUtime=1061.56 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1045558 0 0 0 105929 227 0 0 25 0 5 0 520544797 12669239296 3074015 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4241.22
Current children cumulated vsize (KiB) 12372304

[startup+1122.3 s]
/proc/loadavg: 4.09 5.52 6.90 5/176 25514
/proc/meminfo: memFree=4730908/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=12790080 CPUtime=4480.93 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 3157643 0 0 0 447379 714 0 0 25 0 5 0 520544723 13097041920 3157286 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 3197520 3157286 176 36 0 3195425 0
[pid=25461/tid=25463] ppid=25459 vsize=12790080 CPUtime=1120.57 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 588849 0 0 0 111908 149 0 0 25 0 5 0 520544797 13097041920 3157286 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=12790080 CPUtime=1121.07 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 823555 0 0 0 111935 172 0 0 25 0 5 0 520544797 13097041920 3157286 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=12790080 CPUtime=1117.06 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 648258 0 0 0 111552 154 0 0 25 0 5 0 520544797 13097041920 3157286 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=12790080 CPUtime=1121.46 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1075084 0 0 0 111912 234 0 0 25 0 5 0 520544797 13097041920 3157286 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4480.93
Current children cumulated vsize (KiB) 12790080

[startup+1182.3 s]
/proc/loadavg: 4.03 5.24 6.71 5/176 25515
/proc/meminfo: memFree=4339956/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=13183192 CPUtime=4720.67 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 3256822 0 0 0 471324 743 0 0 25 0 5 0 520544723 13499588608 3256465 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 3295798 3256465 176 36 0 3293703 0
[pid=25461/tid=25463] ppid=25459 vsize=13183192 CPUtime=1180.57 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 603035 0 0 0 117904 153 0 0 25 0 5 0 520544797 13499588608 3256465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=13183192 CPUtime=1181.07 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 874470 0 0 0 117921 186 0 0 25 0 5 0 520544797 13499588608 3256465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=13183192 CPUtime=1177.06 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 648547 0 0 0 117551 155 0 0 25 0 5 0 520544797 13499588608 3256465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=13183192 CPUtime=1181.19 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1108873 0 0 0 117876 243 0 0 25 0 5 0 520544797 13499588608 3256465 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4720.67
Current children cumulated vsize (KiB) 13183192



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 4.02 5.16 6.65 5/176 25516
/proc/meminfo: memFree=4207756/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=13305976 CPUtime=4800.2 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) S 25459 25461 25048 0 -1 4202496 3288103 0 0 0 479267 753 0 0 25 0 5 0 520544723 13625319424 3287746 33554432000 4194304 4338100 140735993333328 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 3326494 3287746 176 36 0 3324399 0
[pid=25461/tid=25463] ppid=25459 vsize=13305976 CPUtime=1200.47 cores=0,2,4,6
/proc/25461/task/25463/stat : 25463 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 607945 0 0 0 119893 154 0 0 25 0 5 0 520544797 13625319424 3287746 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=25461/tid=25464] ppid=25459 vsize=13305976 CPUtime=1200.98 cores=0,2,4,6
/proc/25461/task/25464/stat : 25464 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 890134 0 0 0 119907 191 0 0 25 0 5 0 520544797 13625319424 3287746 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=25461/tid=25465] ppid=25459 vsize=13305976 CPUtime=1196.97 cores=0,2,4,6
/proc/25461/task/25465/stat : 25465 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 648649 0 0 0 119541 156 0 0 25 0 5 0 520544797 13625319424 3287746 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=13305976 CPUtime=1201 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4202560 1119478 0 0 0 119853 247 0 0 25 0 5 0 520544797 13625319424 3287746 33554432000 4194304 4338100 140735993333328 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 13305976

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.6 s]
/proc/loadavg: 4.02 5.16 6.65 5/176 25516
/proc/meminfo: memFree=4207756/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=0 CPUtime=4800.62 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) Z 25459 25461 25048 0 -1 4203548 3288103 0 0 0 479270 792 0 0 18 0 2 0 520544723 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 0 0 0 0 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=0 CPUtime=1201.39 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4203588 1119478 0 0 0 119853 286 0 0 25 0 2 0 520544797 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.62
Current children cumulated vsize (KiB) 0

[startup+1202.8 s]
/proc/loadavg: 4.02 5.16 6.65 5/176 25516
/proc/meminfo: memFree=4207756/32951124 swapFree=67111528/67111528
[pid=25461] ppid=25459 vsize=0 CPUtime=4800.82 cores=0,2,4,6
/proc/25461/stat : 25461 (ttsth-5-4) Z 25459 25461 25048 0 -1 4203548 3288103 0 0 0 479270 812 0 0 18 0 2 0 520544723 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/25461/statm: 0 0 0 0 0 0 0
[pid=25461/tid=25466] ppid=25459 vsize=0 CPUtime=1201.59 cores=0,2,4,6
/proc/25461/task/25466/stat : 25466 (ttsth-5-4) R 25459 25461 25048 0 -1 4203588 1119478 0 0 0 119853 306 0 0 25 0 2 0 520544797 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.82
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.83
CPU time (s): 4800.85
CPU user time (s): 4792.7
CPU system time (s): 8.14676
CPU usage (%): 399.131
Max. virtual memory (cumulated for all children) (KiB): 13305976

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4792.7
system time used= 8.14676
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3288103
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5905
involuntary context switches= 28742

runsolver used 2.10068 second user time and 4.21036 second system time

The end

Launcher Data 

Begin job on node108 at 2011-04-29 17:03:58
IDJOB=3250252
IDBENCH=83248
IDSOLVER=1654
FILE ID=node108/3250252-1304089438
RUNJOBID= node108-1304086608-25066
PBS_JOBID= 13170551
Free space on /tmp= 72120 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-4-5_4231.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250252-1304089438/watcher-3250252-1304089438 -o /tmp/evaluation-result-3250252-1304089438/solver-3250252-1304089438 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250252-1304089438.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a52d60bf6e72caa628ad712a5fe0fb7a
RANDOM SEED=1718414200

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15706088 kB
Buffers:       2164636 kB
Cached:       12758572 kB
SwapCached:          0 kB
Active:        5880540 kB
Inactive:     10779784 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15706088 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            3928 kB
Writeback:           0 kB
AnonPages:     1736776 kB
Mapped:          15220 kB
Slab:           517912 kB
PageTables:       8080 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  1955100 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72120 MiB
End job on node108 at 2011-04-29 17:24:02