Trace number 3264378

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.16 1201.87

General information on the benchmark

NameSAT04/handmade/anton/l4k4/
lksat-n900-m6174-k4-l4-s819398222.used-as.sat04-929.cnf
MD5SUM82170fa4021e7a6c932a31c197f0f64c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.070989
Satisfiable
(Un)Satisfiability was proved
Number of variables880
Number of clauses6174
Sum of the clauses size24696
Maximum clause length4
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 46174
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264378-1304099709.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 880
0.00/0.00	c Clauses = 6174
0.00/0.00	c Literals = 24696
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 880
0.00/0.00	c Clauses = 6174
0.00/0.00	c Literals = 24696
0.00/0.00	c a2
0.00/0.01	c a1
0.10/0.13	c Independent sub-problems = 1
0.10/0.13	c Sub-problem with 880 variables and 6174 clauses
0.10/0.13	c using permutation method 4
0.10/0.13	c count = 2
0.10/0.13	c Ordering finished 0.130 seconds
0.10/0.13	c mapped 0.130 seconds
0.10/0.13	c permuted 0.130 seconds
0.10/0.13	c about to sort
0.10/0.13	c sorted
0.10/0.15	c clauses now 6174
0.10/0.15	c init finished 0.140 seconds
0.10/0.15	c tree built 0.150 seconds
0.10/0.15	c 
0.10/0.15	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.10/0.15	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.10/0.15	c ................................ | ..........                    |                                  0.1         0.0             
0.10/0.15	c ................................ | ..........                  1 |                                  0.1         6.7             
0.10/0.15	c ................................ | ..........                  2 |                                  0.1        13.3             
0.10/0.15	c ................................ | ..........                  3 |                                  0.1        20.0             
0.10/0.15	c ................................ | ..........                  4 |                                  0.1        26.7             
0.10/0.15	c ................................ | ..........                  5 |                                  0.1        33.3             
0.10/0.15	c ................................ | ..........               1000 | 8            207                 0.1      6666.7           65
95.81/24.17	c ................................ | ..........            1827000 | 180923       1071484            96.1     19015.4      6796687
191.66/48.18	c ................................ | ..........            2779000 | 291178       1699214           192.0     14474.0     12033488
287.92/72.23	c ................................ | ..........            3673000 | 395724       2250169           288.1     12751.3     17144027
383.80/96.26	c ................................ | ..........            4334000 | 477312       2616993           384.0     11285.3     21952147
480.07/120.30	c ................................ | *.........            4911000 | 555338       2941684           480.1     10229.5     25251170
575.93/144.36	c ................................ | *.........            5401000 | 619731       3195777           576.2      9374.0     28934977
671.82/168.36	c ................................ | *.........            6034000 | 683678       3522823           672.0      8978.5     32193114
768.09/192.41	c ................................ | *.........            6540000 | 739540       3764288           768.1      8514.2     34313730
863.94/216.44	c ................................ | *.........            6975000 | 788127       3974770           864.1      8071.8     36182498
960.22/240.50	c ................................ | **........            7377000 | 835371       4151708           960.2      7682.6     38657764
1056.10/264.51	c ................................ | **........            7744000 | 879268       4332950          1056.1      7332.4     40930641
1151.97/288.53	c ................................ | **........            8061000 | 915051       4478135          1152.1      6996.8     43017150
1247.84/312.57	c ................................ | **........            8368000 | 949075       4618619          1248.1      6704.4     45237701
1344.13/336.62	c ................................ | **........            8707000 | 983339       4755730          1344.2      6477.5     46991360
1439.98/360.66	c ................................ | ***.......            9039000 | 1018903      4883814          1440.2      6276.0     48552861
1535.83/384.68	c ................................ | ***.......            9278000 | 1052054      4992564          1536.2      6039.7     50135401
1632.10/408.78	c ................................ | ***.......            9480000 | 1080286      5090718          1632.5      5807.2     52113107
1727.96/432.79	c ................................ | ***.......            9662000 | 1108719      5176061          1728.3      5590.4     53423304
1824.63/456.91	c ................................ | ***.......            9857000 | 1136144      5271367          1824.7      5402.0     54790665
1920.11/480.82	c ................................ | ****......           10051000 | 1163651      5358503          1920.2      5234.4     56561748
2016.40/504.93	c ................................ | ****......           10218000 | 1190611      5435979          2016.5      5067.1     58607788
2111.84/528.84	c ................................ | ****......           10429000 | 1216823      5526915          2112.0      4938.0     59679937
2208.10/552.94	c ................................ | ****......           10706000 | 1248621      5648407          2208.3      4848.1     60733758
2303.96/576.92	c ................................ | ****......           11036000 | 1283741      5760274          2304.1      4789.8     61634696
2399.80/600.95	c ................................ | *****.....           11364000 | 1323094      5870420          2400.0      4734.9     62994444
2495.68/625.00	c ................................ | *****.....           11649000 | 1356627      5968982          2496.1      4666.9     64262649
2591.97/649.04	c ................................ | *****.....           11921000 | 1391636      6071948          2592.1      4599.0     65644854
2687.82/673.07	c ................................ | *****.....           12157000 | 1426477      6161306          2688.1      4522.5     67054444
2783.69/697.08	c ................................ | *****.....           12353000 | 1455117      6243304          2784.0      4437.1     68677734
2879.97/721.19	c ................................ | ******....           12585000 | 1488637      6344696          2880.4      4369.2     70448387
2975.84/745.17	c ................................ | ******....           12783000 | 1515749      6446097          2976.2      4295.1     71909411
3072.11/769.22	c ................................ | ******....           12964000 | 1540029      6540092          3072.2      4219.8     73030464
3167.99/793.22	c ................................ | ******....           13137000 | 1564799      6632382          3168.1      4146.6     74200066
3263.89/817.28	c ................................ | ******....           13344000 | 1592132      6726696          3264.2      4088.0     75445104
3360.14/841.33	c ................................ | *******...           13560000 | 1616474      6817362          3360.3      4035.4     76757914
3456.00/865.32	c ................................ | *******...           13748000 | 1641890      6896983          3456.1      3977.9     77896010
3551.87/889.34	c ................................ | *******...           13965000 | 1668736      6978738          3552.0      3931.5     78789585
3648.14/913.46	c ................................ | *******...           14201000 | 1698122      7073782          3648.4      3892.4     79853577
3744.01/937.49	c ................................ | *******...           14447000 | 1729541      7172783          3744.4      3858.3     80968484
3839.92/961.49	c ................................ | ********..           14700000 | 1761111      7269154          3840.3      3827.8     82065946
3935.79/985.48	c ................................ | ********..           14911000 | 1784671      7374399          3936.1      3788.2     83385309
4032.06/1009.53	c ................................ | ********..           15121000 | 1809350      7457751          4032.2      3750.1     84655022
4127.94/1033.59	c ................................ | ********..           15294000 | 1832567      7548190          4128.3      3704.7     85932590
4224.22/1057.68	c ................................ | ********..           15468000 | 1858280      7650421          4224.5      3661.5     87198169
4320.09/1081.62	c ................................ | *********.           15641000 | 1881400      7722318          4320.2      3620.4     88117694
4415.97/1105.63	c ................................ | *********.           15808000 | 1904009      7801092          4416.1      3579.6     88911152
4512.27/1129.71	c ................................ | *********.           16043000 | 1933473      7888490          4512.3      3555.4     89981534
4608.12/1153.74	c ................................ | *********.           16237000 | 1958194      7973697          4608.3      3523.4     91283510
4703.99/1177.78	c ................................ | *********.           16453000 | 1988611      8072724          4704.3      3497.4     92719557
4799.86/1201.73	c CPU limit (4800) reached
4799.86/1201.73	c Time = 4800.02 seconds
4799.86/1201.73	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264378-1304099709/watcher-3264378-1304099709 -o /tmp/evaluation-result-3264378-1304099709/solver-3264378-1304099709 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264378-1304099709.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.93 7.95 7.91 5/179 13131
/proc/meminfo: memFree=21667776/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=9244 CPUtime=0 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) R 13129 13131 12928 0 -1 4202496 383 0 0 0 0 0 0 0 25 0 1 0 521569510 9465856 326 33554432000 4194304 4338100 140737164093392 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/13131/statm: 2311 327 147 36 0 216 0

[startup+0.0477959 s]
/proc/loadavg: 7.93 7.95 7.91 5/179 13131
/proc/meminfo: memFree=21667776/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=11124 CPUtime=0.04 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) R 13129 13131 12928 0 -1 4202496 877 0 0 0 4 0 0 0 25 0 1 0 521569510 11390976 819 33554432000 4194304 4338100 140737164093392 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/13131/statm: 2781 819 155 36 0 686 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 11124

[startup+0.100786 s]
/proc/loadavg: 7.93 7.95 7.91 5/179 13131
/proc/meminfo: memFree=21667776/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=11124 CPUtime=0.1 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) R 13129 13131 12928 0 -1 4202496 877 0 0 0 10 0 0 0 25 0 1 0 521569510 11390976 819 33554432000 4194304 4338100 140737164093392 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/13131/statm: 2781 819 155 36 0 686 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 11124

[startup+0.300707 s]
/proc/loadavg: 7.93 7.95 7.91 5/179 13131
/proc/meminfo: memFree=21667776/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=207444 CPUtime=0.73 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 4529 0 0 0 73 0 0 0 25 0 5 0 521569510 212422656 4471 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 51861 4471 175 36 0 49766 0
Current children cumulated CPU time (s) 0.73
Current children cumulated vsize (KiB) 207444

[startup+0.700638 s]
/proc/loadavg: 7.93 7.95 7.91 5/179 13131
/proc/meminfo: memFree=21667776/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=211740 CPUtime=2.33 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 6824 0 0 0 232 1 0 0 25 0 5 0 521569510 216821760 6766 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 52935 6766 175 36 0 50840 0
Current children cumulated CPU time (s) 2.33
Current children cumulated vsize (KiB) 211740

[startup+1.5005 s]
/proc/loadavg: 8.01 7.97 7.91 10/184 13136
/proc/meminfo: memFree=21636000/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=222164 CPUtime=5.52 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 10138 0 0 0 549 3 0 0 25 0 5 0 521569510 227495936 10080 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 55541 10080 175 36 0 53446 0
[pid=13131/tid=13133] ppid=13129 vsize=222164 CPUtime=1.34 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 1100 0 0 0 133 1 0 0 25 0 5 0 521569525 227495936 10080 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=222164 CPUtime=1.34 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 1520 0 0 0 134 0 0 0 25 0 5 0 521569525 227495936 10080 33554432000 4194304 4338100 140737164093392 18446744073709551615 242969945569 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=222164 CPUtime=1.33 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 1580 0 0 0 133 0 0 0 25 0 5 0 521569525 227495936 10080 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=222164 CPUtime=1.33 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 2422 0 0 0 133 0 0 0 25 0 5 0 521569525 227495936 10080 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5.52
Current children cumulated vsize (KiB) 222164

[startup+3.10023 s]
/proc/loadavg: 8.01 7.97 7.91 10/184 13136
/proc/meminfo: memFree=21616660/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=242480 CPUtime=11.91 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 16190 0 0 0 1180 11 0 0 25 0 5 0 521569510 248299520 16132 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 60620 16132 175 36 0 58525 0
[pid=13131/tid=13133] ppid=13129 vsize=242480 CPUtime=2.94 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 2187 0 0 0 289 5 0 0 25 0 5 0 521569525 248299520 16132 33554432000 4194304 4338100 140737164093392 18446744073709551615 242970593463 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=242480 CPUtime=2.93 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 2870 0 0 0 293 0 0 0 25 0 5 0 521569525 248299520 16133 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982362830 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=242480 CPUtime=2.93 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 2752 0 0 0 289 4 0 0 25 0 5 0 521569525 248299520 16133 33554432000 4194304 4338100 140737164093392 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=242480 CPUtime=2.92 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 4865 0 0 0 292 0 0 0 25 0 5 0 521569525 248299520 16133 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.91
Current children cumulated vsize (KiB) 242480

[startup+6.30068 s]
/proc/loadavg: 8.01 7.97 7.91 10/184 13136
/proc/meminfo: memFree=21574548/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=278940 CPUtime=24.69 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 25283 0 0 0 2441 28 0 0 25 0 5 0 521569510 285634560 25225 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 69735 25225 175 36 0 67640 0
[pid=13131/tid=13133] ppid=13129 vsize=278940 CPUtime=6.13 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 3942 0 0 0 601 12 0 0 25 0 5 0 521569525 285634560 25225 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=278940 CPUtime=6.14 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 5153 0 0 0 613 1 0 0 25 0 5 0 521569525 285634560 25225 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=278940 CPUtime=6.13 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 4480 0 0 0 601 12 0 0 25 0 5 0 521569525 285634560 25225 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=278940 CPUtime=6.1 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 8192 0 0 0 609 1 0 0 25 0 5 0 521569525 285634560 25225 33554432000 4194304 4338100 140737164093392 18446744073709551615 242970218924 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.69
Current children cumulated vsize (KiB) 278940

[startup+12.7006 s]
/proc/loadavg: 8.09 7.98 7.92 9/184 13168
/proc/meminfo: memFree=26413904/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=350972 CPUtime=50.25 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 43831 0 0 0 4967 58 0 0 25 0 5 0 521569510 359395328 43773 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 87743 43773 175 36 0 85648 0
[pid=13131/tid=13133] ppid=13129 vsize=350972 CPUtime=12.53 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 7782 0 0 0 1227 26 0 0 25 0 5 0 521569525 359395328 43773 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=350972 CPUtime=12.53 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 9391 0 0 0 1251 2 0 0 25 0 5 0 521569525 359395328 43773 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=350972 CPUtime=12.54 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 8188 0 0 0 1228 26 0 0 25 0 5 0 521569525 359395328 43773 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=350972 CPUtime=12.47 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 14954 0 0 0 1244 3 0 0 25 0 5 0 521569525 359395328 43773 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.25
Current children cumulated vsize (KiB) 350972

[startup+25.5004 s]
/proc/loadavg: 8.08 7.98 7.92 9/184 13168
/proc/meminfo: memFree=26106884/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=435800 CPUtime=101.39 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 66477 0 0 0 10038 101 0 0 25 0 5 0 521569510 446259200 66419 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 108950 66419 175 36 0 106855 0
[pid=13131/tid=13133] ppid=13129 vsize=435800 CPUtime=25.33 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 12585 0 0 0 2487 46 0 0 25 0 5 0 521569525 446259200 66419 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=435800 CPUtime=25.33 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 15075 0 0 0 2529 4 0 0 25 0 5 0 521569525 446259200 66419 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=435800 CPUtime=25.33 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 12583 0 0 0 2489 44 0 0 25 0 5 0 521569525 446259200 66419 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=435800 CPUtime=25.2 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 22718 0 0 0 2515 5 0 0 25 0 5 0 521569525 446259200 66419 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.39
Current children cumulated vsize (KiB) 435800

[startup+51.101 s]
/proc/loadavg: 8.05 7.98 7.92 9/184 13169
/proc/meminfo: memFree=25540192/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=688796 CPUtime=203.64 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 116678 0 0 0 20171 193 0 0 25 0 5 0 521569510 705327104 116620 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 172199 116620 175 36 0 170104 0
[pid=13131/tid=13133] ppid=13129 vsize=688796 CPUtime=50.87 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 21183 0 0 0 5001 86 0 0 25 0 5 0 521569525 705327104 116620 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=688796 CPUtime=50.93 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 26938 0 0 0 5086 7 0 0 25 0 5 0 521569525 705327104 116620 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=688796 CPUtime=50.93 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 21282 0 0 0 5008 85 0 0 25 0 5 0 521569525 705327104 116620 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=688796 CPUtime=50.71 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 43759 0 0 0 5059 12 0 0 25 0 5 0 521569525 705327104 116620 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.64
Current children cumulated vsize (KiB) 688796

[startup+102.306 s]
/proc/loadavg: 8.06 8.00 7.93 9/184 13171
/proc/meminfo: memFree=24726368/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=1005492 CPUtime=408.18 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 200873 0 0 0 40475 343 0 0 25 0 5 0 521569510 1029623808 200807 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 251373 200807 176 36 0 249278 0
[pid=13131/tid=13133] ppid=13129 vsize=1005492 CPUtime=101.81 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 37868 0 0 0 10028 153 0 0 25 0 5 0 521569525 1029623808 200807 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=1005492 CPUtime=102.14 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 43694 0 0 0 10201 13 0 0 25 0 5 0 521569525 1029623808 200807 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=1005492 CPUtime=102.13 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 38214 0 0 0 10060 153 0 0 25 0 5 0 521569525 1029623808 200807 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=1005492 CPUtime=101.92 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 77581 0 0 0 10169 23 0 0 25 0 5 0 521569525 1029623808 200807 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 408.18
Current children cumulated vsize (KiB) 1005492

[startup+162.301 s]
/proc/loadavg: 8.06 8.01 7.93 9/184 13172
/proc/meminfo: memFree=23987800/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=1292572 CPUtime=647.84 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 276015 0 0 0 64297 487 0 0 25 0 5 0 521569510 1323593728 275949 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 323143 275949 176 36 0 321048 0
[pid=13131/tid=13133] ppid=13129 vsize=1292572 CPUtime=161.49 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 49547 0 0 0 15933 216 0 0 25 0 5 0 521569525 1323593728 275949 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=1292572 CPUtime=162.13 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 61477 0 0 0 16195 18 0 0 25 0 5 0 521569525 1323593728 275949 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=1292572 CPUtime=162.13 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 48779 0 0 0 15996 217 0 0 25 0 5 0 521569525 1323593728 275949 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=1292572 CPUtime=161.92 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 112696 0 0 0 16157 35 0 0 25 0 5 0 521569525 1323593728 275949 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.84
Current children cumulated vsize (KiB) 1292572


################
# More data... #
################

/proc/meminfo: memFree=19632564/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=2491648 CPUtime=2804.86 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 589118 0 0 0 279241 1245 0 0 25 0 5 0 521569510 2551447552 589052 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 622912 589052 176 36 0 620817 0
[pid=13131/tid=13133] ppid=13129 vsize=2491648 CPUtime=699.59 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 110316 0 0 0 69394 565 0 0 25 0 5 0 521569525 2551447552 589052 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=2491648 CPUtime=702.1 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 160008 0 0 0 70159 51 0 0 25 0 5 0 521569525 2551447552 589052 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=2491648 CPUtime=701.1 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 103775 0 0 0 69552 558 0 0 25 0 5 0 521569525 2551447552 589052 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=2491648 CPUtime=701.89 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 211503 0 0 0 70119 70 0 0 25 0 5 0 521569525 2551447552 589052 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2804.86
Current children cumulated vsize (KiB) 2491648

[startup+762.306 s]
/proc/loadavg: 8.27 8.18 8.03 9/184 13200
/proc/meminfo: memFree=19194088/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=2612304 CPUtime=3044.57 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 620240 0 0 0 303150 1307 0 0 25 0 5 0 521569510 2674999296 620174 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 653076 620174 176 36 0 650981 0
[pid=13131/tid=13133] ppid=13129 vsize=2612304 CPUtime=759.59 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 117503 0 0 0 75368 591 0 0 25 0 5 0 521569525 2674999296 620174 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=2612304 CPUtime=762.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 166754 0 0 0 76157 54 0 0 25 0 5 0 521569525 2674999296 620174 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=2612304 CPUtime=760.8 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 110335 0 0 0 75492 588 0 0 25 0 5 0 521569525 2674999296 620174 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=2612304 CPUtime=761.89 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 222132 0 0 0 76116 73 0 0 25 0 5 0 521569525 2674999296 620174 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3044.57
Current children cumulated vsize (KiB) 2612304

[startup+822.301 s]
/proc/loadavg: 8.20 8.17 8.03 9/184 13202
/proc/meminfo: memFree=18816960/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=2708064 CPUtime=3284.24 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 644794 0 0 0 327055 1369 0 0 25 0 5 0 521569510 2773057536 644728 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 677016 644728 176 36 0 674921 0
[pid=13131/tid=13133] ppid=13129 vsize=2708064 CPUtime=819.59 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 122209 0 0 0 81339 620 0 0 25 0 5 0 521569525 2773057536 644728 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=2708064 CPUtime=822.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 174488 0 0 0 82154 57 0 0 25 0 5 0 521569525 2773057536 644728 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=2708064 CPUtime=820.48 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 114461 0 0 0 81433 615 0 0 25 0 5 0 521569525 2773057536 644728 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=2708064 CPUtime=821.88 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 230120 0 0 0 82112 76 0 0 25 0 5 0 521569525 2773057536 644728 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3284.24
Current children cumulated vsize (KiB) 2708064

[startup+882.301 s]
/proc/loadavg: 8.15 8.17 8.04 9/184 13203
/proc/meminfo: memFree=18442348/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=2804736 CPUtime=3523.91 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 670279 0 0 0 350962 1429 0 0 25 0 5 0 521569510 2872049664 670213 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 701184 670213 176 36 0 699089 0
[pid=13131/tid=13133] ppid=13129 vsize=2804736 CPUtime=879.58 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 126480 0 0 0 87312 646 0 0 25 0 5 0 521569525 2872049664 670213 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=2804736 CPUtime=882.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 186885 0 0 0 88149 62 0 0 25 0 5 0 521569525 2872049664 670213 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=2804736 CPUtime=880.16 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 117933 0 0 0 87374 642 0 0 25 0 5 0 521569525 2872049664 670213 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=2804736 CPUtime=881.89 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 235465 0 0 0 88111 78 0 0 25 0 5 0 521569525 2872049664 670213 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3523.91
Current children cumulated vsize (KiB) 2804736

[startup+942.3 s]
/proc/loadavg: 8.09 8.15 8.04 9/184 13205
/proc/meminfo: memFree=18057920/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=2888244 CPUtime=3763.6 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 692341 0 0 0 374860 1500 0 0 25 0 5 0 521569510 2957561856 692275 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 722061 692275 176 36 0 719966 0
[pid=13131/tid=13133] ppid=13129 vsize=2888244 CPUtime=939.28 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 130939 0 0 0 93249 679 0 0 25 0 5 0 521569525 2957561856 692275 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=2888244 CPUtime=942.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 195332 0 0 0 94146 65 0 0 25 0 5 0 521569525 2957561856 692275 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=2888244 CPUtime=940.15 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 122128 0 0 0 93340 675 0 0 25 0 5 0 521569525 2957561856 692275 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=2888244 CPUtime=941.88 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 240426 0 0 0 94109 79 0 0 25 0 5 0 521569525 2957561856 692275 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3763.6
Current children cumulated vsize (KiB) 2888244

[startup+1002.3 s]
/proc/loadavg: 8.15 8.15 8.04 9/184 13207
/proc/meminfo: memFree=17694340/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=2987400 CPUtime=4003.3 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 717683 0 0 0 398761 1569 0 0 25 0 5 0 521569510 3059097600 717617 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 746850 717617 176 36 0 744755 0
[pid=13131/tid=13133] ppid=13129 vsize=2987400 CPUtime=998.98 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 134539 0 0 0 99188 710 0 0 25 0 5 0 521569525 3059097600 717617 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=2987400 CPUtime=1002.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 203337 0 0 0 100143 68 0 0 25 0 5 0 521569525 3059097600 717617 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=2987400 CPUtime=1000.15 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 125793 0 0 0 99308 707 0 0 25 0 5 0 521569525 3059097600 717617 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=2987400 CPUtime=1001.88 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 250498 0 0 0 100106 82 0 0 25 0 5 0 521569525 3059097600 717617 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4003.3
Current children cumulated vsize (KiB) 2987400

[startup+1062.3 s]
/proc/loadavg: 8.13 8.14 8.04 9/184 13208
/proc/meminfo: memFree=17311412/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=3086288 CPUtime=4243 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 744660 0 0 0 422671 1629 0 0 25 0 5 0 521569510 3160358912 744594 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 771572 744594 176 36 0 769477 0
[pid=13131/tid=13133] ppid=13129 vsize=3086288 CPUtime=1058.67 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 138767 0 0 0 105132 735 0 0 25 0 5 0 521569525 3160358912 744594 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=3086288 CPUtime=1062.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 210503 0 0 0 106140 71 0 0 25 0 5 0 521569525 3160358912 744594 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=3086288 CPUtime=1060.15 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 130037 0 0 0 105280 735 0 0 25 0 5 0 521569525 3160358912 744594 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=3086288 CPUtime=1061.88 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 261837 0 0 0 106102 86 0 0 25 0 5 0 521569525 3160358912 744594 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4243
Current children cumulated vsize (KiB) 3086288

[startup+1122.3 s]
/proc/loadavg: 8.11 8.13 8.04 9/184 13210
/proc/meminfo: memFree=16966912/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=3156392 CPUtime=4482.69 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 762856 0 0 0 446580 1689 0 0 25 0 5 0 521569510 3232145408 762790 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 789098 762790 176 36 0 787003 0
[pid=13131/tid=13133] ppid=13129 vsize=3156392 CPUtime=1118.36 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 142848 0 0 0 111073 763 0 0 25 0 5 0 521569525 3232145408 762790 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=3156392 CPUtime=1122.11 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 218792 0 0 0 112138 73 0 0 25 0 5 0 521569525 3232145408 762790 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=3156392 CPUtime=1120.14 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 133532 0 0 0 111251 763 0 0 25 0 5 0 521569525 3232145408 762790 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=3156392 CPUtime=1121.88 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 264168 0 0 0 112101 87 0 0 25 0 5 0 521569525 3232145408 762790 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4482.69
Current children cumulated vsize (KiB) 3156392

[startup+1182.3 s]
/proc/loadavg: 8.10 8.12 8.04 9/184 13211
/proc/meminfo: memFree=16614476/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=3267408 CPUtime=4722.37 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) S 13129 13131 12928 0 -1 4202496 791169 0 0 0 470476 1761 0 0 25 0 5 0 521569510 3345825792 791103 33554432000 4194304 4338100 140737164093392 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 816852 791103 176 36 0 814757 0
[pid=13131/tid=13133] ppid=13129 vsize=3267408 CPUtime=1178.14 cores=1,3,5,7
/proc/13131/task/13133/stat : 13133 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 147713 0 0 0 117017 797 0 0 25 0 5 0 521569525 3345825792 791103 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13131/tid=13134] ppid=13129 vsize=3267408 CPUtime=1182.12 cores=1,3,5,7
/proc/13131/task/13134/stat : 13134 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 228608 0 0 0 118135 77 0 0 25 0 5 0 521569525 3345825792 791103 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13131/tid=13135] ppid=13129 vsize=3267408 CPUtime=1180.05 cores=1,3,5,7
/proc/13131/task/13135/stat : 13135 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 136872 0 0 0 117209 796 0 0 25 0 5 0 521569525 3345825792 791103 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=3267408 CPUtime=1181.88 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4202560 274460 0 0 0 118098 90 0 0 25 0 5 0 521569525 3345825792 791103 33554432000 4194304 4338100 140737164093392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4722.37
Current children cumulated vsize (KiB) 3267408



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.8 s]
/proc/loadavg: 8.13 8.12 8.05 6/181 13212
/proc/meminfo: memFree=17860440/32951132 swapFree=67111364/67111528
[pid=13131] ppid=13129 vsize=0 CPUtime=4800.08 cores=1,3,5,7
/proc/13131/stat : 13131 (ttsth-5-4) Z 13129 13131 12928 0 -1 4203548 800381 0 0 0 478220 1788 0 0 18 0 2 0 521569510 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13131/statm: 0 0 0 0 0 0 0
[pid=13131/tid=13136] ppid=13129 vsize=0 CPUtime=1201.39 cores=1,3,5,7
/proc/13131/task/13136/stat : 13136 (ttsth-5-4) R 13129 13131 12928 0 -1 4203588 277550 0 0 0 120041 98 0 0 25 0 2 0 521569525 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 1201.87
CPU time (s): 4800.16
CPU user time (s): 4782.2
CPU system time (s): 17.9553
CPU usage (%): 399.389
Max. virtual memory (cumulated for all children) (KiB): 3305200

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4782.2
system time used= 17.9553
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 800381
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3985
involuntary context switches= 31169

runsolver used 2.03069 second user time and 4.29535 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-29 19:55:09
IDJOB=3264378
IDBENCH=84580
IDSOLVER=1654
FILE ID=node128/3264378-1304099709
RUNJOBID= node128-1304097280-12947
PBS_JOBID= 13170611
Free space on /tmp= 66976 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/anton/l4k4/lksat-n900-m6174-k4-l4-s819398222.used-as.sat04-929.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264378-1304099709/watcher-3264378-1304099709 -o /tmp/evaluation-result-3264378-1304099709/solver-3264378-1304099709 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264378-1304099709.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 82170fa4021e7a6c932a31c197f0f64c
RANDOM SEED=1649339970

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      21668164 kB
Buffers:       1282456 kB
Cached:        4391604 kB
SwapCached:        160 kB
Active:        5287336 kB
Inactive:      5400720 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      21668164 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:              28 kB
Writeback:           0 kB
AnonPages:     5013332 kB
Mapped:          15504 kB
Slab:           523752 kB
PageTables:      14456 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:  5485308 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66972 MiB
End job on node128 at 2011-04-29 20:15:12