Trace number 3260154

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.34 1205.35

General information on the benchmark

Namecrafted/mosoi/
289/289-sat-9x13.cnf
MD5SUMd1a00ea2cfc43a784948a7238a6910b3
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.011997
Satisfiable
(Un)Satisfiability was proved
Number of variables468
Number of clauses12051
Sum of the clauses size46800
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2702
Number of clauses of size 30
Number of clauses of size 411349
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260154-1304063077.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 468
0.00/0.01	c Clauses = 12051
0.00/0.01	c Literals = 46800
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 468
0.00/0.01	c Clauses = 12051
0.00/0.01	c Literals = 46800
0.00/0.01	c a2
0.00/0.03	c a1
0.59/0.66	c Independent sub-problems = 1
0.59/0.66	c Sub-problem with 468 variables and 12051 clauses
0.59/0.66	c using permutation method 4
0.59/0.66	c count = 2
0.59/0.66	c Ordering finished 0.660 seconds
0.59/0.66	c mapped 0.660 seconds
0.59/0.66	c permuted 0.660 seconds
0.59/0.66	c about to sort
0.59/0.66	c sorted
0.59/0.68	c clauses now 12051
0.59/0.68	c init finished 0.670 seconds
0.59/0.68	c tree built 0.680 seconds
0.59/0.68	c 
0.59/0.68	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.59/0.68	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.59/0.68	c ................................ | ..........                    |                                  0.7         0.0             
0.59/0.68	c ................................ | ..........                  1 |                                  0.7         1.5             
0.59/0.68	c ................................ | ..........                  2 |                                  0.7         2.9             
0.59/0.68	c ................................ | ..........                  3 |                                  0.7         4.4             
0.59/0.68	c ................................ | ..........                  4 |                                  0.7         5.9             
0.59/0.68	c ................................ | ..........                  5 |                                  0.7         7.4             
0.59/0.69	c ................................ | ..........               1000 | 31           3291                0.7      1408.5         5378
96.05/24.63	c ................................ | ..........             697000 | 156135       1380001            96.2      7247.6     13518759
192.07/48.72	c ................................ | ..........            1147000 | 259161       1964381           192.2      5968.7     18225612
288.03/72.81	c ................................ | ..........            1502000 | 350349       2365002           288.1      5213.8     22030854
384.03/96.93	c ................................ | ..........            1786000 | 438661       2681155           384.1      4649.2     24385390
480.05/121.06	c ................................ | *.........            2035000 | 511456       2937141           480.3      4237.2     26674153
576.10/145.16	c ................................ | *.........            2282000 | 588026       3163238           576.3      3959.7     28110329
671.74/169.18	c ................................ | *.........            2518000 | 654898       3418359           672.1      3746.7     29708359
767.90/193.29	c ................................ | *.........            2725000 | 723590       3600279           768.3      3546.9     31843713
863.86/217.38	c ................................ | *.........            2921000 | 788920       3759340           864.2      3380.0     33136037
959.89/241.47	c ................................ | **........            3130000 | 856846       3947429           960.2      3259.7     34491399
1055.98/265.51	c ................................ | **........            3341000 | 928259       4105443          1056.0      3163.7     36255219
1151.91/289.67	c ................................ | **........            3644000 | 1038214      4252582          1152.2      3162.6     38869047
1248.25/313.82	c ................................ | **........            3933000 | 1139229      4402085          1248.3      3150.6     40554999
1344.14/337.93	c ................................ | **........            4290000 | 1263894      4528366          1344.3      3191.3     44407420
1440.19/362.03	c ................................ | ***.......            4565000 | 1351908      4720963          1440.3      3169.4     48239585
1536.02/386.08	c ................................ | ***.......            4766000 | 1413405      4898570          1536.3      3102.2     49904788
1632.06/410.11	c ................................ | ***.......            4945000 | 1472976      5063198          1632.1      3029.8     52215101
1728.41/434.33	c ................................ | ***.......            5113000 | 1525546      5217197          1728.5      2958.0     53841130
1824.50/458.40	c ................................ | ***.......            5282000 | 1578923      5405821          1824.5      2895.0     54940961
1920.30/482.41	c ................................ | ****......            5437000 | 1630626      5549888          1920.3      2831.3     55293647
2016.03/506.47	c ................................ | ****......            5593000 | 1683847      5680393          2016.3      2773.9     56836569
2111.75/530.46	c ................................ | ****......            5765000 | 1738004      5826086          2112.0      2729.6     57752663
2207.92/554.55	c ................................ | ****......            5952000 | 1799933      5975108          2208.1      2695.5     57807788
2304.01/578.66	c ................................ | ****......            6111000 | 1854192      6104764          2304.2      2652.1     58902891
2399.71/602.68	c ................................ | *****.....            6281000 | 1912781      6242265          2400.1      2617.0     59523731
2496.18/626.80	c ................................ | *****.....            6484000 | 1982182      6387748          2496.2      2597.6     61266026
2591.81/650.86	c ................................ | *****.....            6698000 | 2054681      6543797          2592.1      2584.1     64555699
2687.86/674.96	c ................................ | *****.....            6929000 | 2134497      6689890          2688.1      2577.6     67077790
2783.69/699.08	c ................................ | *****.....            7158000 | 2212207      6837910          2784.0      2571.1     68578499
2880.04/723.24	c ................................ | ******....            7394000 | 2291383      7013174          2880.2      2567.2     69541758
2975.97/747.34	c ................................ | ******....            7706000 | 2397945      7191892          2976.2      2589.3     71376755
3072.23/771.57	c ................................ | ******....            7945000 | 2473542      7367415          3072.5      2585.8     72818379
3167.66/795.49	c ................................ | ******....            8087000 | 2517424      7527049          3168.0      2552.7     75354702
3264.13/819.68	c ................................ | ******....            8221000 | 2559579      7703284          3264.5      2518.3     76875166
3360.47/843.80	c ................................ | *******...            8387000 | 2607754      7888252          3360.5      2495.8     77140577
3455.78/867.80	c ................................ | *******...            8564000 | 2660215      8071312          3456.2      2477.9     77140577
3552.34/891.95	c ................................ | *******...            8710000 | 2706051      8221134          3552.5      2451.8     77167926
3648.34/916.05	c ................................ | *******...            8847000 | 2747809      8389405          3648.6      2424.8     78911054
3743.97/940.04	c ................................ | *******...            9000000 | 2792257      8568796          3744.1      2403.8     79770580
3840.51/964.20	c ................................ | ********..            9166000 | 2843600      8715385          3840.5      2386.6     81672658
3935.78/988.19	c ................................ | ********..            9315000 | 2888892      8861067          3936.1      2366.5     82418625
4032.02/1012.26	c ................................ | ********..            9484000 | 2942467      9024143          4032.3      2352.0     84260119
4128.05/1036.34	c ................................ | ********..            9681000 | 3005228      9201409          4128.2      2345.1     84800977
4224.06/1060.46	c ................................ | ********..            9877000 | 3069500      9360944          4224.3      2338.1     85423378
4319.76/1084.49	c ................................ | *********.           10056000 | 3128216      9523221          4320.1      2327.7     85888875
4415.95/1108.67	c ................................ | *********.           10264000 | 3199102      9674202          4416.3      2324.1     86929353
4511.86/1132.76	c ................................ | *********.           10504000 | 3284108      9774604          4512.1      2327.9     88664186
4607.98/1156.87	c ................................ | *********.           10630000 | 3327442      9866810          4608.2      2306.7     89435969
4704.50/1181.28	c ................................ | *********.           10778000 | 3376173      9991001          4704.8      2290.8     89813301

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3260154-1304063077/watcher-3260154-1304063077 -o /tmp/evaluation-result-3260154-1304063077/solver-3260154-1304063077 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260154-1304063077.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.18 7.81 6.87 2/174 30485
/proc/meminfo: memFree=28725300/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) R 30483 30485 27559 0 -1 4202496 402 0 0 0 0 0 0 0 22 0 1 0 517908424 9633792 344 33554432000 4194304 4338100 140736666599600 18446744073709551615 222921474159 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/30485/statm: 2352 345 147 36 0 257 0

[startup+0.0934279 s]
/proc/loadavg: 7.18 7.81 6.87 2/174 30485
/proc/meminfo: memFree=28725300/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=12328 CPUtime=0.09 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) R 30483 30485 27559 0 -1 4202496 1147 0 0 0 9 0 0 0 22 0 1 0 517908424 12623872 1088 33554432000 4194304 4338100 140736666599600 18446744073709551615 4295932 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/30485/statm: 3082 1088 155 36 0 987 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12328

[startup+0.100426 s]
/proc/loadavg: 7.18 7.81 6.87 2/174 30485
/proc/meminfo: memFree=28725300/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=12328 CPUtime=0.09 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) R 30483 30485 27559 0 -1 4202496 1147 0 0 0 9 0 0 0 22 0 1 0 517908424 12623872 1088 33554432000 4194304 4338100 140736666599600 18446744073709551615 4295900 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/30485/statm: 3082 1088 155 36 0 987 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12328

[startup+0.300401 s]
/proc/loadavg: 7.18 7.81 6.87 2/174 30485
/proc/meminfo: memFree=28725300/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=12328 CPUtime=0.29 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) R 30483 30485 27559 0 -1 4202496 1147 0 0 0 29 0 0 0 23 0 1 0 517908424 12623872 1088 33554432000 4194304 4338100 140736666599600 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/30485/statm: 3082 1088 155 36 0 987 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 12328

[startup+0.700223 s]
/proc/loadavg: 7.18 7.81 6.87 2/174 30485
/proc/meminfo: memFree=28725300/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=136164 CPUtime=0.72 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 2630 0 0 0 72 0 0 0 25 0 5 0 517908424 139431936 2460 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 34041 2460 175 36 0 31946 0
Current children cumulated CPU time (s) 0.72
Current children cumulated vsize (KiB) 136164

[startup+1.50108 s]
/proc/loadavg: 7.18 7.81 6.87 9/186 30500
/proc/meminfo: memFree=28653912/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=231040 CPUtime=3.91 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 12177 0 0 0 390 1 0 0 25 0 5 0 517908424 236584960 12007 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 57760 12007 175 36 0 55665 0
[pid=30485/tid=30497] ppid=30483 vsize=231040 CPUtime=0.8 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 2363 0 0 0 80 0 0 0 25 0 5 0 517908493 236584960 12007 33554432000 4194304 4338100 140736666599600 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=231040 CPUtime=0.8 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 2363 0 0 0 80 0 0 0 25 0 5 0 517908493 236584960 12007 33554432000 4194304 4338100 140736666599600 18446744073709551615 4256104 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=231040 CPUtime=0.8 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 2397 0 0 0 80 0 0 0 25 0 5 0 517908493 236584960 12008 33554432000 4194304 4338100 140736666599600 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=231040 CPUtime=0.8 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 2589 0 0 0 80 0 0 0 25 0 5 0 517908493 236584960 12008 33554432000 4194304 4338100 140736666599600 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3.91
Current children cumulated vsize (KiB) 231040

[startup+3.10079 s]
/proc/loadavg: 7.25 7.82 6.88 9/186 30500
/proc/meminfo: memFree=28586004/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=275888 CPUtime=10.31 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 23408 0 0 0 1027 4 0 0 25 0 5 0 517908424 282509312 23238 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 68972 23238 175 36 0 66877 0
[pid=30485/tid=30497] ppid=30483 vsize=275888 CPUtime=2.4 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) S 30483 30485 27559 0 -1 4202560 4816 0 0 0 239 1 0 0 25 0 5 0 517908493 282509312 23238 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933602676 0 0 4096 0 18446604453846571136 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=275888 CPUtime=2.39 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 5099 0 0 0 239 0 0 0 25 0 5 0 517908493 282509312 23238 33554432000 4194304 4338100 140736666599600 18446744073709551615 222921172374 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=275888 CPUtime=2.39 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 5198 0 0 0 239 0 0 0 25 0 5 0 517908493 282509312 23238 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933589710 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=275888 CPUtime=2.4 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 5829 0 0 0 239 1 0 0 25 0 5 0 517908493 282509312 23238 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 10.31
Current children cumulated vsize (KiB) 275888

[startup+6.3012 s]
/proc/loadavg: 7.25 7.82 6.88 9/186 30501
/proc/meminfo: memFree=28416472/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=343564 CPUtime=23.08 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 40728 0 0 0 2299 9 0 0 25 0 5 0 517908424 351809536 40510 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 85891 40510 175 36 0 83796 0
[pid=30485/tid=30497] ppid=30483 vsize=343564 CPUtime=5.59 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 8712 0 0 0 557 2 0 0 25 0 5 0 517908493 351809536 40510 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=343564 CPUtime=5.6 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 9397 0 0 0 558 2 0 0 25 0 5 0 517908493 351809536 40510 33554432000 4194304 4338100 140736666599600 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=343564 CPUtime=5.57 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 9484 0 0 0 557 0 0 0 25 0 5 0 517908493 351809536 40510 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=343564 CPUtime=5.6 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 10669 0 0 0 557 3 0 0 25 0 5 0 517908493 351809536 40510 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.08
Current children cumulated vsize (KiB) 343564

[startup+12.701 s]
/proc/loadavg: 7.43 7.84 6.89 9/186 30501
/proc/meminfo: memFree=28149660/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=455808 CPUtime=48.6 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 70298 0 0 0 4844 16 0 0 25 0 5 0 517908424 466747392 70080 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 113952 70080 175 36 0 111857 0
[pid=30485/tid=30497] ppid=30483 vsize=455808 CPUtime=11.98 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 14493 0 0 0 1193 5 0 0 25 0 5 0 517908493 466747392 70080 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=455808 CPUtime=11.98 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 15602 0 0 0 1195 3 0 0 25 0 5 0 517908493 466747392 70080 33554432000 4194304 4338100 140736666599600 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=455808 CPUtime=11.94 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 16605 0 0 0 1192 2 0 0 25 0 5 0 517908493 466747392 70080 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=455808 CPUtime=11.99 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 21132 0 0 0 1194 5 0 0 25 0 5 0 517908493 466747392 70080 33554432000 4194304 4338100 140736666599600 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 48.6
Current children cumulated vsize (KiB) 455808

[startup+25.5007 s]
/proc/loadavg: 7.52 7.84 6.91 9/186 30501
/proc/meminfo: memFree=27795908/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=640776 CPUtime=99.64 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 116903 0 0 0 9936 28 0 0 25 0 5 0 517908424 656154624 116685 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 160194 116685 176 36 0 158099 0
[pid=30485/tid=30497] ppid=30483 vsize=640776 CPUtime=24.75 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 24199 0 0 0 2468 7 0 0 25 0 5 0 517908493 656154624 116685 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=640776 CPUtime=24.76 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 27034 0 0 0 2470 6 0 0 25 0 5 0 517908493 656154624 116685 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=640776 CPUtime=24.66 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 28629 0 0 0 2461 5 0 0 25 0 5 0 517908493 656154624 116685 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=640776 CPUtime=24.75 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 34575 0 0 0 2467 8 0 0 25 0 5 0 517908493 656154624 116685 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 99.64
Current children cumulated vsize (KiB) 640776

[startup+51.101 s]
/proc/loadavg: 7.68 7.86 6.93 9/186 30502
/proc/meminfo: memFree=27416768/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=808536 CPUtime=201.65 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 160716 0 0 0 20123 42 0 0 25 0 5 0 517908424 827940864 160498 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 202134 160498 176 36 0 200039 0
[pid=30485/tid=30497] ppid=30483 vsize=808536 CPUtime=50.28 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 37239 0 0 0 5016 12 0 0 25 0 5 0 517908493 827940864 160498 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=808536 CPUtime=50.3 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 34596 0 0 0 5022 8 0 0 25 0 5 0 517908493 827940864 160498 33554432000 4194304 4338100 140736666599600 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=808536 CPUtime=50.06 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 38519 0 0 0 4999 7 0 0 25 0 5 0 517908493 827940864 160498 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=808536 CPUtime=50.29 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 47896 0 0 0 5016 13 0 0 25 0 5 0 517908493 827940864 160498 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 201.65
Current children cumulated vsize (KiB) 808536

[startup+102.307 s]
/proc/loadavg: 7.91 7.89 7.00 7/186 30503
/proc/meminfo: memFree=26835308/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=1008860 CPUtime=405.54 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 211175 0 0 0 40494 60 0 0 25 0 5 0 517908424 1033072640 210957 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 252215 210957 176 36 0 250120 0
[pid=30485/tid=30497] ppid=30483 vsize=1008860 CPUtime=101.28 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 51200 0 0 0 10112 16 0 0 25 0 5 0 517908493 1033072640 210957 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=1008860 CPUtime=101.33 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 46742 0 0 0 10120 13 0 0 25 0 5 0 517908493 1033072640 210957 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=1008860 CPUtime=100.84 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 51261 0 0 0 10072 12 0 0 25 0 5 0 517908493 1033072640 210957 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=1008860 CPUtime=101.37 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 59506 0 0 0 10119 18 0 0 25 0 5 0 517908493 1033072640 210957 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 405.54
Current children cumulated vsize (KiB) 1008860

[startup+162.301 s]
/proc/loadavg: 7.99 7.92 7.06 9/186 30505
/proc/meminfo: memFree=26349684/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=1133320 CPUtime=644.63 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 245343 0 0 0 64387 76 0 0 25 0 5 0 517908424 1160519680 245125 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 283330 245125 176 36 0 281235 0
[pid=30485/tid=30497] ppid=30483 vsize=1133320 CPUtime=161.1 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 61442 0 0 0 16090 20 0 0 25 0 5 0 517908493 1160519680 245125 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=1133320 CPUtime=161.08 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 55610 0 0 0 16091 17 0 0 25 0 5 0 517908493 1160519680 245125 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=1133320 CPUtime=160.5 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 57133 0 0 0 16035 15 0 0 25 0 5 0 517908493 1160519680 245125 33554432000 4194304 4338100 140736666599600 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=1133320 CPUtime=161.22 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 68692 0 0 0 16100 22 0 0 25 0 5 0 517908493 1160519680 245125 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 644.63
Current children cumulated vsize (KiB) 1133320


################
# More data... #
################

/proc/meminfo: memFree=23425400/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=2609200 CPUtime=3035.52 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 605732 0 0 0 303317 235 0 0 25 0 5 0 517908424 2671820800 605514 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 652300 605514 176 36 0 650205 0
[pid=30485/tid=30497] ppid=30483 vsize=2609200 CPUtime=758.08 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 172473 0 0 0 75738 70 0 0 25 0 5 0 517908493 2671820800 605514 33554432000 4194304 4338100 140736666599600 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=2609200 CPUtime=758.82 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 132672 0 0 0 75829 53 0 0 25 0 5 0 517908493 2671820800 605514 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=2609200 CPUtime=758.71 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 145279 0 0 0 75819 52 0 0 25 0 5 0 517908493 2671820800 605514 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=2609200 CPUtime=759.19 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 152842 0 0 0 75861 58 0 0 25 0 5 0 517908493 2671820800 605514 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3035.52
Current children cumulated vsize (KiB) 2609200

[startup+822.301 s]
/proc/loadavg: 8.09 8.07 7.56 9/186 30522
/proc/meminfo: memFree=23123288/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=2741764 CPUtime=3274.92 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 640733 0 0 0 327244 248 0 0 25 0 5 0 517908424 2807566336 640515 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 685441 640515 176 36 0 683346 0
[pid=30485/tid=30497] ppid=30483 vsize=2741764 CPUtime=817.98 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 180835 0 0 0 81725 73 0 0 25 0 5 0 517908493 2807566336 640515 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=2741764 CPUtime=818.58 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 140358 0 0 0 81802 56 0 0 25 0 5 0 517908493 2807566336 640515 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=2741764 CPUtime=818.65 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 153978 0 0 0 81810 55 0 0 25 0 5 0 517908493 2807566336 640515 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=2741764 CPUtime=818.99 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 163096 0 0 0 81837 62 0 0 25 0 5 0 517908493 2807566336 640515 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3274.92
Current children cumulated vsize (KiB) 2741764

[startup+882.301 s]
/proc/loadavg: 8.07 8.06 7.59 9/186 30524
/proc/meminfo: memFree=22938848/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=2754196 CPUtime=3514.03 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 644685 0 0 0 351150 253 0 0 25 0 5 0 517908424 2820296704 644467 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 688549 644467 176 36 0 686454 0
[pid=30485/tid=30497] ppid=30483 vsize=2754196 CPUtime=877.8 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 181772 0 0 0 87705 75 0 0 25 0 5 0 517908493 2820296704 644467 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=2754196 CPUtime=878.45 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 141276 0 0 0 87788 57 0 0 25 0 5 0 517908493 2820296704 644467 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=2754196 CPUtime=878.49 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 155225 0 0 0 87792 57 0 0 25 0 5 0 517908493 2820296704 644467 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=2754196 CPUtime=878.57 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 163946 0 0 0 87794 63 0 0 25 0 5 0 517908493 2820296704 644467 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3514.03
Current children cumulated vsize (KiB) 2754196

[startup+942.301 s]
/proc/loadavg: 8.06 8.06 7.62 9/186 30525
/proc/meminfo: memFree=22656456/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=2843496 CPUtime=3753.16 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 667753 0 0 0 375049 267 0 0 25 0 5 0 517908424 2911739904 667535 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 710874 667535 176 36 0 708779 0
[pid=30485/tid=30497] ppid=30483 vsize=2843496 CPUtime=937.52 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 188078 0 0 0 93674 78 0 0 25 0 5 0 517908493 2911739904 667535 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=2843496 CPUtime=938.23 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 148114 0 0 0 93763 60 0 0 25 0 5 0 517908493 2911739904 667535 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=2843496 CPUtime=938.42 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 159133 0 0 0 93782 60 0 0 25 0 5 0 517908493 2911739904 667535 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=2843496 CPUtime=938.27 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 169962 0 0 0 93760 67 0 0 25 0 5 0 517908493 2911739904 667535 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3753.16
Current children cumulated vsize (KiB) 2843496

[startup+1002.3 s]
/proc/loadavg: 8.09 8.08 7.65 9/186 30536
/proc/meminfo: memFree=22428264/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=2963812 CPUtime=3992.51 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 699380 0 0 0 398966 285 0 0 25 0 5 0 517908424 3034943488 699162 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 740953 699162 176 36 0 738858 0
[pid=30485/tid=30497] ppid=30483 vsize=2963812 CPUtime=997.15 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 195503 0 0 0 99632 83 0 0 25 0 5 0 517908493 3034943488 699162 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=2963812 CPUtime=998.18 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 155849 0 0 0 99754 64 0 0 25 0 5 0 517908493 3034943488 699162 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=2963812 CPUtime=998.28 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 167249 0 0 0 99763 65 0 0 25 0 5 0 517908493 3034943488 699162 33554432000 4194304 4338100 140736666599600 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=2963812 CPUtime=998.17 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 178313 0 0 0 99747 70 0 0 25 0 5 0 517908493 3034943488 699162 33554432000 4194304 4338100 140736666599600 18446744073709551615 222921172718 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3992.51
Current children cumulated vsize (KiB) 2963812

[startup+1062.3 s]
/proc/loadavg: 8.07 8.08 7.67 9/186 30538
/proc/meminfo: memFree=22312556/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=3021568 CPUtime=4231.63 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 715099 0 0 0 422866 297 0 0 25 0 5 0 517908424 3094085632 714881 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 755392 714881 176 36 0 753297 0
[pid=30485/tid=30497] ppid=30483 vsize=3021568 CPUtime=1056.71 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 201375 0 0 0 105584 87 0 0 25 0 5 0 517908493 3094085632 714881 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=3021568 CPUtime=1058.08 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 159001 0 0 0 105741 67 0 0 25 0 5 0 517908493 3094085632 714881 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=3021568 CPUtime=1058.1 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 170356 0 0 0 105743 67 0 0 25 0 5 0 517908493 3094085632 714881 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=3021568 CPUtime=1058.01 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 181901 0 0 0 105728 73 0 0 25 0 5 0 517908493 3094085632 714881 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4231.63
Current children cumulated vsize (KiB) 3021568

[startup+1122.3 s]
/proc/loadavg: 8.12 8.09 7.70 9/186 30539
/proc/meminfo: memFree=22148960/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=3107344 CPUtime=4470.57 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 738361 0 0 0 446747 310 0 0 25 0 5 0 517908424 3181920256 738143 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 776836 738143 176 36 0 774741 0
[pid=30485/tid=30497] ppid=30483 vsize=3107344 CPUtime=1116.29 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 205291 0 0 0 111540 89 0 0 25 0 5 0 517908493 3181920256 738143 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=3107344 CPUtime=1117.91 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 161638 0 0 0 111723 68 0 0 25 0 5 0 517908493 3181920256 738143 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=3107344 CPUtime=1117.98 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 179958 0 0 0 111725 73 0 0 25 0 5 0 517908493 3181920256 738143 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=3107344 CPUtime=1117.66 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 189008 0 0 0 111689 77 0 0 25 0 5 0 517908493 3181920256 738143 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4470.57
Current children cumulated vsize (KiB) 3107344

[startup+1182.3 s]
/proc/loadavg: 8.25 8.12 7.73 9/186 30544
/proc/meminfo: memFree=22000956/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=3158184 CPUtime=4708.91 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 750937 0 0 0 470572 319 0 0 25 0 5 0 517908424 3233980416 750719 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 789546 750719 176 36 0 787451 0
[pid=30485/tid=30497] ppid=30483 vsize=3158184 CPUtime=1175.87 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 206342 0 0 0 117497 90 0 0 25 0 5 0 517908493 3233980416 750719 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=3158184 CPUtime=1177.17 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 163639 0 0 0 117647 70 0 0 25 0 5 0 517908493 3233980416 750719 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=3158184 CPUtime=1177.84 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 185137 0 0 0 117708 76 0 0 25 0 5 0 517908493 3233980416 750719 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=3158184 CPUtime=1177.31 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 193353 0 0 0 117651 80 0 0 25 0 5 0 517908493 3233980416 750719 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4708.91
Current children cumulated vsize (KiB) 3158184



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.2 s]
/proc/loadavg: 8.09 8.09 7.73 5/177 30563
/proc/meminfo: memFree=25710292/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=3167964 CPUtime=4800.17 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) S 30483 30485 27559 0 -1 4202496 753061 0 0 0 479695 322 0 0 25 0 5 0 517908424 3243995136 752843 33554432000 4194304 4338100 140736666599600 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 791991 752843 176 36 0 789896 0
[pid=30485/tid=30497] ppid=30483 vsize=3167964 CPUtime=1198.71 cores=0,2,4,6
/proc/30485/task/30497/stat : 30497 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 206392 0 0 0 119780 91 0 0 25 0 5 0 517908493 3243995136 752843 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=3167964 CPUtime=1199.92 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 163743 0 0 0 119921 71 0 0 25 0 5 0 517908493 3243995136 752843 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30485/tid=30499] ppid=30483 vsize=3167964 CPUtime=1200.67 cores=0,2,4,6
/proc/30485/task/30499/stat : 30499 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 186932 0 0 0 119990 77 0 0 25 0 5 0 517908493 3243995136 752843 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30485/tid=30500] ppid=30483 vsize=3167964 CPUtime=1200.15 cores=0,2,4,6
/proc/30485/task/30500/stat : 30500 (ttsth-5-4) R 30483 30485 27559 0 -1 4202560 193528 0 0 0 119934 81 0 0 25 0 5 0 517908493 3243995136 752843 33554432000 4194304 4338100 140736666599600 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 3167964

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.31 s]
/proc/loadavg: 8.09 8.09 7.73 5/177 30563
/proc/meminfo: memFree=25710292/32951124 swapFree=67111356/67111528
[pid=30485] ppid=30483 vsize=0 CPUtime=4800.29 cores=0,2,4,6
/proc/30485/stat : 30485 (ttsth-5-4) Z 30483 30485 27559 0 -1 4203548 753061 0 0 0 479697 332 0 0 18 0 2 0 517908424 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/30485/statm: 0 0 0 0 0 0 0
[pid=30485/tid=30498] ppid=30483 vsize=0 CPUtime=1200.03 cores=0,2,4,6
/proc/30485/task/30498/stat : 30498 (ttsth-5-4) R 30483 30485 27559 0 -1 4203588 163743 0 0 0 119922 81 0 0 25 0 2 0 517908493 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1205.35
CPU time (s): 4800.34
CPU user time (s): 4796.98
CPU system time (s): 3.36449
CPU usage (%): 398.254
Max. virtual memory (cumulated for all children) (KiB): 3167964

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.98
system time used= 3.36449
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 753061
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9323
involuntary context switches= 23356

runsolver used 1.78973 second user time and 4.11237 second system time

The end

Launcher Data 

Begin job on node109 at 2011-04-29 09:44:37
IDJOB=3260154
IDBENCH=83171
IDSOLVER=1654
FILE ID=node109/3260154-1304063077
RUNJOBID= node109-1304063077-30471
PBS_JOBID= 13170356
Free space on /tmp= 72384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-9x13.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3260154-1304063077/watcher-3260154-1304063077 -o /tmp/evaluation-result-3260154-1304063077/solver-3260154-1304063077 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260154-1304063077.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d1a00ea2cfc43a784948a7238a6910b3
RANDOM SEED=1880984569

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28725828 kB
Buffers:        885608 kB
Cached:        2846480 kB
SwapCached:        168 kB
Active:         820568 kB
Inactive:      2979428 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28725828 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:             504 kB
Writeback:           0 kB
AnonPages:       67684 kB
Mapped:          14496 kB
Slab:           360900 kB
PageTables:       4728 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   193216 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72384 MiB
End job on node109 at 2011-04-29 10:04:44