Trace number 3263938

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.16 1201.78

General information on the benchmark

NameSAT03/handmade/markstrom/UNSATISFIABLE/
mm-2x2-6-6-s.1.shuffled-as.sat03-1499.cnf
MD5SUM5fbbe7100f38217499e8b5ea555a4e4d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark201.305
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses2576
Sum of the clauses size8480
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2400
Number of clauses of size 31536
Number of clauses of size 4128
Number of clauses of size 5512
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263938-1304098540.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 400
0.00/0.00	c Clauses = 2576
0.00/0.00	c Literals = 8480
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 400
0.00/0.00	c Clauses = 2576
0.00/0.00	c Literals = 8480
0.00/0.00	c a2
0.00/0.00	c a1
0.00/0.04	c Independent sub-problems = 1
0.00/0.04	c Sub-problem with 400 variables and 2576 clauses
0.00/0.04	c using permutation method 4
0.00/0.04	c count = 2
0.00/0.04	c Ordering finished 0.030 seconds
0.00/0.04	c mapped 0.040 seconds
0.00/0.04	c permuted 0.040 seconds
0.00/0.04	c about to sort
0.00/0.04	c sorted
0.00/0.04	c clauses now 2576
0.00/0.04	c init finished 0.040 seconds
0.00/0.04	c tree built 0.040 seconds
0.00/0.04	c 
0.00/0.04	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.00/0.04	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.00/0.04	c ................................ | ..........                    |                                  0.1         0.0             
0.00/0.04	c ................................ | ..........                  1 |                                  0.1        10.0             
0.00/0.04	c ................................ | ..........                  2 |                                  0.1        20.0             
0.00/0.04	c ................................ | ..........                  3 |                                  0.1        30.0             
0.00/0.04	c ................................ | ..........                  4 |                                  0.1        40.0             
0.00/0.04	c ................................ | ..........                  5 |                                  0.1        50.0             
0.00/0.05	c ................................ | ..........               1000 | 59           165                 0.1     10000.0          150
96.11/24.12	c ................................ | ..........             785000 | 90390        765464             96.2      8160.9      6904183
191.96/48.17	c ................................ | ..........            1085000 | 120265       1067372           192.2      5643.7     10261335
288.23/72.25	c ................................ | ..........            1305000 | 144381       1282825           288.5      4524.0     11615601
384.10/96.27	c ................................ | ..........            1485000 | 166646       1462147           384.4      3863.1     13707407
479.96/120.29	c ................................ | *.........            1637000 | 181043       1615005           480.3      3408.0     14910964
575.83/144.26	c ................................ | *.........            1758000 | 194701       1742613           576.1      3051.6     16410414
673.30/168.60	c ................................ | *.........            1868000 | 205991       1871000           673.3      2774.4     17895152
768.80/192.56	c ................................ | *.........            1974000 | 214034       1988558           769.0      2566.9     19083207
863.84/216.35	c ................................ | *.........            2090000 | 225390       2115790           864.1      2418.8     20820394
959.71/240.37	c ................................ | **........            2180000 | 234040       2215747           960.0      2270.8     21994911
1057.17/264.73	c ................................ | **........            2279000 | 243124       2322185          1057.3      2155.4     23005004
1152.64/288.69	c ................................ | **........            2360000 | 250150       2428069          1153.0      2046.8     23957906
1248.51/312.60	c ................................ | **........            2441000 | 257336       2522862          1248.5      1955.1     24725946
1344.39/336.62	c ................................ | **........            2550000 | 269947       2627698          1344.4      1896.7     25749090
1440.24/360.60	c ................................ | ***.......            2635000 | 279747       2711459          1440.3      1829.5     26472550
1536.11/384.60	c ................................ | ***.......            2716000 | 286664       2793818          1536.1      1768.1     27152825
1631.98/408.65	c ................................ | ***.......            2783000 | 293616       2882473          1632.2      1705.1     27852791
1728.65/432.82	c ................................ | ***.......            2863000 | 300565       2971038          1728.7      1656.1     28850106
1824.92/456.98	c ................................ | ***.......            2937000 | 308339       3049721          1825.2      1609.1     29564530
1920.79/480.93	c ................................ | ****......            2994000 | 312748       3119502          1920.9      1558.6     30270165
2016.27/504.84	c ................................ | ****......            3067000 | 319590       3186281          2016.4      1521.0     30959229
2113.72/529.27	c ................................ | ****......            3122000 | 324966       3260568          2114.0      1476.8     31825269
2208.80/553.02	c ................................ | ****......            3173000 | 329620       3326866          2208.9      1436.5     32625401
2303.86/576.84	c ................................ | ****......            3223000 | 334202       3396059          2304.0      1398.9     33432402
2401.73/601.32	c ................................ | *****.....            3282000 | 340392       3466997          2401.8      1366.5     34253382
2496.80/625.19	c ................................ | *****.....            3340000 | 345544       3534443          2497.2      1337.5     35041861
2593.47/649.33	c ................................ | *****.....            3385000 | 349252       3594681          2593.6      1305.1     35335987
2688.95/673.27	c ................................ | *****.....            3433000 | 353706       3652557          2689.2      1276.6     35776427
2785.62/697.48	c ................................ | *****.....            3480000 | 358042       3709122          2785.9      1249.1     36325375
2880.67/721.29	c ................................ | ******....            3521000 | 361599       3762436          2881.1      1222.1     36877888
2976.54/745.22	c ................................ | ******....            3568000 | 365689       3820603          2976.7      1198.7     37622439
3072.01/769.17	c ................................ | ******....            3612000 | 369756       3879215          3072.3      1175.7     38283786
3169.48/793.54	c ................................ | ******....            3650000 | 373322       3929982          3169.6      1151.6     38701204
3264.57/817.31	c ................................ | ******....            3699000 | 377698       3980417          3264.6      1133.1     39060011
3360.84/841.40	c ................................ | *******...            3750000 | 383169       4039684          3360.8      1115.8     39346789
3460.28/866.31	c ................................ | *******...            3809000 | 389870       4101150          3460.4      1100.8     39704254
3552.96/889.51	c ................................ | *******...            3853000 | 394098       4151911          3553.0      1084.4     40066278
3648.03/913.38	c ................................ | *******...            3900000 | 398964       4204324          3648.4      1069.0     40400667
3747.51/938.23	c ................................ | *******...            3945000 | 403264       4256539          3747.6      1052.7     40690497
3841.37/961.79	c ................................ | ********..            3987000 | 406896       4310582          3841.7      1037.8     40976963
3936.85/985.68	c ................................ | ********..            4032000 | 411202       4368647          3937.2      1024.1     41279725
4032.30/1009.59	c ................................ | ********..            4070000 | 414832       4418485          4032.7      1009.3     41604616
4128.97/1033.77	c ................................ | ********..            4121000 | 418911       4475617          4129.2       998.0     42001263
4224.04/1057.57	c ................................ | ********..            4168000 | 423095       4526336          4224.3       986.7     42456749
4319.91/1081.59	c ................................ | *********.            4225000 | 427964       4582253          4320.3       977.9     43065179
4417.36/1105.92	c ................................ | *********.            4279000 | 432225       4639383          4417.5       968.7     43535271
4512.84/1129.82	c ................................ | *********.            4342000 | 438356       4702349          4512.9       962.1     43918924
4610.70/1154.33	c ................................ | *********.            4386000 | 443006       4746877          4610.9       951.2     44276169
4703.78/1177.67	c ................................ | *********.            4420000 | 446280       4785145          4704.1       939.6     44627110

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263938-1304098540/watcher-3263938-1304098540 -o /tmp/evaluation-result-3263938-1304098540/solver-3263938-1304098540 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263938-1304098540.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.16 8.09 7.92 6/181 21767
/proc/meminfo: memFree=25630496/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=9508 CPUtime=0 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) R 21765 21767 21560 0 -1 4202496 465 0 0 0 0 0 0 0 25 0 1 0 600084605 9736192 407 33554432000 4194304 4338100 140734370679984 18446744073709551615 260019791569 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21767/statm: 2377 407 152 36 0 282 0

[startup+0.086308 s]
/proc/loadavg: 8.16 8.09 7.92 6/181 21767
/proc/meminfo: memFree=25630496/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=199244 CPUtime=0.18 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 1920 0 0 0 18 0 0 0 25 0 5 0 600084605 204025856 1862 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 49811 1862 175 36 0 47716 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 199244

[startup+0.166271 s]
/proc/loadavg: 8.16 8.09 7.92 6/181 21767
/proc/meminfo: memFree=25630496/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=199504 CPUtime=0.5 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 2433 0 0 0 50 0 0 0 25 0 5 0 600084605 204292096 2375 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 49876 2376 175 36 0 47781 0
Current children cumulated CPU time (s) 0.5
Current children cumulated vsize (KiB) 199504

[startup+0.300248 s]
/proc/loadavg: 8.16 8.09 7.92 6/181 21767
/proc/meminfo: memFree=25630496/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=199896 CPUtime=1.03 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 3477 0 0 0 103 0 0 0 25 0 5 0 600084605 204693504 3419 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 49974 3419 175 36 0 47879 0
Current children cumulated CPU time (s) 1.03
Current children cumulated vsize (KiB) 199896

[startup+0.700178 s]
/proc/loadavg: 8.16 8.09 7.92 6/181 21767
/proc/meminfo: memFree=25630496/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=216060 CPUtime=2.63 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 6872 0 0 0 262 1 0 0 25 0 5 0 600084605 221245440 6814 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 54015 6814 175 36 0 51920 0
Current children cumulated CPU time (s) 2.63
Current children cumulated vsize (KiB) 216060

[startup+1.50103 s]
/proc/loadavg: 8.16 8.09 7.92 9/186 21772
/proc/meminfo: memFree=25596544/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=229620 CPUtime=5.83 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 11685 0 0 0 581 2 0 0 25 0 5 0 600084605 235130880 11627 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 57405 11627 175 36 0 55310 0
[pid=21767/tid=21769] ppid=21765 vsize=229620 CPUtime=1.44 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 2558 0 0 0 144 0 0 0 25 0 5 0 600084610 235130880 11627 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=229620 CPUtime=1.43 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 2143 0 0 0 143 0 0 0 25 0 5 0 600084610 235130880 11627 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=229620 CPUtime=1.44 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 1976 0 0 0 144 0 0 0 25 0 5 0 600084610 235130880 11627 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=229620 CPUtime=1.44 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 3291 0 0 0 144 0 0 0 20 0 5 0 600084610 235130880 11627 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5.83
Current children cumulated vsize (KiB) 229620

[startup+3.10074 s]
/proc/loadavg: 8.16 8.09 7.92 9/186 21772
/proc/meminfo: memFree=25573120/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=255684 CPUtime=12.22 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 19148 0 0 0 1218 4 0 0 25 0 5 0 600084605 261820416 19090 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 63921 19090 175 36 0 61826 0
[pid=21767/tid=21769] ppid=21765 vsize=255684 CPUtime=3.04 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 4852 0 0 0 303 1 0 0 25 0 5 0 600084610 261820416 19090 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=255684 CPUtime=3.02 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 3553 0 0 0 302 0 0 0 25 0 5 0 600084610 261820416 19090 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=255684 CPUtime=3.04 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 3279 0 0 0 304 0 0 0 25 0 5 0 600084610 261820416 19090 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=255684 CPUtime=3.04 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 5747 0 0 0 303 1 0 0 25 0 5 0 600084610 261820416 19090 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.22
Current children cumulated vsize (KiB) 255684

[startup+6.30115 s]
/proc/loadavg: 8.15 8.09 7.92 9/186 21772
/proc/meminfo: memFree=25508108/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=303868 CPUtime=25.01 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 30737 0 0 0 2494 7 0 0 25 0 5 0 600084605 311160832 30679 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 75967 30679 176 36 0 73872 0
[pid=21767/tid=21769] ppid=21765 vsize=303868 CPUtime=6.24 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 8704 0 0 0 622 2 0 0 25 0 5 0 600084610 311160832 30679 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=303868 CPUtime=6.21 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 5694 0 0 0 620 1 0 0 25 0 5 0 600084610 311160832 30679 33554432000 4194304 4338100 140734370679984 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=303868 CPUtime=6.24 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 5341 0 0 0 623 1 0 0 25 0 5 0 600084610 311160832 30679 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=303868 CPUtime=6.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 9281 0 0 0 623 1 0 0 25 0 5 0 600084610 311160832 30679 33554432000 4194304 4338100 140734370679984 18446744073709551615 4240519 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.01
Current children cumulated vsize (KiB) 303868

[startup+12.701 s]
/proc/loadavg: 8.13 8.09 7.92 9/186 21772
/proc/meminfo: memFree=25429248/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=355328 CPUtime=50.56 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 43971 0 0 0 5045 11 0 0 25 0 5 0 600084605 363855872 43913 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 88832 43913 176 36 0 86737 0
[pid=21767/tid=21769] ppid=21765 vsize=355328 CPUtime=12.64 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 11415 0 0 0 1261 3 0 0 25 0 5 0 600084610 363855872 43913 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=355328 CPUtime=12.57 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 10240 0 0 0 1255 2 0 0 25 0 5 0 600084610 363855872 43913 33554432000 4194304 4338100 140734370679984 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=355328 CPUtime=12.64 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 8527 0 0 0 1261 3 0 0 25 0 5 0 600084610 363855872 43913 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032208590 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=355328 CPUtime=12.64 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 12072 0 0 0 1261 3 0 0 25 0 5 0 600084610 363855872 43913 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.56
Current children cumulated vsize (KiB) 355328

[startup+25.5006 s]
/proc/loadavg: 8.25 8.12 7.93 9/186 21773
/proc/meminfo: memFree=25308708/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=423220 CPUtime=101.68 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 62478 0 0 0 10149 19 0 0 25 0 5 0 600084605 433377280 62420 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 105805 62420 176 36 0 103710 0
[pid=21767/tid=21769] ppid=21765 vsize=423220 CPUtime=25.44 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 16065 0 0 0 2539 5 0 0 25 0 5 0 600084610 433377280 62420 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=423220 CPUtime=25.3 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 15836 0 0 0 2526 4 0 0 25 0 5 0 600084610 433377280 62420 33554432000 4194304 4338100 140734370679984 18446744073709551615 260020092528 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=423220 CPUtime=25.43 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 12757 0 0 0 2538 5 0 0 25 0 5 0 600084610 433377280 62421 33554432000 4194304 4338100 140734370679984 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=423220 CPUtime=25.43 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 16104 0 0 0 2539 4 0 0 25 0 5 0 600084610 433377280 62421 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.68
Current children cumulated vsize (KiB) 423220

[startup+51.1009 s]
/proc/loadavg: 8.16 8.11 7.93 9/186 21773
/proc/meminfo: memFree=25162840/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=536236 CPUtime=203.94 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 89799 0 0 0 20365 29 0 0 25 0 5 0 600084605 549105664 89741 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 134059 89741 176 36 0 131964 0
[pid=21767/tid=21769] ppid=21765 vsize=536236 CPUtime=51.04 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 22856 0 0 0 5096 8 0 0 25 0 5 0 600084610 549105664 89741 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=536236 CPUtime=50.76 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 20436 0 0 0 5071 5 0 0 25 0 5 0 600084610 549105664 89741 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=536236 CPUtime=51.03 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 22393 0 0 0 5095 8 0 0 25 0 5 0 600084610 549105664 89741 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=536236 CPUtime=51.04 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 22397 0 0 0 5097 7 0 0 25 0 5 0 600084610 549105664 89741 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 203.94
Current children cumulated vsize (KiB) 536236

[startup+102.306 s]
/proc/loadavg: 8.13 8.10 7.94 9/186 21775
/proc/meminfo: memFree=24999724/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=655052 CPUtime=408.48 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 120672 0 0 0 40803 45 0 0 25 0 5 0 600084605 670773248 120614 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 163763 120614 176 36 0 161668 0
[pid=21767/tid=21769] ppid=21765 vsize=655052 CPUtime=102.25 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 31235 0 0 0 10212 13 0 0 25 0 5 0 600084610 670773248 120614 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=655052 CPUtime=101.69 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 29596 0 0 0 10160 9 0 0 25 0 5 0 600084610 670773248 120614 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=655052 CPUtime=102.23 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 26395 0 0 0 10213 10 0 0 25 0 5 0 600084610 670773248 120614 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=655052 CPUtime=102.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 31729 0 0 0 10212 12 0 0 25 0 5 0 600084610 670773248 120614 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 408.48
Current children cumulated vsize (KiB) 655052

[startup+162.3 s]
/proc/loadavg: 7.99 8.07 7.93 9/186 21870
/proc/meminfo: memFree=29704312/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=769840 CPUtime=648.14 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 150533 0 0 0 64753 61 0 0 25 0 5 0 600084605 788316160 150475 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 192460 150475 176 36 0 190365 0
[pid=21767/tid=21769] ppid=21765 vsize=769840 CPUtime=162.24 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 39059 0 0 0 16206 18 0 0 25 0 5 0 600084610 788316160 150475 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=769840 CPUtime=161.36 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 37854 0 0 0 16123 13 0 0 25 0 5 0 600084610 788316160 150475 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=769840 CPUtime=162.22 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 33168 0 0 0 16209 13 0 0 25 0 5 0 600084610 788316160 150475 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=769840 CPUtime=162.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 38735 0 0 0 16208 16 0 0 25 0 5 0 600084610 788316160 150475 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 648.14
Current children cumulated vsize (KiB) 769840


################
# More data... #
################

[pid=21767/tid=21770] ppid=21765 vsize=1425032 CPUtime=758.11 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 87442 0 0 0 75778 33 0 0 25 0 5 0 600084610 1459232768 321155 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1425032 CPUtime=762.17 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 77021 0 0 0 76184 33 0 0 25 0 5 0 600084610 1459232768 321155 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1425032 CPUtime=762.25 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 76973 0 0 0 76186 39 0 0 25 0 5 0 600084610 1459232768 321155 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3044.87
Current children cumulated vsize (KiB) 1425032

[startup+822.301 s]
/proc/loadavg: 8.16 8.12 8.01 9/186 21891
/proc/meminfo: memFree=25223380/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1459904 CPUtime=3284.53 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 330670 0 0 0 328300 153 0 0 25 0 5 0 600084605 1494941696 330612 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 364976 330612 176 36 0 362881 0
[pid=21767/tid=21769] ppid=21765 vsize=1459904 CPUtime=822.26 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 81253 0 0 0 82183 43 0 0 25 0 5 0 600084610 1494941696 330612 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1459904 CPUtime=817.77 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 88957 0 0 0 81743 34 0 0 25 0 5 0 600084610 1494941696 330612 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1459904 CPUtime=822.17 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 80212 0 0 0 82182 35 0 0 25 0 5 0 600084610 1494941696 330612 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1459904 CPUtime=822.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 78531 0 0 0 82185 39 0 0 25 0 5 0 600084610 1494941696 330612 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3284.53
Current children cumulated vsize (KiB) 1459904

[startup+882.301 s]
/proc/loadavg: 8.09 8.11 8.01 9/186 21892
/proc/meminfo: memFree=24885548/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1488824 CPUtime=3524.19 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 337817 0 0 0 352263 156 0 0 25 0 5 0 600084605 1524555776 337759 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 372206 337759 176 36 0 370111 0
[pid=21767/tid=21769] ppid=21765 vsize=1488824 CPUtime=882.26 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 81429 0 0 0 88183 43 0 0 25 0 5 0 600084610 1524555776 337759 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1488824 CPUtime=877.45 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 90546 0 0 0 87710 35 0 0 25 0 5 0 600084610 1524555776 337759 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1488824 CPUtime=882.16 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 82771 0 0 0 88180 36 0 0 25 0 5 0 600084610 1524555776 337759 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1488824 CPUtime=882.25 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 81354 0 0 0 88184 41 0 0 25 0 5 0 600084610 1524555776 337759 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3524.19
Current children cumulated vsize (KiB) 1488824

[startup+942.301 s]
/proc/loadavg: 7.99 8.07 8.00 9/186 21894
/proc/meminfo: memFree=24531736/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1511744 CPUtime=3763.88 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 344510 0 0 0 376227 161 0 0 25 0 5 0 600084605 1548025856 344452 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 377936 344452 176 36 0 375841 0
[pid=21767/tid=21769] ppid=21765 vsize=1511744 CPUtime=942.27 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 81603 0 0 0 94183 44 0 0 25 0 5 0 600084610 1548025856 344452 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1511744 CPUtime=937.12 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 92130 0 0 0 93676 36 0 0 25 0 5 0 600084610 1548025856 344452 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1511744 CPUtime=942.16 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 85779 0 0 0 94179 37 0 0 25 0 5 0 600084610 1548025856 344452 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1511744 CPUtime=942.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 83281 0 0 0 94183 41 0 0 25 0 5 0 600084610 1548025856 344452 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3763.88
Current children cumulated vsize (KiB) 1511744

[startup+1002.3 s]
/proc/loadavg: 7.99 8.06 8.00 9/186 21896
/proc/meminfo: memFree=24185360/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1543608 CPUtime=4003.54 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 351421 0 0 0 400190 164 0 0 25 0 5 0 600084605 1580654592 351363 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 385902 351363 176 36 0 383807 0
[pid=21767/tid=21769] ppid=21765 vsize=1543608 CPUtime=1002.26 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 81777 0 0 0 100182 44 0 0 25 0 5 0 600084610 1580654592 351363 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1543608 CPUtime=996.8 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 92387 0 0 0 99643 37 0 0 25 0 5 0 600084610 1580654592 351363 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1543608 CPUtime=1002.16 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 89273 0 0 0 100177 39 0 0 25 0 5 0 600084610 1580654592 351363 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1543608 CPUtime=1002.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 86267 0 0 0 100181 43 0 0 25 0 5 0 600084610 1580654592 351363 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4003.54
Current children cumulated vsize (KiB) 1543608

[startup+1062.3 s]
/proc/loadavg: 7.93 8.03 7.99 9/186 21897
/proc/meminfo: memFree=23998800/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1575784 CPUtime=4243.21 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 360311 0 0 0 424150 171 0 0 25 0 5 0 600084605 1613602816 360253 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 393946 360253 176 36 0 391851 0
[pid=21767/tid=21769] ppid=21765 vsize=1575784 CPUtime=1062.27 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 83943 0 0 0 106181 46 0 0 25 0 5 0 600084610 1613602816 360253 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1575784 CPUtime=1056.47 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 92548 0 0 0 105610 37 0 0 25 0 5 0 600084610 1613602816 360253 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1575784 CPUtime=1062.15 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 92897 0 0 0 106174 41 0 0 25 0 5 0 600084610 1613602816 360253 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1575784 CPUtime=1062.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 89206 0 0 0 106179 45 0 0 25 0 5 0 600084610 1613602816 360253 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4243.21
Current children cumulated vsize (KiB) 1575784

[startup+1122.3 s]
/proc/loadavg: 7.97 8.02 7.99 9/186 21899
/proc/meminfo: memFree=23876484/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1614224 CPUtime=4482.88 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 370547 0 0 0 448111 177 0 0 25 0 5 0 600084605 1652965376 370489 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 403556 370489 176 36 0 401461 0
[pid=21767/tid=21769] ppid=21765 vsize=1614224 CPUtime=1122.27 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 86349 0 0 0 112180 47 0 0 25 0 5 0 600084610 1652965376 370489 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1614224 CPUtime=1116.14 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 93113 0 0 0 111577 37 0 0 25 0 5 0 600084610 1652965376 370489 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1614224 CPUtime=1122.14 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 96819 0 0 0 112171 43 0 0 25 0 5 0 600084610 1652965376 370489 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1614224 CPUtime=1122.24 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 92549 0 0 0 112177 47 0 0 25 0 5 0 600084610 1652965376 370489 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4482.88
Current children cumulated vsize (KiB) 1614224

[startup+1182.3 s]
/proc/loadavg: 8.10 8.05 8.00 9/186 21900
/proc/meminfo: memFree=23703192/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1643412 CPUtime=4722.56 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 378340 0 0 0 472074 182 0 0 25 0 5 0 600084605 1682853888 378282 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 410853 378282 176 36 0 408758 0
[pid=21767/tid=21769] ppid=21765 vsize=1643412 CPUtime=1182.27 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 87860 0 0 0 118179 48 0 0 25 0 5 0 600084610 1682853888 378282 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1643412 CPUtime=1175.83 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 94483 0 0 0 117544 39 0 0 25 0 5 0 600084610 1682853888 378282 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1643412 CPUtime=1182.15 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 99139 0 0 0 118170 45 0 0 25 0 5 0 600084610 1682853888 378282 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1643412 CPUtime=1182.23 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 95141 0 0 0 118175 48 0 0 25 0 5 0 600084610 1682853888 378282 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4722.56
Current children cumulated vsize (KiB) 1643412



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 8.15 8.06 8.01 9/186 21901
/proc/meminfo: memFree=23635972/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1653056 CPUtime=4800.06 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 381052 0 0 0 479823 183 0 0 25 0 5 0 600084605 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 413264 380994 176 36 0 411169 0
[pid=21767/tid=21769] ppid=21765 vsize=1653056 CPUtime=1201.68 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 88693 0 0 0 120119 49 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1653056 CPUtime=1195.12 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 95005 0 0 0 119473 39 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1653056 CPUtime=1201.55 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 99775 0 0 0 120110 45 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1653056 CPUtime=1201.65 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 95862 0 0 0 120116 49 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.06
Current children cumulated vsize (KiB) 1653056

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 8.15 8.06 8.01 9/186 21901
/proc/meminfo: memFree=23635972/32951124 swapFree=67111464/67111528
[pid=21767] ppid=21765 vsize=1653056 CPUtime=4800.06 cores=1,3,5,7
/proc/21767/stat : 21767 (ttsth-5-4) S 21765 21767 21560 0 -1 4202496 381052 0 0 0 479823 183 0 0 25 0 5 0 600084605 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21767/statm: 413264 380994 176 36 0 411169 0
[pid=21767/tid=21769] ppid=21765 vsize=1653056 CPUtime=1201.68 cores=1,3,5,7
/proc/21767/task/21769/stat : 21769 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 88693 0 0 0 120119 49 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21767/tid=21770] ppid=21765 vsize=1653056 CPUtime=1195.12 cores=1,3,5,7
/proc/21767/task/21770/stat : 21770 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 95005 0 0 0 119473 39 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21767/tid=21771] ppid=21765 vsize=1653056 CPUtime=1201.55 cores=1,3,5,7
/proc/21767/task/21771/stat : 21771 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 99775 0 0 0 120110 45 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21767/tid=21772] ppid=21765 vsize=1653056 CPUtime=1201.65 cores=1,3,5,7
/proc/21767/task/21772/stat : 21772 (ttsth-5-4) R 21765 21767 21560 0 -1 4202560 95862 0 0 0 120116 49 0 0 25 0 5 0 600084610 1692729344 380994 33554432000 4194304 4338100 140734370679984 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.06
Current children cumulated vsize (KiB) 1653056

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.78
CPU time (s): 4800.16
CPU user time (s): 4798.26
CPU system time (s): 1.90271
CPU usage (%): 399.422
Max. virtual memory (cumulated for all children) (KiB): 1653056

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4798.26
system time used= 1.90271
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 381052
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 389
involuntary context switches= 33977

runsolver used 1.93471 second user time and 4.6303 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 19:35:40
IDJOB=3263938
IDBENCH=84062
IDSOLVER=1654
FILE ID=node110/3263938-1304098540
RUNJOBID= node110-1304096259-21579
PBS_JOBID= 13170606
Free space on /tmp= 72840 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/markstrom/UNSATISFIABLE/mm-2x2-6-6-s.1.shuffled-as.sat03-1499.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263938-1304098540/watcher-3263938-1304098540 -o /tmp/evaluation-result-3263938-1304098540/solver-3263938-1304098540 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263938-1304098540.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 5fbbe7100f38217499e8b5ea555a4e4d
RANDOM SEED=1636968239

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25630636 kB
Buffers:        905644 kB
Cached:         695580 kB
SwapCached:         64 kB
Active:        5359316 kB
Inactive:      1545940 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25630636 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:              48 kB
Writeback:           0 kB
AnonPages:     5303140 kB
Mapped:          15324 kB
Slab:           341708 kB
PageTables:      15008 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  5585388 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72840 MiB
End job on node110 at 2011-04-29 19:55:44