Trace number 3259802

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.46 1208.83

General information on the benchmark

NameCRAFTED/parity-games/
instance_n9_i9_pp.cnf
MD5SUM3e04f934e4f74a8e89587f6c1327f77e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark269.553
Satisfiable
(Un)Satisfiability was proved
Number of variables33867
Number of clauses454832
Sum of the clauses size1512036
Maximum clause length72
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 270326
Number of clauses of size 3228663
Number of clauses of size 4122769
Number of clauses of size 525920
Number of clauses of size over 57154

Solver Data 

0.49/0.54	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259802-1304056481.cnf 
0.49/0.54	c Using SATTIMEOUT of 4800
0.49/0.54	c Using SATRAM of 15500
0.49/0.54	c Size of subset node = 32
0.49/0.54	c Variables = 33867
0.49/0.54	c Clauses = 454832
0.49/0.54	c Literals = 1512036
0.49/0.54	c Reduced size:
0.49/0.54	c Variables = 33867
0.49/0.54	c Clauses = 454832
0.49/0.54	c Literals = 1512036
0.49/0.54	c a2
1.29/1.34	c a1
6.78/6.85	c Independent sub-problems = 1
6.78/6.85	c Sub-problem with 33867 variables and 454832 clauses
6.78/6.85	c using permutation method 4
6.78/6.85	c count = 1
6.78/6.85	c Ordering finished 6.740 seconds
6.78/6.85	c mapped 6.840 seconds
6.78/6.85	c permuted 6.840 seconds
6.78/6.85	c about to sort
6.89/6.91	c sorted
8.39/8.46	c clauses now 454832
8.39/8.46	c init finished 7.260 seconds
8.39/8.46	c tree built 8.450 seconds
8.39/8.46	c 
8.39/8.46	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
8.39/8.46	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
8.39/8.46	c ................................ | ..........                    |                                  8.4         0.0             
8.39/8.48	c ................................ | ..........                  1 |                                  8.5         0.1             
8.39/8.48	c ................................ | ..........                  2 |                                  8.5         0.2             
8.39/8.48	c ................................ | ..........                  3 |                                  8.5         0.4             
8.39/8.48	c ................................ | ..........                  4 |                                  8.5         0.5             
8.39/8.48	c ................................ | ..........                  5 |                                  8.5         0.6             
8.39/8.50	c ................................ | ..........               1000 | 2            183                 8.6       117.0           50
95.98/30.52	c ................................ | ..........             750000 | 50162        400068             96.1      7806.0      6925822
191.79/54.56	c ................................ | ..........            1150000 | 86170        607635            192.1      5988.0     10428198
288.01/78.64	c ................................ | ..........            1469000 | 110322       762407            288.2      5097.5     12924988
383.85/102.68	c ................................ | ..........            1754000 | 133460       918503            384.2      4565.4     15251731
479.69/126.68	c ................................ | *.........            2013000 | 156929       1044651           480.0      4193.5     17571446
575.95/150.75	c ................................ | *.........            2192000 | 174917       1134792           576.2      3804.5     18963016
672.20/174.86	c ................................ | *.........            2394000 | 194408       1245277           672.4      3560.3     20291067
768.03/198.87	c ................................ | *.........            2576000 | 209348       1325469           768.3      3352.8     21463867
863.86/222.84	c ................................ | *.........            2771000 | 225971       1408889           864.0      3207.0     22261594
960.09/246.98	c ................................ | **........            2967000 | 243687       1492729           960.5      3089.2     23025836
1056.34/271.00	c ................................ | **........            3131000 | 257822       1575293          1056.4      2964.0     24655377
1152.17/295.05	c ................................ | **........            3298000 | 271745       1646839          1152.4      2861.8     25802333
1248.41/319.17	c ................................ | **........            3456000 | 287941       1723489          1248.7      2767.7     27392964
1344.67/343.21	c ................................ | **........            3606000 | 302559       1800500          1344.7      2681.7     29030753
1440.50/367.22	c ................................ | ***.......            3759000 | 315755       1864556          1440.6      2609.3     30367903
1536.26/391.26	c ................................ | ***.......            3937000 | 330490       1937005          1536.5      2562.3     31256003
1632.50/415.33	c ................................ | ***.......            4107000 | 345733       2007864          1632.6      2515.6     32276160
1727.93/439.23	c ................................ | ***.......            4281000 | 360435       2077638          1728.1      2477.3     33213267
1824.57/463.40	c ................................ | ***.......            4446000 | 373704       2140709          1824.6      2436.7     34258690
1920.03/487.36	c ................................ | ****......            4591000 | 385038       2196746          1920.3      2390.8     34795468
2015.86/511.40	c ................................ | ****......            4740000 | 396910       2250504          2016.2      2350.9     35317156
2111.68/535.38	c ................................ | ****......            4903000 | 410173       2319933          2112.0      2321.5     36175235
2207.93/559.49	c ................................ | ****......            5048000 | 421785       2388066          2208.3      2285.9     36508329
2304.17/583.54	c ................................ | ****......            5193000 | 433144       2453420          2304.3      2253.6     36989991
2400.01/607.53	c ................................ | *****.....            5337000 | 444181       2525005          2400.2      2223.6     37403730
2496.25/631.63	c ................................ | *****.....            5492000 | 456400       2606606          2496.4      2200.0     37776189
2592.11/655.60	c ................................ | *****.....            5636000 | 467589       2668840          2592.1      2174.3     38186992
2688.35/679.74	c ................................ | *****.....            5840000 | 484120       2781193          2688.5      2172.2     39441964
2783.95/703.77	c ................................ | *****.....            6009000 | 498513       2877904          2784.3      2158.2     40331220
2880.19/727.90	c ................................ | ******....            6177000 | 514493       2993674          2880.2      2144.6     42144174
2976.03/751.93	c ................................ | ******....            6337000 | 528039       3070402          2976.2      2129.2     43286568
3072.27/776.07	c ................................ | ******....            6522000 | 543375       3162134          3072.6      2122.7     44488775
3168.10/800.01	c ................................ | ******....            6692000 | 557390       3242532          3168.2      2112.3     45211899
3264.34/824.13	c ................................ | ******....            6852000 | 571574       3316532          3264.5      2098.9     46246713
3360.61/848.21	c ................................ | *******...            7006000 | 583392       3380759          3360.7      2084.7     46701771
3456.05/872.15	c ................................ | *******...            7217000 | 601624       3496181          3456.3      2088.1     48140181
3551.88/896.18	c ................................ | *******...            7412000 | 617024       3601416          3552.2      2086.6     49780941
3648.14/920.27	c ................................ | *******...            7581000 | 631310       3695790          3648.4      2077.9     51391140
3744.38/944.31	c ................................ | *******...            7736000 | 644613       3774045          3744.4      2066.0     52495565
3840.63/968.43	c ................................ | ********..            7907000 | 658962       3874722          3840.8      2058.7     53373911
3936.08/992.30	c ................................ | ********..            8063000 | 671789       3955259          3936.1      2048.5     54294508
4032.35/1016.45	c ................................ | ********..            8197000 | 683493       4027770          4032.6      2032.7     55322970
4128.17/1040.40	c ................................ | ********..            8308000 | 694012       4092689          4128.2      2012.5     56073791
4224.02/1064.42	c ................................ | ********..            8439000 | 705377       4162038          4224.1      1997.8     56725226
4320.28/1088.59	c ................................ | *********.            8565000 | 716006       4228125          4320.6      1982.3     57071391
4416.52/1112.61	c ................................ | *********.            8672000 | 726043       4286191          4416.6      1963.5     57588290
4512.36/1136.68	c ................................ | *********.            8806000 | 736715       4348441          4512.7      1951.4     58109657
4607.83/1160.59	c ................................ | *********.            8936000 | 749044       4416576          4608.2      1939.2     58871926
4704.08/1184.64	c ................................ | *********.            9051000 | 758963       4472285          4704.2      1924.0     59368937
4800.32/1208.70	c CPU limit (4800) reached
4800.32/1208.70	c Time = 4800.32 seconds
4800.32/1208.70	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259802-1304056481/watcher-3259802-1304056481 -o /tmp/evaluation-result-3259802-1304056481/solver-3259802-1304056481 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259802-1304056481.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.67 8.02 7.91 3/175 21589
/proc/meminfo: memFree=25055484/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=9244 CPUtime=0 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 369 0 0 0 0 0 0 0 25 0 1 0 517247731 9465856 311 33554432000 4194304 4338100 140734048668112 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 2311 312 147 36 0 216 0

[startup+0.0654069 s]
/proc/loadavg: 7.67 8.02 7.91 3/175 21589
/proc/meminfo: memFree=25055484/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=15388 CPUtime=0.05 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 1924 0 0 0 5 0 0 0 25 0 1 0 517247731 15757312 1866 33554432000 4194304 4338100 140734048668112 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 3847 1867 148 36 0 1752 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 15388

[startup+0.100399 s]
/proc/loadavg: 7.67 8.02 7.91 3/175 21589
/proc/meminfo: memFree=25055484/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=19060 CPUtime=0.09 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 2815 0 0 0 9 0 0 0 25 0 1 0 517247731 19517440 2757 33554432000 4194304 4338100 140734048668112 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 4765 2758 148 36 0 2670 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19060

[startup+0.300372 s]
/proc/loadavg: 7.67 8.02 7.91 3/175 21589
/proc/meminfo: memFree=25055484/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=41888 CPUtime=0.28 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 8493 0 0 0 27 1 0 0 25 0 1 0 517247731 42893312 8435 33554432000 4194304 4338100 140734048668112 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 10472 8437 148 36 0 8377 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 41888

[startup+0.700285 s]
/proc/loadavg: 7.67 8.02 7.91 3/175 21589
/proc/meminfo: memFree=25055484/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=147788 CPUtime=0.69 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 33440 0 0 0 65 4 0 0 25 0 1 0 517247731 151334912 32907 33554432000 4194304 4338100 140734048668112 18446744073709551615 274664834154 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 36947 32912 260 36 0 34852 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 147788

[startup+1.50115 s]
/proc/loadavg: 7.67 8.02 7.91 3/176 21590
/proc/meminfo: memFree=24920572/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=151548 CPUtime=1.49 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 36583 0 0 0 144 5 0 0 25 0 1 0 517247731 155185152 33815 33554432000 4194304 4338100 140734048668112 18446744073709551615 4245168 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 37887 33815 227 36 0 35792 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 151548

[startup+3.10079 s]
/proc/loadavg: 7.67 8.02 7.91 6/180 21594
/proc/meminfo: memFree=24626980/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=151548 CPUtime=3.09 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 37371 0 0 0 304 5 0 0 25 0 1 0 517247731 155185152 34550 33554432000 4194304 4338100 140734048668112 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 37887 34550 175 36 0 35792 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 151548

[startup+6.30025 s]
/proc/loadavg: 7.62 8.00 7.91 7/180 21594
/proc/meminfo: memFree=24533496/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=151548 CPUtime=6.29 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) R 21587 21589 18837 0 -1 4202496 37392 0 0 0 623 6 0 0 25 0 1 0 517247731 155185152 34571 33554432000 4194304 4338100 140734048668112 18446744073709551615 4297815 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/21589/statm: 37887 34571 175 36 0 35792 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 151548

[startup+12.7011 s]
/proc/loadavg: 7.65 8.00 7.91 9/184 21598
/proc/meminfo: memFree=23827520/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=893884 CPUtime=25.2 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 178736 0 0 0 2475 45 0 0 25 0 5 0 517247731 915337216 174119 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 223471 174120 178 36 0 221376 0
[pid=21589/tid=21595] ppid=21587 vsize=893884 CPUtime=4.16 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 5512 0 0 0 408 8 0 0 18 0 5 0 517248578 915337216 174120 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=893884 CPUtime=4.18 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 9608 0 0 0 417 1 0 0 25 0 5 0 517248578 915337216 174120 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=893884 CPUtime=4.15 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 4673 0 0 0 408 7 0 0 18 0 5 0 517248578 915337216 174120 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=893884 CPUtime=4.22 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 5415 0 0 0 416 6 0 0 18 0 5 0 517248578 915337216 174120 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 25.2
Current children cumulated vsize (KiB) 893884

[startup+25.5009 s]
/proc/loadavg: 7.73 8.00 7.91 9/184 21599
/proc/meminfo: memFree=23480480/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=930084 CPUtime=76.11 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 205748 0 0 0 7496 115 0 0 25 0 5 0 517247731 952406016 201131 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 232521 201131 178 36 0 230426 0
[pid=21589/tid=21595] ppid=21587 vsize=930084 CPUtime=16.86 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 11664 0 0 0 1654 32 0 0 18 0 5 0 517248578 952406016 201131 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=930084 CPUtime=16.9 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 17580 0 0 0 1687 3 0 0 25 0 5 0 517248578 952406016 201131 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=930084 CPUtime=16.84 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 10873 0 0 0 1652 32 0 0 18 0 5 0 517248578 952406016 201131 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=930084 CPUtime=17.01 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 12102 0 0 0 1677 24 0 0 23 0 5 0 517248578 952406016 201131 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 76.11
Current children cumulated vsize (KiB) 930084

[startup+51.1005 s]
/proc/loadavg: 7.82 8.00 7.91 9/184 21600
/proc/meminfo: memFree=23058176/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=1085608 CPUtime=178.22 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 236187 0 0 0 17565 257 0 0 25 0 5 0 517247731 1111662592 231570 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 271402 231570 178 36 0 269307 0
[pid=21589/tid=21595] ppid=21587 vsize=1085608 CPUtime=42.31 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 20218 0 0 0 4151 80 0 0 25 0 5 0 517248578 1111662592 231570 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=1085608 CPUtime=42.47 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 23171 0 0 0 4242 5 0 0 25 0 5 0 517248578 1111662592 231570 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=1085608 CPUtime=42.36 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 18837 0 0 0 4157 79 0 0 25 0 5 0 517248578 1111662592 231570 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=1085608 CPUtime=42.58 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 20432 0 0 0 4188 70 0 0 25 0 5 0 517248578 1111662592 231570 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 178.22
Current children cumulated vsize (KiB) 1085608

[startup+102.307 s]
/proc/loadavg: 7.96 8.01 7.91 9/184 21601
/proc/meminfo: memFree=22278456/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=1282880 CPUtime=382.69 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 282388 0 0 0 37775 494 0 0 25 0 5 0 517247731 1313669120 277771 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 320720 277771 178 36 0 318625 0
[pid=21589/tid=21595] ppid=21587 vsize=1282880 CPUtime=93.24 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 32143 0 0 0 9166 158 0 0 25 0 5 0 517248578 1313669120 277771 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=1282880 CPUtime=93.68 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 34486 0 0 0 9360 8 0 0 25 0 5 0 517248578 1313669120 277771 33554432000 4194304 4338100 140734048668112 18446744073709551615 274664204014 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=1282880 CPUtime=93.53 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 30325 0 0 0 9194 159 0 0 25 0 5 0 517248578 1313669120 277771 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=1282880 CPUtime=93.75 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 31905 0 0 0 9229 146 0 0 25 0 5 0 517248578 1313669120 277771 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 382.69
Current children cumulated vsize (KiB) 1282880

[startup+162.3 s]
/proc/loadavg: 7.98 8.00 7.91 9/184 21602
/proc/meminfo: memFree=21549328/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=1397792 CPUtime=622.26 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 318424 0 0 0 61513 713 0 0 25 0 5 0 517247731 1431339008 313807 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 349448 313807 178 36 0 347353 0
[pid=21589/tid=21595] ppid=21587 vsize=1397792 CPUtime=152.91 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 40715 0 0 0 15061 230 0 0 25 0 5 0 517248578 1431339008 313807 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=1397792 CPUtime=153.67 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 45255 0 0 0 15356 11 0 0 25 0 5 0 517248578 1431339008 313807 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=1397792 CPUtime=153.47 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 38729 0 0 0 15119 228 0 0 25 0 5 0 517248578 1431339008 313807 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=1397792 CPUtime=153.7 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 40196 0 0 0 15150 220 0 0 25 0 5 0 517248578 1431339008 313807 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 622.26
Current children cumulated vsize (KiB) 1397792

[startup+222.301 s]
/proc/loadavg: 8.07 8.02 7.92 9/184 21604
/proc/meminfo: memFree=21011896/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=1503368 CPUtime=861.86 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 341027 0 0 0 85281 905 0 0 25 0 5 0 517247731 1539448832 336410 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 375842 336410 178 36 0 373747 0
[pid=21589/tid=21595] ppid=21587 vsize=1503368 CPUtime=212.59 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 44676 0 0 0 20975 284 0 0 25 0 5 0 517248578 1539448832 336410 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=1503368 CPUtime=213.67 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 54896 0 0 0 21353 14 0 0 25 0 5 0 517248578 1539448832 336410 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=1503368 CPUtime=213.44 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 43304 0 0 0 21048 296 0 0 25 0 5 0 517248578 1539448832 336410 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=1503368 CPUtime=213.67 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 44622 0 0 0 21079 288 0 0 25 0 5 0 517248578 1539448832 336410 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676633972 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 861.86
Current children cumulated vsize (KiB) 1503368

[startup+282.3 s]
/proc/loadavg: 8.08 8.03 7.93 9/184 21606
/proc/meminfo: memFree=20538324/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=1630504 CPUtime=1101.46 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 366810 0 0 0 109046 1100 0 0 25 0 5 0 517247731 1669636096 362193 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 407626 362193 178 36 0 405531 0
[pid=21589/tid=21595] ppid=21587 vsize=1630504 CPUtime=272.27 cores=0,2,4,6

################
# More data... #
################

/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 62905 0 0 0 68574 755 0 0 25 0 5 0 517248578 2217361408 489572 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2778.36
Current children cumulated vsize (KiB) 2165392

[startup+762.307 s]
/proc/loadavg: 8.30 8.20 8.03 9/184 21630
/proc/meminfo: memFree=17719096/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2246792 CPUtime=3017.58 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 522264 0 0 0 299273 2485 0 0 25 0 5 0 517247731 2300715008 517647 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 561698 517648 178 36 0 559603 0
[pid=21589/tid=21595] ppid=21587 vsize=2246792 CPUtime=749.7 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 63348 0 0 0 74214 756 0 0 25 0 5 0 517248578 2300715008 517648 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2246792 CPUtime=753.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 175347 0 0 0 75321 48 0 0 25 0 5 0 517248578 2300715008 517648 33554432000 4194304 4338100 140734048668112 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2246792 CPUtime=752.65 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 64887 0 0 0 74436 829 0 0 25 0 5 0 517248578 2300715008 517649 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2246792 CPUtime=753.05 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 65154 0 0 0 74476 829 0 0 25 0 5 0 517248578 2300715008 517649 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3017.58
Current children cumulated vsize (KiB) 2246792

[startup+822.301 s]
/proc/loadavg: 8.23 8.19 8.04 9/184 21632
/proc/meminfo: memFree=17362208/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2331864 CPUtime=3257.16 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 543455 0 0 0 323011 2705 0 0 25 0 5 0 517247731 2387828736 538838 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 582966 538838 178 36 0 580871 0
[pid=21589/tid=21595] ppid=21587 vsize=2331864 CPUtime=809.64 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 67623 0 0 0 80133 831 0 0 25 0 5 0 517248578 2387828736 538838 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2331864 CPUtime=813.68 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 185116 0 0 0 81317 51 0 0 25 0 5 0 517248578 2387828736 538838 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2331864 CPUtime=812.32 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 68203 0 0 0 80335 897 0 0 25 0 5 0 517248578 2387828736 538838 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2331864 CPUtime=813.01 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 68984 0 0 0 80399 902 0 0 25 0 5 0 517248578 2387828736 538838 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3257.16
Current children cumulated vsize (KiB) 2331864

[startup+882.301 s]
/proc/loadavg: 8.15 8.17 8.04 9/184 21634
/proc/meminfo: memFree=16981256/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2434240 CPUtime=3496.78 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 566450 0 0 0 346762 2916 0 0 25 0 5 0 517247731 2492661760 561833 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 608560 561833 178 36 0 606465 0
[pid=21589/tid=21595] ppid=21587 vsize=2434240 CPUtime=869.56 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 73611 0 0 0 86050 906 0 0 25 0 5 0 517248578 2492661760 561833 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2434240 CPUtime=873.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 190287 0 0 0 87316 53 0 0 25 0 5 0 517248578 2492661760 561833 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2434240 CPUtime=872.06 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 73840 0 0 0 86246 960 0 0 25 0 5 0 517248578 2492661760 561833 33554432000 4194304 4338100 140734048668112 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2434240 CPUtime=872.97 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 75183 0 0 0 86323 974 0 0 25 0 5 0 517248578 2492661760 561833 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3496.78
Current children cumulated vsize (KiB) 2434240

[startup+942.3 s]
/proc/loadavg: 8.16 8.17 8.04 9/184 21635
/proc/meminfo: memFree=16698396/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2522568 CPUtime=3736.39 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 595503 0 0 0 370564 3075 0 0 25 0 5 0 517247731 2583109632 590886 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 630642 590886 178 36 0 628547 0
[pid=21589/tid=21595] ppid=21587 vsize=2522568 CPUtime=929.52 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 83002 0 0 0 91990 962 0 0 25 0 5 0 517248578 2583109632 590886 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2522568 CPUtime=933.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 193177 0 0 0 93315 54 0 0 25 0 5 0 517248578 2583109632 590886 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2522568 CPUtime=931.82 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 82064 0 0 0 92172 1010 0 0 25 0 5 0 517248578 2583109632 590886 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2522568 CPUtime=932.86 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 83731 0 0 0 92261 1025 0 0 25 0 5 0 517248578 2583109632 590886 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3736.39
Current children cumulated vsize (KiB) 2522568

[startup+1002.3 s]
/proc/loadavg: 8.15 8.16 8.05 9/184 21637
/proc/meminfo: memFree=16368428/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2668556 CPUtime=3976.02 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 617462 0 0 0 394368 3234 0 0 25 0 5 0 517247731 2732601344 612845 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 667139 612846 178 36 0 665044 0
[pid=21589/tid=21595] ppid=21587 vsize=2668556 CPUtime=989.49 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 89732 0 0 0 97933 1016 0 0 25 0 5 0 517248578 2732601344 612846 33554432000 4194304 4338100 140734048668112 18446744073709551615 4241581 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2668556 CPUtime=993.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 194398 0 0 0 99314 55 0 0 25 0 5 0 517248578 2732601344 612846 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2668556 CPUtime=991.6 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 88971 0 0 0 98100 1060 0 0 25 0 5 0 517248578 2732601344 612846 33554432000 4194304 4338100 140734048668112 18446744073709551615 4286229 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2668556 CPUtime=992.74 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 90833 0 0 0 98195 1079 0 0 25 0 5 0 517248578 2732601344 612846 33554432000 4194304 4338100 140734048668112 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3976.02
Current children cumulated vsize (KiB) 2668556

[startup+1062.3 s]
/proc/loadavg: 8.14 8.16 8.05 9/184 21638
/proc/meminfo: memFree=16060632/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2668556 CPUtime=4215.64 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 633892 0 0 0 418208 3356 0 0 25 0 5 0 517247731 2732601344 629275 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 667139 629275 178 36 0 665044 0
[pid=21589/tid=21595] ppid=21587 vsize=2668556 CPUtime=1049.45 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 95564 0 0 0 103886 1059 0 0 25 0 5 0 517248578 2732601344 629275 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2668556 CPUtime=1053.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 194673 0 0 0 105314 55 0 0 25 0 5 0 517248578 2732601344 629275 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2668556 CPUtime=1051.31 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 94251 0 0 0 104032 1099 0 0 25 0 5 0 517248578 2732601344 629275 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2668556 CPUtime=1052.71 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 95875 0 0 0 104150 1121 0 0 25 0 5 0 517248578 2732601344 629275 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4215.64
Current children cumulated vsize (KiB) 2668556

[startup+1122.3 s]
/proc/loadavg: 8.21 8.18 8.06 9/184 21640
/proc/meminfo: memFree=15860484/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2740604 CPUtime=4455.25 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 642982 0 0 0 442037 3488 0 0 25 0 5 0 517247731 2806378496 638365 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 685151 638365 178 36 0 683056 0
[pid=21589/tid=21595] ppid=21587 vsize=2740604 CPUtime=1109.4 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 99108 0 0 0 109836 1104 0 0 25 0 5 0 517248578 2806378496 638365 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2740604 CPUtime=1113.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 195597 0 0 0 111313 56 0 0 25 0 5 0 517248578 2806378496 638365 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2740604 CPUtime=1110.99 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 96918 0 0 0 109962 1137 0 0 25 0 5 0 517248578 2806378496 638365 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2740604 CPUtime=1112.67 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 97830 0 0 0 110099 1168 0 0 25 0 5 0 517248578 2806378496 638365 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4455.25
Current children cumulated vsize (KiB) 2740604

[startup+1182.3 s]
/proc/loadavg: 8.08 8.14 8.06 9/184 21642
/proc/meminfo: memFree=15596836/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=2756884 CPUtime=4694.89 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) S 21587 21589 18837 0 -1 4202496 656474 0 0 0 465856 3633 0 0 25 0 5 0 517247731 2823049216 651857 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 689221 651857 178 36 0 687126 0
[pid=21589/tid=21595] ppid=21587 vsize=2756884 CPUtime=1169.38 cores=0,2,4,6
/proc/21589/task/21595/stat : 21595 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 103750 0 0 0 115785 1153 0 0 25 0 5 0 517248578 2823049216 651857 33554432000 4194304 4338100 140734048668112 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21589/tid=21596] ppid=21587 vsize=2756884 CPUtime=1173.69 cores=0,2,4,6
/proc/21589/task/21596/stat : 21596 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 196641 0 0 0 117312 57 0 0 25 0 5 0 517248578 2823049216 651857 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21589/tid=21597] ppid=21587 vsize=2756884 CPUtime=1170.7 cores=0,2,4,6
/proc/21589/task/21597/stat : 21597 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 100550 0 0 0 115887 1183 0 0 25 0 5 0 517248578 2823049216 651857 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=2756884 CPUtime=1172.63 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4202560 102004 0 0 0 116045 1218 0 0 25 0 5 0 517248578 2823049216 651857 33554432000 4194304 4338100 140734048668112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4694.89
Current children cumulated vsize (KiB) 2756884



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1208.7 s]
/proc/loadavg: 7.73 8.06 8.03 5/175 21660
/proc/meminfo: memFree=22510908/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=0 CPUtime=4800.32 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) Z 21587 21589 18837 0 -1 4203548 659985 0 0 0 476354 3678 0 0 18 0 2 0 517247731 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 0 0 0 0 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=0 CPUtime=1199 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4203588 102964 0 0 0 118668 1232 0 0 25 0 2 0 517248578 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.32
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1208.81 s]
/proc/loadavg: 7.73 8.06 8.03 2/173 21661
/proc/meminfo: memFree=24731400/32951132 swapFree=67111524/67111528
[pid=21589] ppid=21587 vsize=0 CPUtime=4800.43 cores=0,2,4,6
/proc/21589/stat : 21589 (ttsth-5-4) Z 21587 21589 18837 0 -1 4203548 659985 0 0 0 476354 3689 0 0 18 0 2 0 517247731 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/21589/statm: 0 0 0 0 0 0 0
[pid=21589/tid=21598] ppid=21587 vsize=0 CPUtime=1199.11 cores=0,2,4,6
/proc/21589/task/21598/stat : 21598 (ttsth-5-4) R 21587 21589 18837 0 -1 4203588 102964 0 0 0 118668 1243 0 0 25 0 2 0 517248578 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.43
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1208.83
CPU time (s): 4800.46
CPU user time (s): 4763.55
CPU system time (s): 36.9114
CPU usage (%): 397.117
Max. virtual memory (cumulated for all children) (KiB): 2763396

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4763.55
system time used= 36.9114
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 659985
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 71800
involuntary context switches= 21598

runsolver used 1.79973 second user time and 4.41533 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 07:54:41
IDJOB=3259802
IDBENCH=71799
IDSOLVER=1654
FILE ID=node131/3259802-1304056481
RUNJOBID= node131-1304054069-21377
PBS_JOBID= 13170364
Free space on /tmp= 71372 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/parity-games/instance_n9_i9_pp.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259802-1304056481/watcher-3259802-1304056481 -o /tmp/evaluation-result-3259802-1304056481/solver-3259802-1304056481 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259802-1304056481.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3e04f934e4f74a8e89587f6c1327f77e
RANDOM SEED=1165668314

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      25055640 kB
Buffers:        957108 kB
Cached:        6300892 kB
SwapCached:          4 kB
Active:         374028 kB
Inactive:      7026792 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      25055640 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           15724 kB
Writeback:          36 kB
AnonPages:      141944 kB
Mapped:          15316 kB
Slab:           431444 kB
PageTables:       4160 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   290032 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71376 MiB
End job on node131 at 2011-04-29 08:14:52