Trace number 3264642

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.44 1202.13

General information on the benchmark

NameSAT04/handmade/marijn/equilarge-l/
equilarge_l2.used-as.sat04-894.cnf
MD5SUMb3ddb2ee2bb2b83e1c98b2a9018351fc
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark590.41
Satisfiable
(Un)Satisfiability was proved
Number of variables4487
Number of clauses18555
Sum of the clauses size54310
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21355
Number of clauses of size 317200
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264642-1304100239.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 4487
0.00/0.01	c Clauses = 18555
0.00/0.01	c Literals = 54310
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 4487
0.00/0.01	c Clauses = 18555
0.00/0.01	c Literals = 54310
0.00/0.01	c a2
0.00/0.03	c a1
0.19/0.25	c Independent sub-problems = 1
0.19/0.25	c Sub-problem with 4487 variables and 18555 clauses
0.19/0.25	c using permutation method 4
0.19/0.25	c count = 2
0.19/0.25	c Ordering finished 0.250 seconds
0.19/0.25	c mapped 0.250 seconds
0.19/0.25	c permuted 0.250 seconds
0.19/0.25	c about to sort
0.19/0.26	c sorted
0.29/0.33	c clauses now 18555
0.29/0.33	c init finished 0.300 seconds
0.29/0.33	c tree built 0.320 seconds
0.29/0.33	c 
0.29/0.33	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.33	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.33	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.33	c ................................ | ..........                  1 |                                  0.3         3.1             
0.29/0.33	c ................................ | ..........                  2 |                                  0.3         6.2             
0.29/0.33	c ................................ | ..........                  3 |                                  0.3         9.4             
0.29/0.33	c ................................ | ..........                  4 |                                  0.3        12.5             
0.29/0.33	c ................................ | ..........                  5 |                                  0.3        15.6             
0.29/0.33	c ................................ | ..........               1000 |              184                 0.3      3030.3           22
96.05/24.31	c ................................ | ..........            1147000 | 20950        341112             96.1     11935.5      9172024
191.94/48.34	c ................................ | ..........            1749000 | 35037        411884            192.1      9105.1     12495312
287.79/72.38	c ................................ | ..........            2461000 | 47220        605050            288.1      8541.6     14884490
383.64/96.40	c ................................ | ..........            3079000 | 59097        660318            384.0      8017.4     16112627
479.92/120.45	c ................................ | *.........            3796000 | 72751        804380            480.1      7906.4     20301684
575.79/144.48	c ................................ | *.........            4390000 | 86132        855506            576.1      7619.9     22962695
672.06/168.50	c ................................ | *.........            5116000 | 99447        971654            672.1      7612.4     25231903
767.96/192.52	c ................................ | *.........            5866000 | 112524       1088259           768.0      7637.9     26970352
863.80/216.59	c ................................ | *.........            6539000 | 124241       1240414           864.2      7566.8     29492563
960.06/240.61	c ................................ | **........            7163000 | 135017       1324830           960.1      7460.6     31128944
1055.91/264.65	c ................................ | **........            7824000 | 146966       1473317          1056.1      7408.2     33063622
1151.78/288.67	c ................................ | **........            8434000 | 158765       1528353          1152.1      7320.6     33588190
1248.05/312.71	c ................................ | **........            9156000 | 172343       1613413          1248.1      7336.0     35069832
1343.91/336.73	c ................................ | **........            9877000 | 186338       1695057          1344.0      7348.7     36557333
1439.79/360.76	c ................................ | ***.......           10597000 | 199913       1757999          1440.0      7358.9     37757663
1536.05/384.82	c ................................ | ***.......           11144000 | 211366       1911334          1536.1      7254.6     43222031
1631.92/408.88	c ................................ | ***.......           11496000 | 219705       2010800          1632.3      7043.0     48090948
1728.19/432.90	c ................................ | ***.......           11894000 | 228223       2169409          1728.2      6882.3     52150605
1824.04/456.92	c ................................ | ***.......           12255000 | 235250       2287287          1824.1      6718.2     55882380
1919.90/480.96	c ................................ | ****......           12551000 | 241776       2359920          1920.2      6536.4     59282102
2016.18/505.02	c ................................ | ****......           12883000 | 248681       2464990          2016.3      6389.5     63614494
2112.06/529.01	c ................................ | ****......           13223000 | 254975       2550757          2112.1      6260.6     67009299
2207.93/553.07	c ................................ | ****......           13501000 | 260722       2626649          2208.2      6114.0     70475192
2303.79/577.07	c ................................ | ****......           13769000 | 266459       2717091          2304.1      5975.9     73330147
2400.06/601.13	c ................................ | *****.....           14079000 | 271986       2821882          2400.2      5865.7     76345940
2495.93/625.19	c ................................ | *****.....           14364000 | 277309       2910677          2496.3      5754.1     79470748
2592.20/649.28	c ................................ | *****.....           14637000 | 282076       3002204          2592.5      5645.9     82647161
2688.08/673.26	c ................................ | *****.....           14864000 | 286863       3053874          2688.3      5529.1     85175978
2783.94/697.26	c ................................ | *****.....           15077000 | 291384       3098960          2784.2      5415.2     87145683
2879.81/721.26	c ................................ | ******....           15225000 | 295584       3176706          2880.1      5286.3     89553326
2976.08/745.30	c ................................ | ******....           15370000 | 299676       3211263          2976.1      5164.5     91766639
3072.35/769.42	c ................................ | ******....           15514000 | 303294       3246247          3072.5      5049.4     93826762
3168.23/793.41	c ................................ | ******....           15667000 | 307133       3280707          3168.3      4945.0     95752627
3264.10/817.42	c ................................ | ******....           15818000 | 310995       3316176          3264.2      4845.9     97986571
3359.99/841.49	c ................................ | *******...           16001000 | 315453       3361280          3360.3      4761.7    100207858
3456.23/865.50	c ................................ | *******...           16198000 | 319503       3414026          3456.3      4686.6    102689815
3552.50/889.61	c ................................ | *******...           16409000 | 323555       3451407          3552.6      4618.9    105062256
3648.29/913.65	c ................................ | *******...           16626000 | 327721       3508977          3648.5      4556.9    105901341
3744.16/937.63	c ................................ | *******...           16808000 | 331697       3548357          3744.3      4489.0    105901341
3840.03/961.64	c ................................ | ********..           16999000 | 335498       3606534          3840.2      4426.6    106744534
3935.92/985.67	c ................................ | ********..           17175000 | 339443       3644096          3936.2      4363.4    107520600
4031.78/1009.68	c ................................ | ********..           17387000 | 343296       3684091          4032.1      4312.1    108823586
4128.04/1033.75	c ................................ | ********..           17590000 | 346872       3721052          4128.3      4260.9    110005398
4223.92/1057.76	c ................................ | ********..           17788000 | 350535       3750600          4224.2      4211.0    110832769
4320.18/1081.86	c ................................ | *********.           17987000 | 353922       3791090          4320.5      4163.2    112048157
4416.06/1105.88	c ................................ | *********.           18153000 | 357241       3826153          4416.4      4110.4    113392362
4511.93/1129.84	c ................................ | *********.           18301000 | 360318       3855873          4512.1      4056.0    114610017
4607.83/1153.86	c ................................ | *********.           18486000 | 363625       3894618          4608.0      4011.7    116088924
4704.07/1177.96	c ................................ | *********.           18616000 | 366656       3932761          4704.3      3957.2    117439151
4799.93/1201.98	c CPU limit (4800) reached
4799.93/1201.98	c Time = 4800.29 seconds
4799.93/1201.98	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264642-1304100239/watcher-3264642-1304100239 -o /tmp/evaluation-result-3264642-1304100239/solver-3264642-1304100239 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264642-1304100239.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.17 8.11 8.03 1/175 5579
/proc/meminfo: memFree=23622364/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) R 5577 5579 5016 0 -1 4202496 421 0 0 0 0 0 0 0 25 0 1 0 600254608 9633792 363 33554432000 4194304 4338100 140734488679232 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/5579/statm: 2352 363 147 36 0 257 0

[startup+0.0374049 s]
/proc/loadavg: 8.17 8.11 8.03 1/175 5579
/proc/meminfo: memFree=23622364/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=13640 CPUtime=0.03 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) R 5577 5579 5016 0 -1 4202496 1518 0 0 0 3 0 0 0 25 0 1 0 600254608 13967360 1385 33554432000 4194304 4338100 140734488679232 18446744073709551615 260905053215 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/5579/statm: 3410 1385 154 36 0 1315 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 13640

[startup+0.100393 s]
/proc/loadavg: 8.17 8.11 8.03 1/175 5579
/proc/meminfo: memFree=23622364/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=14036 CPUtime=0.09 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) R 5577 5579 5016 0 -1 4202496 1647 0 0 0 9 0 0 0 25 0 1 0 600254608 14372864 1514 33554432000 4194304 4338100 140734488679232 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/5579/statm: 3509 1514 156 36 0 1414 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 14036

[startup+0.300426 s]
/proc/loadavg: 8.17 8.11 8.03 1/175 5579
/proc/meminfo: memFree=23622364/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=58416 CPUtime=0.29 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) R 5577 5579 5016 0 -1 4202496 12620 0 0 0 28 1 0 0 25 0 1 0 600254608 59817984 12450 33554432000 4194304 4338100 140734488679232 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/5579/statm: 14604 12453 163 36 0 12509 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 58416

[startup+0.700292 s]
/proc/loadavg: 8.17 8.11 8.03 1/175 5579
/proc/meminfo: memFree=23622364/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=200316 CPUtime=1.78 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 20339 0 0 0 176 2 0 0 25 0 5 0 600254608 205123584 20169 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 50079 20169 176 36 0 47984 0
Current children cumulated CPU time (s) 1.78
Current children cumulated vsize (KiB) 200316

[startup+1.50115 s]
/proc/loadavg: 7.83 8.04 8.01 5/180 5584
/proc/meminfo: memFree=23529236/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=294664 CPUtime=4.98 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 28363 0 0 0 494 4 0 0 25 0 5 0 600254608 301735936 28193 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 73666 28193 176 36 0 71571 0
[pid=5579/tid=5581] ppid=5577 vsize=294664 CPUtime=1.15 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 3420 0 0 0 115 0 0 0 25 0 5 0 600254641 301735936 28193 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=294664 CPUtime=1.15 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 3228 0 0 0 115 0 0 0 19 0 5 0 600254641 301735936 28193 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=294664 CPUtime=1.16 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 4085 0 0 0 116 0 0 0 19 0 5 0 600254641 301735936 28193 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=294664 CPUtime=1.16 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 2816 0 0 0 116 0 0 0 19 0 5 0 600254641 301735936 28193 33554432000 4194304 4338100 140734488679232 18446744073709551615 260905090720 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4.98
Current children cumulated vsize (KiB) 294664

[startup+3.10085 s]
/proc/loadavg: 7.83 8.04 8.01 9/186 5615
/proc/meminfo: memFree=23432568/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=345768 CPUtime=11.37 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 39977 0 0 0 1132 5 0 0 25 0 5 0 600254608 354066432 39807 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 86442 39807 176 36 0 84347 0
[pid=5579/tid=5581] ppid=5577 vsize=345768 CPUtime=2.74 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 7203 0 0 0 273 1 0 0 25 0 5 0 600254641 354066432 39807 33554432000 4194304 4338100 140734488679232 18446744073709551615 260905090778 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=345768 CPUtime=2.75 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 5447 0 0 0 274 1 0 0 23 0 5 0 600254641 354066432 39807 33554432000 4194304 4338100 140734488679232 18446744073709551615 260904789398 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=345768 CPUtime=2.75 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 6797 0 0 0 275 0 0 0 23 0 5 0 600254641 354066432 39807 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=345768 CPUtime=2.75 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 5716 0 0 0 275 0 0 0 23 0 5 0 600254641 354066432 39807 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.37
Current children cumulated vsize (KiB) 345768

[startup+6.30027 s]
/proc/loadavg: 7.83 8.04 8.01 9/186 5615
/proc/meminfo: memFree=23170452/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=396740 CPUtime=24.15 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 53099 0 0 0 2407 8 0 0 25 0 5 0 600254608 406261760 52929 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 99185 52929 176 36 0 97090 0
[pid=5579/tid=5581] ppid=5577 vsize=396740 CPUtime=5.92 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 11115 0 0 0 590 2 0 0 25 0 5 0 600254641 406261760 52929 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=396740 CPUtime=5.95 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 8905 0 0 0 594 1 0 0 25 0 5 0 600254641 406261760 52929 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=396740 CPUtime=5.96 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 9373 0 0 0 595 1 0 0 25 0 5 0 600254641 406261760 52929 33554432000 4194304 4338100 140734488679232 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=396740 CPUtime=5.96 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 8892 0 0 0 595 1 0 0 25 0 5 0 600254641 406261760 52929 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.15
Current children cumulated vsize (KiB) 396740

[startup+12.7011 s]
/proc/loadavg: 7.86 8.04 8.01 9/186 5615
/proc/meminfo: memFree=22912700/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=465384 CPUtime=49.72 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 70241 0 0 0 4958 14 0 0 25 0 5 0 600254608 476553216 70071 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 116346 70071 176 36 0 114251 0
[pid=5579/tid=5581] ppid=5577 vsize=465384 CPUtime=12.32 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 14637 0 0 0 1228 4 0 0 25 0 5 0 600254641 476553216 70071 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=465384 CPUtime=12.33 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 11555 0 0 0 1231 2 0 0 25 0 5 0 600254641 476553216 70071 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=465384 CPUtime=12.35 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 15081 0 0 0 1233 2 0 0 25 0 5 0 600254641 476553216 70071 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=465384 CPUtime=12.36 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 14154 0 0 0 1233 3 0 0 25 0 5 0 600254641 476553216 70071 33554432000 4194304 4338100 140734488679232 18446744073709551615 260904789742 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.72
Current children cumulated vsize (KiB) 465384

[startup+25.5007 s]
/proc/loadavg: 7.88 8.04 8.01 9/186 5616
/proc/meminfo: memFree=22580652/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=563672 CPUtime=100.84 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 95116 0 0 0 10063 21 0 0 25 0 5 0 600254608 577200128 94946 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 140918 94946 176 36 0 138823 0
[pid=5579/tid=5581] ppid=5577 vsize=563672 CPUtime=25.12 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 21200 0 0 0 2506 6 0 0 25 0 5 0 600254641 577200128 94946 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=563672 CPUtime=25.05 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 16573 0 0 0 2502 3 0 0 25 0 5 0 600254641 577200128 94946 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=563672 CPUtime=25.15 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 22266 0 0 0 2511 4 0 0 25 0 5 0 600254641 577200128 94946 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=563672 CPUtime=25.15 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 20263 0 0 0 2510 5 0 0 25 0 5 0 600254641 577200128 94946 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 100.84
Current children cumulated vsize (KiB) 563672

[startup+51.1011 s]
/proc/loadavg: 8.06 8.07 8.02 9/186 5616
/proc/meminfo: memFree=22169948/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=668168 CPUtime=203.11 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 121748 0 0 0 20276 35 0 0 25 0 5 0 600254608 684204032 121578 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 167042 121578 176 36 0 164947 0
[pid=5579/tid=5581] ppid=5577 vsize=668168 CPUtime=50.72 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 27062 0 0 0 5063 9 0 0 25 0 5 0 600254641 684204032 121578 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=668168 CPUtime=50.52 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 23685 0 0 0 5045 7 0 0 25 0 5 0 600254641 684204032 121578 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=668168 CPUtime=50.75 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 29535 0 0 0 5067 8 0 0 25 0 5 0 600254641 684204032 121578 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=668168 CPUtime=50.76 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 26652 0 0 0 5068 8 0 0 25 0 5 0 600254641 684204032 121578 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.11
Current children cumulated vsize (KiB) 668168

[startup+102.307 s]
/proc/loadavg: 8.17 8.09 8.02 9/186 5618
/proc/meminfo: memFree=21550900/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=808796 CPUtime=407.64 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 158567 0 0 0 40707 57 0 0 25 0 5 0 600254608 828207104 158397 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 202199 158397 176 36 0 200104 0
[pid=5579/tid=5581] ppid=5577 vsize=808796 CPUtime=101.91 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 33745 0 0 0 10176 15 0 0 25 0 5 0 600254641 828207104 158397 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=808796 CPUtime=101.44 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 33947 0 0 0 10132 12 0 0 25 0 5 0 600254641 828207104 158397 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=808796 CPUtime=101.94 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 43211 0 0 0 10180 14 0 0 25 0 5 0 600254641 828207104 158397 33554432000 4194304 4338100 140734488679232 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=808796 CPUtime=101.96 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 32850 0 0 0 10184 12 0 0 25 0 5 0 600254641 828207104 158397 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 407.64
Current children cumulated vsize (KiB) 808796

[startup+162.301 s]
/proc/loadavg: 8.19 8.10 8.03 9/186 5619
/proc/meminfo: memFree=20917952/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=1052556 CPUtime=647.29 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 219544 0 0 0 64648 81 0 0 25 0 5 0 600254608 1077817344 219374 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 263139 219374 176 36 0 261044 0
[pid=5579/tid=5581] ppid=5577 vsize=1052556 CPUtime=161.91 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 49561 0 0 0 16167 24 0 0 25 0 5 0 600254641 1077817344 219374 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=1052556 CPUtime=161.12 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 48361 0 0 0 16095 17 0 0 25 0 5 0 600254641 1077817344 219374 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=1052556 CPUtime=161.95 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 60487 0 0 0 16174 21 0 0 25 0 5 0 600254641 1077817344 219374 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=1052556 CPUtime=161.96 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 46321 0 0 0 16180 16 0 0 25 0 5 0 600254641 1077817344 219374 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 647.29
Current children cumulated vsize (KiB) 1052556


################
# More data... #
################

/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 178091 0 0 0 70145 51 0 0 25 0 5 0 600254641 3186393088 728965 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2804.31
Current children cumulated vsize (KiB) 3111712

[startup+762.307 s]
/proc/loadavg: 8.12 8.08 8.03 9/186 5635
/proc/meminfo: memFree=16645652/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=3287804 CPUtime=3044.01 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 772778 0 0 0 304122 279 0 0 25 0 5 0 600254608 3366711296 772608 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 821951 772608 176 36 0 819856 0
[pid=5579/tid=5581] ppid=5577 vsize=3287804 CPUtime=761.92 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 178548 0 0 0 76101 91 0 0 25 0 5 0 600254641 3366711296 772608 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=3287804 CPUtime=760.68 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 178143 0 0 0 76013 55 0 0 25 0 5 0 600254641 3366711296 772608 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=3287804 CPUtime=759.08 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 209112 0 0 0 75833 75 0 0 25 0 5 0 600254641 3366711296 772608 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=3287804 CPUtime=761.96 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 192161 0 0 0 76142 54 0 0 25 0 5 0 600254641 3366711296 772608 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3044.01
Current children cumulated vsize (KiB) 3287804

[startup+822.301 s]
/proc/loadavg: 8.11 8.08 8.03 9/186 5637
/proc/meminfo: memFree=16305856/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=3460372 CPUtime=3283.67 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 815640 0 0 0 328079 288 0 0 25 0 5 0 600254608 3543420928 815470 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 865093 815470 176 36 0 862998 0
[pid=5579/tid=5581] ppid=5577 vsize=3460372 CPUtime=821.92 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 185556 0 0 0 82098 94 0 0 25 0 5 0 600254641 3543420928 815470 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=3460372 CPUtime=820.68 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 189220 0 0 0 82010 58 0 0 25 0 5 0 600254641 3543420928 815470 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254421 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=3460372 CPUtime=818.75 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 221860 0 0 0 81798 77 0 0 25 0 5 0 600254641 3543420928 815470 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=3460372 CPUtime=821.96 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 204190 0 0 0 82139 57 0 0 25 0 5 0 600254641 3543420928 815470 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3283.67
Current children cumulated vsize (KiB) 3460372

[startup+882.301 s]
/proc/loadavg: 8.21 8.11 8.04 9/186 5639
/proc/meminfo: memFree=15948208/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=3640384 CPUtime=3523.33 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 862499 0 0 0 352032 301 0 0 25 0 5 0 600254608 3727753216 862329 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 910096 862329 176 36 0 908001 0
[pid=5579/tid=5581] ppid=5577 vsize=3640384 CPUtime=881.9 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 194972 0 0 0 88094 96 0 0 25 0 5 0 600254641 3727753216 862329 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=3640384 CPUtime=880.68 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 198353 0 0 0 88007 61 0 0 25 0 5 0 600254641 3727753216 862329 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=3640384 CPUtime=878.42 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 240184 0 0 0 87760 82 0 0 25 0 5 0 600254641 3727753216 862329 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=3640384 CPUtime=881.96 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 214176 0 0 0 88137 59 0 0 25 0 5 0 600254641 3727753216 862329 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3523.33
Current children cumulated vsize (KiB) 3640384

[startup+942.301 s]
/proc/loadavg: 8.21 8.12 8.05 9/186 5640
/proc/meminfo: memFree=15632224/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=3830996 CPUtime=3762.93 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 900238 0 0 0 375983 310 0 0 25 0 5 0 600254608 3922939904 900068 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 957749 900068 176 36 0 955654 0
[pid=5579/tid=5581] ppid=5577 vsize=3830996 CPUtime=941.9 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 201096 0 0 0 94091 99 0 0 25 0 5 0 600254641 3922939904 900068 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=3830996 CPUtime=940.65 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 204923 0 0 0 94003 62 0 0 25 0 5 0 600254641 3922939904 900068 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=3830996 CPUtime=938.1 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 248019 0 0 0 93726 84 0 0 25 0 5 0 600254641 3922939904 900068 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=3830996 CPUtime=941.92 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 231386 0 0 0 94130 62 0 0 25 0 5 0 600254641 3922939904 900068 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3762.93
Current children cumulated vsize (KiB) 3830996

[startup+1002.3 s]
/proc/loadavg: 8.13 8.11 8.05 9/186 5642
/proc/meminfo: memFree=15394624/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=3907124 CPUtime=4002.61 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 919489 0 0 0 399945 316 0 0 25 0 5 0 600254608 4000894976 919319 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 976781 919319 176 36 0 974686 0
[pid=5579/tid=5581] ppid=5577 vsize=3907124 CPUtime=1001.9 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 206756 0 0 0 100089 101 0 0 25 0 5 0 600254641 4000894976 919319 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917208323 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=3907124 CPUtime=1000.65 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 218143 0 0 0 100001 64 0 0 25 0 5 0 600254641 4000894976 919319 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=3907124 CPUtime=997.76 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 248210 0 0 0 99692 84 0 0 25 0 5 0 600254641 4000894976 919319 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254231 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=3907124 CPUtime=1001.93 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 231566 0 0 0 100130 63 0 0 25 0 5 0 600254641 4000894976 919319 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4002.61
Current children cumulated vsize (KiB) 3907124

[startup+1062.3 s]
/proc/loadavg: 8.04 8.09 8.04 9/186 5643
/proc/meminfo: memFree=15168532/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=3986248 CPUtime=4242.3 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 939744 0 0 0 423909 321 0 0 25 0 5 0 600254608 4081917952 939574 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 996562 939574 176 36 0 994467 0
[pid=5579/tid=5581] ppid=5577 vsize=3986248 CPUtime=1061.9 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 216656 0 0 0 106086 104 0 0 25 0 5 0 600254641 4081917952 939574 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=3986248 CPUtime=1060.66 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 228321 0 0 0 105999 67 0 0 25 0 5 0 600254641 4081917952 939574 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=3986248 CPUtime=1057.45 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 248296 0 0 0 105660 85 0 0 25 0 5 0 600254641 4081917952 939574 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=3986248 CPUtime=1061.93 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 231657 0 0 0 106130 63 0 0 25 0 5 0 600254641 4081917952 939574 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4242.3
Current children cumulated vsize (KiB) 3986248

[startup+1122.3 s]
/proc/loadavg: 8.08 8.08 8.04 9/186 5645
/proc/meminfo: memFree=14913820/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=4093524 CPUtime=4481.97 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 966251 0 0 0 447869 328 0 0 25 0 5 0 600254608 4191768576 966081 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 1023381 966081 176 36 0 1021286 0
[pid=5579/tid=5581] ppid=5577 vsize=4093524 CPUtime=1121.9 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 226783 0 0 0 112082 108 0 0 25 0 5 0 600254641 4191768576 966081 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=4093524 CPUtime=1120.66 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 236883 0 0 0 111997 69 0 0 25 0 5 0 600254641 4191768576 966081 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=4093524 CPUtime=1117.12 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 255919 0 0 0 111626 86 0 0 25 0 5 0 600254641 4191768576 966081 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=4093524 CPUtime=1121.93 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 231852 0 0 0 112130 63 0 0 25 0 5 0 600254641 4191768576 966081 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4481.97
Current children cumulated vsize (KiB) 4093524

[startup+1182.3 s]
/proc/loadavg: 8.09 8.08 8.04 9/186 5647
/proc/meminfo: memFree=14652272/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=4194940 CPUtime=4721.64 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) S 5577 5579 5016 0 -1 4202496 993058 0 0 0 471829 335 0 0 25 0 5 0 600254608 4295618560 992888 33554432000 4194304 4338100 140734488679232 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 1048735 992888 176 36 0 1046640 0
[pid=5579/tid=5581] ppid=5577 vsize=4194940 CPUtime=1181.91 cores=1,3,5,7
/proc/5579/task/5581/stat : 5581 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 236560 0 0 0 118080 111 0 0 25 0 5 0 600254641 4295618560 992888 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5579/tid=5582] ppid=5577 vsize=4194940 CPUtime=1180.65 cores=1,3,5,7
/proc/5579/task/5582/stat : 5582 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 246312 0 0 0 117994 71 0 0 25 0 5 0 600254641 4295618560 992888 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5579/tid=5583] ppid=5577 vsize=4194940 CPUtime=1176.79 cores=1,3,5,7
/proc/5579/task/5583/stat : 5583 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 263346 0 0 0 117592 87 0 0 25 0 5 0 600254641 4295618560 992888 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=4194940 CPUtime=1181.92 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4202560 232026 0 0 0 118129 63 0 0 25 0 5 0 600254641 4295618560 992888 33554432000 4194304 4338100 140734488679232 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4721.64
Current children cumulated vsize (KiB) 4194940



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.06 8.08 8.04 6/183 5647
/proc/meminfo: memFree=14845212/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=0 CPUtime=4800.31 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) Z 5577 5579 5016 0 -1 4203548 1000137 0 0 0 479692 339 0 0 18 0 2 0 600254608 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 0 0 0 0 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=0 CPUtime=1201.63 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4203588 232083 0 0 0 120098 65 0 0 25 0 2 0 600254641 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.31
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.11 s]
/proc/loadavg: 8.06 8.08 8.04 6/183 5647
/proc/meminfo: memFree=14845212/32950928 swapFree=67071984/67111528
[pid=5579] ppid=5577 vsize=0 CPUtime=4800.41 cores=1,3,5,7
/proc/5579/stat : 5579 (ttsth-5-4) Z 5577 5579 5016 0 -1 4203548 1000137 0 0 0 479692 349 0 0 18 0 2 0 600254608 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5579/statm: 0 0 0 0 0 0 0
[pid=5579/tid=5584] ppid=5577 vsize=0 CPUtime=1201.74 cores=1,3,5,7
/proc/5579/task/5584/stat : 5584 (ttsth-5-4) R 5577 5579 5016 0 -1 4203588 232083 0 0 0 120098 76 0 0 25 0 2 0 600254641 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.41
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.13
CPU time (s): 4800.44
CPU user time (s): 4796.92
CPU system time (s): 3.51946
CPU usage (%): 399.327
Max. virtual memory (cumulated for all children) (KiB): 4223356

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.92
system time used= 3.51946
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1000137
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1364
involuntary context switches= 19184

runsolver used 1.86372 second user time and 4.70728 second system time

The end

Launcher Data 

Begin job on node116 at 2011-04-29 20:03:59
IDJOB=3264642
IDBENCH=84761
IDSOLVER=1654
FILE ID=node116/3264642-1304100239
RUNJOBID= node116-1304099035-5482
PBS_JOBID= 13170597
Free space on /tmp= 72516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/marijn/equilarge-l/equilarge_l2.used-as.sat04-894.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264642-1304100239/watcher-3264642-1304100239 -o /tmp/evaluation-result-3264642-1304100239/solver-3264642-1304100239 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264642-1304100239.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b3ddb2ee2bb2b83e1c98b2a9018351fc
RANDOM SEED=1593962888

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23622644 kB
Buffers:       1905076 kB
Cached:        6953652 kB
SwapCached:       9732 kB
Active:        4401704 kB
Inactive:      4498004 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23622644 kB
SwapTotal:    67111528 kB
SwapFree:     67071984 kB
Dirty:             412 kB
Writeback:           0 kB
AnonPages:       34624 kB
Mapped:          14192 kB
Slab:           365488 kB
PageTables:       4752 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   221044 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72516 MiB
End job on node116 at 2011-04-29 20:24:03