Trace number 3258218

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.29 1202.5

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v1375c111739g6.cnf
MD5SUMc7c04151ac14a13eb4a03259d055e119
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1375
Number of clauses111739
Sum of the clauses size224743
Maximum clause length25
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2111684
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 555

Solver Data 

0.00/0.07	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258218-1304049739.cnf 
0.00/0.07	c Using SATTIMEOUT of 4800
0.00/0.07	c Using SATRAM of 15500
0.00/0.07	c Size of subset node = 32
0.00/0.07	c Variables = 1375
0.00/0.07	c Clauses = 111739
0.00/0.07	c Literals = 224743
0.00/0.07	c Reduced size:
0.00/0.07	c Variables = 1375
0.00/0.07	c Clauses = 111739
0.00/0.07	c Literals = 224743
0.00/0.07	c a2
0.09/0.14	c a1
0.19/0.27	c Independent sub-problems = 1
0.19/0.27	c Sub-problem with 1375 variables and 111739 clauses
0.19/0.27	c using permutation method 4
0.19/0.27	c count = 1
0.19/0.27	c Ordering finished 0.260 seconds
0.19/0.27	c mapped 0.270 seconds
0.19/0.27	c permuted 0.270 seconds
0.19/0.27	c about to sort
0.19/0.28	c sorted
0.49/0.53	c clauses now 111739
0.49/0.53	c init finished 0.300 seconds
0.49/0.53	c tree built 0.520 seconds
0.49/0.53	c 
0.49/0.53	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.53	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.53	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.53	c ................................ | ..........                  1 |                                  0.5         1.9             
0.49/0.53	c ................................ | ..........                  2 |                                  0.5         3.8             
0.49/0.53	c ................................ | ..........                  3 |                                  0.5         5.8             
0.49/0.53	c ................................ | ..........                  4 |                                  0.5         7.7             
0.49/0.53	c ................................ | ..........                  5 |                                  0.5         9.6             
0.49/0.53	c ................................ | ..........               1000 | 5            145                 0.5      1886.8         3840
95.86/24.44	c ................................ | ..........            3416000 | 61057        424887             96.0     35575.9     18231559
191.72/48.46	c ................................ | ..........            5993000 | 112349       715730            192.0     31213.5     31134876
287.99/72.50	c ................................ | ..........            8329000 | 160425       949272            288.0     28920.1     37916010
383.85/96.55	c ................................ | ..........           10349000 | 205163       1170593           384.0     26947.7     46982648
479.73/120.58	c ................................ | *.........           12183000 | 245245       1371183           480.1     25378.6     52476157
575.98/144.61	c ................................ | *.........           13798000 | 285682       1566556           576.0     23953.6     58846583
671.80/168.66	c ................................ | *.........           15859000 | 328737       1766039           672.0     23597.9     60596404
767.65/192.69	c ................................ | *.........           17829000 | 374421       1952202           768.0     23214.5     66518994
863.90/216.72	c ................................ | *.........           19481000 | 416885       2117741           864.0     22547.2     72115481
959.75/240.76	c ................................ | **........           20954000 | 456894       2265932           960.0     21827.1     77149613
1055.65/264.79	c ................................ | **........           22560000 | 496768       2413578          1056.0     21363.2     81398799
1151.91/288.83	c ................................ | **........           23997000 | 534953       2552722          1152.0     20829.8     84454109
1247.74/312.87	c ................................ | **........           25266000 | 571687       2691515          1248.0     20244.9     87042072
1343.98/336.91	c ................................ | **........           26553000 | 603397       2857520          1344.0     19756.4     89824098
1439.84/360.94	c ................................ | ***.......           27755000 | 635374       3011435          1440.0     19274.2     94333513
1535.71/384.99	c ................................ | ***.......           28889000 | 669827       3169365          1536.1     18807.2    102056714
1631.96/409.01	c ................................ | ***.......           30000000 | 700500       3327920          1632.0     18382.4    108051193
1727.84/433.08	c ................................ | ***.......           31050000 | 730039       3461692          1728.2     17967.1    111078231
1823.68/457.09	c ................................ | ***.......           32007000 | 759214       3584445          1824.1     17547.1    114069797
1919.93/481.12	c ................................ | ****......           33360000 | 793990       3721475          1920.0     17374.9    119402711
2015.75/505.16	c ................................ | ****......           34595000 | 827653       3856752          2016.0     17160.2    119418889
2111.99/529.21	c ................................ | ****......           35723000 | 858668       3982514          2112.0     16914.1    122064712
2207.82/553.26	c ................................ | ****......           36858000 | 889053       4100465          2208.1     16692.5    122064712
2303.67/577.28	c ................................ | ****......           37927000 | 920256       4217825          2304.0     16461.4    122064712
2399.96/601.33	c ................................ | *****.....           38967000 | 949972       4331615          2400.1     16235.7    123164701
2495.81/625.35	c ................................ | *****.....           40044000 | 980653       4449345          2496.0     16043.1    127224033
2591.68/649.38	c ................................ | *****.....           41071000 | 1011099      4557227          2592.0     15845.2    130535788
2687.92/673.44	c ................................ | *****.....           42021000 | 1038658      4669842          2688.1     15632.3    133774065
2783.79/697.47	c ................................ | *****.....           42955000 | 1063741      4784541          2784.1     15428.7    137396658
2880.04/721.51	c ................................ | ******....           43855000 | 1089991      4900585          2880.1     15227.0    139829684
2975.90/745.53	c ................................ | ******....           44747000 | 1115340      5018854          2976.0     15035.9    141736751
3071.73/769.59	c ................................ | ******....           45592000 | 1140927      5131830          3072.1     14840.8    144742283
3167.92/793.63	c ................................ | ******....           46410000 | 1166230      5244512          3168.1     14649.4    150308312
3263.77/817.68	c ................................ | ******....           47218000 | 1188960      5354442          3264.1     14465.8    153444407
3359.64/841.70	c ................................ | *******...           48229000 | 1213260      5463679          3360.0     14353.7    155228890
3455.90/865.73	c ................................ | *******...           49703000 | 1245169      5598176          3456.0     14381.5    158342391
3551.74/889.77	c ................................ | *******...           51154000 | 1279293      5724588          3552.0     14401.4    163503955
3647.98/913.82	c ................................ | *******...           52369000 | 1311081      5835050          3648.1     14355.3    166644972
3743.85/937.84	c ................................ | *******...           53545000 | 1345818      5941337          3744.0     14301.4    171084857
3839.71/961.89	c ................................ | ********..           54665000 | 1375580      6040584          3840.1     14235.4    172712378
3935.96/985.92	c ................................ | ********..           55854000 | 1403500      6147464          3936.1     14190.3    173256827
4031.81/1009.95	c ................................ | ********..           57164000 | 1434520      6276083          4032.0     14177.5    173256827
4127.65/1033.99	c ................................ | ********..           58425000 | 1466322      6399651          4128.0     14153.2    175434495
4223.86/1058.05	c ................................ | ********..           59541000 | 1496885      6509340          4224.1     14095.6    178194702
4319.76/1082.08	c ................................ | *********.           60576000 | 1525105      6610310          4320.1     14022.0    180749533
4415.98/1106.11	c ................................ | *********.           61575000 | 1550086      6706417          4416.0     13943.6    183342831
4511.77/1130.16	c ................................ | *********.           62674000 | 1573950      6805058          4512.0     13890.5    186880308
4608.04/1154.20	c ................................ | *********.           63725000 | 1599141      6908579          4608.1     13829.1    189719964
4703.90/1178.24	c ................................ | *********.           64685000 | 1624224      7011021          4704.1     13750.8    190428019
4799.73/1202.28	c CPU limit (4800) reached
4799.73/1202.28	c Time = 4800.06 seconds
4799.73/1202.28	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258218-1304049739/watcher-3258218-1304049739 -o /tmp/evaluation-result-3258218-1304049739/solver-3258218-1304049739 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258218-1304049739.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.82 7.72 6.86 2/174 30400
/proc/meminfo: memFree=15158020/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=10024 CPUtime=0 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) R 30398 30400 30368 0 -1 4202496 565 0 0 0 0 0 0 0 23 0 1 0 516573567 10264576 507 33554432000 4194304 4338100 140733399285328 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30400/statm: 2506 508 147 36 0 411 0

[startup+0.0968979 s]
/proc/loadavg: 6.82 7.72 6.86 2/174 30400
/proc/meminfo: memFree=15158020/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=31532 CPUtime=0.09 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) R 30398 30400 30368 0 -1 4202496 5905 0 0 0 9 0 0 0 23 0 1 0 516573567 32288768 5846 33554432000 4194304 4338100 140733399285328 18446744073709551615 4206987 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30400/statm: 7883 5846 153 36 0 5788 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31532

[startup+0.100894 s]
/proc/loadavg: 6.82 7.72 6.86 2/174 30400
/proc/meminfo: memFree=15158020/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=31532 CPUtime=0.09 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) R 30398 30400 30368 0 -1 4202496 5905 0 0 0 9 0 0 0 23 0 1 0 516573567 32288768 5846 33554432000 4194304 4338100 140733399285328 18446744073709551615 4206993 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30400/statm: 7883 5846 153 36 0 5788 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31532

[startup+0.300862 s]
/proc/loadavg: 6.82 7.72 6.86 2/174 30400
/proc/meminfo: memFree=15158020/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=34056 CPUtime=0.29 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) R 30398 30400 30368 0 -1 4202496 8665 0 0 0 29 0 0 0 25 0 1 0 516573567 34873344 6527 33554432000 4194304 4338100 140733399285328 18446744073709551615 4274560 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30400/statm: 8514 6527 163 36 0 6419 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 34056

[startup+1.4326 s]
/proc/loadavg: 6.82 7.72 6.86 2/174 30400
/proc/meminfo: memFree=15158020/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=299396 CPUtime=4.11 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 31648 0 0 0 400 11 0 0 25 0 5 0 516573567 306581504 29291 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 74849 29293 176 36 0 72754 0
Current children cumulated CPU time (s) 4.11
Current children cumulated vsize (KiB) 299396

[startup+1.50059 s]
/proc/loadavg: 6.82 7.72 6.86 2/174 30400
/proc/meminfo: memFree=15158020/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=308912 CPUtime=4.39 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 32342 0 0 0 427 12 0 0 25 0 5 0 516573567 316325888 29985 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 77228 29985 176 36 0 75133 0
Current children cumulated CPU time (s) 4.39
Current children cumulated vsize (KiB) 308912

[startup+3.10031 s]
/proc/loadavg: 6.82 7.72 6.86 9/186 30415
/proc/meminfo: memFree=14788412/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=359092 CPUtime=10.77 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 46630 0 0 0 1048 29 0 0 25 0 5 0 516573567 367710208 44273 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 89773 44273 176 36 0 87678 0
[pid=30400/tid=30408] ppid=30398 vsize=359092 CPUtime=2.55 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 9619 0 0 0 253 2 0 0 22 0 5 0 516573621 367710208 44273 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=359092 CPUtime=2.55 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 4013 0 0 0 244 11 0 0 22 0 5 0 516573621 367710208 44273 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=359092 CPUtime=2.54 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 8120 0 0 0 253 1 0 0 22 0 5 0 516573621 367710208 44273 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=359092 CPUtime=2.55 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 4099 0 0 0 244 11 0 0 22 0 5 0 516573621 367710208 44273 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.77
Current children cumulated vsize (KiB) 359092

[startup+6.30075 s]
/proc/loadavg: 6.91 7.73 6.87 9/186 30415
/proc/meminfo: memFree=14592408/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=460020 CPUtime=23.56 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 71060 0 0 0 2295 61 0 0 25 0 5 0 516573567 471060480 68703 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 115005 68703 176 36 0 112910 0
[pid=30400/tid=30408] ppid=30398 vsize=460020 CPUtime=5.76 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 19014 0 0 0 571 5 0 0 25 0 5 0 516573621 471060480 68703 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=460020 CPUtime=5.76 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 7071 0 0 0 551 25 0 0 25 0 5 0 516573621 471060480 68703 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=460020 CPUtime=5.73 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 16659 0 0 0 570 3 0 0 25 0 5 0 516573621 471060480 68703 33554432000 4194304 4338100 140733399285328 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=460020 CPUtime=5.76 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 7537 0 0 0 551 25 0 0 25 0 5 0 516573621 471060480 68703 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.56
Current children cumulated vsize (KiB) 460020

[startup+12.7006 s]
/proc/loadavg: 7.00 7.73 6.87 9/186 30415
/proc/meminfo: memFree=14292468/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=597352 CPUtime=49.14 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 105656 0 0 0 4793 121 0 0 25 0 5 0 516573567 611688448 103299 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 149338 103299 176 36 0 147243 0
[pid=30400/tid=30408] ppid=30398 vsize=597352 CPUtime=12.16 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 33940 0 0 0 1208 8 0 0 25 0 5 0 516573621 611688448 103301 33554432000 4194304 4338100 140733399285328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=597352 CPUtime=12.16 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 10764 0 0 0 1165 51 0 0 25 0 5 0 516573621 611688448 103301 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=597352 CPUtime=12.1 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 28744 0 0 0 1205 5 0 0 25 0 5 0 516573621 611688448 103301 33554432000 4194304 4338100 140733399285328 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=597352 CPUtime=12.16 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 11432 0 0 0 1163 53 0 0 25 0 5 0 516573621 611688448 103302 33554432000 4194304 4338100 140733399285328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.14
Current children cumulated vsize (KiB) 597352

[startup+25.5004 s]
/proc/loadavg: 7.22 7.74 6.89 9/186 30415
/proc/meminfo: memFree=13829616/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=863232 CPUtime=100.25 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 171837 0 0 0 9790 235 0 0 25 0 5 0 516573567 883949568 169480 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 215808 169480 176 36 0 213713 0
[pid=30400/tid=30408] ppid=30398 vsize=863232 CPUtime=24.9 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 60407 0 0 0 2475 15 0 0 25 0 5 0 516573621 883949568 169480 33554432000 4194304 4338100 140733399285328 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=863232 CPUtime=24.95 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 18148 0 0 0 2394 101 0 0 25 0 5 0 516573621 883949568 169480 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=863232 CPUtime=24.88 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 51942 0 0 0 2478 10 0 0 25 0 5 0 516573621 883949568 169480 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=863232 CPUtime=24.96 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 20561 0 0 0 2391 105 0 0 25 0 5 0 516573621 883949568 169480 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.25
Current children cumulated vsize (KiB) 863232

[startup+51.1009 s]
/proc/loadavg: 7.76 7.83 6.94 9/186 30416
/proc/meminfo: memFree=13108664/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=1277852 CPUtime=202.51 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 274588 0 0 0 19836 415 0 0 25 0 5 0 516573567 1308520448 272231 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 319463 272231 176 36 0 317368 0
[pid=30400/tid=30408] ppid=30398 vsize=1277852 CPUtime=50.35 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 99653 0 0 0 5009 26 0 0 25 0 5 0 516573621 1308520448 272231 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=1277852 CPUtime=50.56 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 30256 0 0 0 4871 185 0 0 25 0 5 0 516573621 1308520448 272231 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=1277852 CPUtime=50.48 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 91272 0 0 0 5032 16 0 0 25 0 5 0 516573621 1308520448 272231 33554432000 4194304 4338100 140733399285328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=1277852 CPUtime=50.56 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 32628 0 0 0 4871 185 0 0 25 0 5 0 516573621 1308520448 272231 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.51
Current children cumulated vsize (KiB) 1277852

[startup+102.301 s]
/proc/loadavg: 7.96 7.87 7.00 9/186 30419
/proc/meminfo: memFree=12083928/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=1789996 CPUtime=407.01 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 399868 0 0 0 39967 734 0 0 25 0 5 0 516573567 1832955904 397511 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 447499 397511 176 36 0 445404 0
[pid=30400/tid=30408] ppid=30398 vsize=1789996 CPUtime=101.28 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 156681 0 0 0 10083 45 0 0 25 0 5 0 516573621 1832955904 397511 33554432000 4194304 4338100 140733399285328 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=1789996 CPUtime=101.74 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 40178 0 0 0 9845 329 0 0 25 0 5 0 516573621 1832955904 397511 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=1789996 CPUtime=101.68 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 142346 0 0 0 10142 26 0 0 25 0 5 0 516573621 1832955904 397511 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=1789996 CPUtime=101.75 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 39884 0 0 0 9845 330 0 0 25 0 5 0 516573621 1832955904 397511 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.01
Current children cumulated vsize (KiB) 1789996

[startup+162.3 s]
/proc/loadavg: 8.39 8.03 7.11 9/186 30422
/proc/meminfo: memFree=11108540/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=2222812 CPUtime=646.65 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 505088 0 0 0 63548 1117 0 0 25 0 5 0 516573567 2276159488 502731 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 555703 502731 176 36 0 553608 0
[pid=30400/tid=30408] ppid=30398 vsize=2222812 CPUtime=160.94 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 197182 0 0 0 16033 61 0 0 25 0 5 0 516573621 2276159488 502731 33554432000 4194304 4338100 140733399285328 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=2222812 CPUtime=161.73 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 55525 0 0 0 15664 509 0 0 25 0 5 0 516573621 2276159488 502731 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=2222812 CPUtime=161.68 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 177214 0 0 0 16134 34 0 0 25 0 5 0 516573621 2276159488 502731 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=2222812 CPUtime=161.74 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 54388 0 0 0 15665 509 0 0 25 0 5 0 516573621 2276159488 502731 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.65
Current children cumulated vsize (KiB) 2222812

[startup+222.301 s]
/proc/loadavg: 8.24 8.05 7.17 9/186 30423
/proc/meminfo: memFree=10351512/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=2660732 CPUtime=886.27 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 615291 0 0 0 87180 1447 0 0 25 0 5 0 516573567 2724589568 612934 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 665183 612934 176 36 0 663088 0
[pid=30400/tid=30408] ppid=30398 vsize=2660732 CPUtime=220.6 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 267616 0 0 0 21976 84 0 0 25 0 5 0 516573621 2724589568 612934 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 92856 0 0 0 68377 1717 0 0 25 0 5 0 516573621 4851429376 1125675 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2803.35
Current children cumulated vsize (KiB) 4737724

[startup+762.301 s]
/proc/loadavg: 8.23 8.11 7.57 9/186 30438
/proc/meminfo: memFree=6641652/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=4922540 CPUtime=3042.96 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1174660 0 0 0 300314 3982 0 0 25 0 5 0 516573567 5040680960 1172303 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1230635 1172303 176 36 0 1228540 0
[pid=30400/tid=30408] ppid=30398 vsize=4922540 CPUtime=759.85 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 498906 0 0 0 75806 179 0 0 25 0 5 0 516573621 5040680960 1172303 33554432000 4194304 4338100 140733399285328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=4922540 CPUtime=760.35 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 93114 0 0 0 74181 1854 0 0 25 0 5 0 516573621 5040680960 1172303 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=4922540 CPUtime=761.6 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 465902 0 0 0 76056 104 0 0 25 0 5 0 516573621 5040680960 1172303 33554432000 4194304 4338100 140733399285328 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=4922540 CPUtime=760.61 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 95959 0 0 0 74219 1842 0 0 25 0 5 0 516573621 5040680960 1172303 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3042.96
Current children cumulated vsize (KiB) 4922540

[startup+822.301 s]
/proc/loadavg: 8.15 8.10 7.60 9/186 30439
/proc/meminfo: memFree=6101872/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=5269880 CPUtime=3282.57 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1260897 0 0 0 324005 4252 0 0 25 0 5 0 516573567 5396357120 1258540 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1317470 1258540 176 36 0 1315375 0
[pid=30400/tid=30408] ppid=30398 vsize=5269880 CPUtime=819.81 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 526741 0 0 0 81794 187 0 0 25 0 5 0 516573621 5396357120 1258540 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=5269880 CPUtime=820.34 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 102335 0 0 0 80053 1981 0 0 25 0 5 0 516573621 5396357120 1258540 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=5269880 CPUtime=821.59 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 500857 0 0 0 82047 112 0 0 25 0 5 0 516573621 5396357120 1258540 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254112 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=5269880 CPUtime=820.26 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 110185 0 0 0 80057 1969 0 0 25 0 5 0 516573621 5396357120 1258540 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3282.57
Current children cumulated vsize (KiB) 5269880

[startup+882.301 s]
/proc/loadavg: 8.12 8.10 7.63 9/186 30441
/proc/meminfo: memFree=5778700/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=5511132 CPUtime=3522.2 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1320992 0 0 0 347714 4506 0 0 25 0 5 0 516573567 5643399168 1318635 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1377783 1318635 176 36 0 1375688 0
[pid=30400/tid=30408] ppid=30398 vsize=5511132 CPUtime=879.79 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 564218 0 0 0 87778 201 0 0 25 0 5 0 516573621 5643399168 1318635 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=5511132 CPUtime=880.33 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 105162 0 0 0 85934 2099 0 0 25 0 5 0 516573621 5643399168 1318635 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=5511132 CPUtime=881.58 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 518024 0 0 0 88042 116 0 0 25 0 5 0 516573621 5643399168 1318635 33554432000 4194304 4338100 140733399285328 18446744073709551615 4287708 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=5511132 CPUtime=879.92 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 112809 0 0 0 85906 2086 0 0 25 0 5 0 516573621 5643399168 1318635 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3522.2
Current children cumulated vsize (KiB) 5511132

[startup+942.307 s]
/proc/loadavg: 8.11 8.09 7.65 9/186 30443
/proc/meminfo: memFree=5338744/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=5811552 CPUtime=3761.83 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1396671 0 0 0 371431 4752 0 0 25 0 5 0 516573567 5951029248 1394314 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1452888 1394314 176 36 0 1450793 0
[pid=30400/tid=30408] ppid=30398 vsize=5811552 CPUtime=939.76 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 604300 0 0 0 93760 216 0 0 25 0 5 0 516573621 5951029248 1394314 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=5811552 CPUtime=940.02 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 112631 0 0 0 91793 2209 0 0 25 0 5 0 516573621 5951029248 1394314 33554432000 4194304 4338100 140733399285328 18446744073709551615 4230463 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=5811552 CPUtime=941.59 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 537177 0 0 0 94038 121 0 0 25 0 5 0 516573621 5951029248 1394314 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=5811552 CPUtime=939.9 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 121784 0 0 0 91787 2203 0 0 25 0 5 0 516573621 5951029248 1394314 33554432000 4194304 4338100 140733399285328 18446744073709551615 4256695 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3761.83
Current children cumulated vsize (KiB) 5811552

[startup+1002.3 s]
/proc/loadavg: 8.09 8.09 7.68 9/186 30444
/proc/meminfo: memFree=5243988/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=5858736 CPUtime=4001.46 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1413038 0 0 0 395183 4963 0 0 25 0 5 0 516573567 5999345664 1410681 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1464684 1410681 176 36 0 1462589 0
[pid=30400/tid=30408] ppid=30398 vsize=5858736 CPUtime=999.75 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 615385 0 0 0 99753 222 0 0 25 0 5 0 516573621 5999345664 1410681 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=5858736 CPUtime=999.69 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 112655 0 0 0 97659 2310 0 0 25 0 5 0 516573621 5999345664 1410681 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=5858736 CPUtime=1001.57 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 542099 0 0 0 100035 122 0 0 25 0 5 0 516573621 5999345664 1410681 33554432000 4194304 4338100 140733399285328 18446744073709551615 274473363170 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=5858736 CPUtime=999.89 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 122120 0 0 0 97684 2305 0 0 25 0 5 0 516573621 5999345664 1410681 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4001.46
Current children cumulated vsize (KiB) 5858736

[startup+1062.3 s]
/proc/loadavg: 8.20 8.12 7.71 9/186 30446
/proc/meminfo: memFree=4931128/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=6024784 CPUtime=4241.05 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1454636 0 0 0 418922 5183 0 0 25 0 5 0 516573567 6169378816 1452279 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1506196 1452279 176 36 0 1504101 0
[pid=30400/tid=30408] ppid=30398 vsize=6024784 CPUtime=1059.73 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 647623 0 0 0 105739 234 0 0 25 0 5 0 516573621 6169378816 1452279 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=6024784 CPUtime=1059.34 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 112678 0 0 0 103521 2413 0 0 25 0 5 0 516573621 6169378816 1452279 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=6024784 CPUtime=1061.55 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 551417 0 0 0 106030 125 0 0 25 0 5 0 516573621 6169378816 1452279 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=6024784 CPUtime=1059.87 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 122139 0 0 0 103580 2407 0 0 25 0 5 0 516573621 6169378816 1452279 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4241.05
Current children cumulated vsize (KiB) 6024784

[startup+1122.3 s]
/proc/loadavg: 8.17 8.12 7.74 9/186 30447
/proc/meminfo: memFree=4558108/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=6268900 CPUtime=4480.62 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1516002 0 0 0 442674 5388 0 0 25 0 5 0 516573567 6419353600 1513645 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1567225 1513646 176 36 0 1565130 0
[pid=30400/tid=30408] ppid=30398 vsize=6268900 CPUtime=1119.69 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 673383 0 0 0 111725 244 0 0 25 0 5 0 516573621 6419353600 1513646 33554432000 4194304 4338100 140733399285328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=6268900 CPUtime=1119 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 112697 0 0 0 109395 2505 0 0 25 0 5 0 516573621 6419353600 1513646 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=6268900 CPUtime=1121.52 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 586990 0 0 0 112017 135 0 0 25 0 5 0 516573621 6419353600 1513646 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=6268900 CPUtime=1119.85 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 122154 0 0 0 109484 2501 0 0 25 0 5 0 516573621 6419353600 1513646 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4480.62
Current children cumulated vsize (KiB) 6268900

[startup+1182.3 s]
/proc/loadavg: 8.10 8.11 7.75 9/186 30449
/proc/meminfo: memFree=4327960/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=6397280 CPUtime=4720.27 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) S 30398 30400 30368 0 -1 4202496 1549516 0 0 0 466434 5593 0 0 25 0 5 0 516573567 6550814720 1547159 33554432000 4194304 4338100 140733399285328 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 1599320 1547159 176 36 0 1597225 0
[pid=30400/tid=30408] ppid=30398 vsize=6397280 CPUtime=1179.68 cores=1,3,5,7
/proc/30400/task/30408/stat : 30408 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 691573 0 0 0 117716 252 0 0 25 0 5 0 516573621 6550814720 1547159 33554432000 4194304 4338100 140733399285328 18446744073709551615 4288429 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30400/tid=30409] ppid=30398 vsize=6397280 CPUtime=1178.65 cores=1,3,5,7
/proc/30400/task/30409/stat : 30409 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 114506 0 0 0 115264 2601 0 0 25 0 5 0 516573621 6550814720 1547159 33554432000 4194304 4338100 140733399285328 18446744073709551615 274473362828 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30400/tid=30410] ppid=30398 vsize=6397280 CPUtime=1181.52 cores=1,3,5,7
/proc/30400/task/30410/stat : 30410 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 600488 0 0 0 118014 138 0 0 25 0 5 0 516573621 6550814720 1547159 33554432000 4194304 4338100 140733399285328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=6397280 CPUtime=1179.85 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4202560 122170 0 0 0 115387 2598 0 0 25 0 5 0 516573621 6550814720 1547159 33554432000 4194304 4338100 140733399285328 18446744073709551615 274473362900 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4720.27
Current children cumulated vsize (KiB) 6397280



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.3 s]
/proc/loadavg: 8.15 8.12 7.76 6/183 30450
/proc/meminfo: memFree=4643028/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=0 CPUtime=4800.08 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) Z 30398 30400 30368 0 -1 4203548 1562960 0 0 0 474346 5662 0 0 18 0 2 0 516573567 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 0 0 0 0 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=0 CPUtime=1199.85 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) R 30398 30400 30368 0 -1 4203588 122173 0 0 0 117355 2630 0 0 25 0 2 0 516573621 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.5 s]
/proc/loadavg: 8.15 8.12 7.76 6/183 30450
/proc/meminfo: memFree=4643028/32950928 swapFree=67111528/67111528
[pid=30400] ppid=30398 vsize=0 CPUtime=4800.27 cores=1,3,5,7
/proc/30400/stat : 30400 (ttsth-5-4) Z 30398 30400 30368 0 -1 4203548 1562960 0 0 0 474346 5681 0 0 18 0 2 0 516573567 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30400/statm: 0 0 0 0 0 0 0
[pid=30400/tid=30411] ppid=30398 vsize=0 CPUtime=1200.04 cores=1,3,5,7
/proc/30400/task/30411/stat : 30411 (ttsth-5-4) S 30398 30400 30368 0 -1 4203588 122173 0 0 0 117355 2649 0 0 25 0 2 0 516573621 0 0 33554432000 0 0 0 0 0 0 2147483391 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.27
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.5
CPU time (s): 4800.29
CPU user time (s): 4743.47
CPU system time (s): 56.8194
CPU usage (%): 399.192
Max. virtual memory (cumulated for all children) (KiB): 6450588

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4743.47
system time used= 56.8194
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1562960
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3768
involuntary context switches= 17452

runsolver used 1.9617 second user time and 4.6063 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 06:02:19
IDJOB=3258218
IDBENCH=71315
IDSOLVER=1654
FILE ID=node125/3258218-1304049739
RUNJOBID= node125-1304049739-30387
PBS_JOBID= 13170412
Free space on /tmp= 72760 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v1375c111739g6.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258218-1304049739/watcher-3258218-1304049739 -o /tmp/evaluation-result-3258218-1304049739/solver-3258218-1304049739 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258218-1304049739.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c7c04151ac14a13eb4a03259d055e119
RANDOM SEED=1721483152

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15158300 kB
Buffers:       2218104 kB
Cached:       14973176 kB
SwapCached:          0 kB
Active:        9043688 kB
Inactive:      8215356 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15158300 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            2860 kB
Writeback:           0 kB
AnonPages:       66652 kB
Mapped:          14720 kB
Slab:           468792 kB
PageTables:       5008 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186204 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72760 MiB
End job on node125 at 2011-04-29 06:22:23