Trace number 3258966

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.29 1204.4

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v760c43649g9.cnf
MD5SUMaa71b1c0a084106bad56c0f4d6c26bee
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.595909
Satisfiable
(Un)Satisfiability was proved
Number of variables760
Number of clauses43649
Sum of the clauses size87978
Maximum clause length19
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243609
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258966-1304053738.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c a2
0.00/0.05	c a1
0.09/0.10	c Independent sub-problems = 1
0.09/0.10	c Sub-problem with 760 variables and 43649 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 1
0.09/0.10	c Ordering finished 0.090 seconds
0.09/0.10	c mapped 0.090 seconds
0.09/0.10	c permuted 0.090 seconds
0.09/0.10	c about to sort
0.09/0.10	c sorted
0.09/0.19	c clauses now 43649
0.09/0.19	c init finished 0.110 seconds
0.09/0.19	c tree built 0.180 seconds
0.09/0.19	c 
0.09/0.19	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.19	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.19	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.19	c ................................ | ..........                  1 |                                  0.2         5.6             
0.09/0.19	c ................................ | ..........                  2 |                                  0.2        11.1             
0.09/0.19	c ................................ | ..........                  3 |                                  0.2        16.7             
0.09/0.19	c ................................ | ..........                  4 |                                  0.2        22.2             
0.09/0.19	c ................................ | ..........                  5 |                                  0.2        27.8             
0.09/0.19	c ................................ | ..........               1000 | 8            173                 0.2      5263.2         7819
96.16/24.32	c ................................ | ..........            3218000 | 69661        691211             96.0     33513.9     20381009
191.66/48.39	c ................................ | ..........            5697000 | 123252       1195287           192.0     29670.3     33246419
288.04/72.50	c ................................ | ..........            7697000 | 170134       1591078           288.0     26724.8     40002501
384.03/96.61	c ................................ | ..........            9507000 | 204606       1956845           384.0     24755.2     44641051
480.02/120.79	c ................................ | *.........           11272000 | 239312       2315318           480.0     23481.9     51015268
576.06/144.85	c ................................ | *.........           12932000 | 271493       2650069           576.0     22450.6     57414559
672.13/168.96	c ................................ | *.........           14443000 | 307965       2943817           672.0     21491.0     67933344
767.76/192.98	c ................................ | *.........           15918000 | 339728       3230817           768.0     20725.8     77452533
863.87/217.04	c ................................ | *.........           17301000 | 368417       3497395           864.0     20023.6     84332549
959.95/241.12	c ................................ | **........           18543000 | 398296       3726044           960.0     19314.6     88414280
1056.00/265.21	c ................................ | **........           19765000 | 424519       3965761          1056.0     18716.1     90530763
1151.65/289.29	c ................................ | **........           20886000 | 452445       4192988          1152.0     18129.7     92964745
1247.66/313.38	c ................................ | **........           22009000 | 477092       4463802          1248.0     17635.4     95651211
1343.74/337.47	c ................................ | **........           23109000 | 506571       4727536          1344.0     17193.7    100065532
1439.78/361.55	c ................................ | ***.......           24115000 | 536405       4964646          1440.0     16746.5    101970256
1535.86/385.63	c ................................ | ***.......           25162000 | 565092       5202335          1536.0     16381.5    106660833
1631.97/409.72	c ................................ | ***.......           26197000 | 593336       5432375          1632.1     16051.5    112973742
1727.81/433.77	c ................................ | ***.......           27153000 | 616041       5652127          1728.1     15712.9    114427919
1823.90/457.85	c ................................ | ***.......           28124000 | 639111       5864048          1824.1     15418.3    116813166
1920.04/481.91	c ................................ | ****......           29041000 | 661307       6057655          1920.1     15124.9    118171440
2015.83/505.95	c ................................ | ****......           30032000 | 687489       6277282          2016.0     14896.5    120961013
2111.94/530.01	c ................................ | ****......           31348000 | 718277       6501915          2112.0     14842.7    125544312
2207.66/554.09	c ................................ | ****......           32616000 | 749889       6713535          2208.0     14771.5    130148881
2303.77/578.16	c ................................ | ****......           33829000 | 782733       6919300          2304.0     14682.5    133614823
2399.90/602.23	c ................................ | *****.....           35034000 | 808099       7117134          2400.0     14597.3    140012802
2495.96/626.32	c ................................ | *****.....           36124000 | 833988       7321110          2496.0     14472.5    142065355
2591.66/650.38	c ................................ | *****.....           37212000 | 859711       7513188          2592.0     14356.5    145047855
2687.72/674.48	c ................................ | *****.....           38291000 | 883169       7699289          2688.0     14245.0    147148361
2783.84/698.55	c ................................ | *****.....           39318000 | 907947       7887005          2784.1     14122.6    148965774
2879.89/722.65	c ................................ | ******....           40384000 | 930426       8064923          2880.1     14021.7    150329700
2975.98/746.70	c ................................ | ******....           41413000 | 954956       8242975          2976.0     13915.7    152100951
3072.03/770.81	c ................................ | ******....           42381000 | 978754       8425601          3072.1     13795.6    156030788
3167.75/794.88	c ................................ | ******....           43293000 | 1002973      8598764          3168.1     13665.4    158418533
3263.86/818.94	c ................................ | ******....           44284000 | 1026559      8786809          3264.0     13567.2    162709410
3359.90/843.03	c ................................ | *******...           45136000 | 1048211      8959713          3360.0     13433.3    164426634
3456.00/867.12	c ................................ | *******...           46043000 | 1070313      9129549          3456.1     13322.4    165353841
3551.67/891.20	c ................................ | *******...           46879000 | 1092939      9296100          3552.1     13197.6    167737992
3647.77/915.26	c ................................ | *******...           47787000 | 1117196      9460511          3648.0     13099.5    168914166
3743.92/939.32	c ................................ | *******...           48679000 | 1138618      9618395          3744.0     13001.9    172051673
3840.02/963.40	c ................................ | ********..           49617000 | 1160083      9783414          3840.0     12921.0    172051673
3935.73/987.47	c ................................ | ********..           50571000 | 1181448      9946346          3936.0     12848.3    172721714
4031.83/1011.56	c ................................ | ********..           51521000 | 1202678      10112787         4032.1     12777.8    174151181
4127.95/1035.64	c ................................ | ********..           52421000 | 1228080      10272500         4128.1     12698.6    175659424
4224.04/1059.71	c ................................ | ********..           53277000 | 1251602      10417617         4224.1     12612.6    176244721
4319.76/1083.77	c ................................ | *********.           54125000 | 1275729      10562577         4320.0     12528.8    176288889
4415.88/1107.83	c ................................ | *********.           55014000 | 1299863      10709804         4416.0     12457.8    179437002
4512.02/1131.91	c ................................ | *********.           55889000 | 1325782      10853138         4512.1     12386.6    181771798
4607.75/1155.96	c ................................ | *********.           56746000 | 1346070      11002048         4608.0     12314.6    184792795
4703.79/1180.05	c ................................ | *********.           57592000 | 1368920      11208412         4704.0     12243.2    185410508
4799.84/1204.14	c CPU limit (4800) reached
4799.84/1204.14	c Time = 4800.03 seconds
4799.84/1204.14	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258966-1304053738/watcher-3258966-1304053738 -o /tmp/evaluation-result-3258966-1304053738/solver-3258966-1304053738 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258966-1304053738.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.50 7.99 7.99 2/175 2428
/proc/meminfo: memFree=18301764/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=9112 CPUtime=0 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) R 2426 2428 1994 0 -1 4202496 354 0 0 0 0 0 0 0 25 0 1 0 516974292 9330688 296 33554432000 4194304 4338100 140734080556480 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2428/statm: 2278 297 147 36 0 183 0

[startup+0.0859719 s]
/proc/loadavg: 7.50 7.99 7.99 2/175 2428
/proc/meminfo: memFree=18301764/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=18496 CPUtime=0.08 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) R 2426 2428 1994 0 -1 4202496 3461 0 0 0 8 0 0 0 25 0 1 0 516974292 18939904 2654 33554432000 4194304 4338100 140734080556480 18446744073709551615 230808820761 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2428/statm: 4624 2654 156 36 0 2529 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 18496

[startup+0.100994 s]
/proc/loadavg: 7.50 7.99 7.99 2/175 2428
/proc/meminfo: memFree=18301764/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=18496 CPUtime=0.09 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) R 2426 2428 1994 0 -1 4202496 3471 0 0 0 9 0 0 0 25 0 1 0 516974292 18939904 2664 33554432000 4194304 4338100 140734080556480 18446744073709551615 4211136 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2428/statm: 4624 2664 161 36 0 2529 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18496

[startup+0.315745 s]
/proc/loadavg: 7.50 7.99 7.99 2/175 2428
/proc/meminfo: memFree=18301764/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=231664 CPUtime=0.94 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 12033 0 0 0 93 1 0 0 25 0 5 0 516974292 237223936 11053 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 57916 11122 176 36 0 55821 0
Current children cumulated CPU time (s) 0.94
Current children cumulated vsize (KiB) 231664

[startup+0.72852 s]
/proc/loadavg: 7.50 7.99 7.99 2/175 2428
/proc/meminfo: memFree=18301764/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=244572 CPUtime=2.47 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 16703 0 0 0 245 2 0 0 25 0 5 0 516974292 250441728 15723 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 63489 17788 176 36 0 61394 0
Current children cumulated CPU time (s) 2.47
Current children cumulated vsize (KiB) 244572

[startup+1.62237 s]
/proc/loadavg: 7.50 7.99 7.99 2/175 2428
/proc/meminfo: memFree=18301764/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=292164 CPUtime=6.46 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 27203 0 0 0 641 5 0 0 25 0 5 0 516974292 299175936 26223 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 73041 26895 176 36 0 70946 0
Current children cumulated CPU time (s) 6.46
Current children cumulated vsize (KiB) 292164

[startup+3.10036 s]
/proc/loadavg: 7.50 7.99 7.99 9/186 2464
/proc/meminfo: memFree=17998840/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=346000 CPUtime=11.76 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 40964 0 0 0 1167 9 0 0 25 0 5 0 516974292 354304000 39984 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 86500 39985 176 36 0 84405 0
[pid=2428/tid=2430] ppid=2426 vsize=346000 CPUtime=2.9 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 10569 0 0 0 289 1 0 0 24 0 5 0 516974311 354304000 39985 33554432000 4194304 4338100 140734080556480 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=346000 CPUtime=2.84 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 8481 0 0 0 282 2 0 0 18 0 5 0 516974311 354304000 39985 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=346000 CPUtime=2.9 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 10067 0 0 0 289 1 0 0 24 0 5 0 516974311 354304000 39985 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=346000 CPUtime=2.9 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 2875 0 0 0 287 3 0 0 24 0 5 0 516974311 354304000 39985 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.76
Current children cumulated vsize (KiB) 346000

[startup+6.30077 s]
/proc/loadavg: 7.54 7.99 7.99 9/186 2464
/proc/meminfo: memFree=17790452/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=462556 CPUtime=24.5 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 69859 0 0 0 2429 21 0 0 25 0 5 0 516974292 473657344 68879 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 115639 68879 176 36 0 113544 0
[pid=2428/tid=2430] ppid=2426 vsize=462556 CPUtime=6.1 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 20155 0 0 0 606 4 0 0 25 0 5 0 516974311 473657344 68880 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=462556 CPUtime=5.98 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 13042 0 0 0 594 4 0 0 18 0 5 0 516974311 473657344 68882 33554432000 4194304 4338100 140734080556480 18446744073709551615 4256358 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=462556 CPUtime=6.1 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 19494 0 0 0 608 2 0 0 25 0 5 0 516974311 473657344 68883 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=462556 CPUtime=6.1 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 8198 0 0 0 601 9 0 0 25 0 5 0 516974311 473657344 68884 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.5
Current children cumulated vsize (KiB) 462556

[startup+12.7006 s]
/proc/loadavg: 7.58 7.99 7.99 9/186 2464
/proc/meminfo: memFree=17443596/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=652336 CPUtime=49.98 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 117353 0 0 0 4958 40 0 0 25 0 5 0 516974292 667992064 116373 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 163084 116373 176 36 0 160989 0
[pid=2428/tid=2430] ppid=2426 vsize=652336 CPUtime=12.5 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 36942 0 0 0 1242 8 0 0 25 0 5 0 516974311 667992064 116373 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=652336 CPUtime=12.27 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 19757 0 0 0 1219 8 0 0 18 0 5 0 516974311 667992064 116373 33554432000 4194304 4338100 140734080556480 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=652336 CPUtime=12.5 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 34944 0 0 0 1246 4 0 0 25 0 5 0 516974311 667992064 116373 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=652336 CPUtime=12.5 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 16737 0 0 0 1232 18 0 0 25 0 5 0 516974311 667992064 116373 33554432000 4194304 4338100 140734080556480 18446744073709551615 4286342 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.98
Current children cumulated vsize (KiB) 652336

[startup+25.5003 s]
/proc/loadavg: 7.67 7.99 7.99 9/186 2465
/proc/meminfo: memFree=16945260/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=901856 CPUtime=100.94 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 179870 0 0 0 10017 77 0 0 25 0 5 0 516974292 923500544 178602 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 225464 178602 176 36 0 223369 0
[pid=2428/tid=2430] ppid=2426 vsize=901856 CPUtime=25.3 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 60000 0 0 0 2514 16 0 0 25 0 5 0 516974311 923500544 178615 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=901856 CPUtime=24.83 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 27939 0 0 0 2469 14 0 0 18 0 5 0 516974311 923500544 178615 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=901856 CPUtime=25.3 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 57645 0 0 0 2522 8 0 0 25 0 5 0 516974311 923500544 178615 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=901856 CPUtime=25.3 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 25326 0 0 0 2493 37 0 0 22 0 5 0 516974311 923500544 178615 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 100.94
Current children cumulated vsize (KiB) 901856

[startup+51.1006 s]
/proc/loadavg: 7.84 8.01 8.00 9/186 2465
/proc/meminfo: memFree=16176344/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=1316996 CPUtime=202.81 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 285610 0 0 0 20140 141 0 0 25 0 5 0 516974292 1348603904 284342 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 329249 284342 176 36 0 327154 0
[pid=2428/tid=2430] ppid=2426 vsize=1316996 CPUtime=50.88 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 106087 0 0 0 5060 28 0 0 25 0 5 0 516974311 1348603904 284342 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=1316996 CPUtime=49.93 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 36351 0 0 0 4972 21 0 0 18 0 5 0 516974311 1348603904 284343 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=1316996 CPUtime=50.89 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 95758 0 0 0 5073 16 0 0 25 0 5 0 516974311 1348603904 284343 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=1316996 CPUtime=50.88 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 38441 0 0 0 5015 73 0 0 25 0 5 0 516974311 1348603904 284343 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.81
Current children cumulated vsize (KiB) 1316996

[startup+102.301 s]
/proc/loadavg: 7.99 8.02 8.00 8/186 2467
/proc/meminfo: memFree=15201432/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=1775004 CPUtime=406.73 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 380364 0 0 0 40436 237 0 0 25 0 5 0 516974292 1817604096 379096 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 443751 379096 176 36 0 441656 0
[pid=2428/tid=2430] ppid=2426 vsize=1775004 CPUtime=102.07 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 150344 0 0 0 10165 42 0 0 25 0 5 0 516974311 1817604096 379096 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=1775004 CPUtime=100.3 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 41578 0 0 0 9998 32 0 0 19 0 5 0 516974311 1817604096 379096 33554432000 4194304 4338100 140734080556480 18446744073709551615 4253906 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=1775004 CPUtime=102.08 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 132149 0 0 0 10184 24 0 0 25 0 5 0 516974311 1817604096 379096 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=1775004 CPUtime=102.07 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 47320 0 0 0 10069 138 0 0 18 0 5 0 516974311 1817604096 379096 33554432000 4194304 4338100 140734080556480 18446744073709551615 230808560656 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.73
Current children cumulated vsize (KiB) 1775004

[startup+162.3 s]
/proc/loadavg: 8.24 8.08 8.02 9/186 2468
/proc/meminfo: memFree=14163788/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=2388000 CPUtime=645.81 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 535633 0 0 0 64239 342 0 0 25 0 5 0 516974292 2445312000 534365 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 597000 534365 176 36 0 594905 0
[pid=2428/tid=2430] ppid=2426 vsize=2388000 CPUtime=162.05 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 222463 0 0 0 16145 60 0 0 25 0 5 0 516974311 2445312000 534365 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254218 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=2388000 CPUtime=160.29 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 51375 0 0 0 15933 96 0 0 25 0 5 0 516974311 2445312000 534365 33554432000 4194304 4338100 140734080556480 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=2388000 CPUtime=162.07 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 189802 0 0 0 16171 36 0 0 25 0 5 0 516974311 2445312000 534365 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254041 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=2388000 CPUtime=161.19 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 63020 0 0 0 15970 149 0 0 18 0 5 0 516974311 2445312000 534365 33554432000 4194304 4338100 140734080556480 18446744073709551615 4256104 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 645.81
Current children cumulated vsize (KiB) 2388000

[startup+222.3 s]
/proc/loadavg: 8.33 8.14 8.04 9/186 2470
/proc/meminfo: memFree=13010440/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=3048112 CPUtime=885 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 699110 0 0 0 88050 450 0 0 25 0 5 0 516974292 3121266688 697842 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 762028 697842 176 36 0 759933 0
[pid=2428/tid=2430] ppid=2426 vsize=3048112 CPUtime=221.87 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 291377 0 0 0 22106 81 0 0 25 0 5 0 516974311 3121266688 697842 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 113853 0 0 0 69394 475 0 0 25 0 5 0 516974311 5271027712 1223827 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2798.99
Current children cumulated vsize (KiB) 5147488

[startup+762.3 s]
/proc/loadavg: 8.13 8.10 8.05 9/186 2484
/proc/meminfo: memFree=8356432/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=5333228 CPUtime=3038.14 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1271425 0 0 0 302495 1319 0 0 25 0 5 0 516974292 5461225472 1270157 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1333307 1270157 176 36 0 1331212 0
[pid=2428/tid=2430] ppid=2426 vsize=5333228 CPUtime=761.18 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 575776 0 0 0 75942 176 0 0 25 0 5 0 516974311 5461225472 1270157 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=5333228 CPUtime=758.18 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 85396 0 0 0 75301 517 0 0 18 0 5 0 516974311 5461225472 1270157 33554432000 4194304 4338100 140734080556480 18446744073709551615 4241466 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=5333228 CPUtime=759.93 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 487244 0 0 0 75891 102 0 0 25 0 5 0 516974311 5461225472 1270157 33554432000 4194304 4338100 140734080556480 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=5333228 CPUtime=758.63 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 114036 0 0 0 75341 522 0 0 25 0 5 0 516974311 5461225472 1270157 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3038.14
Current children cumulated vsize (KiB) 5333228

[startup+822.3 s]
/proc/loadavg: 8.08 8.10 8.05 9/186 2486
/proc/meminfo: memFree=7844300/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=5575404 CPUtime=3277.4 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1332512 0 0 0 326357 1383 0 0 25 0 5 0 516974292 5709213696 1331244 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1393851 1331244 176 36 0 1391756 0
[pid=2428/tid=2430] ppid=2426 vsize=5575404 CPUtime=821.15 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 607401 0 0 0 81930 185 0 0 25 0 5 0 516974311 5709213696 1331244 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=5575404 CPUtime=817.81 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 85608 0 0 0 81259 522 0 0 18 0 5 0 516974311 5709213696 1331244 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=5575404 CPUtime=819.59 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 516281 0 0 0 81850 109 0 0 25 0 5 0 516974311 5709213696 1331244 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=5575404 CPUtime=818.62 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 114249 0 0 0 81297 565 0 0 25 0 5 0 516974311 5709213696 1331244 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3277.4
Current children cumulated vsize (KiB) 5575404

[startup+882.3 s]
/proc/loadavg: 8.14 8.11 8.06 9/186 2488
/proc/meminfo: memFree=7426776/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=5698584 CPUtime=3516.58 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1364463 0 0 0 350217 1441 0 0 25 0 5 0 516974292 5835350016 1363195 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1424646 1363195 176 36 0 1422551 0
[pid=2428/tid=2430] ppid=2426 vsize=5698584 CPUtime=881.1 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 627497 0 0 0 87917 193 0 0 25 0 5 0 516974311 5835350016 1363195 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=5698584 CPUtime=877.42 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 85789 0 0 0 87217 525 0 0 18 0 5 0 516974311 5835350016 1363195 33554432000 4194304 4338100 140734080556480 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=5698584 CPUtime=879.25 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 527788 0 0 0 87813 112 0 0 25 0 5 0 516974311 5835350016 1363195 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=5698584 CPUtime=878.59 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 114416 0 0 0 87250 609 0 0 25 0 5 0 516974311 5835350016 1363195 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3516.58
Current children cumulated vsize (KiB) 5698584

[startup+942.3 s]
/proc/loadavg: 8.12 8.10 8.06 9/186 2489
/proc/meminfo: memFree=7045952/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=5852780 CPUtime=3755.88 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1404151 0 0 0 374090 1498 0 0 25 0 5 0 516974292 5993246720 1402883 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1463195 1402883 176 36 0 1461100 0
[pid=2428/tid=2430] ppid=2426 vsize=5852780 CPUtime=941.02 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 645690 0 0 0 93902 200 0 0 25 0 5 0 516974311 5993246720 1402883 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=5852780 CPUtime=937.1 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 86424 0 0 0 93180 530 0 0 18 0 5 0 516974311 5993246720 1402883 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=5852780 CPUtime=938.96 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 547762 0 0 0 93779 117 0 0 25 0 5 0 516974311 5993246720 1402883 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=5852780 CPUtime=938.58 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 115302 0 0 0 93208 650 0 0 25 0 5 0 516974311 5993246720 1402883 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3755.88
Current children cumulated vsize (KiB) 5852780

[startup+1002.3 s]
/proc/loadavg: 8.08 8.08 8.05 9/186 2491
/proc/meminfo: memFree=6719560/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=5979680 CPUtime=3995.13 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1421269 0 0 0 397961 1552 0 0 25 0 5 0 516974292 6123192320 1420001 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1494920 1420001 176 36 0 1492825 0
[pid=2428/tid=2430] ppid=2426 vsize=5979680 CPUtime=1000.75 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 657066 0 0 0 99870 205 0 0 25 0 5 0 516974311 6123192320 1420003 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=5979680 CPUtime=996.77 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 89526 0 0 0 99142 535 0 0 18 0 5 0 516974311 6123192320 1420003 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=5979680 CPUtime=998.85 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 548135 0 0 0 99768 117 0 0 25 0 5 0 516974311 6123192320 1420003 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=5979680 CPUtime=998.53 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 117571 0 0 0 99161 692 0 0 25 0 5 0 516974311 6123192320 1420003 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3995.13
Current children cumulated vsize (KiB) 5979680

[startup+1062.3 s]
/proc/loadavg: 8.08 8.08 8.05 9/186 2492
/proc/meminfo: memFree=6336616/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=6068188 CPUtime=4234.41 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1449185 0 0 0 421832 1609 0 0 25 0 5 0 516974292 6213824512 1447917 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1517047 1447917 176 36 0 1514952 0
[pid=2428/tid=2430] ppid=2426 vsize=6068188 CPUtime=1060.73 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 678424 0 0 0 105857 216 0 0 25 0 5 0 516974311 6213824512 1447917 33554432000 4194304 4338100 140734080556480 18446744073709551615 230808561082 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=6068188 CPUtime=1056.43 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 92946 0 0 0 105104 539 0 0 18 0 5 0 516974311 6213824512 1447917 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=6068188 CPUtime=1058.53 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 549081 0 0 0 105735 118 0 0 25 0 5 0 516974311 6213824512 1447917 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=6068188 CPUtime=1058.51 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 119761 0 0 0 105117 734 0 0 25 0 5 0 516974311 6213824512 1447917 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4234.41
Current children cumulated vsize (KiB) 6068188

[startup+1122.31 s]
/proc/loadavg: 8.08 8.08 8.05 9/186 2494
/proc/meminfo: memFree=5915600/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=6197228 CPUtime=4473.74 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1483403 0 0 0 445710 1664 0 0 25 0 5 0 516974292 6345961472 1482135 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1549307 1482135 176 36 0 1547212 0
[pid=2428/tid=2430] ppid=2426 vsize=6197228 CPUtime=1120.69 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 694179 0 0 0 111847 222 0 0 25 0 5 0 516974311 6345961472 1482135 33554432000 4194304 4338100 140734080556480 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=6197228 CPUtime=1116.12 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 93488 0 0 0 111069 543 0 0 18 0 5 0 516974311 6345961472 1482135 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=6197228 CPUtime=1118.19 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 565875 0 0 0 111698 121 0 0 25 0 5 0 516974311 6345961472 1482135 33554432000 4194304 4338100 140734080556480 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=6197228 CPUtime=1118.51 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 120888 0 0 0 111076 775 0 0 25 0 5 0 516974311 6345961472 1482135 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4473.74
Current children cumulated vsize (KiB) 6197228

[startup+1182.3 s]
/proc/loadavg: 8.11 8.08 8.05 9/186 2496
/proc/meminfo: memFree=5686420/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=6354948 CPUtime=4712.96 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) S 2426 2428 1994 0 -1 4202496 1528273 0 0 0 469570 1726 0 0 25 0 5 0 516974292 6507466752 1527005 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 1588737 1527005 176 36 0 1586642 0
[pid=2428/tid=2430] ppid=2426 vsize=6354948 CPUtime=1180.66 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 716501 0 0 0 117834 232 0 0 25 0 5 0 516974311 6507466752 1527005 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2428/tid=2431] ppid=2426 vsize=6354948 CPUtime=1175.74 cores=0,2,4,6
/proc/2428/task/2431/stat : 2431 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 94797 0 0 0 117026 548 0 0 18 0 5 0 516974311 6507466752 1527005 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2428/tid=2432] ppid=2426 vsize=6354948 CPUtime=1177.86 cores=0,2,4,6
/proc/2428/task/2432/stat : 2432 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 583503 0 0 0 117660 126 0 0 25 0 5 0 516974311 6507466752 1527005 33554432000 4194304 4338100 140734080556480 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2428/tid=2433] ppid=2426 vsize=6354948 CPUtime=1178.49 cores=0,2,4,6
/proc/2428/task/2433/stat : 2433 (ttsth-5-4) R 2426 2428 1994 0 -1 4202560 124499 0 0 0 117030 819 0 0 25 0 5 0 516974311 6507466752 1527005 33554432000 4194304 4338100 140734080556480 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4712.96
Current children cumulated vsize (KiB) 6354948



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.2 s]
/proc/loadavg: 8.08 8.07 8.05 2/177 2497
/proc/meminfo: memFree=13223560/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=0 CPUtime=4800.09 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) Z 2426 2428 1994 0 -1 4203548 1543092 0 0 0 478251 1758 0 0 18 0 2 0 516974292 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 0 0 0 0 0 0 0
[pid=2428/tid=2430] ppid=2426 vsize=0 CPUtime=1202.54 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202564 721408 0 0 0 120015 239 0 0 25 0 2 0 516974311 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.31 s]
/proc/loadavg: 8.08 8.07 8.05 2/177 2497
/proc/meminfo: memFree=13223560/32951124 swapFree=67111528/67111528
[pid=2428] ppid=2426 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/2428/stat : 2428 (ttsth-5-4) Z 2426 2428 1994 0 -1 4203548 1543092 0 0 0 478251 1769 0 0 18 0 2 0 516974292 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2428/statm: 0 0 0 0 0 0 0
[pid=2428/tid=2430] ppid=2426 vsize=0 CPUtime=1202.65 cores=0,2,4,6
/proc/2428/task/2430/stat : 2430 (ttsth-5-4) R 2426 2428 1994 0 -1 4202564 721408 0 0 0 120015 250 0 0 25 0 2 0 516974311 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.4
CPU time (s): 4800.29
CPU user time (s): 4782.51
CPU system time (s): 17.7763
CPU usage (%): 398.564
Max. virtual memory (cumulated for all children) (KiB): 6411780

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4782.51
system time used= 17.7763
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1543092
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 13602
involuntary context switches= 22595

runsolver used 1.9637 second user time and 4.19036 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 07:08:58
IDJOB=3258966
IDBENCH=71444
IDSOLVER=1654
FILE ID=node104/3258966-1304053738
RUNJOBID= node104-1304052534-2326
PBS_JOBID= 13170409
Free space on /tmp= 71544 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v760c43649g9.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258966-1304053738/watcher-3258966-1304053738 -o /tmp/evaluation-result-3258966-1304053738/solver-3258966-1304053738 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258966-1304053738.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= aa71b1c0a084106bad56c0f4d6c26bee
RANDOM SEED=644976086

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18302044 kB
Buffers:       2195984 kB
Cached:       11801048 kB
SwapCached:          0 kB
Active:        2344836 kB
Inactive:     11718820 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18302044 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             816 kB
Writeback:           0 kB
AnonPages:       65692 kB
Mapped:          15188 kB
Slab:           522724 kB
PageTables:       4124 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   218768 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71548 MiB
End job on node104 at 2011-04-29 07:29:05