Trace number 3262002

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.51 1202.38

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-22_464.cnf
MD5SUMa722a199fd84ba0b319251daa47aa19c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables232
Number of clauses28991
Sum of the clauses size130855
Maximum clause length22
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 277
Number of clauses of size 326594
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52320

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262002-1304090519.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 232
0.00/0.03	c Clauses = 28991
0.00/0.03	c Literals = 130855
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 232
0.00/0.03	c Clauses = 28991
0.00/0.03	c Literals = 130855
0.00/0.03	c a2
0.09/0.19	c a1
0.39/0.47	c Independent sub-problems = 1
0.39/0.47	c Sub-problem with 232 variables and 28991 clauses
0.39/0.47	c using permutation method 4
0.39/0.47	c count = 1
0.39/0.47	c Ordering finished 0.460 seconds
0.39/0.47	c mapped 0.460 seconds
0.39/0.47	c permuted 0.460 seconds
0.39/0.47	c about to sort
0.39/0.47	c sorted
0.49/0.57	c clauses now 28991
0.49/0.57	c init finished 0.470 seconds
0.49/0.57	c tree built 0.570 seconds
0.49/0.57	c 
0.49/0.57	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.57	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.57	c ................................ | ..........                    |                                  0.6         0.0             
0.49/0.57	c ................................ | ..........                  1 |                                  0.6         1.8             
0.49/0.57	c ................................ | ..........                  2 |                                  0.6         3.5             
0.49/0.57	c ................................ | ..........                  3 |                                  0.6         5.3             
0.49/0.57	c ................................ | ..........                  4 |                                  0.6         7.0             
0.49/0.57	c ................................ | ..........                  5 |                                  0.6         8.8             
0.49/0.59	c ................................ | ..........               1000 | 37           341                 0.7      1538.5        26733
96.14/24.54	c ................................ | ..........             376000 | 32892        132754             96.3      3903.7     21332912
192.01/48.59	c ................................ | ..........             568000 | 50551        203585            192.4      2952.2     35482278
288.29/72.65	c ................................ | ..........             738000 | 68983        258485            288.5      2558.1     45910281
383.75/96.57	c ................................ | ..........             862000 | 83238        301660            384.0      2244.6     54812623
480.02/120.60	c ................................ | *.........             984000 | 96682        350843            480.0      2049.8     64834313
575.89/144.64	c ................................ | *.........            1086000 | 108667       389808            576.0      1885.3     72473662
671.76/168.67	c ................................ | *.........            1196000 | 119332       422534            672.0      1779.6     79654109
768.03/192.76	c ................................ | *.........            1297000 | 128615       454903            768.3      1688.2     86940863
865.09/217.04	c ................................ | *.........            1381000 | 137181       488790            865.3      1596.0     92800649
960.19/240.86	c ................................ | **........            1455000 | 145640       513517            960.4      1514.9     98064683
1056.45/265.00	c ................................ | **........            1533000 | 154819       539397           1056.8      1450.5    103100484
1152.72/289.02	c ................................ | **........            1613000 | 164025       567127           1152.8      1399.2    107386168
1248.58/313.05	c ................................ | **........            1687000 | 172945       591231           1248.8      1350.9    112174000
1345.26/337.20	c ................................ | **........            1760000 | 179836       612386           1345.3      1308.3    117749258
1440.73/361.15	c ................................ | ***.......            1830000 | 187847       634009           1440.9      1270.0    123043036
1535.82/384.95	c ................................ | ***.......            1896000 | 195455       657219           1536.0      1234.4    127410830
1632.89/409.23	c ................................ | ***.......            1974000 | 202405       685116           1633.0      1208.8    131198018
1728.34/433.18	c ................................ | ***.......            2045000 | 211267       709300           1728.7      1183.0    135471201
1825.02/457.37	c ................................ | ***.......            2110000 | 219539       732211           1825.3      1156.0    139132107
1920.08/481.11	c ................................ | ****......            2161000 | 227310       753553           1920.2      1125.4    142539245
2016.75/505.36	c ................................ | ****......            2219000 | 234563       774318           2017.0      1100.2    144363185
2113.02/529.40	c ................................ | ****......            2276000 | 241051       795925           2113.0      1077.1    145160769
2209.29/553.56	c ................................ | ****......            2335000 | 247347       818868           2209.5      1056.8    147599093
2303.97/577.25	c ................................ | ****......            2396000 | 253732       837470           2304.2      1039.8    150649433
2401.04/601.60	c ................................ | *****.....            2456000 | 259435       859588           2401.4      1022.7    154133050
2496.91/625.59	c ................................ | *****.....            2518000 | 266433       878962           2497.3      1008.3    157896223
2593.18/649.64	c ................................ | *****.....            2580000 | 273584       897018           2593.3       994.9    161102186
2688.64/673.52	c ................................ | *****.....            2636000 | 278694       913848           2688.7       980.4    164116223
2784.92/697.69	c ................................ | *****.....            2696000 | 283925       932303           2785.3       967.9    167623152
2880.81/721.65	c ................................ | ******....            2755000 | 289481       952773           2881.0       956.3    170919946
2977.08/745.77	c ................................ | ******....            2809000 | 294733       969600           2977.3       943.5    173640848
3073.75/769.91	c ................................ | ******....            2859000 | 300736       985001           3073.8       930.1    174608918
3170.01/794.07	c ................................ | ******....            2908000 | 305674       1000245          3170.3       917.3    177544895
3263.88/817.54	c ................................ | ******....            2955000 | 311164       1017352          3264.0       905.3    180760414
3359.77/841.57	c ................................ | *******...            2999000 | 316324       1034049          3360.1       892.5    183924965
3456.04/865.66	c ................................ | *******...            3041000 | 322294       1047672          3456.3       879.9    186249091
3552.29/889.75	c ................................ | *******...            3088000 | 328158       1062482          3552.5       869.2    188622453
3648.57/913.86	c ................................ | *******...            3133000 | 334406       1079214          3648.8       858.6    190934589
3744.03/937.70	c ................................ | *******...            3170000 | 339399       1092910          3744.1       846.7    192191912
3841.10/962.04	c ................................ | ********..            3208000 | 344205       1106902          3841.3       835.1    193593529
3938.57/986.49	c ................................ | ********..            3245000 | 348807       1120293          3938.9       823.8    194659749
4034.05/1010.39	c ................................ | ********..            3283000 | 354006       1134705          4034.4       813.7    197221938
4129.11/1034.11	c ................................ | ********..            3322000 | 359495       1149106          4129.2       804.5    199870811
4225.78/1058.33	c ................................ | ********..            3355000 | 364202       1162107          4225.9       793.9    202099303
4320.05/1081.92	c ................................ | *********.            3390000 | 369252       1174992          4320.2       784.7    204554090
4417.51/1106.31	c ................................ | *********.            3428000 | 374093       1187651          4417.6       776.0    207041590
4513.38/1130.34	c ................................ | *********.            3464000 | 379607       1199544          4513.6       767.5    209125997
4608.85/1154.25	c ................................ | *********.            3503000 | 384942       1213129          4609.1       760.0    210226298
4705.53/1178.40	c ................................ | *********.            3544000 | 390559       1227316          4705.6       753.2    212410054

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262002-1304090519/watcher-3262002-1304090519 -o /tmp/evaluation-result-3262002-1304090519/solver-3262002-1304090519 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262002-1304090519.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.13 7.85 7.94 2/177 6933
/proc/meminfo: memFree=26101448/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) R 6931 6933 6568 0 -1 4202496 436 0 0 0 0 0 0 0 23 0 1 0 520652309 9633792 378 33554432000 4194304 4338100 140737423354384 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/6933/statm: 2352 378 146 36 0 257 0

[startup+0.0547629 s]
/proc/loadavg: 7.13 7.85 7.94 2/177 6933
/proc/meminfo: memFree=26101448/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=27964 CPUtime=0.04 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) R 6931 6933 6568 0 -1 4202496 4182 0 0 0 4 0 0 0 24 0 1 0 520652309 28635136 4019 33554432000 4194304 4338100 140737423354384 18446744073709551615 230808862475 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/6933/statm: 7024 4039 383 36 0 4929 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 27964

[startup+0.100752 s]
/proc/loadavg: 7.13 7.85 7.94 2/177 6933
/proc/meminfo: memFree=26101448/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=32188 CPUtime=0.09 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) R 6931 6933 6568 0 -1 4202496 4905 0 0 0 8 1 0 0 24 0 1 0 520652309 32960512 4709 33554432000 4194304 4338100 140737423354384 18446744073709551615 230808560870 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/6933/statm: 8047 4709 487 36 0 5952 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 32188

[startup+0.300714 s]
/proc/loadavg: 7.13 7.85 7.94 2/177 6933
/proc/meminfo: memFree=26101448/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=15564 CPUtime=0.29 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) R 6931 6933 6568 0 -1 4202496 5490 0 0 0 28 1 0 0 25 0 1 0 520652309 15937536 1901 33554432000 4194304 4338100 140737423354384 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/6933/statm: 3891 1901 155 36 0 1796 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 15564

[startup+0.70061 s]
/proc/loadavg: 7.13 7.85 7.94 2/177 6933
/proc/meminfo: memFree=26101448/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=158592 CPUtime=1.06 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 13430 0 0 0 104 2 0 0 25 0 5 0 520652309 162398208 9784 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 39648 9784 176 36 0 37553 0
Current children cumulated CPU time (s) 1.06
Current children cumulated vsize (KiB) 158592

[startup+1.50046 s]
/proc/loadavg: 7.13 7.85 7.94 9/186 6942
/proc/meminfo: memFree=26011856/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=203044 CPUtime=4.26 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 24105 0 0 0 421 5 0 0 25 0 5 0 520652309 207917056 20459 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 50761 20459 176 36 0 48666 0
[pid=6933/tid=6935] ppid=6931 vsize=203044 CPUtime=0.92 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 2663 0 0 0 91 1 0 0 19 0 5 0 520652367 207917056 20459 33554432000 4194304 4338100 140737423354384 18446744073709551615 4256400 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=203044 CPUtime=0.91 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 3297 0 0 0 91 0 0 0 19 0 5 0 520652367 207917056 20459 33554432000 4194304 4338100 140737423354384 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=203044 CPUtime=0.91 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 3170 0 0 0 91 0 0 0 25 0 5 0 520652367 207917056 20460 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=203044 CPUtime=0.91 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 3900 0 0 0 91 0 0 0 25 0 5 0 520652367 207917056 20460 33554432000 4194304 4338100 140737423354384 18446744073709551615 4206899 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4.26
Current children cumulated vsize (KiB) 203044

[startup+3.10117 s]
/proc/loadavg: 7.20 7.86 7.94 10/186 6942
/proc/meminfo: memFree=25905580/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=285032 CPUtime=10.66 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 44790 0 0 0 1058 8 0 0 25 0 5 0 520652309 291872768 41144 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 71258 41145 176 36 0 69163 0
[pid=6933/tid=6935] ppid=6931 vsize=285032 CPUtime=2.51 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 6679 0 0 0 250 1 0 0 22 0 5 0 520652367 291872768 41145 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=285032 CPUtime=2.52 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 8273 0 0 0 251 1 0 0 22 0 5 0 520652367 291872768 41145 33554432000 4194304 4338100 140737423354384 18446744073709551615 4198984 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=285032 CPUtime=2.52 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 7925 0 0 0 251 1 0 0 25 0 5 0 520652367 291872768 41145 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=285032 CPUtime=2.51 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 10838 0 0 0 250 1 0 0 25 0 5 0 520652367 291872768 41145 33554432000 4194304 4338100 140737423354384 18446744073709551615 4284198 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 10.66
Current children cumulated vsize (KiB) 285032

[startup+6.30058 s]
/proc/loadavg: 7.20 7.86 7.94 10/186 6942
/proc/meminfo: memFree=25649772/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=404688 CPUtime=23.43 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 75835 0 0 0 2329 14 0 0 25 0 5 0 520652309 414400512 72189 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 101172 72190 176 36 0 99077 0
[pid=6933/tid=6935] ppid=6931 vsize=404688 CPUtime=5.72 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 13463 0 0 0 569 3 0 0 25 0 5 0 520652367 414400512 72190 33554432000 4194304 4338100 140737423354384 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=404688 CPUtime=5.72 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 15749 0 0 0 569 3 0 0 25 0 5 0 520652367 414400512 72190 33554432000 4194304 4338100 140737423354384 18446744073709551615 4239472 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=404688 CPUtime=5.71 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 15086 0 0 0 569 2 0 0 25 0 5 0 520652367 414400512 72190 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=404688 CPUtime=5.68 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 20462 0 0 0 566 2 0 0 25 0 5 0 520652367 414400512 72191 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.43
Current children cumulated vsize (KiB) 404688

[startup+12.7004 s]
/proc/loadavg: 7.40 7.88 7.95 9/186 6942
/proc/meminfo: memFree=25258088/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=655568 CPUtime=49 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 120729 0 0 0 4875 25 0 0 25 0 5 0 520652309 671301632 117083 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 163892 117083 176 36 0 161797 0
[pid=6933/tid=6935] ppid=6931 vsize=655568 CPUtime=12.12 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 24012 0 0 0 1205 7 0 0 25 0 5 0 520652367 671301632 117083 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=655568 CPUtime=12.11 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 24591 0 0 0 1206 5 0 0 25 0 5 0 520652367 671301632 117083 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=655568 CPUtime=12.11 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 24889 0 0 0 1207 4 0 0 25 0 5 0 520652367 671301632 117083 33554432000 4194304 4338100 140737423354384 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=655568 CPUtime=12.05 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 36161 0 0 0 1200 5 0 0 25 0 5 0 520652367 671301632 117083 33554432000 4194304 4338100 140737423354384 18446744073709551615 230808862128 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 49
Current children cumulated vsize (KiB) 655568

[startup+25.5011 s]
/proc/loadavg: 7.57 7.90 7.95 9/186 6943
/proc/meminfo: memFree=24706412/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=909860 CPUtime=100.14 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 184484 0 0 0 9977 37 0 0 25 0 5 0 520652309 931696640 180838 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 227465 180838 176 36 0 225370 0
[pid=6933/tid=6935] ppid=6931 vsize=909860 CPUtime=24.92 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 39755 0 0 0 2482 10 0 0 25 0 5 0 520652367 931696640 180838 33554432000 4194304 4338100 140737423354384 18446744073709551615 4229577 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=909860 CPUtime=24.91 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 37188 0 0 0 2483 8 0 0 25 0 5 0 520652367 931696640 180838 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=909860 CPUtime=24.92 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 40909 0 0 0 2485 7 0 0 25 0 5 0 520652367 931696640 180838 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=909860 CPUtime=24.78 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 55556 0 0 0 2470 8 0 0 25 0 5 0 520652367 931696640 180838 33554432000 4194304 4338100 140737423354384 18446744073709551615 4240519 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.14
Current children cumulated vsize (KiB) 909860

[startup+51.1004 s]
/proc/loadavg: 7.72 7.90 7.95 9/186 6943
/proc/meminfo: memFree=23837760/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=1374256 CPUtime=202.39 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 300944 0 0 0 20179 60 0 0 25 0 5 0 520652309 1407238144 297298 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 343564 297299 176 36 0 341469 0
[pid=6933/tid=6935] ppid=6931 vsize=1374256 CPUtime=50.52 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 68566 0 0 0 5036 16 0 0 25 0 5 0 520652367 1407238144 297299 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=1374256 CPUtime=50.52 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 59879 0 0 0 5038 14 0 0 25 0 5 0 520652367 1407238144 297300 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=1374256 CPUtime=50.52 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 72131 0 0 0 5038 14 0 0 25 0 5 0 520652367 1407238144 297300 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=1374256 CPUtime=50.24 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 89294 0 0 0 5011 13 0 0 25 0 5 0 520652367 1407238144 297301 33554432000 4194304 4338100 140737423354384 18446744073709551615 230808560656 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 202.39
Current children cumulated vsize (KiB) 1374256

[startup+102.307 s]
/proc/loadavg: 7.98 7.95 7.96 10/186 6945
/proc/meminfo: memFree=22512412/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=2020472 CPUtime=406.94 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 462623 0 0 0 40599 95 0 0 25 0 5 0 520652309 2068963328 458977 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 505118 458977 176 36 0 503023 0
[pid=6933/tid=6935] ppid=6931 vsize=2020472 CPUtime=101.72 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 110670 0 0 0 10147 25 0 0 25 0 5 0 520652367 2068963328 458977 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=2020472 CPUtime=101.73 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 98248 0 0 0 10151 22 0 0 25 0 5 0 520652367 2068963328 458977 33554432000 4194304 4338100 140737423354384 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=2020472 CPUtime=101.73 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 108016 0 0 0 10151 22 0 0 25 0 5 0 520652367 2068963328 458977 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=2020472 CPUtime=101.16 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 134613 0 0 0 10094 22 0 0 25 0 5 0 520652367 2068963328 458977 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.94
Current children cumulated vsize (KiB) 2020472

[startup+162.301 s]
/proc/loadavg: 8.04 7.97 7.97 10/186 6946
/proc/meminfo: memFree=21113788/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=2669684 CPUtime=646.59 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 625375 0 0 0 64528 131 0 0 25 0 5 0 520652309 2733756416 621729 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 667421 621729 176 36 0 665326 0
[pid=6933/tid=6935] ppid=6931 vsize=2669684 CPUtime=161.72 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 151741 0 0 0 16137 35 0 0 25 0 5 0 520652367 2733756416 621729 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=2669684 CPUtime=161.72 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 130694 0 0 0 16143 29 0 0 25 0 5 0 520652367 2733756416 621729 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=2669684 CPUtime=161.72 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 140284 0 0 0 16144 28 0 0 25 0 5 0 520652367 2733756416 621729 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=2669684 CPUtime=160.83 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 191580 0 0 0 16047 36 0 0 25 0 5 0 520652367 2733756416 621729 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 646.59
Current children cumulated vsize (KiB) 2669684


################
# More data... #
################

/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 430075 0 0 0 75663 93 0 0 25 0 5 0 520652367 5891870720 1390111 33554432000 4194304 4338100 140737423354384 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3043.4
Current children cumulated vsize (KiB) 5753780

[startup+822.301 s]
/proc/loadavg: 8.14 8.11 8.03 9/186 6964
/proc/meminfo: memFree=13410776/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=5989040 CPUtime=3283.06 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1451384 0 0 0 327974 332 0 0 25 0 5 0 520652309 6132776960 1447738 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1497260 1447738 176 36 0 1495165 0
[pid=6933/tid=6935] ppid=6931 vsize=5989040 CPUtime=821.73 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 358825 0 0 0 82087 86 0 0 25 0 5 0 520652367 6132776960 1447738 33554432000 4194304 4338100 140737423354384 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=5989040 CPUtime=821.74 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 309472 0 0 0 82099 75 0 0 25 0 5 0 520652367 6132776960 1447738 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=5989040 CPUtime=821.74 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 321203 0 0 0 82105 69 0 0 25 0 5 0 520652367 6132776960 1447738 33554432000 4194304 4338100 140737423354384 18446744073709551615 4263517 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=5989040 CPUtime=817.24 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 450808 0 0 0 81626 98 0 0 25 0 5 0 520652367 6132776960 1447738 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3283.06
Current children cumulated vsize (KiB) 5989040

[startup+882.301 s]
/proc/loadavg: 8.05 8.09 8.02 10/186 6965
/proc/meminfo: memFree=12899128/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=6186500 CPUtime=3522.73 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1501615 0 0 0 351929 344 0 0 25 0 5 0 520652309 6334976000 1497969 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1546625 1497969 176 36 0 1544530 0
[pid=6933/tid=6935] ppid=6931 vsize=6186500 CPUtime=881.73 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 371012 0 0 0 88084 89 0 0 25 0 5 0 520652367 6334976000 1497969 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=6186500 CPUtime=881.74 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 319027 0 0 0 88097 77 0 0 25 0 5 0 520652367 6334976000 1497969 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=6186500 CPUtime=881.74 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 332214 0 0 0 88102 72 0 0 25 0 5 0 520652367 6334976000 1497969 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=6186500 CPUtime=876.9 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 468286 0 0 0 87589 101 0 0 25 0 5 0 520652367 6334976000 1497969 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3522.73
Current children cumulated vsize (KiB) 6186500

[startup+942.301 s]
/proc/loadavg: 8.02 8.07 8.02 10/186 6967
/proc/meminfo: memFree=12488924/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=6335904 CPUtime=3762.41 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1539916 0 0 0 375888 353 0 0 25 0 5 0 520652309 6487965696 1536270 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1583976 1536270 176 36 0 1581881 0
[pid=6933/tid=6935] ppid=6931 vsize=6335904 CPUtime=941.74 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 378084 0 0 0 94083 91 0 0 25 0 5 0 520652367 6487965696 1536270 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=6335904 CPUtime=941.74 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 324857 0 0 0 94095 79 0 0 25 0 5 0 520652367 6487965696 1536270 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=6335904 CPUtime=941.74 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 344290 0 0 0 94099 75 0 0 25 0 5 0 520652367 6487965696 1536270 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=6335904 CPUtime=936.58 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 481609 0 0 0 93554 104 0 0 25 0 5 0 520652367 6487965696 1536270 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3762.41
Current children cumulated vsize (KiB) 6335904

[startup+1002.3 s]
/proc/loadavg: 8.00 8.05 8.01 10/186 6969
/proc/meminfo: memFree=12036528/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=6458424 CPUtime=4002.08 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1570183 0 0 0 399847 361 0 0 25 0 5 0 520652309 6613426176 1566537 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1614606 1566537 176 36 0 1612511 0
[pid=6933/tid=6935] ppid=6931 vsize=6458424 CPUtime=1001.74 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 380118 0 0 0 100082 92 0 0 25 0 5 0 520652367 6613426176 1566537 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=6458424 CPUtime=1001.74 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 328224 0 0 0 100094 80 0 0 25 0 5 0 520652367 6613426176 1566537 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=6458424 CPUtime=1001.74 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 350040 0 0 0 100098 76 0 0 25 0 5 0 520652367 6613426176 1566537 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=6458424 CPUtime=996.25 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 500725 0 0 0 99516 109 0 0 25 0 5 0 520652367 6613426176 1566537 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4002.08
Current children cumulated vsize (KiB) 6458424

[startup+1062.3 s]
/proc/loadavg: 8.04 8.06 8.01 9/186 6970
/proc/meminfo: memFree=11740744/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=6655884 CPUtime=4241.76 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1618746 0 0 0 423803 373 0 0 25 0 5 0 520652309 6815625216 1615100 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1663971 1615100 176 36 0 1661876 0
[pid=6933/tid=6935] ppid=6931 vsize=6655884 CPUtime=1061.74 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 391267 0 0 0 106079 95 0 0 25 0 5 0 520652367 6815625216 1615100 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=6655884 CPUtime=1061.74 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 335002 0 0 0 106092 82 0 0 25 0 5 0 520652367 6815625216 1615100 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=6655884 CPUtime=1061.75 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 365692 0 0 0 106096 79 0 0 25 0 5 0 520652367 6815625216 1615100 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=6655884 CPUtime=1055.92 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 515709 0 0 0 105479 113 0 0 25 0 5 0 520652367 6815625216 1615100 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4241.76
Current children cumulated vsize (KiB) 6655884

[startup+1122.3 s]
/proc/loadavg: 8.10 8.07 8.01 9/186 6972
/proc/meminfo: memFree=11319748/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=6850216 CPUtime=4481.42 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1669201 0 0 0 447755 387 0 0 25 0 5 0 520652309 7014621184 1665555 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1712554 1665555 176 36 0 1710459 0
[pid=6933/tid=6935] ppid=6931 vsize=6850216 CPUtime=1121.74 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 404551 0 0 0 112076 98 0 0 25 0 5 0 520652367 7014621184 1665555 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=6850216 CPUtime=1121.75 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 341873 0 0 0 112091 84 0 0 25 0 5 0 520652367 7014621184 1665555 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=6850216 CPUtime=1121.74 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 379607 0 0 0 112091 83 0 0 25 0 5 0 520652367 7014621184 1665555 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=6850216 CPUtime=1115.59 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 532094 0 0 0 111440 119 0 0 25 0 5 0 520652367 7014621184 1665555 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4481.42
Current children cumulated vsize (KiB) 6850216

[startup+1182.3 s]
/proc/loadavg: 8.12 8.09 8.02 9/186 6973
/proc/meminfo: memFree=10957400/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=6969476 CPUtime=4721.11 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1699584 0 0 0 471715 396 0 0 25 0 5 0 520652309 7136743424 1695938 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1742369 1695938 176 36 0 1740274 0
[pid=6933/tid=6935] ppid=6931 vsize=6969476 CPUtime=1181.75 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 407261 0 0 0 118076 99 0 0 25 0 5 0 520652367 7136743424 1695938 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=6969476 CPUtime=1181.75 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 343537 0 0 0 118090 85 0 0 25 0 5 0 520652367 7136743424 1695938 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=6969476 CPUtime=1181.74 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 389168 0 0 0 118089 85 0 0 25 0 5 0 520652367 7136743424 1695938 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=6969476 CPUtime=1175.27 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 548542 0 0 0 117404 123 0 0 25 0 5 0 520652367 7136743424 1695938 33554432000 4194304 4338100 140737423354384 18446744073709551615 4257106 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4721.11
Current children cumulated vsize (KiB) 6969476



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.1 s]
/proc/loadavg: 8.15 8.10 8.02 9/186 6974
/proc/meminfo: memFree=10803128/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=7044680 CPUtime=4800.21 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) S 6931 6933 6568 0 -1 4202496 1717233 0 0 0 479621 400 0 0 25 0 5 0 520652309 7213752320 1713587 33554432000 4194304 4338100 140737423354384 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 1761170 1713587 176 36 0 1759075 0
[pid=6933/tid=6935] ppid=6931 vsize=7044680 CPUtime=1201.55 cores=1,3,5,7
/proc/6933/task/6935/stat : 6935 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 411371 0 0 0 120055 100 0 0 25 0 5 0 520652367 7213752320 1713589 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=6933/tid=6936] ppid=6931 vsize=7044680 CPUtime=1201.55 cores=1,3,5,7
/proc/6933/task/6936/stat : 6936 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 345724 0 0 0 120070 85 0 0 25 0 5 0 520652367 7213752320 1713589 33554432000 4194304 4338100 140737423354384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=6933/tid=6937] ppid=6931 vsize=7044680 CPUtime=1201.55 cores=1,3,5,7
/proc/6933/task/6937/stat : 6937 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 393360 0 0 0 120068 87 0 0 25 0 5 0 520652367 7213752320 1713589 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=7044680 CPUtime=1194.96 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4202560 555704 0 0 0 119371 125 0 0 25 0 5 0 520652367 7213752320 1713589 33554432000 4194304 4338100 140737423354384 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 7044680

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.21 s]
/proc/loadavg: 8.15 8.10 8.02 6/184 6975
/proc/meminfo: memFree=13047232/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=0 CPUtime=4800.34 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) Z 6931 6933 6568 0 -1 4203548 1717240 0 0 0 479623 411 0 0 18 0 2 0 520652309 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 0 0 0 0 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=0 CPUtime=1195.06 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4203588 555709 0 0 0 119371 135 0 0 25 0 2 0 520652367 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

[startup+1202.31 s]
/proc/loadavg: 8.15 8.10 8.02 6/184 6975
/proc/meminfo: memFree=13047232/32951124 swapFree=67111524/67111528
[pid=6933] ppid=6931 vsize=0 CPUtime=4800.43 cores=1,3,5,7
/proc/6933/stat : 6933 (ttsth-5-4) Z 6931 6933 6568 0 -1 4203548 1717240 0 0 0 479623 420 0 0 18 0 2 0 520652309 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/6933/statm: 0 0 0 0 0 0 0
[pid=6933/tid=6938] ppid=6931 vsize=0 CPUtime=1195.15 cores=1,3,5,7
/proc/6933/task/6938/stat : 6938 (ttsth-5-4) R 6931 6933 6568 0 -1 4203588 555709 0 0 0 119371 144 0 0 25 0 2 0 520652367 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.43
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.38
CPU time (s): 4800.51
CPU user time (s): 4796.24
CPU system time (s): 4.27635
CPU usage (%): 399.25
Max. virtual memory (cumulated for all children) (KiB): 7044680

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.24
system time used= 4.27635
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1717240
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 617
involuntary context switches= 32134

runsolver used 1.80872 second user time and 4.76328 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 17:21:59
IDJOB=3262002
IDBENCH=83268
IDSOLVER=1654
FILE ID=node104/3262002-1304090519
RUNJOBID= node104-1304090518-6914
PBS_JOBID= 13170552
Free space on /tmp= 71548 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-22_464.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262002-1304090519/watcher-3262002-1304090519 -o /tmp/evaluation-result-3262002-1304090519/solver-3262002-1304090519 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262002-1304090519.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a722a199fd84ba0b319251daa47aa19c
RANDOM SEED=59802743

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      26102596 kB
Buffers:       1881488 kB
Cached:        4344288 kB
SwapCached:          4 kB
Active:         173680 kB
Inactive:      6132288 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      26102596 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1660 kB
Writeback:           0 kB
AnonPages:       80296 kB
Mapped:          14784 kB
Slab:           480080 kB
PageTables:       4192 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   228984 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71548 MiB
End job on node104 at 2011-04-29 17:42:03