Trace number 3264818

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.2 1211.61

General information on the benchmark

NameSAT05/crafted/jarvisalo05/mod2c-rand3bip-sat/
mod2c-rand3bip-sat-250-2.shuffled-as.sat05-2534.cnf
MD5SUM9a6126d13f733e360922c2ae93b7b6b9
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark173.141
Satisfiable
(Un)Satisfiability was proved
Number of variables354
Number of clauses2596
Sum of the clauses size11752
Maximum clause length6
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 41592
Number of clauses of size 5640
Number of clauses of size over 5364

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264818-1304100568.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 354
0.00/0.00	c Clauses = 2596
0.00/0.00	c Literals = 11752
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 354
0.00/0.00	c Clauses = 2596
0.00/0.00	c Literals = 11752
0.00/0.00	c a2
0.00/0.01	c a1
0.01/0.09	c Independent sub-problems = 1
0.01/0.09	c Sub-problem with 354 variables and 2596 clauses
0.01/0.09	c using permutation method 4
0.01/0.09	c count = 2
0.01/0.09	c Ordering finished 0.090 seconds
0.01/0.09	c mapped 0.090 seconds
0.01/0.09	c permuted 0.090 seconds
0.01/0.09	c about to sort
0.01/0.09	c sorted
0.09/0.10	c clauses now 2596
0.09/0.10	c init finished 0.090 seconds
0.09/0.10	c tree built 0.100 seconds
0.09/0.10	c 
0.09/0.10	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.10	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.10	c ................................ | ..........                    |                                  0.1         0.0             
0.09/0.10	c ................................ | ..........                  1 |                                  0.1        10.0             
0.09/0.10	c ................................ | ..........                  2 |                                  0.1        20.0             
0.09/0.10	c ................................ | ..........                  3 |                                  0.1        30.0             
0.09/0.10	c ................................ | ..........                  4 |                                  0.1        40.0             
0.09/0.10	c ................................ | ..........                  5 |                                  0.1        50.0             
0.09/0.10	c ................................ | ..........               1000 | 93           146                 0.1     10000.0         2086
95.72/24.19	c ................................ | ..........            1404000 | 348418       703329             96.1     14612.8     16769990
191.72/48.28	c ................................ | ..........            2166000 | 554914       1104736           192.0     11278.9     24104305
287.98/72.42	c ................................ | ..........            3012000 | 772568       1473120           288.1     10456.2     34626858
383.94/96.52	c ................................ | ..........            3543000 | 911953       1712198           384.0      9225.6     39992206
479.91/120.66	c ................................ | *.........            3968000 | 1027321      1912828           480.2      8263.6     43324318
575.95/144.75	c ................................ | *.........            4334000 | 1126376      2091661           576.2      7522.0     45622333
671.96/168.83	c ................................ | *.........            4730000 | 1230046      2297679           672.1      7037.6     51381359
767.76/192.97	c ................................ | *.........            5095000 | 1326039      2495178           768.1      6633.4     53491400
864.04/217.13	c ................................ | *.........            5457000 | 1423855      2678476           864.2      6314.7     55325871
960.04/241.22	c ................................ | **........            5781000 | 1510896      2842368           960.1      6021.0     57679084
1055.76/265.28	c ................................ | **........            6099000 | 1596642      2995307          1056.1      5775.1     60652847
1151.73/289.49	c ................................ | **........            6444000 | 1686816      3165215          1152.1      5593.2     62112823
1247.99/313.63	c ................................ | **........            6768000 | 1773758      3324883          1248.1      5422.6     64876374
1343.62/337.70	c ................................ | **........            7062000 | 1854594      3474681          1344.0      5254.5     67895601
1439.98/361.82	c ................................ | ***.......            7337000 | 1929664      3616520          1440.0      5095.0     70610540
1535.96/386.25	c ................................ | ***.......            7840000 | 2064444      3878406          1536.1      5103.8     70610540
1632.09/410.41	c ................................ | ***.......            8277000 | 2183188      4102686          1632.1      5071.3     70878819
1727.88/434.68	c ................................ | ***.......            8710000 | 2299671      4323958          1728.2      5040.0     71861703
1824.08/458.93	c ................................ | ***.......            9059000 | 2393871      4506452          1824.2      4966.0     73280396
1920.05/483.13	c ................................ | ****......            9384000 | 2482730      4663678          1920.2      4887.1     73409425
2015.95/507.38	c ................................ | ****......            9739000 | 2578438      4841898          2016.3      4830.2     75133932
2111.73/531.58	c ................................ | ****......           10061000 | 2665991      5001696          2112.1      4763.6     77194409
2208.03/555.85	c ................................ | ****......           10367000 | 2748723      5151842          2208.2      4694.7     78392192
2304.05/580.14	c ................................ | ****......           10681000 | 2833549      5314664          2304.2      4635.4     79062140
2400.19/604.53	c ................................ | *****.....           10986000 | 2916033      5479314          2400.3      4576.9     79108474
2495.73/628.58	c ................................ | *****.....           11293000 | 2998954      5634381          2496.1      4524.3     80117474
2592.01/653.03	c ................................ | *****.....           11615000 | 3086081      5788504          2592.1      4480.9     80117474
2687.93/677.21	c ................................ | *****.....           11937000 | 3173361      5951668          2688.0      4440.8     80266369
2783.88/701.44	c ................................ | *****.....           12228000 | 3253335      6094445          2784.1      4392.2     82761032
2879.84/725.65	c ................................ | ******....           12535000 | 3337497      6244522          2880.0      4352.4     86522463
2975.90/749.88	c ................................ | ******....           12824000 | 3416347      6390642          2976.2      4308.8     89881919
3072.19/774.20	c ................................ | ******....           13105000 | 3493408      6537768          3072.2      4265.7     89881919
3168.17/798.51	c ................................ | ******....           13398000 | 3573601      6687207          3168.2      4228.9     91217405
3264.00/822.80	c ................................ | ******....           13673000 | 3648490      6833237          3264.0      4189.0     91889456
3360.02/847.07	c ................................ | *******...           13934000 | 3720230      6968615          3360.3      4146.6     92451186
3455.90/871.36	c ................................ | *******...           14201000 | 3792622      7106838          3456.2      4108.9     93614100
3551.98/895.66	c ................................ | *******...           14446000 | 3859529      7232030          3552.2      4066.7     94376742
3647.81/919.87	c ................................ | *******...           14671000 | 3921086      7347765          3648.1      4021.5     96438949
3743.77/943.99	c ................................ | *******...           14939000 | 3994366      7472843          3744.2      3990.0     99712524
3839.98/968.42	c ................................ | ********..           15260000 | 4083362      7627620          3840.1      3973.9    101615276
3935.97/992.82	c ................................ | ********..           15531000 | 4158637      7759353          3936.1      3945.8    101640503
4031.74/1016.99	c ................................ | ********..           15819000 | 4237553      7903033          4032.1      3923.3    101640503
4127.84/1041.36	c ................................ | ********..           16084000 | 4310526      8040135          4128.1      3896.2    102399745
4224.20/1065.92	c ................................ | ********..           16341000 | 4381702      8166418          4224.3      3868.4    102655458
4319.99/1090.18	c ................................ | *********.           16597000 | 4451564      8300995          4320.3      3841.6    103311904
4416.01/1114.58	c ................................ | *********.           16842000 | 4518194      8429064          4416.4      3813.5    104388961
4512.25/1138.82	c ................................ | *********.           17079000 | 4583676      8548697          4512.3      3785.0    104919353
4607.89/1162.93	c ................................ | *********.           17305000 | 4645964      8663984          4608.0      3755.4    105581039
4704.13/1187.22	c ................................ | *********.           17519000 | 4704621      8770407          4704.2      3724.1    106526898
4799.78/1211.47	c CPU limit (4800) reached
4799.78/1211.47	c Time = 4800.05 seconds
4799.78/1211.47	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264818-1304100568/watcher-3264818-1304100568 -o /tmp/evaluation-result-3264818-1304100568/solver-3264818-1304100568 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264818-1304100568.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.41 9.08 9.57 3/173 29849
/proc/meminfo: memFree=17441472/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=8980 CPUtime=0 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) R 29847 29849 28861 0 -1 4202496 328 0 0 0 0 0 0 0 23 0 1 0 600287582 9195520 270 33554432000 4194304 4338100 140735946791536 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29849/statm: 2245 272 150 36 0 150 0

[startup+0.019035 s]
/proc/loadavg: 6.41 9.08 9.57 3/173 29849
/proc/meminfo: memFree=17441472/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=9772 CPUtime=0.01 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) R 29847 29849 28861 0 -1 4202496 530 0 0 0 1 0 0 0 23 0 1 0 600287582 10006528 471 33554432000 4194304 4338100 140735946791536 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29849/statm: 2443 471 155 36 0 348 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 9772

[startup+0.101022 s]
/proc/loadavg: 6.41 9.08 9.57 3/173 29849
/proc/meminfo: memFree=17441472/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=12684 CPUtime=0.09 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) R 29847 29849 28861 0 -1 4202496 1308 0 0 0 9 0 0 0 23 0 1 0 600287582 12988416 1196 33554432000 4194304 4338100 140735946791536 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29849/statm: 3171 1205 162 36 0 1076 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12684

[startup+0.300989 s]
/proc/loadavg: 6.41 9.08 9.57 3/173 29849
/proc/meminfo: memFree=17441472/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=198840 CPUtime=0.77 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 4512 0 0 0 77 0 0 0 23 0 5 0 600287582 203612160 4400 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 49710 4400 175 36 0 47615 0
Current children cumulated CPU time (s) 0.77
Current children cumulated vsize (KiB) 198840

[startup+0.700919 s]
/proc/loadavg: 6.41 9.08 9.57 3/173 29849
/proc/meminfo: memFree=17441472/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=286660 CPUtime=2.37 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 9292 0 0 0 236 1 0 0 23 0 5 0 600287582 293539840 9180 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 71665 9180 175 36 0 69570 0
Current children cumulated CPU time (s) 2.37
Current children cumulated vsize (KiB) 286660

[startup+1.50078 s]
/proc/loadavg: 6.41 9.08 9.57 10/186 29863
/proc/meminfo: memFree=17337232/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=318328 CPUtime=5.56 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 17455 0 0 0 550 6 0 0 23 0 5 0 600287582 325967872 17343 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 79582 17344 175 36 0 77487 0
[pid=29849/tid=29860] ppid=29847 vsize=318328 CPUtime=1.39 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 3281 0 0 0 138 1 0 0 25 0 5 0 600287593 325967872 17345 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=318328 CPUtime=1.37 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 4678 0 0 0 137 0 0 0 18 0 5 0 600287593 325967872 17346 33554432000 4194304 4338100 140735946791536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=318328 CPUtime=1.35 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 3816 0 0 0 134 1 0 0 25 0 5 0 600287593 325967872 17346 33554432000 4194304 4338100 140735946791536 18446744073709551615 215987988326 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=318328 CPUtime=1.31 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 4152 0 0 0 130 1 0 0 25 0 5 0 600287593 325967872 17347 33554432000 4194304 4338100 140735946791536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.56
Current children cumulated vsize (KiB) 318328

[startup+3.1005 s]
/proc/loadavg: 6.41 9.08 9.57 9/186 29863
/proc/meminfo: memFree=17249088/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=362644 CPUtime=11.95 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 29410 0 0 0 1184 11 0 0 23 0 5 0 600287582 371347456 29298 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 90661 29299 175 36 0 88566 0
[pid=29849/tid=29860] ppid=29847 vsize=362644 CPUtime=2.98 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 5569 0 0 0 296 2 0 0 25 0 5 0 600287593 371347456 29299 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000406767 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=362644 CPUtime=2.97 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 8434 0 0 0 296 1 0 0 21 0 5 0 600287593 371347456 29299 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000405254 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=362644 CPUtime=2.95 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 7350 0 0 0 292 3 0 0 25 0 5 0 600287593 371347456 29299 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=362644 CPUtime=2.92 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 6527 0 0 0 289 3 0 0 25 0 5 0 600287593 371347456 29299 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.95
Current children cumulated vsize (KiB) 362644

[startup+6.30094 s]
/proc/loadavg: 6.54 9.06 9.56 9/186 29863
/proc/meminfo: memFree=17013004/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=468292 CPUtime=24.7 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 55945 0 0 0 2452 18 0 0 23 0 5 0 600287582 479531008 55833 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 117073 55833 176 36 0 114978 0
[pid=29849/tid=29860] ppid=29847 vsize=468292 CPUtime=6.17 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 9553 0 0 0 614 3 0 0 25 0 5 0 600287593 479531008 55833 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=468292 CPUtime=6.17 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 17737 0 0 0 613 4 0 0 18 0 5 0 600287593 479531008 55834 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=468292 CPUtime=6.12 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 15821 0 0 0 607 5 0 0 25 0 5 0 600287593 479531008 55834 33554432000 4194304 4338100 140735946791536 18446744073709551615 215987987686 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=468292 CPUtime=6.11 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 11304 0 0 0 606 5 0 0 25 0 5 0 600287593 479531008 55834 33554432000 4194304 4338100 140735946791536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.7
Current children cumulated vsize (KiB) 468292

[startup+12.7008 s]
/proc/loadavg: 6.65 9.05 9.55 9/186 29864
/proc/meminfo: memFree=16705404/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=599688 CPUtime=50.23 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 89840 0 0 0 4992 31 0 0 23 0 5 0 600287582 614080512 89728 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 149922 89729 176 36 0 147827 0
[pid=29849/tid=29860] ppid=29847 vsize=599688 CPUtime=12.56 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 15639 0 0 0 1249 7 0 0 25 0 5 0 600287593 614080512 89729 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=599688 CPUtime=12.55 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 28459 0 0 0 1248 7 0 0 25 0 5 0 600287593 614080512 89730 33554432000 4194304 4338100 140735946791536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=599688 CPUtime=12.48 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 27067 0 0 0 1240 8 0 0 25 0 5 0 600287593 614080512 89731 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254816 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=599688 CPUtime=12.5 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 17147 0 0 0 1243 7 0 0 25 0 5 0 600287593 614080512 89731 33554432000 4194304 4338100 140735946791536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 50.23
Current children cumulated vsize (KiB) 599688

[startup+25.5006 s]
/proc/loadavg: 6.95 8.99 9.53 9/186 29864
/proc/meminfo: memFree=16205540/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=815184 CPUtime=101.3 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 144782 0 0 0 10077 53 0 0 23 0 5 0 600287582 834748416 144662 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 203796 144662 176 36 0 201701 0
[pid=29849/tid=29860] ppid=29847 vsize=815184 CPUtime=25.35 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 25550 0 0 0 2523 12 0 0 25 0 5 0 600287593 834748416 144663 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=815184 CPUtime=25.34 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 45567 0 0 0 2523 11 0 0 25 0 5 0 600287593 834748416 144663 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=815184 CPUtime=25.2 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 43970 0 0 0 2505 15 0 0 25 0 5 0 600287593 834748416 144663 33554432000 4194304 4338100 140735946791536 18446744073709551615 215987987681 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=815184 CPUtime=25.27 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 28166 0 0 0 2514 13 0 0 25 0 5 0 600287593 834748416 144664 33554432000 4194304 4338100 140735946791536 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 101.3
Current children cumulated vsize (KiB) 815184

[startup+51.1011 s]
/proc/loadavg: 7.31 8.91 9.49 7/186 29865
/proc/meminfo: memFree=15505440/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=1124200 CPUtime=203.28 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 223134 0 0 0 20239 89 0 0 23 0 5 0 600287582 1151180800 223014 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 281050 223014 176 36 0 278955 0
[pid=29849/tid=29860] ppid=29847 vsize=1124200 CPUtime=50.85 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 44628 0 0 0 5064 21 0 0 25 0 5 0 600287593 1151180800 223014 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=1124200 CPUtime=50.85 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 70285 0 0 0 5063 22 0 0 20 0 5 0 600287593 1151180800 223014 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=1124200 CPUtime=50.75 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 63797 0 0 0 5051 24 0 0 25 0 5 0 600287593 1151180800 223014 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=1124200 CPUtime=50.7 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 42893 0 0 0 5049 21 0 0 25 0 5 0 600287593 1151180800 223014 33554432000 4194304 4338100 140735946791536 18446744073709551615 4253990 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.28
Current children cumulated vsize (KiB) 1124200

[startup+102.306 s]
/proc/loadavg: 7.70 8.77 9.41 9/186 29866
/proc/meminfo: memFree=14480208/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=1620192 CPUtime=407.05 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 350450 0 0 0 40568 137 0 0 23 0 5 0 600287582 1659076608 350330 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 405048 350330 176 36 0 402953 0
[pid=29849/tid=29860] ppid=29847 vsize=1620192 CPUtime=101.8 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 75787 0 0 0 10146 34 0 0 25 0 5 0 600287593 1659076608 350330 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254410 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=1620192 CPUtime=101.81 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 104472 0 0 0 10146 35 0 0 25 0 5 0 600287593 1659076608 350330 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=1620192 CPUtime=101.79 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 101447 0 0 0 10142 37 0 0 25 0 5 0 600287593 1659076608 350330 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=1620192 CPUtime=101.52 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 67213 0 0 0 10122 30 0 0 25 0 5 0 600287593 1659076608 350330 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 407.05
Current children cumulated vsize (KiB) 1620192

[startup+162.301 s]
/proc/loadavg: 7.89 8.63 9.32 9/186 29868
/proc/meminfo: memFree=13877060/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=1870008 CPUtime=646.13 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 415093 0 0 0 64442 171 0 0 23 0 5 0 600287582 1914888192 414973 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 467502 414973 176 36 0 465407 0
[pid=29849/tid=29860] ppid=29847 vsize=1870008 CPUtime=161.52 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 95879 0 0 0 16109 43 0 0 25 0 5 0 600287593 1914888192 414973 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=1870008 CPUtime=161.65 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 120256 0 0 0 16123 42 0 0 25 0 5 0 600287593 1914888192 414973 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=1870008 CPUtime=161.48 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 115800 0 0 0 16102 46 0 0 25 0 5 0 600287593 1914888192 414973 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=1870008 CPUtime=161.35 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 81627 0 0 0 16097 38 0 0 25 0 5 0 600287593 1914888192 414973 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.13
Current children cumulated vsize (KiB) 1870008


################
# More data... #
################

/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 148892 0 0 0 69690 94 0 0 20 0 5 0 600287593 3081658368 719444 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2787.48
Current children cumulated vsize (KiB) 3009432

[startup+762.307 s]
/proc/loadavg: 7.98 8.05 8.65 9/186 29884
/proc/meminfo: memFree=10594368/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3269168 CPUtime=3025.24 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 778892 0 0 0 302066 458 0 0 23 0 5 0 600287582 3347628032 778772 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 817292 778772 176 36 0 815197 0
[pid=29849/tid=29860] ppid=29847 vsize=3269168 CPUtime=755.05 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) S 29847 29849 28861 0 -1 4202560 242602 0 0 0 75363 142 0 0 25 0 5 0 600287593 3347628032 778772 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000405254 0 0 4096 0 18446604453599654208 0 0 -1 4 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3269168 CPUtime=756.58 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 179376 0 0 0 75550 108 0 0 25 0 5 0 600287593 3347628032 778772 33554432000 4194304 4338100 140735946791536 18446744073709551615 4235669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3269168 CPUtime=756.15 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) S 29847 29849 28861 0 -1 4202560 187758 0 0 0 75513 102 0 0 25 0 5 0 600287593 3347628032 778772 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000405254 0 0 4096 0 18446604453599654208 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3269168 CPUtime=757.32 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) S 29847 29849 28861 0 -1 4202560 167625 0 0 0 75628 104 0 0 25 0 5 0 600287593 3347628032 778772 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000405254 0 0 4096 0 18446604453599654208 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3025.24
Current children cumulated vsize (KiB) 3269168

[startup+822.301 s]
/proc/loadavg: 8.04 8.04 8.61 9/186 29885
/proc/meminfo: memFree=10439960/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3334672 CPUtime=3262.12 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 797814 0 0 0 325729 483 0 0 23 0 5 0 600287582 3414704128 797694 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 833668 797694 176 36 0 831573 0
[pid=29849/tid=29860] ppid=29847 vsize=3334672 CPUtime=814.06 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 259032 0 0 0 81255 151 0 0 25 0 5 0 600287593 3414704128 797694 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3334672 CPUtime=816.01 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 180106 0 0 0 81489 112 0 0 25 0 5 0 600287593 3414704128 797694 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3334672 CPUtime=815.33 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 188792 0 0 0 81425 108 0 0 25 0 5 0 600287593 3414704128 797694 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3334672 CPUtime=816.59 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 168353 0 0 0 81549 110 0 0 25 0 5 0 600287593 3414704128 797694 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3262.12
Current children cumulated vsize (KiB) 3334672

[startup+882.301 s]
/proc/loadavg: 7.90 8.00 8.56 9/186 29887
/proc/meminfo: memFree=10113176/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3412432 CPUtime=3499.36 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 818607 0 0 0 349431 505 0 0 23 0 5 0 600287582 3494330368 818487 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 853108 818487 176 36 0 851013 0
[pid=29849/tid=29860] ppid=29847 vsize=3412432 CPUtime=873.17 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 272649 0 0 0 87157 160 0 0 18 0 5 0 600287593 3494330368 818487 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3412432 CPUtime=875.66 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 180772 0 0 0 87450 116 0 0 25 0 5 0 600287593 3494330368 818487 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3412432 CPUtime=874.46 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 194633 0 0 0 87331 115 0 0 25 0 5 0 600287593 3494330368 818487 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3412432 CPUtime=875.94 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 169022 0 0 0 87481 113 0 0 25 0 5 0 600287593 3494330368 818487 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3499.36
Current children cumulated vsize (KiB) 3412432

[startup+942.301 s]
/proc/loadavg: 8.03 8.01 8.52 6/186 29888
/proc/meminfo: memFree=9824728/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3574392 CPUtime=3737.38 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 860734 0 0 0 373205 533 0 0 23 0 5 0 600287582 3660177408 860614 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 893598 860614 176 36 0 891503 0
[pid=29849/tid=29860] ppid=29847 vsize=3574392 CPUtime=932.59 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 291743 0 0 0 93090 169 0 0 18 0 5 0 600287593 3660177408 860615 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254421 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3574392 CPUtime=935.27 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 187992 0 0 0 93405 122 0 0 25 0 5 0 600287593 3660177408 860615 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3574392 CPUtime=933.74 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 209857 0 0 0 93250 124 0 0 25 0 5 0 600287593 3660177408 860615 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3574392 CPUtime=935.65 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 169612 0 0 0 93448 117 0 0 25 0 5 0 600287593 3660177408 860615 33554432000 4194304 4338100 140735946791536 18446744073709551615 215988253729 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3737.38
Current children cumulated vsize (KiB) 3574392

[startup+1002.3 s]
/proc/loadavg: 7.86 7.97 8.48 9/186 29890
/proc/meminfo: memFree=9600760/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3649152 CPUtime=3973.8 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 882531 0 0 0 396822 558 0 0 23 0 5 0 600287582 3736731648 882411 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 912288 882411 176 36 0 910193 0
[pid=29849/tid=29860] ppid=29847 vsize=3649152 CPUtime=991.28 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 301767 0 0 0 98952 176 0 0 18 0 5 0 600287593 3736731648 882411 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3649152 CPUtime=994.4 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 195834 0 0 0 99313 127 0 0 18 0 5 0 600287593 3736731648 882411 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3649152 CPUtime=992.89 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 211230 0 0 0 99158 131 0 0 25 0 5 0 600287593 3736731648 882411 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3649152 CPUtime=995.08 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 172169 0 0 0 99386 122 0 0 25 0 5 0 600287593 3736731648 882411 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3973.8
Current children cumulated vsize (KiB) 3649152

[startup+1062.3 s]
/proc/loadavg: 8.02 7.99 8.45 6/186 29892
/proc/meminfo: memFree=9439544/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3683120 CPUtime=4210.41 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 894310 0 0 0 420462 579 0 0 23 0 5 0 600287582 3771514880 894190 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 920780 894190 176 36 0 918685 0
[pid=29849/tid=29860] ppid=29847 vsize=3683120 CPUtime=1050.29 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 309532 0 0 0 104847 182 0 0 25 0 5 0 600287593 3771514880 894190 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3683120 CPUtime=1053.86 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 197353 0 0 0 105255 131 0 0 25 0 5 0 600287593 3771514880 894190 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3683120 CPUtime=1051.76 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 213040 0 0 0 105039 137 0 0 25 0 5 0 600287593 3771514880 894190 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3683120 CPUtime=1054.39 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 172854 0 0 0 105311 128 0 0 25 0 5 0 600287593 3771514880 894190 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4210.41
Current children cumulated vsize (KiB) 3683120

[startup+1122.3 s]
/proc/loadavg: 8.06 8.01 8.43 9/186 29893
/proc/meminfo: memFree=9267404/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=3951104 CPUtime=4446.91 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 913713 0 0 0 444090 601 0 0 23 0 5 0 600287582 4045930496 913593 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 987776 913593 176 36 0 985681 0
[pid=29849/tid=29860] ppid=29847 vsize=3951104 CPUtime=1108.92 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 327017 0 0 0 110699 193 0 0 25 0 5 0 600287593 4045930496 913593 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=3951104 CPUtime=1113.23 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 197987 0 0 0 111188 135 0 0 25 0 5 0 600287593 4045930496 913593 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=3951104 CPUtime=1110.86 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 213681 0 0 0 110946 140 0 0 25 0 5 0 600287593 4045930496 913593 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=3951104 CPUtime=1113.77 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 173497 0 0 0 111246 131 0 0 25 0 5 0 600287593 4045930496 913593 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4446.91
Current children cumulated vsize (KiB) 3951104

[startup+1182.3 s]
/proc/loadavg: 8.00 7.99 8.39 6/186 29895
/proc/meminfo: memFree=9031528/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=4003456 CPUtime=4684.85 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) S 29847 29849 28861 0 -1 4202496 928602 0 0 0 467861 624 0 0 23 0 5 0 600287582 4099538944 928482 33554432000 4194304 4338100 140735946791536 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 1000864 928482 176 36 0 998769 0
[pid=29849/tid=29860] ppid=29847 vsize=4003456 CPUtime=1168.34 cores=0,2,4,6
/proc/29849/task/29860/stat : 29860 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 340201 0 0 0 116632 202 0 0 25 0 5 0 600287593 4099538944 928482 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29849/tid=29861] ppid=29847 vsize=4003456 CPUtime=1172.69 cores=0,2,4,6
/proc/29849/task/29861/stat : 29861 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 198551 0 0 0 117130 139 0 0 25 0 5 0 600287593 4099538944 928482 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29849/tid=29862] ppid=29847 vsize=4003456 CPUtime=1170.45 cores=0,2,4,6
/proc/29849/task/29862/stat : 29862 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 214252 0 0 0 116899 146 0 0 25 0 5 0 600287593 4099538944 928482 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=4003456 CPUtime=1173.24 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4202560 174067 0 0 0 117188 136 0 0 25 0 5 0 600287593 4099538944 928482 33554432000 4194304 4338100 140735946791536 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4684.85
Current children cumulated vsize (KiB) 4003456



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1211.5 s]
/proc/loadavg: 8.00 7.99 8.38 6/183 29896
/proc/meminfo: memFree=9615800/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=0 CPUtime=4800.08 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) Z 29847 29849 28861 0 -1 4203548 936405 0 0 0 479371 637 0 0 18 0 2 0 600287582 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 0 0 0 0 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=0 CPUtime=1202.15 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4203588 174316 0 0 0 120073 142 0 0 25 0 2 0 600287593 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1211.61 s]
/proc/loadavg: 8.00 7.99 8.38 6/183 29896
/proc/meminfo: memFree=9615800/32950940 swapFree=67111528/67111528
[pid=29849] ppid=29847 vsize=0 CPUtime=4800.19 cores=0,2,4,6
/proc/29849/stat : 29849 (ttsth-5-4) Z 29847 29849 28861 0 -1 4203548 936405 0 0 0 479371 648 0 0 18 0 2 0 600287582 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29849/statm: 0 0 0 0 0 0 0
[pid=29849/tid=29863] ppid=29847 vsize=0 CPUtime=1202.25 cores=0,2,4,6
/proc/29849/task/29863/stat : 29863 (ttsth-5-4) R 29847 29849 28861 0 -1 4203588 174316 0 0 0 120073 152 0 0 25 0 2 0 600287593 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1211.61
CPU time (s): 4800.2
CPU user time (s): 4793.72
CPU system time (s): 6.48601
CPU usage (%): 396.182
Max. virtual memory (cumulated for all children) (KiB): 4032508

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4793.72
system time used= 6.48601
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 936405
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 95065
involuntary context switches= 22754

runsolver used 2.09168 second user time and 4.35734 second system time

The end

Launcher Data 

Begin job on node117 at 2011-04-29 20:09:28
IDJOB=3264818
IDBENCH=85353
IDSOLVER=1654
FILE ID=node117/3264818-1304100568
RUNJOBID= node117-1304100568-29834
PBS_JOBID= 13170599
Free space on /tmp= 71680 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/jarvisalo05/mod2c-rand3bip-sat/mod2c-rand3bip-sat-250-2.shuffled-as.sat05-2534.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264818-1304100568/watcher-3264818-1304100568 -o /tmp/evaluation-result-3264818-1304100568/solver-3264818-1304100568 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264818-1304100568.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9a6126d13f733e360922c2ae93b7b6b9
RANDOM SEED=824226131

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      17441876 kB
Buffers:       2381484 kB
Cached:       12480924 kB
SwapCached:          0 kB
Active:        8148600 kB
Inactive:      6778020 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      17441876 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:              28 kB
Writeback:           0 kB
AnonPages:       63500 kB
Mapped:          14492 kB
Slab:           518772 kB
PageTables:       4684 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS:   187768 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71680 MiB
End job on node117 at 2011-04-29 20:29:42