Trace number 3251264

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.37 1210.03

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1327.5-n60-p15-s1351253.cnf
MD5SUMb23c52475e3d098cb9d5a92c025d6982
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark7.18991
Satisfiable
(Un)Satisfiability was proved
Number of variables3420
Number of clauses1120760
Sum of the clauses size2244820
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21120700
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data 

0.69/0.77	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251264-1304107233.cnf 
0.69/0.77	c Using SATTIMEOUT of 4800
0.69/0.77	c Using SATRAM of 15500
0.69/0.77	c Size of subset node = 32
0.69/0.77	c Variables = 3420
0.69/0.77	c Clauses = 1120760
0.69/0.77	c Literals = 2244820
0.69/0.77	c Reduced size:
0.69/0.77	c Variables = 3420
0.69/0.77	c Clauses = 1120760
0.69/0.77	c Literals = 2244820
0.69/0.77	c a2
1.29/1.38	c a1
3.99/4.05	c Independent sub-problems = 1
3.99/4.05	c Sub-problem with 3420 variables and 1120760 clauses
3.99/4.05	c using permutation method 4
3.99/4.05	c count = 1
3.99/4.05	c Ordering finished 3.910 seconds
3.99/4.05	c mapped 4.050 seconds
3.99/4.05	c permuted 4.050 seconds
3.99/4.05	c about to sort
4.09/4.14	c sorted
9.71/9.83	c clauses now 1120760
9.71/9.83	c init finished 4.500 seconds
9.71/9.83	c tree built 9.740 seconds
9.71/9.83	c 
9.71/9.83	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
9.71/9.83	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
9.71/9.83	c ................................ | ..........                    |                                  9.7         0.0             
9.71/9.83	c ................................ | ..........                  1 |                                  9.7         0.1             
9.71/9.83	c ................................ | ..........                  2 |                                  9.7         0.2             
9.71/9.83	c ................................ | ..........                  3 |                                  9.7         0.3             
9.71/9.83	c ................................ | ..........                  4 |                                  9.7         0.4             
9.71/9.83	c ................................ | ..........                  5 |                                  9.7         0.5             
9.71/9.84	c ................................ | ..........               1000 |              68                  9.8       102.4         3083
95.80/31.45	c ................................ | ..........            3338000 | 1552         261234             96.0     34770.8     14509360
192.01/55.50	c ................................ | ..........            6934000 | 3657         537004            192.0     36110.8     30083911
287.86/79.53	c ................................ | ..........           10284000 | 5497         806366            288.0     35705.9     39297832
383.72/103.57	c ................................ | ..........           13675000 | 7242         1072844           384.0     35611.1     50271263
479.99/127.61	c ................................ | *.........           17081000 | 9780         1337276           480.0     35585.4     58821085
575.82/151.65	c ................................ | *.........           20426000 | 13354        1594926           576.0     35460.6     65942445
671.66/175.68	c ................................ | *.........           23795000 | 16929        1854513           672.0     35409.2     72637595
767.90/199.72	c ................................ | *.........           27021000 | 20405        2107021           768.0     35183.6     79431372
863.74/223.76	c ................................ | *.........           30193000 | 25191        2357010           864.0     34944.8     83870769
959.98/247.80	c ................................ | **........           33351000 | 28549        2607226           960.0     34740.6     93592662
1055.83/271.84	c ................................ | **........           36521000 | 30427        2853405          1056.0     34584.0     96983013
1151.70/295.88	c ................................ | **........           39872000 | 33145        3110884          1152.0     34610.5    107925969
1247.91/319.93	c ................................ | **........           42985000 | 35519        3354439          1248.0     34442.6    111179520
1343.84/344.14	c ................................ | **........           46309000 | 37825        3601908          1344.0     34456.1    118114304
1439.68/368.18	c ................................ | ***.......           49527000 | 39613        3851602          1440.0     34393.3    128054713
1535.90/392.22	c ................................ | ***.......           52568000 | 43083        4090737          1536.0     34223.7    133972732
1631.73/416.26	c ................................ | ***.......           55687000 | 44846        4334427          1632.0     34121.9    136867699
1727.96/440.31	c ................................ | ***.......           58753000 | 47516        4570993          1728.0     34000.2    143483185
1823.81/464.34	c ................................ | ***.......           61697000 | 49662        4802126          1824.0     33825.1    147025267
1919.67/488.39	c ................................ | ****......           64786000 | 51459        5044127          1920.0     33742.7    153034495
2015.88/512.43	c ................................ | ****......           67587000 | 53287        5279231          2016.0     33525.0    158943488
2111.70/536.47	c ................................ | ****......           70690000 | 55406        5513639          2112.0     33470.6    161051940
2207.93/560.52	c ................................ | ****......           73681000 | 57709        5744141          2208.0     33369.6    163549645
2303.74/584.57	c ................................ | ****......           76812000 | 60014        5977093          2304.0     33338.3    170567687
2399.97/608.61	c ................................ | *****.....           79875000 | 62025        6207065          2400.0     33281.0    177145961
2495.81/632.65	c ................................ | *****.....           82976000 | 63696        6450214          2496.0     33243.5    184603548
2591.67/656.69	c ................................ | *****.....           86039000 | 65796        6691372          2592.0     33193.9    194062660
2687.89/680.73	c ................................ | *****.....           88945000 | 68615        6919110          2688.0     33089.5    200235362
2783.73/704.77	c ................................ | *****.....           91864000 | 70728        7151288          2784.0     32997.0    204010892
2879.95/728.81	c ................................ | ******....           94839000 | 72453        7383258          2880.0     32930.1    211626505
2975.79/752.85	c ................................ | ******....           97820000 | 74216        7609677          2976.0     32869.4    218350105
3073.21/777.22	c ................................ | ******....          100590000 | 75918        7842415          3073.3     32730.4    227617608
3167.81/800.94	c ................................ | ******....          103510000 | 77796        8071314          3168.0     32673.6    229297581
3263.69/824.98	c ................................ | ******....          106569000 | 80227        8304758          3264.0     32649.8    236419977
3359.90/849.03	c ................................ | *******...          109528000 | 82498        8530096          3360.0     32597.4    242343640
3455.71/873.07	c ................................ | *******...          112545000 | 85157        8761300          3456.0     32564.9    248716611
3551.97/897.10	c ................................ | *******...          115514000 | 87221        8985928          3552.0     32520.8    253966455
3647.80/921.15	c ................................ | *******...          118498000 | 89495        9215415          3648.0     32482.8    258959867
3743.64/945.18	c ................................ | *******...          121321000 | 91748        9440025          3744.0     32404.1    262494541
3839.89/969.23	c ................................ | ********..          124192000 | 93340        9659799          3840.0     32341.6    265429336
3935.73/993.27	c ................................ | ********..          127079000 | 94889        9886968          3936.0     32286.2    266572918
4031.96/1017.31	c ................................ | ********..          129796000 | 96528        10117956         4032.0     32191.5    274548133
4127.78/1041.35	c ................................ | ********..          132819000 | 98400        10343112         4128.0     32175.1    280915616
4223.61/1065.39	c ................................ | ********..          135780000 | 100963       10562904         4224.0     32144.9    286084642
4319.85/1089.44	c ................................ | *********.          138712000 | 102573       10785964         4320.0     32109.1    289021481
4416.00/1113.50	c ................................ | *********.          141496000 | 104236       11005132         4416.0     32041.6    293017828
4511.82/1137.55	c ................................ | *********.          144339000 | 105908       11222942         4512.0     31989.9    293017828
4607.67/1161.59	c ................................ | *********.          147201000 | 107913       11444611         4608.0     31944.6    293017828
4703.88/1185.63	c ................................ | *********.          149933000 | 109924       11662802         4704.0     31873.3    293017828
4799.73/1209.66	c CPU limit (4800) reached
4799.73/1209.66	c Time = 4800.00 seconds
4799.73/1209.66	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251264-1304107233/watcher-3251264-1304107233 -o /tmp/evaluation-result-3251264-1304107233/solver-3251264-1304107233 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251264-1304107233.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.86 7.82 7.97 2/175 26641
/proc/meminfo: memFree=30780596/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=9632 CPUtime=0 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 470 0 0 0 0 0 0 0 21 0 1 0 522323003 9863168 412 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 2408 413 147 36 0 313 0

[startup+0.0859921 s]
/proc/loadavg: 6.86 7.82 7.97 2/175 26641
/proc/meminfo: memFree=30780596/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=26056 CPUtime=0.07 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 4464 0 0 0 7 0 0 0 21 0 1 0 522323003 26681344 4406 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 6514 4408 148 36 0 4419 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 26056

[startup+0.100987 s]
/proc/loadavg: 6.86 7.82 7.97 2/175 26641
/proc/meminfo: memFree=30780596/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=28828 CPUtime=0.09 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 5204 0 0 0 9 0 0 0 21 0 1 0 522323003 29519872 5146 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 7207 5149 148 36 0 5112 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28828

[startup+0.300958 s]
/proc/loadavg: 6.86 7.82 7.97 2/175 26641
/proc/meminfo: memFree=30780596/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=69504 CPUtime=0.29 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 15071 0 0 0 28 1 0 0 22 0 1 0 522323003 71172096 15013 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 17376 15015 148 36 0 15281 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 69504

[startup+0.700889 s]
/proc/loadavg: 6.86 7.82 7.97 2/175 26641
/proc/meminfo: memFree=30780596/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=143664 CPUtime=0.69 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 33096 0 0 0 65 4 0 0 23 0 1 0 522323003 147111936 33038 33554432000 4194304 4338100 140737340540048 18446744073709551615 4209589 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 35916 33038 150 36 0 33821 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 143664

[startup+1.50075 s]
/proc/loadavg: 6.86 7.82 7.97 3/176 26642
/proc/meminfo: memFree=30539540/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=243036 CPUtime=1.49 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 53446 0 0 0 143 6 0 0 25 0 1 0 522323003 248868864 53178 33554432000 4194304 4338100 140737340540048 18446744073709551615 4228108 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 60759 53178 693 36 0 58664 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 243036

[startup+3.10047 s]
/proc/loadavg: 6.86 7.82 7.97 6/180 26646
/proc/meminfo: memFree=30451344/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=245228 CPUtime=3.09 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 78446 0 0 0 300 9 0 0 25 0 1 0 522323003 251113472 58881 33554432000 4194304 4338100 140737340540048 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26641/statm: 61307 58882 155 36 0 59212 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 245228

[startup+6.30087 s]
/proc/loadavg: 6.87 7.80 7.97 8/180 26646
/proc/meminfo: memFree=30268900/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=343780 CPUtime=6.21 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) R 26639 26641 26305 0 -1 4202496 105367 0 0 0 607 14 0 0 25 0 1 0 522323003 352030720 83613 33554432000 4194304 4338100 140737340540048 18446744073709551615 4278345 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/26641/statm: 85945 83613 164 36 0 83850 0
Current children cumulated CPU time (s) 6.21
Current children cumulated vsize (KiB) 343780

[startup+12.7008 s]
/proc/loadavg: 6.72 7.76 7.95 10/184 26650
/proc/meminfo: memFree=29715400/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=834316 CPUtime=21.19 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 185035 0 0 0 2056 63 0 0 25 0 5 0 522323003 854339584 163280 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 208579 163280 176 36 0 206484 0
[pid=26641/tid=26647] ppid=26639 vsize=834316 CPUtime=2.84 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 7087 0 0 0 283 1 0 0 24 0 5 0 522323987 854339584 163280 33554432000 4194304 4338100 140737340540048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=834316 CPUtime=2.85 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 3297 0 0 0 267 18 0 0 24 0 5 0 522323987 854339584 163280 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=834316 CPUtime=2.86 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 5209 0 0 0 285 1 0 0 23 0 5 0 522323987 854339584 163280 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676621094 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=834316 CPUtime=2.85 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 2569 0 0 0 265 20 0 0 23 0 5 0 522323987 854339584 163280 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 21.19
Current children cumulated vsize (KiB) 834316

[startup+25.5005 s]
/proc/loadavg: 7.22 7.82 7.97 9/184 26650
/proc/meminfo: memFree=29102360/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=1099352 CPUtime=72.29 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 251216 0 0 0 6960 269 0 0 25 0 5 0 522323003 1125736448 229461 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 274838 229461 176 36 0 272743 0
[pid=26641/tid=26647] ppid=26639 vsize=1099352 CPUtime=15.57 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 30319 0 0 0 1547 10 0 0 25 0 5 0 522323987 1125736448 229461 33554432000 4194304 4338100 140737340540048 18446744073709551615 4286120 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=1099352 CPUtime=15.65 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 12626 0 0 0 1451 114 0 0 25 0 5 0 522323987 1125736448 229461 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=1099352 CPUtime=15.63 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 29083 0 0 0 1556 7 0 0 25 0 5 0 522323987 1125736448 229461 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203689 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=1099352 CPUtime=15.66 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 12315 0 0 0 1451 115 0 0 25 0 5 0 522323987 1125736448 229461 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 72.29
Current children cumulated vsize (KiB) 1099352

[startup+51.1011 s]
/proc/loadavg: 7.56 7.85 7.97 10/184 26660
/proc/meminfo: memFree=28204148/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=1646776 CPUtime=174.44 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 382335 0 0 0 16744 700 0 0 25 0 5 0 522323003 1686298624 360580 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 411694 360580 176 36 0 409599 0
[pid=26641/tid=26647] ppid=26639 vsize=1646776 CPUtime=41.04 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 73125 0 0 0 4080 24 0 0 25 0 5 0 522323987 1686298624 360580 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203419 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=1646776 CPUtime=41.25 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 33482 0 0 0 3808 317 0 0 25 0 5 0 522323987 1686298624 360580 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=1646776 CPUtime=41.13 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 75805 0 0 0 4092 21 0 0 25 0 5 0 522323987 1686298624 360580 33554432000 4194304 4338100 140737340540048 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=1646776 CPUtime=41.24 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 33050 0 0 0 3809 315 0 0 25 0 5 0 522323987 1686298624 360580 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 174.44
Current children cumulated vsize (KiB) 1646776

[startup+102.306 s]
/proc/loadavg: 7.93 7.90 7.98 10/184 26661
/proc/meminfo: memFree=26637248/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=2423320 CPUtime=378.95 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 570608 0 0 0 36317 1578 0 0 25 0 5 0 522323003 2481479680 548853 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 605830 548853 176 36 0 603735 0
[pid=26641/tid=26647] ppid=26639 vsize=2423320 CPUtime=91.96 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 145650 0 0 0 9146 50 0 0 25 0 5 0 522323987 2481479680 548853 33554432000 4194304 4338100 140737340540048 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=2423320 CPUtime=92.46 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 56956 0 0 0 8512 734 0 0 25 0 5 0 522323987 2481479680 548853 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=2423320 CPUtime=92.29 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 145451 0 0 0 9190 39 0 0 25 0 5 0 522323987 2481479680 548853 33554432000 4194304 4338100 140737340540048 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=2423320 CPUtime=92.45 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 55678 0 0 0 8514 731 0 0 25 0 5 0 522323987 2481479680 548853 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 378.95
Current children cumulated vsize (KiB) 2423320

[startup+162.301 s]
/proc/loadavg: 7.83 7.88 7.97 9/184 26663
/proc/meminfo: memFree=25045680/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=3097576 CPUtime=618.55 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 729102 0 0 0 59289 2566 0 0 25 0 5 0 522323003 3171917824 707347 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 774394 707347 176 36 0 772299 0
[pid=26641/tid=26647] ppid=26639 vsize=3097576 CPUtime=151.63 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 217740 0 0 0 15082 81 0 0 25 0 5 0 522323987 3171917824 707347 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203401 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=3097576 CPUtime=152.44 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 68777 0 0 0 14038 1206 0 0 25 0 5 0 522323987 3171917824 707347 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=3097576 CPUtime=152.26 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 200005 0 0 0 15174 52 0 0 25 0 5 0 522323987 3171917824 707347 33554432000 4194304 4338100 140737340540048 18446744073709551615 4286342 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=3097576 CPUtime=152.44 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 75707 0 0 0 14040 1204 0 0 25 0 5 0 522323987 3171917824 707347 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 618.55
Current children cumulated vsize (KiB) 3097576

[startup+222.3 s]
/proc/loadavg: 7.98 7.92 7.98 9/184 26665
/proc/meminfo: memFree=23860448/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=3547852 CPUtime=858.14 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 840564 0 0 0 82219 3595 0 0 25 0 5 0 522323003 3633000448 818809 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 886963 818809 176 36 0 884868 0
[pid=26641/tid=26647] ppid=26639 vsize=3547852 CPUtime=211.29 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 279213 0 0 0 21018 111 0 0 25 0 5 0 522323987 3633000448 818809 33554432000 4194304 4338100 140737340540048 18446744073709551615 4253990 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=3547852 CPUtime=212.44 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 71077 0 0 0 19544 1700 0 0 25 0 5 0 522323987 3633000448 818809 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=3547852 CPUtime=212.2 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 246341 0 0 0 21156 64 0 0 25 0 5 0 522323987 3633000448 818809 33554432000 4194304 4338100 140737340540048 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=3547852 CPUtime=212.43 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 77060 0 0 0 19546 1697 0 0 25 0 5 0 522323987 3633000448 818809 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 858.14
Current children cumulated vsize (KiB) 3547852

[startup+282.301 s]
/proc/loadavg: 8.03 7.94 7.98 9/184 26666
/proc/meminfo: memFree=22831572/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=4108300 CPUtime=1097.76 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 981986 0 0 0 105212 4564 0 0 25 0 5 0 522323003 4206899200 960231 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 1027075 960232 176 36 0 1024980 0
[pid=26641/tid=26647] ppid=26639 vsize=4108300 CPUtime=271.19 cores=1,3,5,7

################
# More data... #
################

/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 880974 0 0 0 74659 368 0 0 25 0 5 0 522323987 8420691968 1931291 33554432000 4194304 4338100 140737340540048 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=8223332 CPUtime=750.39 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 110673 0 0 0 69063 5976 0 0 25 0 5 0 522323987 8420691968 1931291 33554432000 4194304 4338100 140737340540048 18446744073709551615 4288258 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=8223332 CPUtime=751.8 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 675293 0 0 0 75006 174 0 0 25 0 5 0 522323987 8420691968 1931291 33554432000 4194304 4338100 140737340540048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=8223332 CPUtime=751.51 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 119233 0 0 0 69201 5950 0 0 25 0 5 0 522323987 8420691968 1931291 33554432000 4194304 4338100 140737340540048 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3013.75
Current children cumulated vsize (KiB) 8223332

[startup+822.301 s]
/proc/loadavg: 8.06 8.06 8.02 9/184 26684
/proc/meminfo: memFree=13649376/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=8598188 CPUtime=3253.28 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2058606 0 0 0 311791 13537 0 0 25 0 5 0 522323003 8804544512 2036851 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2149547 2036851 176 36 0 2147452 0
[pid=26641/tid=26647] ppid=26639 vsize=8598188 CPUtime=809.89 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 948228 0 0 0 80593 396 0 0 25 0 5 0 522323987 8804544512 2036851 33554432000 4194304 4338100 140737340540048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=8598188 CPUtime=810.35 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 116431 0 0 0 74561 6474 0 0 25 0 5 0 522323987 8804544512 2036851 33554432000 4194304 4338100 140737340540048 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=8598188 CPUtime=811.78 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 703815 0 0 0 80997 181 0 0 25 0 5 0 522323987 8804544512 2036851 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203533 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=8598188 CPUtime=811.47 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 123259 0 0 0 74685 6462 0 0 25 0 5 0 522323987 8804544512 2036851 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3253.28
Current children cumulated vsize (KiB) 8598188

[startup+882.3 s]
/proc/loadavg: 8.05 8.06 8.02 9/184 26685
/proc/meminfo: memFree=12841972/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=9182260 CPUtime=3492.84 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2182688 0 0 0 334742 14542 0 0 25 0 5 0 522323003 9402634240 2160933 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2295565 2160933 176 36 0 2293470 0
[pid=26641/tid=26647] ppid=26639 vsize=9182260 CPUtime=869.71 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 1013559 0 0 0 86550 421 0 0 25 0 5 0 522323987 9402634240 2160933 33554432000 4194304 4338100 140737340540048 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=9182260 CPUtime=870.33 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 123373 0 0 0 80075 6958 0 0 25 0 5 0 522323987 9402634240 2160933 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664204014 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=9182260 CPUtime=871.71 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 749980 0 0 0 86978 193 0 0 25 0 5 0 522323987 9402634240 2160933 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676622480 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=9182260 CPUtime=871.29 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 128903 0 0 0 80183 6946 0 0 25 0 5 0 522323987 9402634240 2160933 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203329 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3492.84
Current children cumulated vsize (KiB) 9182260

[startup+942.301 s]
/proc/loadavg: 8.24 8.11 8.03 9/184 26686
/proc/meminfo: memFree=11911560/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=9514788 CPUtime=3732.47 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2267134 0 0 0 357727 15520 0 0 25 0 5 0 522323003 9743142912 2245379 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2378697 2245379 176 36 0 2376602 0
[pid=26641/tid=26647] ppid=26639 vsize=9514788 CPUtime=929.48 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 1060398 0 0 0 92503 445 0 0 25 0 5 0 522323987 9743142912 2245379 33554432000 4194304 4338100 140737340540048 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=9514788 CPUtime=930.31 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 123375 0 0 0 85607 7424 0 0 25 0 5 0 522323987 9743142912 2245379 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=9514788 CPUtime=931.68 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 787301 0 0 0 92965 203 0 0 25 0 5 0 522323987 9743142912 2245379 33554432000 4194304 4338100 140737340540048 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=9514788 CPUtime=931.21 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 129187 0 0 0 85697 7424 0 0 25 0 5 0 522323987 9743142912 2245379 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3732.47
Current children cumulated vsize (KiB) 9514788

[startup+1002.3 s]
/proc/loadavg: 8.23 8.12 8.04 9/184 26688
/proc/meminfo: memFree=11111480/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=9716660 CPUtime=3972.05 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2327208 0 0 0 380695 16510 0 0 25 0 5 0 522323003 9949859840 2305453 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2429165 2305453 176 36 0 2427070 0
[pid=26641/tid=26647] ppid=26639 vsize=9716660 CPUtime=989.12 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 1093250 0 0 0 98446 466 0 0 25 0 5 0 522323987 9949859840 2305453 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203689 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=9716660 CPUtime=990.28 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 123375 0 0 0 91123 7905 0 0 25 0 5 0 522323987 9949859840 2305453 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=9716660 CPUtime=991.67 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 814522 0 0 0 98956 211 0 0 25 0 5 0 522323987 9949859840 2305453 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203710 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=9716660 CPUtime=991.19 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 129188 0 0 0 91215 7904 0 0 25 0 5 0 522323987 9949859840 2305453 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3972.05
Current children cumulated vsize (KiB) 9716660

[startup+1062.3 s]
/proc/loadavg: 8.15 8.11 8.04 9/184 26690
/proc/meminfo: memFree=10249400/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=10317012 CPUtime=4211.64 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2454897 0 0 0 403657 17507 0 0 25 0 5 0 522323003 10564620288 2433142 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2579253 2433142 176 36 0 2577158 0
[pid=26641/tid=26647] ppid=26639 vsize=10317012 CPUtime=1049.06 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 1174520 0 0 0 104407 499 0 0 25 0 5 0 522323987 10564620288 2433142 33554432000 4194304 4338100 140737340540048 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=10317012 CPUtime=1050.27 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 123376 0 0 0 96643 8384 0 0 25 0 5 0 522323987 10564620288 2433142 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664203373 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=10317012 CPUtime=1051.35 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 860938 0 0 0 104914 221 0 0 25 0 5 0 522323987 10564620288 2433142 33554432000 4194304 4338100 140737340540048 18446744073709551615 274664204002 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=10317012 CPUtime=1051.19 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 129190 0 0 0 96739 8380 0 0 25 0 5 0 522323987 10564620288 2433142 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4211.64
Current children cumulated vsize (KiB) 10317012

[startup+1122.3 s]
/proc/loadavg: 8.05 8.09 8.03 9/184 26692
/proc/meminfo: memFree=9728068/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=10604060 CPUtime=4451.15 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2527011 0 0 0 426650 18465 0 0 25 0 5 0 522323003 10858557440 2505256 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2651015 2505256 176 36 0 2648920 0
[pid=26641/tid=26647] ppid=26639 vsize=10604060 CPUtime=1109.01 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 1218494 0 0 0 110381 520 0 0 25 0 5 0 522323987 10858557440 2505256 33554432000 4194304 4338100 140737340540048 18446744073709551615 4241581 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=10604060 CPUtime=1110.24 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 128147 0 0 0 102176 8848 0 0 25 0 5 0 522323987 10858557440 2505256 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=10604060 CPUtime=1111 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 880837 0 0 0 110874 226 0 0 25 0 5 0 522323987 10858557440 2505256 33554432000 4194304 4338100 140737340540048 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=10604060 CPUtime=1111.13 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 132660 0 0 0 102265 8848 0 0 25 0 5 0 522323987 10858557440 2505256 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4451.15
Current children cumulated vsize (KiB) 10604060

[startup+1182.3 s]
/proc/loadavg: 8.08 8.08 8.03 9/184 26693
/proc/meminfo: memFree=9290552/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=10767900 CPUtime=4690.7 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) S 26639 26641 26305 0 -1 4202496 2552813 0 0 0 449698 19372 0 0 25 0 5 0 522323003 11026329600 2531058 33554432000 4194304 4338100 140737340540048 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 2691975 2531058 176 36 0 2689880 0
[pid=26641/tid=26647] ppid=26639 vsize=10767900 CPUtime=1168.97 cores=1,3,5,7
/proc/26641/task/26647/stat : 26647 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 1221253 0 0 0 116369 528 0 0 25 0 5 0 522323987 11026329600 2531058 33554432000 4194304 4338100 140737340540048 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26641/tid=26648] ppid=26639 vsize=10767900 CPUtime=1170.22 cores=1,3,5,7
/proc/26641/task/26648/stat : 26648 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 137765 0 0 0 107725 9297 0 0 25 0 5 0 522323987 11026329600 2531058 33554432000 4194304 4338100 140737340540048 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26641/tid=26649] ppid=26639 vsize=10767900 CPUtime=1170.66 cores=1,3,5,7
/proc/26641/task/26649/stat : 26649 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 884312 0 0 0 116839 227 0 0 25 0 5 0 522323987 11026329600 2531058 33554432000 4194304 4338100 140737340540048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=10767900 CPUtime=1171.08 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4202560 142610 0 0 0 107811 9297 0 0 25 0 5 0 522323987 11026329600 2531058 33554432000 4194304 4338100 140737340540048 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4690.7
Current children cumulated vsize (KiB) 10767900



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1209.7 s]
/proc/loadavg: 8.04 8.08 8.03 6/181 26694
/proc/meminfo: memFree=9764600/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=0 CPUtime=4800.04 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) Z 26639 26641 26305 0 -1 4203548 2557269 0 0 0 460209 19795 0 0 18 0 2 0 522323003 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 0 0 0 0 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=0 CPUtime=1198.47 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4203588 143855 0 0 0 110337 9510 0 0 25 0 2 0 522323987 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1209.81 s]
/proc/loadavg: 8.04 8.08 8.03 6/181 26694
/proc/meminfo: memFree=9764600/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=0 CPUtime=4800.14 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) Z 26639 26641 26305 0 -1 4203548 2557269 0 0 0 460209 19805 0 0 18 0 2 0 522323003 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 0 0 0 0 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=0 CPUtime=1198.58 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4203588 143855 0 0 0 110337 9521 0 0 25 0 2 0 522323987 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.14
Current children cumulated vsize (KiB) 0

[startup+1210.01 s]
/proc/loadavg: 8.04 8.08 8.03 3/179 26696
/proc/meminfo: memFree=18907364/32951132 swapFree=67111356/67111528
[pid=26641] ppid=26639 vsize=0 CPUtime=4800.34 cores=1,3,5,7
/proc/26641/stat : 26641 (ttsth-5-4) Z 26639 26641 26305 0 -1 4203548 2557269 0 0 0 460209 19825 0 0 18 0 2 0 522323003 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26641/statm: 0 0 0 0 0 0 0
[pid=26641/tid=26650] ppid=26639 vsize=0 CPUtime=1198.77 cores=1,3,5,7
/proc/26641/task/26650/stat : 26650 (ttsth-5-4) R 26639 26641 26305 0 -1 4203588 143855 0 0 0 110337 9540 0 0 25 0 2 0 522323987 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1210.03
CPU time (s): 4800.37
CPU user time (s): 4602.09
CPU system time (s): 198.273
CPU usage (%): 396.716
Max. virtual memory (cumulated for all children) (KiB): 10774412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4602.09
system time used= 198.273
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2557269
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10483
involuntary context switches= 34167

runsolver used 2.03069 second user time and 4.5753 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 22:00:34
IDJOB=3251264
IDBENCH=88217
IDSOLVER=1654
FILE ID=node131/3251264-1304107233
RUNJOBID= node131-1304107233-26622
PBS_JOBID= 13170641
Free space on /tmp= 71368 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1327.5-n60-p15-s1351253.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251264-1304107233/watcher-3251264-1304107233 -o /tmp/evaluation-result-3251264-1304107233/solver-3251264-1304107233 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251264-1304107233.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b23c52475e3d098cb9d5a92c025d6982
RANDOM SEED=2120916740

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30781744 kB
Buffers:        204252 kB
Cached:        1532544 kB
SwapCached:        168 kB
Active:         225432 kB
Inactive:      1637348 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30781744 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:           19460 kB
Writeback:           0 kB
AnonPages:      125736 kB
Mapped:          15488 kB
Slab:           243204 kB
PageTables:       4508 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   269692 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71368 MiB
End job on node131 at 2011-04-29 22:20:45