Trace number 3261914

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.4 1202.38

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-21_399.cnf
MD5SUM693b5fe291cecff45feff3be98a908cf
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark8.87165
Satisfiable
(Un)Satisfiability was proved
Number of variables200
Number of clauses21294
Sum of the clauses size93307
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2265
Number of clauses of size 319364
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 51665

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261914-1304090475.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 200
0.00/0.02	c Clauses = 21294
0.00/0.02	c Literals = 93307
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 200
0.00/0.02	c Clauses = 21294
0.00/0.02	c Literals = 93307
0.00/0.02	c a2
0.09/0.12	c a1
0.19/0.29	c Independent sub-problems = 1
0.19/0.29	c Sub-problem with 200 variables and 21294 clauses
0.19/0.29	c using permutation method 4
0.19/0.29	c count = 1
0.19/0.29	c Ordering finished 0.270 seconds
0.19/0.29	c mapped 0.280 seconds
0.19/0.29	c permuted 0.280 seconds
0.19/0.29	c about to sort
0.19/0.29	c sorted
0.29/0.35	c clauses now 21294
0.29/0.35	c init finished 0.290 seconds
0.29/0.35	c tree built 0.350 seconds
0.29/0.35	c 
0.29/0.35	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.35	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.35	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.35	c ................................ | ..........                  1 |                                  0.3         2.9             
0.29/0.35	c ................................ | ..........                  2 |                                  0.3         5.7             
0.29/0.35	c ................................ | ..........                  3 |                                  0.3         8.6             
0.29/0.35	c ................................ | ..........                  4 |                                  0.3        11.4             
0.29/0.35	c ................................ | ..........                  5 |                                  0.3        14.3             
3.33/1.16	c ................................ | ..........               1000 | 35           355                 0.4      2381.0        32731
95.99/24.34	c ................................ | ..........             422000 | 41465        166970             96.2      4387.6     23476043
191.87/48.35	c ................................ | ..........             654000 | 66420        253318            192.1      3405.4     34854896
288.12/72.42	c ................................ | ..........             843000 | 92424        324036            288.2      2924.7     44665983
384.37/96.58	c ................................ | ..........             992000 | 107934       390911            384.7      2578.6     48378703
480.23/120.55	c ................................ | *.........            1115000 | 122242       447072            480.4      2320.7     50831061
576.50/144.60	c ................................ | *.........            1227000 | 134975       498593            576.5      2128.3     53734274
672.37/168.67	c ................................ | *.........            1334000 | 146526       547198            672.7      1983.2     57670078
767.85/192.55	c ................................ | *.........            1425000 | 160074       589796            768.0      1855.4     61712303
864.52/216.75	c ................................ | *.........            1501000 | 172618       626082            864.7      1735.8     64133239
959.98/240.61	c ................................ | **........            1573000 | 185634       660112            960.0      1638.5     67774411
1056.24/264.75	c ................................ | **........            1645000 | 197535       694783           1056.5      1557.1     72390361
1153.31/289.04	c ................................ | **........            1717000 | 207921       727838           1153.5      1488.5     76207499
1248.38/312.82	c ................................ | **........            1781000 | 217770       756783           1248.5      1426.5     79868667
1344.26/336.84	c ................................ | **........            1845000 | 226383       786024           1344.4      1372.3     83637289
1440.92/361.04	c ................................ | ***.......            1910000 | 235656       817776           1441.1      1325.4     88138720
1536.81/385.09	c ................................ | ***.......            1967000 | 243986       849260           1537.2      1279.6     92729641
1632.67/409.06	c ................................ | ***.......            2012000 | 251596       879101           1632.9      1232.1     96997599
1729.75/433.33	c ................................ | ***.......            2054000 | 259486       903187           1729.9      1187.4    101060540
1824.81/457.14	c ................................ | ***.......            2105000 | 267783       926223           1825.0      1153.4    104281037
1920.68/481.18	c ................................ | ****......            2157000 | 275915       947791           1921.0      1122.8    108166585
2016.15/505.04	c ................................ | ****......            2206000 | 282626       972178           2016.3      1094.1    111361040
2112.81/529.26	c ................................ | ****......            2259000 | 289529       998239           2113.1      1069.1    114411709
2209.48/553.42	c ................................ | ****......            2313000 | 295580       1024099          2209.6      1046.8    117619674
2304.56/577.22	c ................................ | ****......            2364000 | 301806       1047595          2304.7      1025.8    120379871
2401.22/601.44	c ................................ | *****.....            2416000 | 308129       1070873          2401.4      1006.1    123268920
2496.69/625.30	c ................................ | *****.....            2464000 | 315019       1092333          2496.7       986.9    126022652
2592.16/649.29	c ................................ | *****.....            2515000 | 320924       1118438          2592.5       970.1    128880645
2688.05/673.27	c ................................ | *****.....            2567000 | 326632       1144843          2688.3       954.9    131731411
2784.72/697.45	c ................................ | *****.....            2623000 | 332862       1169865          2784.9       941.9    134788470
2880.17/721.34	c ................................ | ******....            2673000 | 340451       1186751          2880.3       928.0    137499527
2976.84/745.57	c ................................ | ******....            2720000 | 348144       1204075          2977.1       913.6    140087775
3072.70/769.54	c ................................ | ******....            2764000 | 355533       1220054          3072.9       899.5    143159854
3169.37/793.75	c ................................ | ******....            2814000 | 362173       1236963          3169.6       887.8    146344468
3265.24/817.75	c ................................ | ******....            2861000 | 369042       1253199          3265.5       876.1    147341182
3361.10/841.78	c ................................ | *******...            2911000 | 376489       1269611          3361.4       866.0    147510758
3457.77/865.98	c ................................ | *******...            2958000 | 384401       1287769          3458.1       855.4    148097001
3552.04/889.51	c ................................ | *******...            3003000 | 390820       1306931          3552.1       845.4    148644241
3649.93/914.03	c ................................ | *******...            3049000 | 397296       1325431          3650.0       835.3    149336791
3745.38/937.97	c ................................ | *******...            3094000 | 403551       1344402          3745.7       826.0    151099678
3840.45/961.77	c ................................ | ********..            3138000 | 410063       1361685          3840.7       817.0    152565123
3937.91/986.11	c ................................ | ********..            3180000 | 417106       1376502          3937.9       807.5    154889793
4032.61/1009.84	c ................................ | ********..            3221000 | 423470       1392467          4032.8       798.7    156698112
4128.07/1033.78	c ................................ | ********..            3264000 | 429191       1406720          4128.4       790.6    159050697
4225.55/1058.14	c ................................ | ********..            3302000 | 434882       1422463          4225.7       781.4    161384234
4321.42/1082.34	c ................................ | *********.            3338000 | 440914       1438011          4321.6       772.4    163571059
4416.07/1106.02	c ................................ | *********.            3372000 | 445918       1454691          4416.2       763.6    165557481
4513.95/1130.53	c ................................ | *********.            3406000 | 450973       1470072          4514.1       754.5    167785423
4608.22/1154.15	c ................................ | *********.            3439000 | 455572       1485879          4608.4       746.2    170184277
4706.49/1178.77	c ................................ | *********.            3466000 | 459694       1499823          4706.8       736.4    172576420
4799.96/1202.14	c CPU limit (4800) reached
4799.96/1202.14	c Time = 4800.16 seconds
4799.96/1202.14	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3261914-1304090475/watcher-3261914-1304090475 -o /tmp/evaluation-result-3261914-1304090475/solver-3261914-1304090475 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261914-1304090475.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.18 8.11 7.97 5/181 21057
/proc/meminfo: memFree=28515316/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) R 21055 21057 20765 0 -1 4202496 442 0 0 0 0 0 0 0 25 0 1 0 599278126 9768960 385 33554432000 4194304 4338100 140737196760880 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21057/statm: 2385 386 146 36 0 290 0

[startup+0.0993231 s]
/proc/loadavg: 8.18 8.11 7.97 5/181 21057
/proc/meminfo: memFree=28515316/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=23864 CPUtime=0.09 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) R 21055 21057 20765 0 -1 4202496 3381 0 0 0 9 0 0 0 25 0 1 0 599278126 24436736 3225 33554432000 4194304 4338100 140737196760880 18446744073709551615 260019791053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21057/statm: 5966 3225 317 36 0 3871 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23864

[startup+0.100302 s]
/proc/loadavg: 8.18 8.11 7.97 5/181 21057
/proc/meminfo: memFree=28515316/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=23864 CPUtime=0.09 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) R 21055 21057 20765 0 -1 4202496 3381 0 0 0 9 0 0 0 25 0 1 0 599278126 24436736 3225 33554432000 4194304 4338100 140737196760880 18446744073709551615 4206980 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21057/statm: 5966 3225 317 36 0 3871 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23864

[startup+0.300285 s]
/proc/loadavg: 8.18 8.11 7.97 5/181 21057
/proc/meminfo: memFree=28515316/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=16672 CPUtime=0.29 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) R 21055 21057 20765 0 -1 4202496 4587 0 0 0 29 0 0 0 25 0 1 0 599278126 17072128 2172 33554432000 4194304 4338100 140737196760880 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21057/statm: 4201 2179 164 36 0 2106 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 16672

[startup+0.700149 s]
/proc/loadavg: 8.18 8.11 7.97 5/181 21057
/proc/meminfo: memFree=28515316/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=165664 CPUtime=1.72 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 13640 0 0 0 170 2 0 0 25 0 5 0 599278126 169639936 11182 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 41416 11182 176 36 0 39321 0
Current children cumulated CPU time (s) 1.72
Current children cumulated vsize (KiB) 165664

[startup+1.501 s]
/proc/loadavg: 8.24 8.12 7.98 10/186 21062
/proc/meminfo: memFree=28453408/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=272392 CPUtime=4.91 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 23829 0 0 0 488 3 0 0 25 0 5 0 599278126 278929408 21371 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 68098 21372 176 36 0 66003 0
[pid=21057/tid=21059] ppid=21055 vsize=272392 CPUtime=1.13 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 3846 0 0 0 113 0 0 0 19 0 5 0 599278162 278929408 21373 33554432000 4194304 4338100 140737196760880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=272392 CPUtime=1.13 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 4198 0 0 0 113 0 0 0 25 0 5 0 599278162 278929408 21373 33554432000 4194304 4338100 140737196760880 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=272392 CPUtime=1.13 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 4137 0 0 0 113 0 0 0 25 0 5 0 599278162 278929408 21373 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032208590 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=272392 CPUtime=1.13 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) S 21055 21057 20765 0 -1 4202560 3928 0 0 0 113 0 0 0 19 0 5 0 599278162 278929408 21374 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032221556 0 0 4096 0 18446604443346815936 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.91
Current children cumulated vsize (KiB) 272392

[startup+3.10071 s]
/proc/loadavg: 8.24 8.12 7.98 9/186 21062
/proc/meminfo: memFree=28392896/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=348256 CPUtime=11.31 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 43227 0 0 0 1124 7 0 0 25 0 5 0 599278126 356614144 40769 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 87064 40769 176 36 0 84969 0
[pid=21057/tid=21059] ppid=21055 vsize=348256 CPUtime=2.73 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 8677 0 0 0 272 1 0 0 23 0 5 0 599278162 356614144 40770 33554432000 4194304 4338100 140737196760880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=348256 CPUtime=2.73 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 9050 0 0 0 272 1 0 0 25 0 5 0 599278162 356614144 40770 33554432000 4194304 4338100 140737196760880 18446744073709551615 4256280 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=348256 CPUtime=2.72 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 10368 0 0 0 271 1 0 0 25 0 5 0 599278162 356614144 40771 33554432000 4194304 4338100 140737196760880 18446744073709551615 4256420 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=348256 CPUtime=2.74 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 7411 0 0 0 273 1 0 0 23 0 5 0 599278162 356614144 40771 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.31
Current children cumulated vsize (KiB) 348256

[startup+6.30112 s]
/proc/loadavg: 8.24 8.12 7.98 9/186 21062
/proc/meminfo: memFree=28245688/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=480292 CPUtime=24.09 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 76175 0 0 0 2396 13 0 0 25 0 5 0 599278126 491819008 73717 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 120073 73717 176 36 0 117978 0
[pid=21057/tid=21059] ppid=21055 vsize=480292 CPUtime=5.93 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 17334 0 0 0 591 2 0 0 25 0 5 0 599278162 491819008 73717 33554432000 4194304 4338100 140737196760880 18446744073709551615 4255960 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=480292 CPUtime=5.93 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 16441 0 0 0 590 3 0 0 25 0 5 0 599278162 491819008 73717 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=480292 CPUtime=5.9 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 19510 0 0 0 587 3 0 0 25 0 5 0 599278162 491819008 73717 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=480292 CPUtime=5.94 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 15168 0 0 0 592 2 0 0 25 0 5 0 599278162 491819008 73717 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032210075 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.09
Current children cumulated vsize (KiB) 480292

[startup+12.7009 s]
/proc/loadavg: 8.28 8.13 7.98 9/186 21062
/proc/meminfo: memFree=28025608/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=668896 CPUtime=49.65 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 123801 0 0 0 4942 23 0 0 25 0 5 0 599278126 684949504 121343 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 167224 121343 176 36 0 165129 0
[pid=21057/tid=21059] ppid=21055 vsize=668896 CPUtime=12.34 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 28975 0 0 0 1229 5 0 0 25 0 5 0 599278162 684949504 121343 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032210179 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=668896 CPUtime=12.33 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 27649 0 0 0 1228 5 0 0 25 0 5 0 599278162 684949504 121343 33554432000 4194304 4338100 140737196760880 18446744073709551615 4285374 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=668896 CPUtime=12.27 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 32659 0 0 0 1221 6 0 0 25 0 5 0 599278162 684949504 121343 33554432000 4194304 4338100 140737196760880 18446744073709551615 4287786 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=668896 CPUtime=12.33 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 26796 0 0 0 1229 4 0 0 25 0 5 0 599278162 684949504 121343 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 49.65
Current children cumulated vsize (KiB) 668896

[startup+25.5006 s]
/proc/loadavg: 8.24 8.13 7.98 9/186 21062
/proc/meminfo: memFree=27686344/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=976496 CPUtime=100.77 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 199987 0 0 0 10039 38 0 0 25 0 5 0 599278126 999931904 197529 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 244124 197529 176 36 0 242029 0
[pid=21057/tid=21059] ppid=21055 vsize=976496 CPUtime=25.13 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 49488 0 0 0 2503 10 0 0 25 0 5 0 599278162 999931904 197529 33554432000 4194304 4338100 140737196760880 18446744073709551615 4229577 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=976496 CPUtime=25.13 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 47811 0 0 0 2504 9 0 0 25 0 5 0 599278162 999931904 197529 33554432000 4194304 4338100 140737196760880 18446744073709551615 4242480 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=976496 CPUtime=24.99 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 50553 0 0 0 2491 8 0 0 25 0 5 0 599278162 999931904 197530 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=976496 CPUtime=25.13 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 44413 0 0 0 2505 8 0 0 25 0 5 0 599278162 999931904 197530 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 100.77
Current children cumulated vsize (KiB) 976496

[startup+51.1009 s]
/proc/loadavg: 8.14 8.11 7.98 9/186 21063
/proc/meminfo: memFree=27265808/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=1342592 CPUtime=203.04 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 292256 0 0 0 20249 55 0 0 25 0 5 0 599278126 1374814208 289798 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 335648 289799 176 36 0 333553 0
[pid=21057/tid=21059] ppid=21055 vsize=1342592 CPUtime=50.73 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 73247 0 0 0 5059 14 0 0 25 0 5 0 599278162 1374814208 289799 33554432000 4194304 4338100 140737196760880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=1342592 CPUtime=50.73 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 70716 0 0 0 5060 13 0 0 25 0 5 0 599278162 1374814208 289799 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=1342592 CPUtime=50.46 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 73851 0 0 0 5033 13 0 0 25 0 5 0 599278162 1374814208 289799 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=1342592 CPUtime=50.73 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 66721 0 0 0 5061 12 0 0 25 0 5 0 599278162 1374814208 289799 33554432000 4194304 4338100 140737196760880 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 203.04
Current children cumulated vsize (KiB) 1342592

[startup+102.306 s]
/proc/loadavg: 8.18 8.13 7.99 9/186 21064
/proc/meminfo: memFree=26649352/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=1862792 CPUtime=407.56 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 412720 0 0 0 40674 82 0 0 25 0 5 0 599278126 1907499008 410262 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 465698 410262 176 36 0 463603 0
[pid=21057/tid=21059] ppid=21055 vsize=1862792 CPUtime=101.93 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 102621 0 0 0 10171 22 0 0 25 0 5 0 599278162 1907499008 410262 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=1862792 CPUtime=101.93 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 101510 0 0 0 10172 21 0 0 25 0 5 0 599278162 1907499008 410262 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=1862792 CPUtime=101.37 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 104562 0 0 0 10119 18 0 0 25 0 5 0 599278162 1907499008 410262 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=1862792 CPUtime=101.93 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 96305 0 0 0 10175 18 0 0 25 0 5 0 599278162 1907499008 410262 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 407.56
Current children cumulated vsize (KiB) 1862792

[startup+162.3 s]
/proc/loadavg: 8.18 8.13 8.00 9/186 21066
/proc/meminfo: memFree=26270096/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=2105100 CPUtime=647.2 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 476083 0 0 0 64623 97 0 0 25 0 5 0 599278126 2155622400 473625 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 526275 473625 176 36 0 524180 0
[pid=21057/tid=21059] ppid=21055 vsize=2105100 CPUtime=161.83 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 105494 0 0 0 16160 23 0 0 25 0 5 0 599278162 2155622400 473625 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=2105100 CPUtime=161.92 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 131980 0 0 0 16163 29 0 0 25 0 5 0 599278162 2155622400 473625 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=2105100 CPUtime=161.32 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 106295 0 0 0 16113 19 0 0 25 0 5 0 599278162 2155622400 473625 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=2105100 CPUtime=161.74 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 124592 0 0 0 16150 24 0 0 25 0 5 0 599278162 2155622400 473625 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 647.2
Current children cumulated vsize (KiB) 2105100


################
# More data... #
################

/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 282654 0 0 0 70109 61 0 0 25 0 5 0 599278162 4690956288 1094306 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2804.27
Current children cumulated vsize (KiB) 4581012

[startup+762.307 s]
/proc/loadavg: 8.10 8.10 8.02 9/186 21114
/proc/meminfo: memFree=23175476/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=4794376 CPUtime=3043.96 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1150145 0 0 0 304141 255 0 0 25 0 5 0 599278126 4909441024 1147687 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1198594 1147687 176 36 0 1196499 0
[pid=21057/tid=21059] ppid=21055 vsize=4794376 CPUtime=759.5 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 218045 0 0 0 75898 52 0 0 25 0 5 0 599278162 4909441024 1147687 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=4794376 CPUtime=761.04 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 339858 0 0 0 76028 76 0 0 25 0 5 0 599278162 4909441024 1147687 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=4794376 CPUtime=761.34 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 286132 0 0 0 76074 60 0 0 25 0 5 0 599278162 4909441024 1147687 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=4794376 CPUtime=761.7 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 298388 0 0 0 76106 64 0 0 25 0 5 0 599278162 4909441024 1147687 33554432000 4194304 4338100 140737196760880 18446744073709551615 4286116 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3043.96
Current children cumulated vsize (KiB) 4794376

[startup+822.301 s]
/proc/loadavg: 8.12 8.11 8.02 9/186 21116
/proc/meminfo: memFree=22649408/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5008628 CPUtime=3283.61 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1197671 0 0 0 328095 266 0 0 25 0 5 0 599278126 5128835072 1195213 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1252157 1195213 176 36 0 1250062 0
[pid=21057/tid=21059] ppid=21055 vsize=5008628 CPUtime=819.49 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 225018 0 0 0 81895 54 0 0 25 0 5 0 599278162 5128835072 1195213 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5008628 CPUtime=820.71 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 352511 0 0 0 81992 79 0 0 25 0 5 0 599278162 5128835072 1195213 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5008628 CPUtime=821.33 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 297462 0 0 0 82071 62 0 0 25 0 5 0 599278162 5128835072 1195213 33554432000 4194304 4338100 140737196760880 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5008628 CPUtime=821.69 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 314958 0 0 0 82101 68 0 0 25 0 5 0 599278162 5128835072 1195213 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3283.61
Current children cumulated vsize (KiB) 5008628

[startup+882.301 s]
/proc/loadavg: 8.08 8.10 8.02 9/186 21117
/proc/meminfo: memFree=22315420/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5043564 CPUtime=3523.28 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1206993 0 0 0 352059 269 0 0 25 0 5 0 599278126 5164609536 1204535 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1260891 1204535 176 36 0 1258796 0
[pid=21057/tid=21059] ppid=21055 vsize=5043564 CPUtime=879.49 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 234093 0 0 0 87892 57 0 0 25 0 5 0 599278162 5164609536 1204535 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5043564 CPUtime=880.38 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 352594 0 0 0 87959 79 0 0 25 0 5 0 599278162 5164609536 1204535 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5043564 CPUtime=881.34 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 297534 0 0 0 88071 63 0 0 25 0 5 0 599278162 5164609536 1204535 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5043564 CPUtime=881.68 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 315050 0 0 0 88100 68 0 0 25 0 5 0 599278162 5164609536 1204535 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3523.28
Current children cumulated vsize (KiB) 5043564

[startup+942.301 s]
/proc/loadavg: 8.11 8.11 8.03 9/186 21119
/proc/meminfo: memFree=22002620/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5134672 CPUtime=3762.96 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1230141 0 0 0 376021 275 0 0 25 0 5 0 599278126 5257904128 1227683 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1283668 1227683 176 36 0 1281573 0
[pid=21057/tid=21059] ppid=21055 vsize=5134672 CPUtime=939.49 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 244513 0 0 0 93890 59 0 0 25 0 5 0 599278162 5257904128 1227683 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5134672 CPUtime=940.06 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 352647 0 0 0 93927 79 0 0 25 0 5 0 599278162 5257904128 1227683 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5134672 CPUtime=941.34 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 304691 0 0 0 94069 65 0 0 25 0 5 0 599278162 5257904128 1227683 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5134672 CPUtime=941.67 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 320568 0 0 0 94098 69 0 0 25 0 5 0 599278162 5257904128 1227683 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3762.96
Current children cumulated vsize (KiB) 5134672

[startup+1002.3 s]
/proc/loadavg: 8.08 8.10 8.03 9/186 21120
/proc/meminfo: memFree=21639096/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5278956 CPUtime=4002.63 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1266052 0 0 0 399978 285 0 0 25 0 5 0 599278126 5405650944 1263594 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1319739 1263594 176 36 0 1317644 0
[pid=21057/tid=21059] ppid=21055 vsize=5278956 CPUtime=999.5 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 254240 0 0 0 99888 62 0 0 25 0 5 0 599278162 5405650944 1263594 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5278956 CPUtime=999.73 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 359709 0 0 0 99892 81 0 0 25 0 5 0 599278162 5405650944 1263594 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5278956 CPUtime=1001.34 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 313674 0 0 0 100067 67 0 0 25 0 5 0 599278162 5405650944 1263594 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5278956 CPUtime=1001.68 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 330707 0 0 0 100095 73 0 0 25 0 5 0 599278162 5405650944 1263594 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4002.63
Current children cumulated vsize (KiB) 5278956

[startup+1062.3 s]
/proc/loadavg: 8.23 8.13 8.04 11/187 21125
/proc/meminfo: memFree=21231336/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5454520 CPUtime=4242.2 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1311392 0 0 0 423925 295 0 0 25 0 5 0 599278126 5585428480 1308934 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1363630 1308935 176 36 0 1361535 0
[pid=21057/tid=21059] ppid=21055 vsize=5454520 CPUtime=1059.5 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 268164 0 0 0 105885 65 0 0 25 0 5 0 599278162 5585428480 1308935 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5454520 CPUtime=1059.56 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 369669 0 0 0 105872 84 0 0 25 0 5 0 599278162 5585428480 1308936 33554432000 4194304 4338100 140737196760880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5454520 CPUtime=1061.34 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 326505 0 0 0 106065 69 0 0 25 0 5 0 599278162 5585428480 1308937 33554432000 4194304 4338100 140737196760880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5454520 CPUtime=1061.42 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 339334 0 0 0 106067 75 0 0 25 0 5 0 599278162 5585428480 1308938 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4242.2
Current children cumulated vsize (KiB) 5454520

[startup+1122.3 s]
/proc/loadavg: 8.33 8.19 8.06 9/186 21127
/proc/meminfo: memFree=20865228/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5623828 CPUtime=4481.18 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1352977 0 0 0 447814 304 0 0 25 0 5 0 599278126 5758799872 1350519 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1405957 1350519 176 36 0 1403862 0
[pid=21057/tid=21059] ppid=21055 vsize=5623828 CPUtime=1119.49 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 279349 0 0 0 111882 67 0 0 25 0 5 0 599278162 5758799872 1350519 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5623828 CPUtime=1119.55 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 380491 0 0 0 111869 86 0 0 25 0 5 0 599278162 5758799872 1350519 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5623828 CPUtime=1120.69 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 337550 0 0 0 111998 71 0 0 25 0 5 0 599278162 5758799872 1350519 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5623828 CPUtime=1121.06 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 347865 0 0 0 112029 77 0 0 25 0 5 0 599278162 5758799872 1350519 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4481.18
Current children cumulated vsize (KiB) 5623828

[startup+1182.3 s]
/proc/loadavg: 8.31 8.20 8.07 9/186 21128
/proc/meminfo: memFree=20468484/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=5808908 CPUtime=4720.87 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) S 21055 21057 20765 0 -1 4202496 1399168 0 0 0 471772 315 0 0 25 0 5 0 599278126 5948321792 1396710 33554432000 4194304 4338100 140737196760880 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 1452227 1396710 176 36 0 1450132 0
[pid=21057/tid=21059] ppid=21055 vsize=5808908 CPUtime=1179.5 cores=0,2,4,6
/proc/21057/task/21059/stat : 21059 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 288862 0 0 0 117880 70 0 0 25 0 5 0 599278162 5948321792 1396710 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21057/tid=21060] ppid=21055 vsize=5808908 CPUtime=1179.56 cores=0,2,4,6
/proc/21057/task/21060/stat : 21060 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 389794 0 0 0 117868 88 0 0 25 0 5 0 599278162 5948321792 1396710 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=5808908 CPUtime=1180.69 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 348445 0 0 0 117995 74 0 0 25 0 5 0 599278162 5948321792 1396710 33554432000 4194304 4338100 140737196760880 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21057/tid=21062] ppid=21055 vsize=5808908 CPUtime=1180.74 cores=0,2,4,6
/proc/21057/task/21062/stat : 21062 (ttsth-5-4) R 21055 21057 20765 0 -1 4202560 364345 0 0 0 117994 80 0 0 25 0 5 0 599278162 5948321792 1396710 33554432000 4194304 4338100 140737196760880 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4720.87
Current children cumulated vsize (KiB) 5808908



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.22 8.18 8.07 6/183 21129
/proc/meminfo: memFree=21552076/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=0 CPUtime=4800.22 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) Z 21055 21057 20765 0 -1 4203548 1412786 0 0 0 479696 326 0 0 18 0 2 0 599278126 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 0 0 0 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=0 CPUtime=1200.6 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4203588 351814 0 0 0 119978 82 0 0 25 0 2 0 599278162 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.32 s]
/proc/loadavg: 8.22 8.18 8.07 6/183 21129
/proc/meminfo: memFree=21552076/32951124 swapFree=67111464/67111528
[pid=21057] ppid=21055 vsize=0 CPUtime=4800.32 cores=0,2,4,6
/proc/21057/stat : 21057 (ttsth-5-4) Z 21055 21057 20765 0 -1 4203548 1412786 0 0 0 479696 336 0 0 18 0 2 0 599278126 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21057/statm: 0 0 0 0 0 0 0
[pid=21057/tid=21061] ppid=21055 vsize=0 CPUtime=1200.71 cores=0,2,4,6
/proc/21057/task/21061/stat : 21061 (ttsth-5-4) R 21055 21057 20765 0 -1 4203588 351814 0 0 0 119978 93 0 0 25 0 2 0 599278162 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.32
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.38
CPU time (s): 4800.4
CPU user time (s): 4796.97
CPU system time (s): 3.43248
CPU usage (%): 399.241
Max. virtual memory (cumulated for all children) (KiB): 5862216

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.97
system time used= 3.43248
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1412786
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 642
involuntary context switches= 18825

runsolver used 2.06369 second user time and 4.47732 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 17:21:15
IDJOB=3261914
IDBENCH=83264
IDSOLVER=1654
FILE ID=node110/3261914-1304090475
RUNJOBID= node110-1304088491-20783
PBS_JOBID= 13170568
Free space on /tmp= 72816 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-21_399.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3261914-1304090475/watcher-3261914-1304090475 -o /tmp/evaluation-result-3261914-1304090475/solver-3261914-1304090475 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261914-1304090475.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 693b5fe291cecff45feff3be98a908cf
RANDOM SEED=233618333

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28515480 kB
Buffers:        896916 kB
Cached:         715860 kB
SwapCached:         64 kB
Active:        2489816 kB
Inactive:      1536544 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28515480 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:             424 kB
Writeback:           0 kB
AnonPages:     2413512 kB
Mapped:          15212 kB
Slab:           341748 kB
PageTables:       9356 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2607824 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72840 MiB
End job on node110 at 2011-04-29 17:41:20