Trace number 3263410

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.33 1210.93

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_21_60_7-log.shuffled-as.sat03-151.cnf
MD5SUM5cab37809f478f7db7fa096e9a155cfe
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables105
Number of clauses40110
Sum of the clauses size400890
Maximum clause length10
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 542
Number of clauses of size over 540068

Solver Data 

0.02/0.09	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263410-1304096259.cnf 
0.02/0.09	c Using SATTIMEOUT of 4800
0.02/0.09	c Using SATRAM of 15500
0.02/0.09	c Size of subset node = 32
0.02/0.09	c Variables = 105
0.02/0.09	c Clauses = 40110
0.02/0.09	c Literals = 400890
0.02/0.09	c Reduced size:
0.02/0.09	c Variables = 105
0.02/0.09	c Clauses = 40110
0.02/0.09	c Literals = 400890
0.02/0.09	c a2
0.69/0.70	c a1
1.79/1.89	c Independent sub-problems = 1
1.79/1.89	c Sub-problem with 105 variables and 40110 clauses
1.79/1.89	c using permutation method 4
1.79/1.89	c count = 1
1.79/1.89	c Ordering finished 1.860 seconds
1.79/1.89	c mapped 1.890 seconds
1.79/1.89	c permuted 1.890 seconds
1.79/1.89	c about to sort
1.89/1.91	c sorted
1.89/1.99	c clauses now 40110
1.89/1.99	c init finished 1.910 seconds
1.89/1.99	c tree built 1.980 seconds
1.89/1.99	c 
1.89/1.99	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.89/1.99	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.89/1.99	c ................................ | ..........                    |                                  2.0         0.0             
1.89/1.99	c ................................ | ..........                  1 |                                  2.0         0.5             
1.89/1.99	c ................................ | ..........                  2 |                                  2.0         1.0             
1.89/1.99	c ................................ | ..........                  3 |                                  2.0         1.5             
1.89/1.99	c ................................ | ..........                  4 |                                  2.0         2.0             
1.89/1.99	c ................................ | ..........                  5 |                                  2.0         2.5             
1.89/2.00	c ................................ | ..........               1000 | 147          1369                2.0       502.5        10127
96.00/25.60	c ................................ | ..........            1864000 | 467937       2585341            96.0     19410.6     24694626
191.92/49.72	c ................................ | ..........            3200000 | 802046       4416955           192.0     16664.9     36575508
287.75/73.87	c ................................ | ..........            4320000 | 1079690      5976022           288.0     14997.9     46275671
383.93/98.03	c ................................ | ..........            5419000 | 1357994      7397945           384.1     14109.8     52485928
480.02/122.20	c ................................ | *.........            6502000 | 1632240      8804052           480.0     13544.7     57723621
575.89/146.35	c ................................ | *.........            7481000 | 1879777      10141904          576.1     12985.6     59492693
672.04/170.50	c ................................ | *.........            8444000 | 2124756      11447360          672.0     12564.7     61329863
767.82/194.66	c ................................ | *.........            9323000 | 2345629      12674737          768.1     12138.4     68087317
863.84/218.87	c ................................ | *.........           10142000 | 2551457      13831382          864.1     11737.1     72224631
959.61/242.99	c ................................ | **........           11005000 | 2767875      14973334          960.0     11463.5     74103278
1055.98/267.22	c ................................ | **........           11875000 | 2987566      16147774         1056.0     11244.8     76853497
1152.02/291.42	c ................................ | **........           12704000 | 3196455      17273609         1152.1     11027.0     81067594
1247.90/315.66	c ................................ | **........           13552000 | 3412636      18387980         1248.1     10857.9     82999042
1343.96/339.84	c ................................ | **........           14355000 | 3615393      19496433         1344.1     10679.9     84422982
1439.88/364.04	c ................................ | ***.......           15093000 | 3800549      20567479         1440.0     10481.1     87002014
1535.85/388.26	c ................................ | ***.......           15880000 | 3999868      21630249         1536.1     10337.9     87932880
1632.06/412.43	c ................................ | ***.......           16660000 | 4195304      22712534         1632.2     10207.4     88698989
1727.83/436.54	c ................................ | ***.......           17520000 | 4413547      23822762         1728.0     10138.8     90115247
1823.74/460.78	c ................................ | ***.......           18303000 | 4611978      24883026         1824.1     10034.2     92670697
1919.89/484.94	c ................................ | ****......           19140000 | 4825576      25922532         1920.1      9968.4     93233047
2015.74/509.08	c ................................ | ****......           19973000 | 5036717      26969098         2016.1      9906.8     94093349
2111.76/533.25	c ................................ | ****......           20750000 | 5234859      27983028         2112.0      9824.8     95476197
2207.86/557.45	c ................................ | ****......           21464000 | 5416538      28983615         2208.1      9720.8     95476197
2303.81/581.58	c ................................ | ****......           22186000 | 5601605      29960985         2304.2      9628.7     97429190
2399.94/605.72	c ................................ | *****.....           22887000 | 5777092      30947180         2400.0      9536.1     98395802
2496.01/629.90	c ................................ | *****.....           23680000 | 5977609      31979765         2496.0      9487.1     99481666
2591.90/654.05	c ................................ | *****.....           24413000 | 6161057      32972577         2592.1      9418.2    100039202
2688.06/678.30	c ................................ | *****.....           25192000 | 6357595      33968598         2688.1      9371.7    100039202
2783.87/702.45	c ................................ | *****.....           25964000 | 6552469      34993120         2784.1      9325.9    102041360
2879.95/726.62	c ................................ | ******....           26716000 | 6740236      35953138         2880.0      9276.3    104387478
2975.70/750.79	c ................................ | ******....           27470000 | 6929708      36934653         2976.1      9230.4    109688416
3072.01/775.01	c ................................ | ******....           28221000 | 7118552      37885470         3072.1      9186.3    110069233
3168.07/799.20	c ................................ | ******....           28981000 | 7311442      38864085         3168.1      9147.8    110453352
3263.73/823.38	c ................................ | ******....           29749000 | 7506209      39843434         3264.1      9114.1    111732122
3359.84/847.54	c ................................ | *******...           30425000 | 7676067      40774457         3360.0      9055.0    113790658
3455.65/871.68	c ................................ | *******...           31159000 | 7862021      41742040         3456.0      9015.9    114755656
3551.92/895.85	c ................................ | *******...           31890000 | 8048449      42668814         3552.1      8977.7    115559127
3647.95/920.02	c ................................ | *******...           32585000 | 8223889      43624591         3648.1      8932.1    115559127
3743.75/944.27	c ................................ | *******...           33257000 | 8393382      44607115         3744.0      8882.7    115559127
3839.76/968.46	c ................................ | ********..           33966000 | 8573198      45614957         3840.0      8845.3    116404637
3935.85/992.66	c ................................ | ********..           34638000 | 8742012      46594521         3936.1      8800.1    116935025
4031.89/1016.84	c ................................ | ********..           35334000 | 8920261      47542932         4032.1      8763.2    117849307
4127.96/1041.06	c ................................ | ********..           36010000 | 9094196      48452971         4128.2      8722.9    118804876
4223.86/1065.25	c ................................ | ********..           36664000 | 9258034      49354970         4224.1      8679.8    120409084
4319.74/1089.37	c ................................ | *********.           37378000 | 9437305      50316159         4320.0      8652.3    122293902
4416.04/1113.81	c ................................ | *********.           38013000 | 9595264      51228117         4416.1      8607.9    122931501
4511.85/1138.05	c ................................ | *********.           38627000 | 9749551      52130095         4512.1      8560.9    123739710
4607.94/1162.23	c ................................ | *********.           39310000 | 9923507      53014044         4608.1      8530.7    124337631
4703.71/1186.50	c ................................ | *********.           39992000 | 10095196     53912216         4704.0      8501.6    125303318
4799.70/1210.69	c CPU limit (4800) reached
4799.70/1210.69	c Time = 4800.09 seconds
4799.70/1210.69	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263410-1304096259/watcher-3263410-1304096259 -o /tmp/evaluation-result-3263410-1304096259/solver-3263410-1304096259 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263410-1304096259.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.43 4.96 6.57 5/176 21597
/proc/meminfo: memFree=30877484/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=9104 CPUtime=0 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) R 21592 21595 21560 0 -1 4202496 347 0 0 0 0 0 0 0 18 0 1 0 599856489 9322496 290 33554432000 4194304 4338100 140734309888896 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21595/statm: 2276 290 147 36 0 181 0

[startup+0.0291 s]
/proc/loadavg: 3.43 4.96 6.57 5/176 21597
/proc/meminfo: memFree=30877484/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=10700 CPUtime=0.02 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) R 21592 21595 21560 0 -1 4202496 748 0 0 0 2 0 0 0 18 0 1 0 599856489 10956800 691 33554432000 4194304 4338100 140734309888896 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21595/statm: 2675 691 147 36 0 580 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 10700

[startup+0.101091 s]
/proc/loadavg: 3.43 4.96 6.57 5/176 21597
/proc/meminfo: memFree=30877484/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=19396 CPUtime=0.09 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) R 21592 21595 21560 0 -1 4202496 2694 0 0 0 9 0 0 0 18 0 1 0 599856489 19861504 2613 33554432000 4194304 4338100 140734309888896 18446744073709551615 260020093039 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21595/statm: 4849 2628 217 36 0 2754 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19396

[startup+0.301049 s]
/proc/loadavg: 3.43 4.96 6.57 5/176 21597
/proc/meminfo: memFree=30877484/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=69244 CPUtime=0.29 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) R 21592 21595 21560 0 -1 4202496 8157 0 0 0 28 1 0 0 18 0 1 0 599856489 70905856 7579 33554432000 4194304 4338100 140734309888896 18446744073709551615 260019791097 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21595/statm: 17311 7579 254 36 0 15216 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 69244

[startup+0.700975 s]
/proc/loadavg: 3.43 4.96 6.57 5/176 21597
/proc/meminfo: memFree=30877484/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=69244 CPUtime=0.69 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) R 21592 21595 21560 0 -1 4202496 8157 0 0 0 68 1 0 0 18 0 1 0 599856489 70905856 7579 33554432000 4194304 4338100 140734309888896 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21595/statm: 17311 7579 254 36 0 15216 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 69244

[startup+1.50083 s]
/proc/loadavg: 3.31 4.91 6.54 3/178 21599
/proc/meminfo: memFree=30822924/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=20804 CPUtime=1.49 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) R 21592 21595 21560 0 -1 4202496 8266 0 0 0 148 1 0 0 20 0 1 0 599856489 21303296 3211 33554432000 4194304 4338100 140734309888896 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21595/statm: 5201 3211 155 36 0 3106 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 20804

[startup+3.1005 s]
/proc/loadavg: 3.31 4.91 6.54 7/182 21603
/proc/meminfo: memFree=30833060/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=357136 CPUtime=6.39 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 32251 0 0 0 633 6 0 0 22 0 5 0 599856489 365707264 27194 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 89284 27195 177 36 0 87189 0
[pid=21595/tid=21600] ppid=21592 vsize=357136 CPUtime=1.1 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 5661 0 0 0 109 1 0 0 25 0 5 0 599856689 365707264 27196 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=357136 CPUtime=1.08 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 5793 0 0 0 108 0 0 0 25 0 5 0 599856689 365707264 27197 33554432000 4194304 4338100 140734309888896 18446744073709551615 260019791290 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=357136 CPUtime=1.09 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 5882 0 0 0 108 1 0 0 19 0 5 0 599856689 365707264 27197 33554432000 4194304 4338100 140734309888896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=357136 CPUtime=1.09 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 6166 0 0 0 109 0 0 0 25 0 5 0 599856689 365707264 27197 33554432000 4194304 4338100 140734309888896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 6.39
Current children cumulated vsize (KiB) 357136

[startup+6.30091 s]
/proc/loadavg: 3.31 4.91 6.54 10/186 21607
/proc/meminfo: memFree=30464152/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=523340 CPUtime=19.14 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 76193 0 0 0 1899 15 0 0 22 0 5 0 599856489 535900160 71136 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 130835 71136 177 36 0 128740 0
[pid=21595/tid=21600] ppid=21592 vsize=523340 CPUtime=4.28 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 16169 0 0 0 425 3 0 0 18 0 5 0 599856689 535900160 71136 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=523340 CPUtime=4.26 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 16458 0 0 0 423 3 0 0 25 0 5 0 599856689 535900160 71136 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=523340 CPUtime=4.28 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 16866 0 0 0 425 3 0 0 18 0 5 0 599856689 535900160 71136 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=523340 CPUtime=4.29 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 17949 0 0 0 426 3 0 0 25 0 5 0 599856689 535900160 71136 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 19.14
Current children cumulated vsize (KiB) 523340

[startup+12.7007 s]
/proc/loadavg: 4.03 5.01 6.56 10/186 21607
/proc/meminfo: memFree=29932376/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=753932 CPUtime=44.65 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 138381 0 0 0 4435 30 0 0 22 0 5 0 599856489 772026368 133324 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 188483 133324 177 36 0 186388 0
[pid=21595/tid=21600] ppid=21592 vsize=753932 CPUtime=10.65 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 31555 0 0 0 1058 7 0 0 25 0 5 0 599856689 772026368 133324 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=753932 CPUtime=10.61 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 31138 0 0 0 1055 6 0 0 25 0 5 0 599856689 772026368 133324 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=753932 CPUtime=10.67 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 31037 0 0 0 1061 6 0 0 25 0 5 0 599856689 772026368 133324 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=753932 CPUtime=10.68 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 35902 0 0 0 1062 6 0 0 25 0 5 0 599856689 772026368 133326 33554432000 4194304 4338100 140734309888896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 44.65
Current children cumulated vsize (KiB) 753932

[startup+25.5004 s]
/proc/loadavg: 4.64 5.11 6.57 10/186 21608
/proc/meminfo: memFree=29269300/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=1066004 CPUtime=95.61 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 223452 0 0 0 9508 53 0 0 22 0 5 0 599856489 1091588096 218395 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 266501 218395 177 36 0 264406 0
[pid=21595/tid=21600] ppid=21592 vsize=1066004 CPUtime=23.31 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 53916 0 0 0 2317 14 0 0 25 0 5 0 599856689 1091588096 218395 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032210075 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=1066004 CPUtime=23.37 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 51422 0 0 0 2325 12 0 0 25 0 5 0 599856689 1091588096 218395 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=1066004 CPUtime=23.43 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 51092 0 0 0 2333 10 0 0 25 0 5 0 599856689 1091588096 218395 33554432000 4194304 4338100 140734309888896 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=1066004 CPUtime=23.46 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 58271 0 0 0 2333 13 0 0 25 0 5 0 599856689 1091588096 218395 33554432000 4194304 4338100 140734309888896 18446744073709551615 260019791316 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 95.61
Current children cumulated vsize (KiB) 1066004

[startup+51.1007 s]
/proc/loadavg: 5.85 5.36 6.61 10/186 21608
/proc/meminfo: memFree=28316460/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=1484708 CPUtime=197.51 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 338884 0 0 0 19662 89 0 0 22 0 5 0 599856489 1520340992 333827 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 371177 333827 177 36 0 369082 0
[pid=21595/tid=21600] ppid=21592 vsize=1484708 CPUtime=48.64 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 89307 0 0 0 4838 26 0 0 25 0 5 0 599856689 1520340992 333827 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=1484708 CPUtime=48.9 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 77269 0 0 0 4870 20 0 0 25 0 5 0 599856689 1520340992 333827 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=1484708 CPUtime=48.97 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 76901 0 0 0 4880 17 0 0 25 0 5 0 599856689 1520340992 333827 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=1484708 CPUtime=48.96 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 86656 0 0 0 4875 21 0 0 25 0 5 0 599856689 1520340992 333827 33554432000 4194304 4338100 140734309888896 18446744073709551615 4287242 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 197.51
Current children cumulated vsize (KiB) 1484708

[startup+102.307 s]
/proc/loadavg: 7.26 5.83 6.70 10/186 21610
/proc/meminfo: memFree=27162496/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=2005384 CPUtime=401.09 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 486548 0 0 0 39964 145 0 0 22 0 5 0 599856489 2053513216 481491 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 501346 481492 177 36 0 499251 0
[pid=21595/tid=21600] ppid=21592 vsize=2005384 CPUtime=99.3 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 140617 0 0 0 9881 49 0 0 25 0 5 0 599856689 2053513216 481492 33554432000 4194304 4338100 140734309888896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=2005384 CPUtime=99.91 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 99881 0 0 0 9963 28 0 0 25 0 5 0 599856689 2053513216 481492 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=2005384 CPUtime=99.96 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 111586 0 0 0 9967 29 0 0 25 0 5 0 599856689 2053513216 481492 33554432000 4194304 4338100 140734309888896 18446744073709551615 260020063163 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=2005384 CPUtime=99.88 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 125714 0 0 0 9953 35 0 0 25 0 5 0 599856689 2053513216 481492 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 401.09
Current children cumulated vsize (KiB) 2005384

[startup+162.301 s]
/proc/loadavg: 7.80 6.24 6.79 9/186 21611
/proc/meminfo: memFree=26295808/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=2478616 CPUtime=639.43 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 575332 0 0 0 63749 194 0 0 22 0 5 0 599856489 2538102784 570275 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 619654 570275 177 36 0 617559 0
[pid=21595/tid=21600] ppid=21592 vsize=2478616 CPUtime=158.68 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 178377 0 0 0 15800 68 0 0 25 0 5 0 599856689 2538102784 570275 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032210159 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=2478616 CPUtime=159.68 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 115319 0 0 0 15931 37 0 0 25 0 5 0 599856689 2538102784 570275 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032208646 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=2478616 CPUtime=159.53 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 124886 0 0 0 15914 39 0 0 25 0 5 0 599856689 2538102784 570275 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=2478616 CPUtime=159.52 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 147999 0 0 0 15905 47 0 0 25 0 5 0 599856689 2538102784 570275 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 639.43
Current children cumulated vsize (KiB) 2478616

[startup+222.301 s]
/proc/loadavg: 7.92 6.56 6.86 10/186 21622
/proc/meminfo: memFree=25460980/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=2901212 CPUtime=877.7 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 694772 0 0 0 87525 245 0 0 22 0 5 0 599856489 2970841088 689715 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 725303 689715 177 36 0 723208 0
[pid=21595/tid=21600] ppid=21592 vsize=2901212 CPUtime=217.86 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 222824 0 0 0 21701 85 0 0 25 0 5 0 599856689 2970841088 689715 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 467951 0 0 0 74957 218 0 0 21 0 5 0 599856689 4995719168 1174993 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=4878632 CPUtime=756.25 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 210311 0 0 0 75528 97 0 0 25 0 5 0 599856689 4995719168 1174993 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=4878632 CPUtime=755.66 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 231864 0 0 0 75449 117 0 0 25 0 5 0 599856689 4995719168 1174993 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=4878632 CPUtime=756.02 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 261173 0 0 0 75472 130 0 0 25 0 5 0 599856689 4995719168 1174993 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3021.71
Current children cumulated vsize (KiB) 4878632

[startup+822.301 s]
/proc/loadavg: 8.06 7.84 7.41 10/186 21638
/proc/meminfo: memFree=21287900/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=4971956 CPUtime=3259.74 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1212230 0 0 0 325380 594 0 0 22 0 5 0 599856489 5091282944 1207173 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1242989 1207173 177 36 0 1240894 0
[pid=21595/tid=21600] ppid=21592 vsize=4971956 CPUtime=810.98 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 485915 0 0 0 80866 232 0 0 25 0 5 0 599856689 5091282944 1207173 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=4971956 CPUtime=815.86 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 215196 0 0 0 81482 104 0 0 25 0 5 0 599856689 5091282944 1207173 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=4971956 CPUtime=815.32 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 236283 0 0 0 81409 123 0 0 25 0 5 0 599856689 5091282944 1207173 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=4971956 CPUtime=815.55 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 266085 0 0 0 81423 132 0 0 25 0 5 0 599856689 5091282944 1207173 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3259.74
Current children cumulated vsize (KiB) 4971956

[startup+882.301 s]
/proc/loadavg: 7.95 7.85 7.44 10/186 21640
/proc/meminfo: memFree=20952928/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=5242220 CPUtime=3498.21 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1260521 0 0 0 349197 624 0 0 22 0 5 0 599856489 5368033280 1255464 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1310555 1255464 177 36 0 1308460 0
[pid=21595/tid=21600] ppid=21592 vsize=5242220 CPUtime=870.47 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 517047 0 0 0 86800 247 0 0 25 0 5 0 599856689 5368033280 1255464 33554432000 4194304 4338100 140734309888896 18446744073709551615 4235756 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=5242220 CPUtime=875.62 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 219621 0 0 0 87453 109 0 0 25 0 5 0 599856689 5368033280 1255464 33554432000 4194304 4338100 140734309888896 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=5242220 CPUtime=874.87 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 240498 0 0 0 87357 130 0 0 25 0 5 0 599856689 5368033280 1255464 33554432000 4194304 4338100 140734309888896 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=5242220 CPUtime=875.23 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 274604 0 0 0 87388 135 0 0 25 0 5 0 599856689 5368033280 1255464 33554432000 4194304 4338100 140734309888896 18446744073709551615 4235661 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3498.21
Current children cumulated vsize (KiB) 5242220

[startup+942.3 s]
/proc/loadavg: 7.98 7.88 7.47 10/186 21641
/proc/meminfo: memFree=20679464/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=5368516 CPUtime=3736.2 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1287180 0 0 0 372969 651 0 0 22 0 5 0 599856489 5497360384 1282123 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1342129 1282123 177 36 0 1340034 0
[pid=21595/tid=21600] ppid=21592 vsize=5368516 CPUtime=929.66 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 521991 0 0 0 92711 255 0 0 25 0 5 0 599856689 5497360384 1282123 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=5368516 CPUtime=935.27 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 227089 0 0 0 93410 117 0 0 25 0 5 0 599856689 5497360384 1282123 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=5368516 CPUtime=934.33 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 246903 0 0 0 93296 137 0 0 25 0 5 0 599856689 5497360384 1282123 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=5368516 CPUtime=934.92 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 282446 0 0 0 93353 139 0 0 25 0 5 0 599856689 5497360384 1282123 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3736.2
Current children cumulated vsize (KiB) 5368516

[startup+1002.3 s]
/proc/loadavg: 8.15 7.94 7.51 10/186 21643
/proc/meminfo: memFree=20414184/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=5439024 CPUtime=3974.31 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1319736 0 0 0 396749 682 0 0 22 0 5 0 599856489 5569560576 1314679 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1359756 1314679 177 36 0 1357661 0
[pid=21595/tid=21600] ppid=21592 vsize=5439024 CPUtime=988.97 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 539406 0 0 0 98631 266 0 0 21 0 5 0 599856689 5569560576 1314679 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=5439024 CPUtime=994.89 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 232273 0 0 0 99363 126 0 0 25 0 5 0 599856689 5569560576 1314679 33554432000 4194304 4338100 140734309888896 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=5439024 CPUtime=993.94 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 251702 0 0 0 99251 143 0 0 25 0 5 0 599856689 5569560576 1314679 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=5439024 CPUtime=994.49 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 287604 0 0 0 99306 143 0 0 25 0 5 0 599856689 5569560576 1314679 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3974.31
Current children cumulated vsize (KiB) 5439024

[startup+1062.3 s]
/proc/loadavg: 8.17 7.98 7.55 10/186 21644
/proc/meminfo: memFree=20150528/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=5553680 CPUtime=4212.34 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1361590 0 0 0 420520 714 0 0 22 0 5 0 599856489 5686968320 1356533 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1388420 1356533 177 36 0 1386325 0
[pid=21595/tid=21600] ppid=21592 vsize=5553680 CPUtime=1048.12 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 567998 0 0 0 104529 283 0 0 25 0 5 0 599856689 5686968320 1356533 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=5553680 CPUtime=1054.43 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 236481 0 0 0 105310 133 0 0 25 0 5 0 599856689 5686968320 1356533 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=5553680 CPUtime=1053.62 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 256138 0 0 0 105215 147 0 0 25 0 5 0 599856689 5686968320 1356533 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=5553680 CPUtime=1054.14 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 292222 0 0 0 105266 148 0 0 25 0 5 0 599856689 5686968320 1356533 33554432000 4194304 4338100 140734309888896 18446744073709551615 4242456 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4212.34
Current children cumulated vsize (KiB) 5553680

[startup+1122.3 s]
/proc/loadavg: 8.23 8.03 7.59 9/186 21646
/proc/meminfo: memFree=19901492/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=5862740 CPUtime=4449.78 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1403520 0 0 0 444231 747 0 0 22 0 5 0 599856489 6003445760 1398463 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1465685 1398463 177 36 0 1463590 0
[pid=21595/tid=21600] ppid=21592 vsize=5862740 CPUtime=1107.13 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 596257 0 0 0 110412 301 0 0 25 0 5 0 599856689 6003445760 1398463 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=5862740 CPUtime=1114.02 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 240912 0 0 0 111264 138 0 0 25 0 5 0 599856689 6003445760 1398463 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=5862740 CPUtime=1112.97 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 260626 0 0 0 111143 154 0 0 25 0 5 0 599856689 6003445760 1398463 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=5862740 CPUtime=1113.63 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 296974 0 0 0 111212 151 0 0 25 0 5 0 599856689 6003445760 1398463 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4449.78
Current children cumulated vsize (KiB) 5862740

[startup+1182.3 s]
/proc/loadavg: 8.11 8.03 7.62 10/186 21648
/proc/meminfo: memFree=19623684/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=5942988 CPUtime=4687.59 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) S 21592 21595 21560 0 -1 4202496 1437385 0 0 0 467981 778 0 0 22 0 5 0 599856489 6085619712 1432328 33554432000 4194304 4338100 140734309888896 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 1485747 1432328 177 36 0 1483652 0
[pid=21595/tid=21600] ppid=21592 vsize=5942988 CPUtime=1166.29 cores=1,3,5,7
/proc/21595/task/21600/stat : 21600 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 615833 0 0 0 116313 316 0 0 25 0 5 0 599856689 6085619712 1432328 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21595/tid=21601] ppid=21592 vsize=5942988 CPUtime=1173.45 cores=1,3,5,7
/proc/21595/task/21601/stat : 21601 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 245086 0 0 0 117201 144 0 0 25 0 5 0 599856689 6085619712 1432328 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=5942988 CPUtime=1172.58 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 266289 0 0 0 117097 161 0 0 25 0 5 0 599856689 6085619712 1432328 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21595/tid=21603] ppid=21592 vsize=5942988 CPUtime=1173.25 cores=1,3,5,7
/proc/21595/task/21603/stat : 21603 (ttsth-5-4) R 21592 21595 21560 0 -1 4202560 301426 0 0 0 117171 154 0 0 25 0 5 0 599856689 6085619712 1432328 33554432000 4194304 4338100 140734309888896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4687.59
Current children cumulated vsize (KiB) 5942988



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1210.7 s]
/proc/loadavg: 8.06 8.03 7.63 7/183 21648
/proc/meminfo: memFree=19586328/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) Z 21592 21595 21560 0 -1 4203548 1448828 0 0 0 479218 792 0 0 18 0 2 0 599856489 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 0 0 0 0 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=0 CPUtime=1200.69 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4203588 268241 0 0 0 119905 164 0 0 25 0 2 0 599856689 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1210.81 s]
/proc/loadavg: 8.06 8.03 7.63 7/183 21648
/proc/meminfo: memFree=19586328/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=0 CPUtime=4800.2 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) Z 21592 21595 21560 0 -1 4203548 1448828 0 0 0 479218 802 0 0 18 0 2 0 599856489 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 0 0 0 0 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=0 CPUtime=1200.8 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4203588 268241 0 0 0 119905 175 0 0 25 0 2 0 599856689 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

[startup+1210.9 s]
/proc/loadavg: 8.06 8.03 7.63 7/183 21648
/proc/meminfo: memFree=19586328/32951124 swapFree=67111464/67111528
[pid=21595] ppid=21592 vsize=0 CPUtime=4800.29 cores=1,3,5,7
/proc/21595/stat : 21595 (ttsth-5-4) Z 21592 21595 21560 0 -1 4203548 1448828 0 0 0 479218 811 0 0 18 0 2 0 599856489 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21595/statm: 0 0 0 0 0 0 0
[pid=21595/tid=21602] ppid=21592 vsize=0 CPUtime=1200.89 cores=1,3,5,7
/proc/21595/task/21602/stat : 21602 (ttsth-5-4) R 21592 21595 21560 0 -1 4203588 268241 0 0 0 119905 184 0 0 25 0 2 0 599856689 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1210.93
CPU time (s): 4800.33
CPU user time (s): 4792.18
CPU system time (s): 8.14276
CPU usage (%): 396.417
Max. virtual memory (cumulated for all children) (KiB): 5963948

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4792.18
system time used= 8.14276
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1448828
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 24649
involuntary context switches= 46508

runsolver used 1.86871 second user time and 4.73628 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 18:57:39
IDJOB=3263410
IDBENCH=83967
IDSOLVER=1654
FILE ID=node110/3263410-1304096259
RUNJOBID= node110-1304096259-21579
PBS_JOBID= 13170606
Free space on /tmp= 72836 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_21_60_7-log.shuffled-as.sat03-151.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263410-1304096259/watcher-3263410-1304096259 -o /tmp/evaluation-result-3263410-1304096259/solver-3263410-1304096259 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263410-1304096259.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 5cab37809f478f7db7fa096e9a155cfe
RANDOM SEED=1350559274

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30877936 kB
Buffers:        902960 kB
Cached:         698164 kB
SwapCached:         64 kB
Active:         122116 kB
Inactive:      1543132 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30877936 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:            4028 kB
Writeback:           0 kB
AnonPages:       63696 kB
Mapped:          14584 kB
Slab:           341824 kB
PageTables:       4488 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   186868 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72836 MiB
End job on node110 at 2011-04-29 19:17:52