Trace number 3264510

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1202.46

General information on the benchmark

NameSAT04/handmade/connamacher/connm-d-0.02/
connm-ue-csp-sat-n1200-d-0.02-s405595518.used-as.sat04-950.cnf
MD5SUM287d3b23c6f72107e24ae4186653564e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark950.907
Satisfiable
(Un)Satisfiability was proved
Number of variables1124
Number of clauses13289
Sum of the clauses size40148
Maximum clause length4
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 313008
Number of clauses of size 4281
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264510-1304100136.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 1124
0.00/0.01	c Clauses = 13289
0.00/0.01	c Literals = 40148
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 1124
0.00/0.01	c Clauses = 13289
0.00/0.01	c Literals = 40148
0.00/0.01	c a2
0.00/0.02	c a1
0.09/0.14	c Independent sub-problems = 1
0.09/0.14	c Sub-problem with 1124 variables and 13289 clauses
0.09/0.14	c using permutation method 4
0.09/0.14	c count = 2
0.09/0.14	c Ordering finished 0.130 seconds
0.09/0.14	c mapped 0.130 seconds
0.09/0.14	c permuted 0.130 seconds
0.09/0.14	c about to sort
0.09/0.14	c sorted
0.09/0.17	c clauses now 13289
0.09/0.17	c init finished 0.140 seconds
0.09/0.17	c tree built 0.160 seconds
0.09/0.17	c 
0.09/0.17	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.17	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.17	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.17	c ................................ | ..........                  1 |                                  0.2         6.2             
0.09/0.17	c ................................ | ..........                  2 |                                  0.2        12.5             
0.09/0.17	c ................................ | ..........                  3 |              3                   0.2        18.8             
0.09/0.17	c ................................ | ..........                  4 |              4                   0.2        25.0             
0.09/0.17	c ................................ | ..........                  5 |              4                   0.2        31.2             
0.09/0.17	c ................................ | ..........               1000 | 11           409                 0.2      6250.0          186
95.72/24.18	c ................................ | ..........            2775000 | 236002       2834862            96.0     28891.2      5296736
191.91/48.23	c ................................ | ..........            4188000 | 366858       4217784           192.0     21808.0      7651845
287.73/72.27	c ................................ | ..........            5346000 | 474810       5348322           288.0     18560.6      8999184
383.93/96.32	c ................................ | ..........            6444000 | 574016       6376388           384.0     16781.2     10394371
479.76/120.39	c ................................ | *.........            7458000 | 668906       7291341           480.1     15533.9     11455984
575.97/144.42	c ................................ | *.........            8380000 | 755886       8129266           576.1     14546.3     11927540
671.80/168.45	c ................................ | *.........            9269000 | 839247       8927767           672.0     13792.7     12663987
768.03/192.51	c ................................ | *.........           10186000 | 927813       9782122           768.0     13262.2     13929295
863.85/216.56	c ................................ | *.........           10896000 | 1013775      10516891          864.1     12609.5     15010913
959.68/240.58	c ................................ | **........           11535000 | 1091310      11158318          960.0     12015.4     15832657
1055.90/264.63	c ................................ | **........           12098000 | 1160901      11721039         1056.0     11456.4     16576001
1152.13/288.73	c ................................ | **........           12623000 | 1228242      12231040         1152.2     10955.9     17663577
1247.96/312.73	c ................................ | **........           13110000 | 1291214      12697809         1248.1     10504.1     18574861
1343.80/336.77	c ................................ | **........           13612000 | 1354114      13166268         1344.1     10127.4     19529915
1440.02/360.85	c ................................ | ***.......           14174000 | 1413477      13675915         1440.2      9841.6     19894710
1535.85/384.86	c ................................ | ***.......           14798000 | 1463619      14234889         1536.1      9633.5     20148047
1632.07/408.90	c ................................ | ***.......           15429000 | 1513572      14798350         1632.0      9453.9     20405550
1727.91/432.94	c ................................ | ***.......           16029000 | 1562929      15335694         1728.1      9275.6     20711735
1823.75/456.97	c ................................ | ***.......           16638000 | 1613369      15851571         1824.0      9121.5     21070894
1919.98/481.01	c ................................ | ****......           17196000 | 1662363      16328514         1920.0      8956.1     21610721
2015.82/505.05	c ................................ | ****......           17746000 | 1711298      16797905         2016.0      8802.4     22008528
2112.07/529.11	c ................................ | ****......           18324000 | 1760226      17300284         2112.1      8675.7     22451245
2207.88/553.16	c ................................ | ****......           18901000 | 1803743      17806861         2208.1      8559.7     22826568
2303.73/577.18	c ................................ | ****......           19467000 | 1847921      18306164         2304.0      8449.1     23214205
2399.95/601.22	c ................................ | *****.....           19999000 | 1892316      18770713         2400.1      8332.7     23521141
2495.79/625.27	c ................................ | *****.....           20520000 | 1938239      19206935         2496.1      8220.9     23987390
2592.02/649.33	c ................................ | *****.....           21036000 | 1984633      19645295         2592.2      8115.2     24468976
2687.81/673.34	c ................................ | *****.....           22797000 | 2147360      21354742         2688.0      8481.0     26453672
2784.01/697.42	c ................................ | *****.....           24042000 | 2262519      22513814         2784.1      8635.6     26778282
2879.80/721.46	c ................................ | ******....           25082000 | 2359578      23463207         2880.1      8708.9     26778282
2976.01/745.51	c ................................ | ******....           26001000 | 2446479      24263763         2976.1      8736.7     27220168
3071.82/769.54	c ................................ | ******....           26838000 | 2526008      24987217         3072.0      8736.3     27712958
3168.04/793.61	c ................................ | ******....           27656000 | 2604240      25688927         3168.1      8729.6     27853691
3263.86/817.66	c ................................ | ******....           28419000 | 2677717      26335126         3264.1      8706.5     28055032
3359.68/841.68	c ................................ | *******...           29141000 | 2761117      27043065         3360.0      8672.9     28727619
3455.93/865.73	c ................................ | *******...           29721000 | 2835884      27632604         3456.0      8599.8     29366858
3551.70/889.79	c ................................ | *******...           30249000 | 2905891      28180662         3552.1      8515.9     29822002
3647.93/913.82	c ................................ | *******...           30775000 | 2974869      28722549         3648.0      8436.1     29822002
3743.76/937.87	c ................................ | *******...           31280000 | 3043846      29216886         3744.1      8354.6     29959088
3839.98/961.92	c ................................ | ********..           31788000 | 3112361      29710510         3840.1      8277.9     30065875
3935.83/985.98	c ................................ | ********..           32285000 | 3178807      30187391         3936.2      8202.1     30485439
4032.04/1010.00	c ................................ | ********..           32926000 | 3238298      30762359         4032.1      8166.0     30934352
4127.90/1034.04	c ................................ | ********..           33579000 | 3296617      31337807         4128.0      8134.4     31469737
4223.68/1058.10	c ................................ | ********..           34204000 | 3349571      31876799         4224.1      8097.3     31981981
4319.92/1082.12	c ................................ | *********.           34800000 | 3400290      32388680         4320.0      8055.5     32344155
4415.76/1106.19	c ................................ | *********.           35386000 | 3452348      32882451         4416.1      8012.9     32662916
4512.00/1130.22	c ................................ | *********.           35960000 | 3503421      33358488         4512.1      7969.7     33072214
4607.85/1154.27	c ................................ | *********.           36525000 | 3555194      33832046         4608.1      7926.2     33318134
4704.04/1178.32	c ................................ | *********.           37096000 | 3606614      34301786         4704.1      7885.9     33640212
4799.88/1202.38	c CPU limit (4800) reached
4799.88/1202.38	c Time = 4800.18 seconds
4799.88/1202.38	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264510-1304100136/watcher-3264510-1304100136 -o /tmp/evaluation-result-3264510-1304100136/solver-3264510-1304100136 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264510-1304100136.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.45 7.96 7.82 3/176 14155
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) R 14151 14155 13368 0 -1 4202496 433 0 0 0 0 0 0 0 20 0 1 0 502429753 9633792 376 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106429637 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/14155/statm: 2352 376 147 36 0 257 0

[startup+0.0968299 s]
/proc/loadavg: 7.45 7.96 7.82 3/176 14155
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=12312 CPUtime=0.09 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) R 14151 14155 13368 0 -1 4202496 1194 0 0 0 9 0 0 0 20 0 1 0 502429753 12607488 1110 33554432000 4194304 4338100 140737126091184 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/14155/statm: 3078 1110 155 36 0 983 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12312

[startup+0.100828 s]
/proc/loadavg: 7.45 7.96 7.82 3/176 14155
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=12312 CPUtime=0.09 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) R 14151 14155 13368 0 -1 4202496 1194 0 0 0 9 0 0 0 20 0 1 0 502429753 12607488 1110 33554432000 4194304 4338100 140737126091184 18446744073709551615 4295940 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/14155/statm: 3078 1110 155 36 0 983 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12312

[startup+0.30076 s]
/proc/loadavg: 7.45 7.96 7.82 3/176 14155
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=211084 CPUtime=0.68 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 5400 0 0 0 67 1 0 0 21 0 5 0 502429753 216150016 5287 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 52771 5287 175 36 0 50676 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 211084

[startup+0.700692 s]
/proc/loadavg: 7.45 7.96 7.82 3/176 14155
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=277148 CPUtime=2.28 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 6890 0 0 0 226 2 0 0 21 0 5 0 502429753 283799552 6777 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 69287 6777 175 36 0 67192 0
Current children cumulated CPU time (s) 2.28
Current children cumulated vsize (KiB) 277148

[startup+1.50055 s]
/proc/loadavg: 7.73 8.01 7.83 10/186 14165
/proc/meminfo: memFree=24170344/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=284064 CPUtime=5.47 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 9680 0 0 0 542 5 0 0 21 0 5 0 502429753 290881536 9567 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 71016 9567 175 36 0 68921 0
[pid=14155/tid=14162] ppid=14151 vsize=284064 CPUtime=1.32 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 1922 0 0 0 131 1 0 0 19 0 5 0 502429770 290881536 9567 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106766907 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=284064 CPUtime=1.31 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 667 0 0 0 131 0 0 0 25 0 5 0 502429770 290881536 9567 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=284064 CPUtime=1.31 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 1119 0 0 0 131 0 0 0 19 0 5 0 502429770 290881536 9567 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=284064 CPUtime=1.32 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 1478 0 0 0 131 1 0 0 19 0 5 0 502429770 290881536 9567 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.47
Current children cumulated vsize (KiB) 284064

[startup+3.10027 s]
/proc/loadavg: 7.73 8.01 7.83 10/186 14165
/proc/meminfo: memFree=24117020/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=307016 CPUtime=11.86 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 15083 0 0 0 1171 15 0 0 21 0 5 0 502429753 314384384 14970 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 76754 14970 175 36 0 74659 0
[pid=14155/tid=14162] ppid=14151 vsize=307016 CPUtime=2.92 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 3978 0 0 0 288 4 0 0 25 0 5 0 502429770 314384384 14970 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=307016 CPUtime=2.9 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 1424 0 0 0 288 2 0 0 25 0 5 0 502429770 314384384 14970 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106163728 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=307016 CPUtime=2.91 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 2346 0 0 0 288 3 0 0 25 0 5 0 502429770 314384384 14970 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=307016 CPUtime=2.92 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 2841 0 0 0 289 3 0 0 25 0 5 0 502429770 314384384 14970 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.86
Current children cumulated vsize (KiB) 307016

[startup+6.30071 s]
/proc/loadavg: 7.73 8.01 7.83 10/186 14165
/proc/meminfo: memFree=23967892/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=345384 CPUtime=24.63 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 25928 0 0 0 2431 32 0 0 21 0 5 0 502429753 353673216 25759 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 86346 25760 176 36 0 84251 0
[pid=14155/tid=14162] ppid=14151 vsize=345384 CPUtime=6.12 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 8407 0 0 0 603 9 0 0 18 0 5 0 502429770 353673216 25760 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=345384 CPUtime=6.08 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 2872 0 0 0 602 6 0 0 18 0 5 0 502429770 353673216 25760 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106164576 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=345384 CPUtime=6.11 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 4877 0 0 0 604 7 0 0 25 0 5 0 502429770 353673216 25760 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=345384 CPUtime=6.12 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 5279 0 0 0 604 8 0 0 25 0 5 0 502429770 353673216 25760 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.63
Current children cumulated vsize (KiB) 345384

[startup+12.7006 s]
/proc/loadavg: 7.77 8.01 7.83 10/186 14165
/proc/meminfo: memFree=23744960/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=402024 CPUtime=50.18 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 42327 0 0 0 4942 76 0 0 21 0 5 0 502429753 411672576 42158 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 100506 42158 176 36 0 98411 0
[pid=14155/tid=14162] ppid=14151 vsize=402024 CPUtime=12.52 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 14772 0 0 0 1229 23 0 0 25 0 5 0 502429770 411672576 42158 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=402024 CPUtime=12.44 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 5049 0 0 0 1229 15 0 0 25 0 5 0 502429770 411672576 42158 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=402024 CPUtime=12.52 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 9783 0 0 0 1234 18 0 0 25 0 5 0 502429770 411672576 42158 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=402024 CPUtime=12.52 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 8229 0 0 0 1233 19 0 0 25 0 5 0 502429770 411672576 42158 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.18
Current children cumulated vsize (KiB) 402024

[startup+25.5003 s]
/proc/loadavg: 7.81 8.01 7.83 10/186 14166
/proc/meminfo: memFree=23450076/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=457196 CPUtime=101.29 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 58946 0 0 0 9939 190 0 0 21 0 5 0 502429753 468168704 58777 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 114299 58777 176 36 0 112204 0
[pid=14155/tid=14162] ppid=14151 vsize=457196 CPUtime=25.31 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 20384 0 0 0 2479 52 0 0 20 0 5 0 502429770 468168704 58777 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106465450 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=457196 CPUtime=25.17 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 8027 0 0 0 2474 43 0 0 25 0 5 0 502429770 468168704 58777 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106465410 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=457196 CPUtime=25.31 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 14541 0 0 0 2485 46 0 0 25 0 5 0 502429770 468168704 58777 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106163751 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=457196 CPUtime=25.32 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 11500 0 0 0 2484 48 0 0 25 0 5 0 502429770 468168704 58777 33554432000 4194304 4338100 140737126091184 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 101.29
Current children cumulated vsize (KiB) 457196

[startup+51.1008 s]
/proc/loadavg: 7.87 8.00 7.84 10/186 14166
/proc/meminfo: memFree=23054136/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=529200 CPUtime=203.49 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 82048 0 0 0 19963 386 0 0 21 0 5 0 502429753 541900800 81879 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 132300 81879 176 36 0 130205 0
[pid=14155/tid=14162] ppid=14151 vsize=529200 CPUtime=50.88 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 28305 0 0 0 4987 101 0 0 25 0 5 0 502429770 541900800 81879 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=529200 CPUtime=50.6 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 12393 0 0 0 4970 90 0 0 19 0 5 0 502429770 541900800 81879 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=529200 CPUtime=50.9 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 20874 0 0 0 4995 95 0 0 25 0 5 0 502429770 541900800 81879 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=529200 CPUtime=50.9 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 15982 0 0 0 4993 97 0 0 25 0 5 0 502429770 541900800 81879 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.49
Current children cumulated vsize (KiB) 529200

[startup+102.307 s]
/proc/loadavg: 7.95 8.00 7.84 10/186 14168
/proc/meminfo: memFree=22477404/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=633428 CPUtime=407.91 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 114690 0 0 0 40063 728 0 0 21 0 5 0 502429753 648630272 114521 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 158357 114521 176 36 0 156262 0
[pid=14155/tid=14162] ppid=14151 vsize=633428 CPUtime=102.06 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 39838 0 0 0 10016 190 0 0 25 0 5 0 502429770 648630272 114521 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=633428 CPUtime=101.5 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 17247 0 0 0 9976 174 0 0 25 0 5 0 502429770 648630272 114521 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=633428 CPUtime=102.07 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 30136 0 0 0 10026 181 0 0 25 0 5 0 502429770 648630272 114521 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=633428 CPUtime=102.07 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 22975 0 0 0 10027 180 0 0 25 0 5 0 502429770 648630272 114521 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.91
Current children cumulated vsize (KiB) 633428

[startup+162.3 s]
/proc/loadavg: 8.04 8.02 7.85 10/186 14169
/proc/meminfo: memFree=22004340/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=702284 CPUtime=647.44 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 138345 0 0 0 63613 1131 0 0 21 0 5 0 502429753 719138816 138176 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 175571 138176 176 36 0 173476 0
[pid=14155/tid=14162] ppid=14151 vsize=702284 CPUtime=162.03 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 48187 0 0 0 15912 291 0 0 25 0 5 0 502429770 719138816 138176 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=702284 CPUtime=161.14 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 21449 0 0 0 15839 275 0 0 25 0 5 0 502429770 719138816 138176 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=702284 CPUtime=162.05 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 37913 0 0 0 15919 286 0 0 18 0 5 0 502429770 719138816 138176 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=702284 CPUtime=162.04 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 26302 0 0 0 15927 277 0 0 25 0 5 0 502429770 719138816 138176 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.44
Current children cumulated vsize (KiB) 702284


################
# More data... #
################

/proc/meminfo: memFree=19790892/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1429032 CPUtime=2803.55 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 311904 0 0 0 276490 3865 0 0 21 0 5 0 502429753 1463328768 311735 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 357258 311735 176 36 0 355163 0
[pid=14155/tid=14162] ppid=14151 vsize=1429032 CPUtime=701.76 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 115390 0 0 0 69195 981 0 0 25 0 5 0 502429770 1463328768 311735 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1429032 CPUtime=697.94 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 52150 0 0 0 68845 949 0 0 25 0 5 0 502429770 1463328768 311735 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1429032 CPUtime=701.81 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 86378 0 0 0 69210 971 0 0 25 0 5 0 502429770 1463328768 311735 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1429032 CPUtime=701.84 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 53492 0 0 0 69224 960 0 0 25 0 5 0 502429770 1463328768 311735 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2803.55
Current children cumulated vsize (KiB) 1429032

[startup+762.307 s]
/proc/loadavg: 8.04 8.03 7.92 10/186 14185
/proc/meminfo: memFree=19565084/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1467492 CPUtime=3043.12 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 327308 0 0 0 300048 4264 0 0 21 0 5 0 502429753 1502711808 327139 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 366873 327139 176 36 0 364778 0
[pid=14155/tid=14162] ppid=14151 vsize=1467492 CPUtime=761.73 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 122552 0 0 0 75089 1084 0 0 25 0 5 0 502429770 1502711808 327139 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1467492 CPUtime=757.77 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 54237 0 0 0 74731 1046 0 0 25 0 5 0 502429770 1502711808 327139 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1467492 CPUtime=761.79 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 89987 0 0 0 75107 1072 0 0 25 0 5 0 502429770 1502711808 327139 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1467492 CPUtime=761.63 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 56038 0 0 0 75103 1060 0 0 22 0 5 0 502429770 1502711808 327139 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3043.12
Current children cumulated vsize (KiB) 1467492

[startup+822.301 s]
/proc/loadavg: 8.01 8.03 7.92 10/186 14187
/proc/meminfo: memFree=19339896/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1497108 CPUtime=3282.62 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 338460 0 0 0 323645 4617 0 0 21 0 5 0 502429753 1533038592 338291 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 374277 338291 176 36 0 372182 0
[pid=14155/tid=14162] ppid=14151 vsize=1497108 CPUtime=821.43 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 125933 0 0 0 80969 1174 0 0 25 0 5 0 502429770 1533038592 338291 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1497108 CPUtime=817.73 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 56420 0 0 0 80637 1136 0 0 25 0 5 0 502429770 1533038592 338291 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1497108 CPUtime=821.75 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 93525 0 0 0 81016 1159 0 0 25 0 5 0 502429770 1533038592 338291 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1497108 CPUtime=821.51 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 58088 0 0 0 81005 1146 0 0 25 0 5 0 502429770 1533038592 338291 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3282.62
Current children cumulated vsize (KiB) 1497108

[startup+882.301 s]
/proc/loadavg: 8.00 8.02 7.92 10/186 14188
/proc/meminfo: memFree=19190104/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1560276 CPUtime=3522.2 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 359023 0 0 0 347346 4874 0 0 21 0 5 0 502429753 1597722624 358854 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 390069 358854 176 36 0 387974 0
[pid=14155/tid=14162] ppid=14151 vsize=1560276 CPUtime=881.06 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 136617 0 0 0 86870 1236 0 0 25 0 5 0 502429770 1597722624 358854 33554432000 4194304 4338100 140737126091184 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1560276 CPUtime=877.72 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 58166 0 0 0 86570 1202 0 0 25 0 5 0 502429770 1597722624 358854 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106164462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1560276 CPUtime=881.74 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 99959 0 0 0 86949 1225 0 0 25 0 5 0 502429770 1597722624 358854 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106164128 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1560276 CPUtime=881.48 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 59787 0 0 0 86940 1208 0 0 25 0 5 0 502429770 1597722624 358854 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106163806 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3522.2
Current children cumulated vsize (KiB) 1560276

[startup+942.301 s]
/proc/loadavg: 8.00 8.01 7.92 10/186 14190
/proc/meminfo: memFree=19000372/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1569212 CPUtime=3761.73 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 367476 0 0 0 371057 5116 0 0 21 0 5 0 502429753 1606873088 367307 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 392303 367307 176 36 0 390208 0
[pid=14155/tid=14162] ppid=14151 vsize=1569212 CPUtime=941.01 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 139167 0 0 0 92803 1298 0 0 25 0 5 0 502429770 1606873088 367307 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1569212 CPUtime=937.36 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 61056 0 0 0 92475 1261 0 0 25 0 5 0 502429770 1606873088 367307 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1569212 CPUtime=941.71 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 101434 0 0 0 92886 1285 0 0 25 0 5 0 502429770 1606873088 367307 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1569212 CPUtime=941.44 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 61325 0 0 0 92875 1269 0 0 25 0 5 0 502429770 1606873088 367307 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3761.73
Current children cumulated vsize (KiB) 1569212

[startup+1002.3 s]
/proc/loadavg: 8.00 8.01 7.92 10/186 14192
/proc/meminfo: memFree=18744312/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1602776 CPUtime=4001.29 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 379650 0 0 0 394764 5365 0 0 21 0 5 0 502429753 1641242624 379481 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 400694 379481 176 36 0 398599 0
[pid=14155/tid=14162] ppid=14151 vsize=1602776 CPUtime=1000.99 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 145423 0 0 0 98741 1358 0 0 25 0 5 0 502429770 1641242624 379481 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1602776 CPUtime=996.99 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 62391 0 0 0 98374 1325 0 0 25 0 5 0 502429770 1641242624 379481 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1602776 CPUtime=1001.69 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 104675 0 0 0 98823 1346 0 0 25 0 5 0 502429770 1641242624 379481 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1602776 CPUtime=1001.41 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 62667 0 0 0 98808 1333 0 0 25 0 5 0 502429770 1641242624 379481 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4001.29
Current children cumulated vsize (KiB) 1602776

[startup+1062.3 s]
/proc/loadavg: 8.05 8.02 7.93 9/186 14193
/proc/meminfo: memFree=18554584/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1646252 CPUtime=4240.86 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 395224 0 0 0 418421 5665 0 0 21 0 5 0 502429753 1685762048 395055 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 411563 395055 176 36 0 409468 0
[pid=14155/tid=14162] ppid=14151 vsize=1646252 CPUtime=1060.95 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 152459 0 0 0 104666 1429 0 0 25 0 5 0 502429770 1685762048 395055 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1646252 CPUtime=1056.64 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 63859 0 0 0 104264 1400 0 0 25 0 5 0 502429770 1685762048 395055 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1646252 CPUtime=1061.66 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 110270 0 0 0 104746 1420 0 0 25 0 5 0 502429770 1685762048 395055 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1646252 CPUtime=1061.39 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 64142 0 0 0 104727 1412 0 0 25 0 5 0 502429770 1685762048 395055 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4240.86
Current children cumulated vsize (KiB) 1646252

[startup+1122.3 s]
/proc/loadavg: 8.02 8.01 7.93 10/186 14195
/proc/meminfo: memFree=18390652/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1876292 CPUtime=4480.45 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 406903 0 0 0 442116 5929 0 0 21 0 5 0 502429753 1921323008 406734 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 469073 406734 176 36 0 466978 0
[pid=14155/tid=14162] ppid=14151 vsize=1876292 CPUtime=1120.94 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 157785 0 0 0 110598 1496 0 0 25 0 5 0 502429770 1921323008 406734 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1876292 CPUtime=1116.29 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 65144 0 0 0 110162 1467 0 0 19 0 5 0 502429770 1921323008 406734 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1876292 CPUtime=1121.66 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 113554 0 0 0 110680 1486 0 0 25 0 5 0 502429770 1921323008 406734 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1876292 CPUtime=1121.36 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 65926 0 0 0 110658 1478 0 0 25 0 5 0 502429770 1921323008 406734 33554432000 4194304 4338100 140737126091184 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4480.45
Current children cumulated vsize (KiB) 1876292

[startup+1182.3 s]
/proc/loadavg: 8.09 8.04 7.94 9/186 14196
/proc/meminfo: memFree=18218284/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=1905936 CPUtime=4720.02 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) S 14151 14155 13368 0 -1 4202496 418184 0 0 0 465815 6187 0 0 21 0 5 0 502429753 1951678464 418015 33554432000 4194304 4338100 140737126091184 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 476484 418015 176 36 0 474389 0
[pid=14155/tid=14162] ppid=14151 vsize=1905936 CPUtime=1180.93 cores=1,3,5,7
/proc/14155/task/14162/stat : 14162 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 161871 0 0 0 116530 1563 0 0 25 0 5 0 502429770 1951678464 418015 33554432000 4194304 4338100 140737126091184 18446744073709551615 4241488 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=14155/tid=14163] ppid=14151 vsize=1905936 CPUtime=1175.92 cores=1,3,5,7
/proc/14155/task/14163/stat : 14163 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 66318 0 0 0 116062 1530 0 0 25 0 5 0 502429770 1951678464 418015 33554432000 4194304 4338100 140737126091184 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=14155/tid=14164] ppid=14151 vsize=1905936 CPUtime=1181.63 cores=1,3,5,7
/proc/14155/task/14164/stat : 14164 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 117935 0 0 0 116611 1552 0 0 25 0 5 0 502429770 1951678464 418015 33554432000 4194304 4338100 140737126091184 18446744073709551615 255106163955 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=1905936 CPUtime=1181.35 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4202560 67566 0 0 0 116595 1540 0 0 25 0 5 0 502429770 1951678464 418015 33554432000 4194304 4338100 140737126091184 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4720.02
Current children cumulated vsize (KiB) 1905936



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.4 s]
/proc/loadavg: 8.06 8.03 7.94 6/183 14197
/proc/meminfo: memFree=18520024/32951124 swapFree=67111524/67111528
[pid=14155] ppid=14151 vsize=0 CPUtime=4800.21 cores=1,3,5,7
/proc/14155/stat : 14155 (ttsth-5-4) Z 14151 14155 13368 0 -1 4203548 423019 0 0 0 473749 6272 0 0 18 0 2 0 502429753 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/14155/statm: 0 0 0 0 0 0 0
[pid=14155/tid=14165] ppid=14151 vsize=0 CPUtime=1201.44 cores=1,3,5,7
/proc/14155/task/14165/stat : 14165 (ttsth-5-4) R 14151 14155 13368 0 -1 4203588 68813 0 0 0 118581 1563 0 0 25 0 2 0 502429770 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 1202.46
CPU time (s): 4800.28
CPU user time (s): 4737.5
CPU system time (s): 62.7775
CPU usage (%): 399.203
Max. virtual memory (cumulated for all children) (KiB): 1920296

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4737.5
system time used= 62.7775
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 423019
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 271072
involuntary context switches= 23731

runsolver used 1.92071 second user time and 4.65429 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-29 20:02:16
IDJOB=3264510
IDBENCH=84643
IDSOLVER=1654
FILE ID=node144/3264510-1304100136
RUNJOBID= node144-1304100136-14137
PBS_JOBID= 13170586
Free space on /tmp= 72408 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/connamacher/connm-d-0.02/connm-ue-csp-sat-n1200-d-0.02-s405595518.used-as.sat04-950.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264510-1304100136/watcher-3264510-1304100136 -o /tmp/evaluation-result-3264510-1304100136/solver-3264510-1304100136 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264510-1304100136.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 287d3b23c6f72107e24ae4186653564e
RANDOM SEED=1221052803

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24247592 kB
Buffers:       2124112 kB
Cached:        5966076 kB
SwapCached:          4 kB
Active:         183788 kB
Inactive:      7970044 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24247592 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             504 kB
Writeback:           0 kB
AnonPages:       62612 kB
Mapped:          14480 kB
Slab:           486248 kB
PageTables:       4868 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187740 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72408 MiB
End job on node144 at 2011-04-29 20:22:21