Trace number 3259098

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.36 1203.05

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v760c43649g7.cnf
MD5SUMbd9e06c08721e6eb7b4cb4a265dbdd48
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark4.14637
Satisfiable
(Un)Satisfiability was proved
Number of variables760
Number of clauses43649
Sum of the clauses size87978
Maximum clause length19
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243609
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259098-1304054069.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c a2
0.03/0.05	c a1
0.03/0.10	c Independent sub-problems = 1
0.03/0.10	c Sub-problem with 760 variables and 43649 clauses
0.03/0.10	c using permutation method 4
0.03/0.10	c count = 1
0.03/0.10	c Ordering finished 0.080 seconds
0.03/0.10	c mapped 0.090 seconds
0.03/0.10	c permuted 0.090 seconds
0.03/0.10	c about to sort
0.09/0.10	c sorted
0.09/0.19	c clauses now 43649
0.09/0.19	c init finished 0.100 seconds
0.09/0.19	c tree built 0.180 seconds
0.09/0.19	c 
0.09/0.19	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.19	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.19	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.19	c ................................ | ..........                  1 |              1                   0.2         5.6             
0.09/0.19	c ................................ | ..........                  2 |              1                   0.2        11.1             
0.09/0.19	c ................................ | ..........                  3 |              1                   0.2        16.7             
0.09/0.19	c ................................ | ..........                  4 |              1                   0.2        22.2             
0.09/0.19	c ................................ | ..........                  5 |              1                   0.2        27.8             
0.09/0.20	c ................................ | ..........               1000 | 5            173                 0.2      5263.2         7620
96.08/24.22	c ................................ | ..........            3440000 | 61341        732207             96.2     35777.4     23044655
191.87/48.23	c ................................ | ..........            5862000 | 112987       1245094           192.0     30529.7     35805068
287.70/72.27	c ................................ | ..........            8242000 | 161305       1689682           288.0     28618.1     45014234
383.95/96.32	c ................................ | ..........           10642000 | 206138       2154348           384.0     27710.7     55738243
479.76/120.36	c ................................ | *.........           13006000 | 249990       2599994           480.0     27094.7     63780694
575.96/144.42	c ................................ | *.........           15200000 | 291659       2983467           576.0     26387.1     74809651
671.79/168.45	c ................................ | *.........           17230000 | 330719       3357441           672.0     25639.1     82873779
767.65/192.49	c ................................ | *.........           19044000 | 368158       3709152           768.0     24796.9     93721244
863.84/216.55	c ................................ | *.........           20994000 | 408377       4055773           864.0     24297.5    100613122
959.67/240.58	c ................................ | **........           22884000 | 448249       4393631           960.0     23837.3    105416543
1055.87/264.67	c ................................ | **........           24590000 | 486723       4722944          1056.2     23282.0    111932221
1151.68/288.69	c ................................ | **........           26470000 | 522439       5055574          1152.0     22976.6    116902094
1247.89/312.73	c ................................ | **........           27990000 | 555761       5362014          1248.0     22427.3    121959902
1343.69/336.78	c ................................ | **........           29979000 | 594933       5682636          1344.0     22305.6    122334209
1439.90/360.83	c ................................ | ***.......           31653000 | 631100       6002792          1440.0     21981.2    123614715
1535.66/384.88	c ................................ | ***.......           33389000 | 668210       6357594          1536.0     21737.6    127539553
1631.86/408.93	c ................................ | ***.......           34994000 | 705942       6669730          1632.0     21442.4    132980616
1727.64/432.99	c ................................ | ***.......           36555000 | 739785       6980588          1728.0     21154.3    135803979
1823.83/457.04	c ................................ | ***.......           37953000 | 771153       7285566          1824.0     20807.6    138728132
1919.98/481.11	c ................................ | ****......           39349000 | 804802       7618968          1920.0     20493.7    143668444
2015.81/505.16	c ................................ | ****......           40667000 | 835812       7940466          2016.1     20171.5    151009695
2111.96/529.21	c ................................ | ****......           41951000 | 867478       8237355          2112.0     19863.0    154699996
2207.79/553.27	c ................................ | ****......           43099000 | 897030       8510091          2208.1     19518.8    160024451
2303.99/577.31	c ................................ | ****......           44275000 | 927306       8789420          2304.0     19216.2    160311475
2399.73/601.37	c ................................ | *****.....           45343000 | 955242       9047864          2400.0     18892.7    160568375
2495.96/625.42	c ................................ | *****.....           46548000 | 985896       9325123          2496.1     18648.7    164440263
2591.75/649.48	c ................................ | *****.....           47645000 | 1013689      9585686          2592.1     18381.0    165008316
2687.93/673.51	c ................................ | *****.....           48829000 | 1041748      9846784          2688.0     18165.6    168828854
2783.66/697.58	c ................................ | *****.....           50694000 | 1074001      10141143         2784.0     18209.1    177442532
2879.83/721.65	c ................................ | ******....           52362000 | 1106251      10439870         2880.0     18181.1    182962125
2975.64/745.70	c ................................ | ******....           53965000 | 1137473      10711304         2976.0     18133.2    186291612
3071.72/769.78	c ................................ | ******....           55457000 | 1169815      10997616         3072.0     18052.2    189767222
3167.89/793.83	c ................................ | ******....           56950000 | 1201352      11271344         3168.0     17976.5    196152064
3263.67/817.89	c ................................ | ******....           58273000 | 1229828      11519333         3264.0     17853.0    198908445
3359.90/841.92	c ................................ | *******...           59650000 | 1260375      11771438         3360.0     17752.9    203452658
3455.72/865.98	c ................................ | *******...           60901000 | 1288510      12004714         3456.1     17621.5    203703715
3551.87/890.03	c ................................ | *******...           62065000 | 1315420      12224974         3552.0     17473.3    203703715
3648.04/914.10	c ................................ | *******...           63443000 | 1342463      12437245         3648.0     17391.0    203937590
3743.79/938.16	c ................................ | *******...           64779000 | 1370791      12657017         3744.0     17301.9    206785847
3839.65/962.19	c ................................ | ********..           66109000 | 1400502      12869821         3840.0     17215.8    209371026
3935.84/986.25	c ................................ | ********..           67308000 | 1425486      13090814         3936.1     17100.3    210818815
4032.04/1010.30	c ................................ | ********..           68529000 | 1452210      13348787         4032.1     16996.0    212315314
4127.72/1034.39	c ................................ | ********..           69654000 | 1475577      13597988         4128.0     16873.4    215775917
4223.88/1058.44	c ................................ | ********..           70720000 | 1500137      13832917         4224.1     16742.2    218808319
4320.05/1082.50	c ................................ | *********.           71728000 | 1520907      14077070         4320.1     16603.5    222991433
4415.85/1106.54	c ................................ | *********.           72638000 | 1541398      14328599         4416.0     16448.7    223431280
4511.69/1130.59	c ................................ | *********.           73733000 | 1564341      14569205         4512.1     16341.4    224667263
4607.81/1154.66	c ................................ | *********.           74892000 | 1590073      14802858         4608.1     16252.4    224870481
4704.03/1178.70	c ................................ | *********.           75958000 | 1613160      15022530         4704.1     16147.4    226313267
4799.80/1202.76	c CPU limit (4800) reached
4799.80/1202.76	c Time = 4800.06 seconds
4799.80/1202.76	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259098-1304054069/watcher-3259098-1304054069 -o /tmp/evaluation-result-3259098-1304054069/solver-3259098-1304054069 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259098-1304054069.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.74 4.91 6.53 2/171 21392
/proc/meminfo: memFree=25147808/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) R 21389 21392 18837 0 -1 4202496 501 0 0 0 0 0 0 0 18 0 1 0 517006589 9994240 443 33554432000 4194304 4338100 140735666653216 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21392/statm: 2440 444 147 36 0 345 0

[startup+0.042608 s]
/proc/loadavg: 3.74 4.91 6.53 2/171 21392
/proc/meminfo: memFree=25147808/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=17376 CPUtime=0.03 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) R 21389 21392 18837 0 -1 4202496 2379 0 0 0 3 0 0 0 18 0 1 0 517006589 17793024 2320 33554432000 4194304 4338100 140735666653216 18446744073709551615 4206987 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21392/statm: 4344 2320 153 36 0 2249 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 17376

[startup+0.100585 s]
/proc/loadavg: 3.74 4.91 6.53 2/171 21392
/proc/meminfo: memFree=25147808/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=18260 CPUtime=0.09 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) R 21389 21392 18837 0 -1 4202496 3504 0 0 0 9 0 0 0 18 0 1 0 517006589 18698240 2579 33554432000 4194304 4338100 140735666653216 18446744073709551615 274664466777 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21392/statm: 4565 2579 161 36 0 2470 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18260

[startup+0.38705 s]
/proc/loadavg: 3.74 4.91 6.53 2/171 21392
/proc/meminfo: memFree=25147808/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=231856 CPUtime=0.99 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 12985 0 0 0 97 2 0 0 18 0 5 0 517006589 237420544 11973 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 57964 11973 176 36 0 55869 0
Current children cumulated CPU time (s) 0.99
Current children cumulated vsize (KiB) 231856

[startup+0.700433 s]
/proc/loadavg: 3.74 4.91 6.53 2/171 21392
/proc/meminfo: memFree=25147808/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=250888 CPUtime=2.21 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 17406 0 0 0 216 5 0 0 18 0 5 0 517006589 256909312 16394 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 62722 16394 176 36 0 60627 0
Current children cumulated CPU time (s) 2.21
Current children cumulated vsize (KiB) 250888

[startup+1.5003 s]
/proc/loadavg: 3.74 4.91 6.53 10/184 21406
/proc/meminfo: memFree=24990240/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=290600 CPUtime=5.4 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 27281 0 0 0 530 10 0 0 18 0 5 0 517006589 297574400 26269 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 72650 26270 176 36 0 70555 0
[pid=21392/tid=21399] ppid=21389 vsize=290600 CPUtime=1.3 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 6138 0 0 0 129 1 0 0 19 0 5 0 517006608 297574400 26270 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=290600 CPUtime=1.3 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 3377 0 0 0 127 3 0 0 19 0 5 0 517006608 297574400 26270 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=290600 CPUtime=1.29 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 6419 0 0 0 129 0 0 0 22 0 5 0 517006608 297574400 26270 33554432000 4194304 4338100 140735666653216 18446744073709551615 4241488 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=290600 CPUtime=1.3 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 2331 0 0 0 127 3 0 0 19 0 5 0 517006608 297574400 26270 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 5.4
Current children cumulated vsize (KiB) 290600

[startup+3.10102 s]
/proc/loadavg: 4.16 4.98 6.54 9/184 21406
/proc/meminfo: memFree=24896372/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=361340 CPUtime=11.8 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 44447 0 0 0 1157 23 0 0 18 0 5 0 517006589 370012160 43435 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 90335 43435 176 36 0 88240 0
[pid=21392/tid=21399] ppid=21389 vsize=361340 CPUtime=2.89 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 12106 0 0 0 287 2 0 0 25 0 5 0 517006608 370012160 43436 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=361340 CPUtime=2.89 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 6327 0 0 0 282 7 0 0 25 0 5 0 517006608 370012160 43437 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=361340 CPUtime=2.89 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 12405 0 0 0 287 2 0 0 25 0 5 0 517006608 370012160 43437 33554432000 4194304 4338100 140735666653216 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=361340 CPUtime=2.9 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 4594 0 0 0 282 8 0 0 25 0 5 0 517006608 370012160 43437 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.8
Current children cumulated vsize (KiB) 361340

[startup+6.30047 s]
/proc/loadavg: 4.16 4.98 6.54 9/184 21406
/proc/meminfo: memFree=24657332/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=480204 CPUtime=24.58 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 74143 0 0 0 2411 47 0 0 18 0 5 0 517006589 491728896 73131 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 120051 73131 176 36 0 117956 0
[pid=21392/tid=21399] ppid=21389 vsize=480204 CPUtime=6.1 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 22010 0 0 0 605 5 0 0 25 0 5 0 517006608 491728896 73131 33554432000 4194304 4338100 140735666653216 18446744073709551615 274664203710 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=480204 CPUtime=6.1 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 11379 0 0 0 593 17 0 0 25 0 5 0 517006608 491728896 73131 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=480204 CPUtime=6.07 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 23036 0 0 0 603 4 0 0 25 0 5 0 517006608 491728896 73131 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676621006 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=480204 CPUtime=6.1 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 8701 0 0 0 592 18 0 0 25 0 5 0 517006608 491728896 73131 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.58
Current children cumulated vsize (KiB) 480204

[startup+12.7004 s]
/proc/loadavg: 4.83 5.09 6.56 9/184 21406
/proc/meminfo: memFree=24241784/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=690196 CPUtime=50.14 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 127171 0 0 0 4912 102 0 0 18 0 5 0 517006589 706760704 126159 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 172549 126159 176 36 0 170454 0
[pid=21392/tid=21399] ppid=21389 vsize=690196 CPUtime=12.49 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 40594 0 0 0 1240 9 0 0 25 0 5 0 517006608 706760704 126159 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=690196 CPUtime=12.49 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 20798 0 0 0 1208 41 0 0 25 0 5 0 517006608 706760704 126160 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=690196 CPUtime=12.43 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 40398 0 0 0 1237 6 0 0 25 0 5 0 517006608 706760704 126160 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=690196 CPUtime=12.49 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 16365 0 0 0 1208 41 0 0 25 0 5 0 517006608 706760704 126160 33554432000 4194304 4338100 140735666653216 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 50.14
Current children cumulated vsize (KiB) 690196

[startup+25.5011 s]
/proc/loadavg: 5.32 5.19 6.58 9/184 21406
/proc/meminfo: memFree=23687136/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=976252 CPUtime=101.24 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 197250 0 0 0 9916 208 0 0 18 0 5 0 517006589 999682048 195950 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 244063 195950 176 36 0 241968 0
[pid=21392/tid=21399] ppid=21389 vsize=976252 CPUtime=25.23 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 69061 0 0 0 2505 18 0 0 25 0 5 0 517006608 999682048 195950 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=976252 CPUtime=25.29 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 29859 0 0 0 2440 89 0 0 25 0 5 0 517006608 999682048 195950 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=976252 CPUtime=25.22 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 64918 0 0 0 2511 11 0 0 25 0 5 0 517006608 999682048 195950 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=976252 CPUtime=25.29 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 24395 0 0 0 2442 87 0 0 25 0 5 0 517006608 999682048 195950 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.24
Current children cumulated vsize (KiB) 976252

[startup+51.1007 s]
/proc/loadavg: 6.30 5.43 6.62 9/184 21407
/proc/meminfo: memFree=22919424/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=1434904 CPUtime=203.44 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 311703 0 0 0 19945 399 0 0 18 0 5 0 517006589 1469341696 310403 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 358726 310403 176 36 0 356631 0
[pid=21392/tid=21399] ppid=21389 vsize=1434904 CPUtime=50.68 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 114842 0 0 0 5037 31 0 0 25 0 5 0 517006608 1469341696 310403 33554432000 4194304 4338100 140735666653216 18446744073709551615 4253888 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=1434904 CPUtime=50.86 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 46847 0 0 0 4911 175 0 0 25 0 5 0 517006608 1469341696 310403 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=1434904 CPUtime=50.81 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 103875 0 0 0 5061 20 0 0 25 0 5 0 517006608 1469341696 310403 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=1434904 CPUtime=50.88 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 37122 0 0 0 4918 170 0 0 25 0 5 0 517006608 1469341696 310403 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.44
Current children cumulated vsize (KiB) 1434904

[startup+102.307 s]
/proc/loadavg: 7.39 5.87 6.70 9/184 21408
/proc/meminfo: memFree=21726048/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=2122596 CPUtime=407.92 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 483541 0 0 0 40034 758 0 0 18 0 5 0 517006589 2173538304 482241 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 530649 482241 176 36 0 528554 0
[pid=21392/tid=21399] ppid=21389 vsize=2122596 CPUtime=101.86 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 204369 0 0 0 10130 56 0 0 25 0 5 0 517006608 2173538304 482241 33554432000 4194304 4338100 140735666653216 18446744073709551615 274664504752 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=2122596 CPUtime=102.04 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 65382 0 0 0 9872 332 0 0 25 0 5 0 517006608 2173538304 482241 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=2122596 CPUtime=101.73 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 161325 0 0 0 10139 34 0 0 25 0 5 0 517006608 2173538304 482241 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=2122596 CPUtime=102.08 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 43448 0 0 0 9875 333 0 0 25 0 5 0 517006608 2173538304 482241 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 407.92
Current children cumulated vsize (KiB) 2122596

[startup+162.301 s]
/proc/loadavg: 7.90 6.26 6.79 9/184 21410
/proc/meminfo: memFree=20538616/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=2859232 CPUtime=647.44 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 666385 0 0 0 63593 1151 0 0 18 0 5 0 517006589 2927853568 665085 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 714808 665085 176 36 0 712713 0
[pid=21392/tid=21399] ppid=21389 vsize=2859232 CPUtime=161.81 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 301641 0 0 0 16097 84 0 0 25 0 5 0 517006608 2927853568 665085 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676622369 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=2859232 CPUtime=161.98 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 82363 0 0 0 15686 512 0 0 25 0 5 0 517006608 2927853568 665085 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=2859232 CPUtime=161.58 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 212729 0 0 0 16113 45 0 0 25 0 5 0 517006608 2927853568 665085 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=2859232 CPUtime=161.86 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 60635 0 0 0 15679 507 0 0 25 0 5 0 517006608 2927853568 665085 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 647.44
Current children cumulated vsize (KiB) 2859232


################
# More data... #
################

/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 99854 0 0 0 68174 1892 0 0 25 0 5 0 517006608 6215000064 1468398 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2802.82
Current children cumulated vsize (KiB) 6069336

[startup+762.307 s]
/proc/loadavg: 8.09 7.85 7.39 9/184 21426
/proc/meminfo: memFree=14469588/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=6390372 CPUtime=3042.22 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1550140 0 0 0 299838 4384 0 0 18 0 5 0 517006589 6543740928 1548840 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1597593 1548840 176 36 0 1595498 0
[pid=21392/tid=21399] ppid=21389 vsize=6390372 CPUtime=760.49 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 786140 0 0 0 75792 257 0 0 25 0 5 0 517006608 6543740928 1548840 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=6390372 CPUtime=761.1 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 155538 0 0 0 74119 1991 0 0 25 0 5 0 517006608 6543740928 1548840 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=6390372 CPUtime=759.79 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 496546 0 0 0 75866 113 0 0 25 0 5 0 517006608 6543740928 1548840 33554432000 4194304 4338100 140735666653216 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=6390372 CPUtime=760.62 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 102899 0 0 0 74042 2020 0 0 25 0 5 0 517006608 6543740928 1548840 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.22
Current children cumulated vsize (KiB) 6390372

[startup+822.301 s]
/proc/loadavg: 8.10 7.90 7.44 9/184 21428
/proc/meminfo: memFree=13971212/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=6676868 CPUtime=3281.65 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1624488 0 0 0 323535 4630 0 0 18 0 5 0 517006589 6837112832 1623188 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1669217 1623188 176 36 0 1667122 0
[pid=21392/tid=21399] ppid=21389 vsize=6676868 CPUtime=820.41 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 826370 0 0 0 81769 272 0 0 25 0 5 0 517006608 6837112832 1623188 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=6676868 CPUtime=821.05 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 156288 0 0 0 80003 2102 0 0 25 0 5 0 517006608 6837112832 1623188 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=6676868 CPUtime=819.39 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 529611 0 0 0 81818 121 0 0 25 0 5 0 517006608 6837112832 1623188 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=6676868 CPUtime=820.57 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 103202 0 0 0 79925 2132 0 0 25 0 5 0 517006608 6837112832 1623188 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3281.65
Current children cumulated vsize (KiB) 6676868

[startup+882.301 s]
/proc/loadavg: 8.06 7.93 7.48 9/184 21429
/proc/meminfo: memFree=13597716/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=6852468 CPUtime=3521.13 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1663846 0 0 0 347254 4859 0 0 18 0 5 0 517006589 7016927232 1662546 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1713117 1662546 176 36 0 1711022 0
[pid=21392/tid=21399] ppid=21389 vsize=6852468 CPUtime=880.35 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 849015 0 0 0 87754 281 0 0 25 0 5 0 517006608 7016927232 1662546 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=6852468 CPUtime=881.01 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 156554 0 0 0 85892 2209 0 0 25 0 5 0 517006608 7016927232 1662546 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=6852468 CPUtime=879.02 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 545795 0 0 0 87778 124 0 0 25 0 5 0 517006608 7016927232 1662546 33554432000 4194304 4338100 140735666653216 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=6852468 CPUtime=880.54 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 103465 0 0 0 85812 2242 0 0 25 0 5 0 517006608 7016927232 1662546 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3521.13
Current children cumulated vsize (KiB) 6852468

[startup+942.301 s]
/proc/loadavg: 8.02 7.94 7.50 9/184 21431
/proc/meminfo: memFree=13245784/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=7001628 CPUtime=3760.57 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1700319 0 0 0 371001 5056 0 0 18 0 5 0 517006589 7169667072 1699019 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1750407 1699019 176 36 0 1748312 0
[pid=21392/tid=21399] ppid=21389 vsize=7001628 CPUtime=940.29 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 876288 0 0 0 93736 293 0 0 25 0 5 0 517006608 7169667072 1699019 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=7001628 CPUtime=940.8 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 159757 0 0 0 91782 2298 0 0 25 0 5 0 517006608 7169667072 1699019 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=7001628 CPUtime=938.77 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 547407 0 0 0 93753 124 0 0 25 0 5 0 517006608 7169667072 1699019 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=7001628 CPUtime=940.47 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 107850 0 0 0 91710 2337 0 0 25 0 5 0 517006608 7169667072 1699019 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3760.57
Current children cumulated vsize (KiB) 7001628

[startup+1002.3 s]
/proc/loadavg: 8.11 7.98 7.55 9/184 21432
/proc/meminfo: memFree=12877496/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=7122472 CPUtime=4000.09 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1733944 0 0 0 394741 5268 0 0 18 0 5 0 517006589 7293411328 1732644 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1780618 1732645 176 36 0 1778523 0
[pid=21392/tid=21399] ppid=21389 vsize=7122472 CPUtime=1000.26 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 904172 0 0 0 99719 307 0 0 25 0 5 0 517006608 7293411328 1732645 33554432000 4194304 4338100 140735666653216 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=7122472 CPUtime=1000.44 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 160733 0 0 0 97647 2397 0 0 25 0 5 0 517006608 7293411328 1732645 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=7122472 CPUtime=998.74 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 550435 0 0 0 99749 125 0 0 25 0 5 0 517006608 7293411328 1732645 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=7122472 CPUtime=1000.42 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 109588 0 0 0 97606 2436 0 0 25 0 5 0 517006608 7293411328 1732645 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4000.09
Current children cumulated vsize (KiB) 7122472

[startup+1062.3 s]
/proc/loadavg: 8.03 7.97 7.56 9/184 21434
/proc/meminfo: memFree=12362512/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=7396456 CPUtime=4239.45 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1802608 0 0 0 418417 5528 0 0 18 0 5 0 517006589 7573970944 1801308 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1849114 1801308 176 36 0 1847019 0
[pid=21392/tid=21399] ppid=21389 vsize=7396456 CPUtime=1060.1 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 949112 0 0 0 105687 323 0 0 25 0 5 0 517006608 7573970944 1801308 33554432000 4194304 4338100 140735666653216 18446744073709551615 274664203706 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=7396456 CPUtime=1060.35 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 161098 0 0 0 103520 2515 0 0 25 0 5 0 517006608 7573970944 1801308 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=7396456 CPUtime=1058.67 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 573458 0 0 0 105736 131 0 0 25 0 5 0 517006608 7573970944 1801308 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=7396456 CPUtime=1060.1 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 109923 0 0 0 103455 2555 0 0 25 0 5 0 517006608 7573970944 1801308 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4239.45
Current children cumulated vsize (KiB) 7396456

[startup+1122.3 s]
/proc/loadavg: 8.13 8.00 7.60 9/184 21436
/proc/meminfo: memFree=11904312/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=7529944 CPUtime=4478.94 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1838427 0 0 0 442127 5767 0 0 18 0 5 0 517006589 7710662656 1837127 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1882486 1837127 176 36 0 1880391 0
[pid=21392/tid=21399] ppid=21389 vsize=7529944 CPUtime=1120.02 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 962489 0 0 0 111673 329 0 0 25 0 5 0 517006608 7710662656 1837127 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=7529944 CPUtime=1120.3 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 161416 0 0 0 109404 2626 0 0 25 0 5 0 517006608 7710662656 1837127 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=7529944 CPUtime=1118.64 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 594967 0 0 0 111728 136 0 0 25 0 5 0 517006608 7710662656 1837127 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=7529944 CPUtime=1119.76 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 110538 0 0 0 109303 2673 0 0 25 0 5 0 517006608 7710662656 1837127 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4478.94
Current children cumulated vsize (KiB) 7529944

[startup+1182.3 s]
/proc/loadavg: 8.18 8.03 7.63 9/184 21437
/proc/meminfo: memFree=11758216/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=7622504 CPUtime=4718.39 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) S 21389 21392 18837 0 -1 4202496 1866146 0 0 0 465865 5974 0 0 18 0 5 0 517006589 7805444096 1864846 33554432000 4194304 4338100 140735666653216 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 1905626 1864846 176 36 0 1903531 0
[pid=21392/tid=21399] ppid=21389 vsize=7622504 CPUtime=1179.99 cores=1,3,5,7
/proc/21392/task/21399/stat : 21399 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 969091 0 0 0 117664 335 0 0 25 0 5 0 517006608 7805444096 1864846 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21392/tid=21400] ppid=21389 vsize=7622504 CPUtime=1180.22 cores=1,3,5,7
/proc/21392/task/21400/stat : 21400 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 162708 0 0 0 115299 2723 0 0 25 0 5 0 517006608 7805444096 1864846 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21392/tid=21401] ppid=21389 vsize=7622504 CPUtime=1178.45 cores=1,3,5,7
/proc/21392/task/21401/stat : 21401 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 612260 0 0 0 117702 143 0 0 25 0 5 0 517006608 7805444096 1864846 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=7622504 CPUtime=1179.5 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4202560 113070 0 0 0 115180 2770 0 0 25 0 5 0 517006608 7805444096 1864846 33554432000 4194304 4338100 140735666653216 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4718.39
Current children cumulated vsize (KiB) 7622504



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.8 s]
/proc/loadavg: 8.11 8.03 7.64 6/181 21438
/proc/meminfo: memFree=12338776/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) Z 21389 21392 18837 0 -1 4203548 1885297 0 0 0 473963 6047 0 0 18 0 2 0 517006589 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 0 0 0 0 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=0 CPUtime=1199.99 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4203588 114056 0 0 0 117193 2806 0 0 25 0 2 0 517006608 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203 s]
/proc/loadavg: 8.11 8.03 7.64 6/181 21438
/proc/meminfo: memFree=12338776/32951132 swapFree=67111524/67111528
[pid=21392] ppid=21389 vsize=0 CPUtime=4800.29 cores=1,3,5,7
/proc/21392/stat : 21392 (ttsth-5-4) Z 21389 21392 18837 0 -1 4203548 1885297 0 0 0 473963 6066 0 0 18 0 2 0 517006589 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21392/statm: 0 0 0 0 0 0 0
[pid=21392/tid=21402] ppid=21389 vsize=0 CPUtime=1200.19 cores=1,3,5,7
/proc/21392/task/21402/stat : 21402 (ttsth-5-4) R 21389 21392 18837 0 -1 4203588 114056 0 0 0 117193 2826 0 0 25 0 2 0 517006608 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.05
CPU time (s): 4800.36
CPU user time (s): 4739.64
CPU system time (s): 60.7188
CPU usage (%): 399.015
Max. virtual memory (cumulated for all children) (KiB): 7695240

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4739.64
system time used= 60.7188
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1885297
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8743
involuntary context switches= 32288

runsolver used 2.02669 second user time and 4.55131 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 07:14:30
IDJOB=3259098
IDBENCH=71487
IDSOLVER=1654
FILE ID=node131/3259098-1304054069
RUNJOBID= node131-1304054069-21378
PBS_JOBID= 13170364
Free space on /tmp= 71384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v760c43649g7.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259098-1304054069/watcher-3259098-1304054069 -o /tmp/evaluation-result-3259098-1304054069/solver-3259098-1304054069 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259098-1304054069.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= bd9e06c08721e6eb7b4cb4a265dbdd48
RANDOM SEED=188512661

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      25147972 kB
Buffers:        956660 kB
Cached:        6285820 kB
SwapCached:          4 kB
Active:         276708 kB
Inactive:      7031968 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      25147972 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1272 kB
Writeback:           0 kB
AnonPages:       65796 kB
Mapped:          14712 kB
Slab:           430876 kB
PageTables:       4752 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   190372 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71384 MiB
End job on node131 at 2011-04-29 07:34:34