Trace number 1780003

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth-5-0 2009-03-19 (4 threads)? (TO) 2181.11 1000.13

General information on the benchmark

NameAPPLICATIONS/bioinfo/
q_query_3_l47_lambda.cnf
MD5SUMed7689c65135a913d97feca3ab3874eb
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark390.06
Satisfiable
(Un)Satisfiability was proved
Number of variables33871
Number of clauses170161
Sum of the clauses size490938
Maximum clause length9
Minimum clause length1
Number of clauses of size 14346
Number of clauses of size 241947
Number of clauses of size 3100545
Number of clauses of size 416116
Number of clauses of size 57066
Number of clauses of size over 5141

Solver Data 

0.40/0.42	c Command: HOME/ttsth-5-0 -num-threads 4 HOME/instance-1780003-1243680640.cnf 
0.40/0.42	c Using SATTIMEOUT of 10000
0.40/0.42	c Using SATRAM of 2047
0.40/0.42	c Size of subset node = 16
0.40/0.42	c Variables = 33871
0.40/0.42	c Clauses = 170161
0.40/0.42	c Literals = 490938
0.40/0.42	c Reduced size:
0.40/0.42	c Variables = 33871
0.40/0.42	c Clauses = 153802
0.40/0.42	c Literals = 449883
0.40/0.42	c a2
0.70/0.80	c a1
11.50/11.51	c Independent sub-problems = 1
11.50/11.51	c Sub-problem with 28617 variables and 153802 clauses
11.50/11.51	c using permutation method 4
11.50/11.51	c count = 1
11.50/11.51	c Ordering finished 11.440 seconds
11.50/11.51	c mapped 11.510 seconds
11.50/11.51	c permuted 11.510 seconds
11.50/11.51	c about to sort
11.50/11.56	c sorted
14.10/14.20	c clauses now 153802
14.10/14.20	c init finished 11.760 seconds
14.10/14.20	c tree built 14.200 seconds
14.10/14.20	c 
14.10/14.20	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
14.10/14.20	c ---------------------------------|----------------------------------|------------------------------------------------------
14.10/14.20	c ................................ | ..........                       |                       14.2           0.0             
14.10/14.20	c ................................ | ..........                     1 |                       14.2           0.1             
14.10/14.20	c ****************................ | ..........                     2 |                       14.2           0.1            1
14.10/14.20	c ****************................ | ..........                     3 |                       14.2           0.2            1
14.10/14.20	c ****************................ | ..........                     4 |                       14.2           0.3            1
14.10/14.20	c ****************................ | ..........                     5 |                       14.2           0.4            1
14.10/14.20	c ****************................ | ..........                     6 |                       14.2           0.4            1
14.21/14.21	c ****************................ | ..........                  1000 | 3                     14.2          70.4          111
200.15/105.26	c ****************................ | ..........               1939000 | 131194               200.3        9682.4     17507074
400.12/196.55	c ****************................ | ..........               2972000 | 201700               400.2        7426.1     27230763
599.93/295.50	c ****************................ | ..........               3931000 | 265183               600.1        6550.4     35383444
800.25/394.01	c ****************................ | ..........               4816000 | 325152               800.3        6018.0     42440856
999.86/493.90	c ****************................ | *.........               5656000 | 382111              1000.0        5655.8     48468265
1200.07/594.00	c ****************................ | *.........               6457000 | 435694              1200.2        5379.8     55190019
1400.17/684.26	c ****************................ | *.........               7202000 | 485791              1400.3        5143.2     61698354
1599.86/764.69	c ****************................ | *.........               7785000 | 525525              1600.1        4865.3     67280603
1800.07/845.63	c ****************................ | *.........               8360000 | 564678              1800.2        4644.1     72429986
2000.21/926.79	c ****************................ | **........               9089000 | 613989              2000.4        4543.5     78658319

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:503) (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w /tmp/evaluation-result-1780003-1243680640/watcher-1780003-1243680640 -o /tmp/evaluation-result-1780003-1243680640/solver-1780003-1243680640 -C 10000 -W 1000 -M 2047 HOME/ttsth-5-0 -num-threads 4 HOME/instance-1780003-1243680640.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 10000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 10030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 2096128 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 2147328 KiB
Current StackSize limit: 8192 KiB


[startup+0 s]
/proc/loadavg: 6.71 6.64 6.19 4/117 19040
/proc/meminfo: memFree=3067940/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1276 CPUtime=0.01
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 219 0 0 0 1 0 0 0 20 0 1 0 1132726805 1306624 174 4294967295 134512640 135237372 3215838752 3215838432 134544641 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 319 174 75 177 0 141 0

[startup+0.066891 s]
/proc/loadavg: 6.71 6.64 6.19 4/117 19040
/proc/meminfo: memFree=3067940/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=3792 CPUtime=0.06
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 810 0 0 0 6 0 0 0 20 0 1 0 1132726805 3883008 765 4294967295 134512640 135237372 3215838752 3215837276 134639534 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 948 766 75 177 0 770 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 3792

[startup+0.106884 s]
/proc/loadavg: 6.71 6.64 6.19 4/117 19040
/proc/meminfo: memFree=3067940/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=5364 CPUtime=0.1
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 1230 0 0 0 9 1 0 0 20 0 1 0 1132726805 5492736 1185 4294967295 134512640 135237372 3215838752 3215837276 134639534 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 1341 1186 75 177 0 1163 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 5364

[startup+0.306874 s]
/proc/loadavg: 6.71 6.64 6.19 4/117 19040
/proc/meminfo: memFree=3067940/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=12848 CPUtime=0.3
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 2991 0 0 0 29 1 0 0 21 0 1 0 1132726805 13156352 2946 4294967295 134512640 135237372 3215838752 3215838444 134534215 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 3212 2946 75 177 0 3034 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 12848

[startup+0.706853 s]
/proc/loadavg: 6.71 6.64 6.19 4/117 19040
/proc/meminfo: memFree=3067940/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=24672 CPUtime=0.7
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 5907 0 0 0 69 1 0 0 22 0 1 0 1132726805 25264128 5832 4294967295 134512640 135237372 3215838752 3215837464 134519981 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 6168 5832 76 177 0 5990 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 24672

[startup+1.50681 s]
/proc/loadavg: 6.41 6.58 6.17 4/118 19041
/proc/meminfo: memFree=3044704/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=29212 CPUtime=1.5
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 7662 0 0 0 149 1 0 0 25 0 1 0 1132726805 29913088 7089 4294967295 134512640 135237372 3215838752 3215838080 134555272 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 7303 7089 77 177 0 7125 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 29212

[startup+3.10673 s]
/proc/loadavg: 6.41 6.58 6.17 4/120 19045
/proc/meminfo: memFree=3038980/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=29456 CPUtime=3.1
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 7674 0 0 0 309 1 0 0 25 0 1 0 1132726805 30162944 7101 4294967295 134512640 135237372 3215838752 3215838048 134520489 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 7364 7101 77 177 0 7186 0
Current children cumulated CPU time (s) 3.1
Current children cumulated vsize (KiB) 29456

[startup+6.30656 s]
/proc/loadavg: 6.41 6.58 6.17 4/118 19049
/proc/meminfo: memFree=3039100/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=29912 CPUtime=6.3
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 7784 0 0 0 629 1 0 0 25 0 1 0 1132726805 30629888 7211 4294967295 134512640 135237372 3215838752 3215838112 134600707 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 7478 7211 77 177 0 7300 0
Current children cumulated CPU time (s) 6.3
Current children cumulated vsize (KiB) 29912

[startup+12.7062 s]
/proc/loadavg: 5.88 6.46 6.14 4/118 19058
/proc/meminfo: memFree=2972340/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=98424 CPUtime=12.7
/proc/19040/stat : 19040 (ttsth-5-0) R 19038 19040 10612 0 -1 4202496 25152 0 0 0 1264 6 0 0 25 0 1 0 1132726805 100786176 24428 4294967295 134512640 135237372 3215838752 3215800288 134551480 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19040/statm: 24606 24428 91 177 0 24428 0
Current children cumulated CPU time (s) 12.7
Current children cumulated vsize (KiB) 98424

[startup+25.5156 s]
/proc/loadavg: 5.98 6.46 6.14 7/122 19082
/proc/meminfo: memFree=2885528/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=217432 CPUtime=36.72
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 47002 0 0 0 3659 13 0 0 25 0 5 0 1132726805 222650368 45998 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 54358 45998 95 177 0 54180 0
[pid=19040/tid=19063] ppid=19038 vsize=217432 CPUtime=6.01
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 3672 0 0 0 600 1 0 0 25 0 5 0 1132728226 222650368 45998 4294967295 134512640 135237372 3215838752 3076614032 134749345 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=217432 CPUtime=5.24
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 2968 0 0 0 523 1 0 0 25 0 5 0 1132728226 222650368 45998 4294967295 134512640 135237372 3215838752 3068179968 134546034 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=217432 CPUtime=5.73
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 2610 0 0 0 573 0 0 0 25 0 5 0 1132728226 222650368 45998 4294967295 134512640 135237372 3215838752 3059860896 134549905 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=217432 CPUtime=5.54
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 3386 0 0 0 554 0 0 0 25 0 5 0 1132728226 222650368 45998 4294967295 134512640 135237372 3215838752 3051375952 134549888 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 36.72
Current children cumulated vsize (KiB) 217432

[startup+51.1142 s]
/proc/loadavg: 6.19 6.47 6.15 7/122 19304
/proc/meminfo: memFree=2811804/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=294184 CPUtime=87.79
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 65509 0 0 0 8756 23 0 0 25 0 5 0 1132726805 301244416 64505 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 73546 64505 95 177 0 73368 0
[pid=19040/tid=19063] ppid=19038 vsize=294184 CPUtime=18.82
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 7617 0 0 0 1878 4 0 0 25 0 5 0 1132728226 301244416 64505 4294967295 134512640 135237372 3215838752 3076501232 134546034 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=294184 CPUtime=17.95
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 8820 0 0 0 1790 5 0 0 25 0 5 0 1132728226 301244416 64505 4294967295 134512640 135237372 3215838752 3068287420 134680502 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=294184 CPUtime=18.33
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 6166 0 0 0 1831 2 0 0 25 0 5 0 1132728226 301244416 64505 4294967295 134512640 135237372 3215838752 3059753952 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=294184 CPUtime=18.49
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 8540 0 0 0 1848 1 0 0 25 0 5 0 1132728226 301244416 64505 4294967295 134512640 135237372 3215838752 3051386080 134562459 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 87.79
Current children cumulated vsize (KiB) 294184

[startup+102.312 s]
/proc/loadavg: 6.48 6.49 6.17 7/122 19566
/proc/meminfo: memFree=2656932/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=455808 CPUtime=194.35
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 103989 0 0 0 19396 39 0 0 25 0 5 0 1132726805 466747392 102985 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 113952 102985 95 177 0 113774 0
[pid=19040/tid=19063] ppid=19038 vsize=455808 CPUtime=45.69
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 12558 0 0 0 4560 9 0 0 25 0 5 0 1132728226 466747392 102985 4294967295 134512640 135237372 3215838752 3076571120 134680326 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=455808 CPUtime=43.25
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 19268 0 0 0 4316 9 0 0 25 0 5 0 1132728226 466747392 102985 4294967295 134512640 135237372 3215838752 3068177008 134680346 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=455808 CPUtime=48.13
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 18698 0 0 0 4806 7 0 0 25 0 5 0 1132728226 466747392 102985 4294967295 134512640 135237372 3215838752 3059958256 134587177 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=455808 CPUtime=43.08
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 19099 0 0 0 4305 3 0 0 25 0 5 0 1132728226 466747392 102985 4294967295 134512640 135237372 3215838752 3051473132 134636382 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 194.35
Current children cumulated vsize (KiB) 455808

[startup+162.308 s]
/proc/loadavg: 6.49 6.49 6.19 8/122 19847
/proc/meminfo: memFree=2564668/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=553220 CPUtime=316.49
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 127180 0 0 0 31602 47 0 0 25 0 5 0 1132726805 566497280 126176 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 138305 126176 95 177 0 138127 0
[pid=19040/tid=19063] ppid=19038 vsize=553220 CPUtime=75.91
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 16362 0 0 0 7581 10 0 0 25 0 5 0 1132728226 566497280 126176 4294967295 134512640 135237372 3215838752 3076543744 134562401 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=553220 CPUtime=73.05
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 24236 0 0 0 7293 12 0 0 25 0 5 0 1132728226 566497280 126176 4294967295 134512640 135237372 3215838752 3068213952 134590160 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=553220 CPUtime=78.23
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 25182 0 0 0 7816 7 0 0 25 0 5 0 1132728226 566497280 126176 4294967295 134512640 135237372 3215838752 3059750720 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=553220 CPUtime=75.1
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 27034 0 0 0 7503 7 0 0 25 0 5 0 1132728226 566497280 126176 4294967295 134512640 135237372 3215838752 3051339788 134636641 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 316.49
Current children cumulated vsize (KiB) 553220

[startup+222.315 s]
/proc/loadavg: 6.63 6.51 6.21 8/123 20070
/proc/meminfo: memFree=2463376/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=659168 CPUtime=451.73
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 152249 0 0 0 45111 62 0 0 25 0 5 0 1132726805 674988032 151245 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 164792 151245 95 177 0 164614 0
[pid=19040/tid=19063] ppid=19038 vsize=659168 CPUtime=108.51
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 19443 0 0 0 10840 11 0 0 25 0 5 0 1132728226 674988032 151245 4294967295 134512640 135237372 3215838752 3076546672 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=659168 CPUtime=103.03
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 31244 0 0 0 10288 15 0 0 25 0 5 0 1132728226 674988032 151245 4294967295 134512640 135237372 3215838752 3068186128 134680720 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=659168 CPUtime=108.38
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 30828 0 0 0 10829 9 0 0 25 0 5 0 1132728226 674988032 151245 4294967295 134512640 135237372 3215838752 3059955760 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=659168 CPUtime=117.61
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 36368 0 0 0 11745 16 0 0 25 0 5 0 1132728226 674988032 151245 4294967295 134512640 135237372 3215838752 3051381856 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 451.73
Current children cumulated vsize (KiB) 659168

[startup+282.312 s]
/proc/loadavg: 6.49 6.49 6.22 9/123 20304
/proc/meminfo: memFree=2396608/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=730468 CPUtime=571.78
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 169074 0 0 0 57108 70 0 0 25 0 5 0 1132726805 747999232 168070 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 182617 168070 95 177 0 182439 0
[pid=19040/tid=19063] ppid=19038 vsize=730468 CPUtime=138.96
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 21908 0 0 0 13882 14 0 0 25 0 5 0 1132728226 747999232 168070 4294967295 134512640 135237372 3215838752 3076526720 134562344 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=730468 CPUtime=132.75
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 36519 0 0 0 13258 17 0 0 25 0 5 0 1132728226 747999232 168070 4294967295 134512640 135237372 3215838752 3068170496 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=730468 CPUtime=138.76
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 34707 0 0 0 13867 9 0 0 25 0 5 0 1132728226 747999232 168070 4294967295 134512640 135237372 3215838752 3059854192 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=730468 CPUtime=147.11
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 41574 0 0 0 14692 19 0 0 25 0 5 0 1132728226 747999232 168070 4294967295 134512640 135237372 3215838752 3051431392 134749335 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 571.78
Current children cumulated vsize (KiB) 730468

[startup+342.309 s]
/proc/loadavg: 6.40 6.47 6.23 8/123 20532
/proc/meminfo: memFree=2311776/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=818120 CPUtime=693.73
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 190064 0 0 0 69295 78 0 0 25 0 5 0 1132726805 837754880 189060 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 204530 189060 95 177 0 204352 0
[pid=19040/tid=19063] ppid=19038 vsize=818120 CPUtime=169.4

################
# More data... #
################

[pid=19040/tid=19064] ppid=19038 vsize=949404 CPUtime=222.75
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 54372 0 0 0 22249 26 0 0 25 0 5 0 1132728226 972189696 219850 4294967295 134512640 135237372 3215838752 3068169072 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=949404 CPUtime=228.73
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 46956 0 0 0 22860 13 0 0 25 0 5 0 1132728226 972189696 219850 4294967295 134512640 135237372 3215838752 3059849344 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=949404 CPUtime=240.87
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 55879 0 0 0 24061 26 0 0 25 0 5 0 1132728226 972189696 219850 4294967295 134512640 135237372 3215838752 3051442376 134519949 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 936.86
Current children cumulated vsize (KiB) 949404

[startup+522.31 s]
/proc/loadavg: 6.55 6.50 6.28 8/122 21325
/proc/meminfo: memFree=2136320/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1005124 CPUtime=1056.85
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 234154 0 0 0 105582 103 0 0 25 0 5 0 1132726805 1029246976 233150 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 251281 233150 95 177 0 251103 0
[pid=19040/tid=19063] ppid=19038 vsize=1005124 CPUtime=260.31
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 31585 0 0 0 26007 24 0 0 25 0 5 0 1132728226 1029246976 233150 4294967295 134512640 135237372 3215838752 3076545408 134562413 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1005124 CPUtime=252.75
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 58713 0 0 0 25248 27 0 0 25 0 5 0 1132728226 1029246976 233150 4294967295 134512640 135237372 3215838752 3068244192 134591082 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1005124 CPUtime=258.88
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 50112 0 0 0 25874 14 0 0 25 0 5 0 1132728226 1029246976 233150 4294967295 134512640 135237372 3215838752 3059777536 134591446 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1005124 CPUtime=270.71
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 59378 0 0 0 27044 27 0 0 25 0 5 0 1132728226 1029246976 233150 4294967295 134512640 135237372 3215838752 3051427088 134593171 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1056.85
Current children cumulated vsize (KiB) 1005124

[startup+582.317 s]
/proc/loadavg: 6.43 6.48 6.28 7/122 21601
/proc/meminfo: memFree=2078260/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1065592 CPUtime=1176.86
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 248507 0 0 0 117576 110 0 0 25 0 5 0 1132726805 1091166208 247503 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 266398 247503 95 177 0 266220 0
[pid=19040/tid=19063] ppid=19038 vsize=1065592 CPUtime=290.51
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 33891 0 0 0 29025 26 0 0 25 0 5 0 1132728226 1091166208 247503 4294967295 134512640 135237372 3215838752 3076548704 134587166 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1065592 CPUtime=282.56
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 62551 0 0 0 28228 28 0 0 25 0 5 0 1132728226 1091166208 247503 4294967295 134512640 135237372 3215838752 3068166768 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1065592 CPUtime=288.93
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 53201 0 0 0 28877 16 0 0 25 0 5 0 1132728226 1091166208 247503 4294967295 134512640 135237372 3215838752 3059785376 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1065592 CPUtime=300.66
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 64498 0 0 0 30037 29 0 0 25 0 5 0 1132728226 1091166208 247503 4294967295 134512640 135237372 3215838752 3051403292 134562272 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1176.86
Current children cumulated vsize (KiB) 1065592

[startup+642.313 s]
/proc/loadavg: 6.75 6.56 6.32 7/122 21876
/proc/meminfo: memFree=2003636/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1144004 CPUtime=1300.13
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 266972 0 0 0 129894 119 0 0 25 0 5 0 1132726805 1171460096 265968 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 286001 265968 95 177 0 285823 0
[pid=19040/tid=19063] ppid=19038 vsize=1144004 CPUtime=320.56
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 36033 0 0 0 32030 26 0 0 25 0 5 0 1132728226 1171460096 265968 4294967295 134512640 135237372 3215838752 3076540800 134749353 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1144004 CPUtime=312.45
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 67672 0 0 0 31214 31 0 0 25 0 5 0 1132728226 1171460096 265968 4294967295 134512640 135237372 3215838752 3068174032 134681633 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1144004 CPUtime=318.83
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 56877 0 0 0 31864 19 0 0 25 0 5 0 1132728226 1171460096 265968 4294967295 134512640 135237372 3215838752 3059809296 134749339 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1144004 CPUtime=334.09
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 72024 0 0 0 33377 32 0 0 25 0 5 0 1132728226 1171460096 265968 4294967295 134512640 135237372 3215838752 3051429328 134680265 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1300.13
Current children cumulated vsize (KiB) 1144004

[startup+702.31 s]
/proc/loadavg: 6.56 6.54 6.32 7/122 22151
/proc/meminfo: memFree=1937052/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1214984 CPUtime=1445.25
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 283737 0 0 0 144394 131 0 0 25 0 5 0 1132726805 1244143616 282733 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 303746 282733 95 177 0 303568 0
[pid=19040/tid=19063] ppid=19038 vsize=1214984 CPUtime=354.51
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 38692 0 0 0 35423 28 0 0 25 0 5 0 1132728226 1244143616 282733 4294967295 134512640 135237372 3215838752 3076524592 134562344 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1214984 CPUtime=341.85
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 71555 0 0 0 34153 32 0 0 25 0 5 0 1132728226 1244143616 282733 4294967295 134512640 135237372 3215838752 3068170400 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1214984 CPUtime=351.52
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 61601 0 0 0 35132 20 0 0 25 0 5 0 1132728226 1244143616 282733 4294967295 134512640 135237372 3215838752 3059841456 134749353 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1214984 CPUtime=383.17
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 77523 0 0 0 38277 40 0 0 25 0 5 0 1132728226 1244143616 282733 4294967295 134512640 135237372 3215838752 3051414896 134546034 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1445.25
Current children cumulated vsize (KiB) 1214984

[startup+762.317 s]
/proc/loadavg: 6.68 6.59 6.35 7/122 22428
/proc/meminfo: memFree=1870820/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1284348 CPUtime=1594.21
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 300296 0 0 0 159279 142 0 0 25 0 5 0 1132726805 1315172352 299292 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 321087 299292 95 177 0 320909 0
[pid=19040/tid=19063] ppid=19038 vsize=1284348 CPUtime=384.81
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 40980 0 0 0 38451 30 0 0 25 0 5 0 1132728226 1315172352 299292 4294967295 134512640 135237372 3215838752 3076498736 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1284348 CPUtime=371.29
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 75343 0 0 0 37097 32 0 0 25 0 5 0 1132728226 1315172352 299292 4294967295 134512640 135237372 3215838752 3068243456 134562344 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1284348 CPUtime=381.23
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 65332 0 0 0 38103 20 0 0 25 0 5 0 1132728226 1315172352 299292 4294967295 134512640 135237372 3215838752 3059807920 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1284348 CPUtime=442.68
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 84275 0 0 0 44219 49 0 0 25 0 5 0 1132728226 1315172352 299292 4294967295 134512640 135237372 3215838752 3051434064 134680467 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1594.21
Current children cumulated vsize (KiB) 1284348

[startup+822.304 s]
/proc/loadavg: 6.53 6.56 6.36 8/122 22704
/proc/meminfo: memFree=1816960/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1341300 CPUtime=1742.35
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 313754 0 0 0 174078 157 0 0 25 0 5 0 1132726805 1373491200 312750 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 335325 312750 95 177 0 335147 0
[pid=19040/tid=19063] ppid=19038 vsize=1341300 CPUtime=415.14
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 43155 0 0 0 41481 33 0 0 25 0 5 0 1132728226 1373491200 312750 4294967295 134512640 135237372 3215838752 3076504976 134562341 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1341300 CPUtime=400.58
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 78270 0 0 0 40023 35 0 0 25 0 5 0 1132728226 1373491200 312750 4294967295 134512640 135237372 3215838752 3068157584 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1341300 CPUtime=424.42
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 71098 0 0 0 42421 21 0 0 25 0 5 0 1132728226 1373491200 312750 4294967295 134512640 135237372 3215838752 3059901952 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1341300 CPUtime=488.01
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 86865 0 0 0 48744 57 0 0 25 0 5 0 1132728226 1373491200 312750 4294967295 134512640 135237372 3215838752 3051432768 134749337 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1742.35
Current children cumulated vsize (KiB) 1341300

[startup+882.301 s]
/proc/loadavg: 6.53 6.56 6.37 7/122 22984
/proc/meminfo: memFree=1754196/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1407416 CPUtime=1890.75
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 329207 0 0 0 188905 170 0 0 25 0 5 0 1132726805 1441193984 328203 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 351854 328203 95 177 0 351676 0
[pid=19040/tid=19063] ppid=19038 vsize=1407416 CPUtime=445.61
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 45396 0 0 0 44528 33 0 0 25 0 5 0 1132728226 1441193984 328203 4294967295 134512640 135237372 3215838752 3076498224 134562278 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1407416 CPUtime=429.65
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 81532 0 0 0 42926 39 0 0 25 0 5 0 1132728226 1441193984 328203 4294967295 134512640 135237372 3215838752 3068361264 134591471 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1407416 CPUtime=461.14
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 75951 0 0 0 46090 24 0 0 25 0 5 0 1132728226 1441193984 328203 4294967295 134512640 135237372 3215838752 3059787712 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1407416 CPUtime=540.15
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 91962 0 0 0 53952 63 0 0 25 0 5 0 1132728226 1441193984 328203 4294967295 134512640 135237372 3215838752 3051410908 134749296 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1890.75
Current children cumulated vsize (KiB) 1407416

[startup+942.318 s]
/proc/loadavg: 6.40 6.51 6.36 7/122 23259
/proc/meminfo: memFree=1675132/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1490596 CPUtime=2038.66
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 349248 0 0 0 203680 186 0 0 25 0 5 0 1132726805 1526370304 348244 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 372649 348244 95 177 0 372471 0
[pid=19040/tid=19063] ppid=19038 vsize=1490596 CPUtime=488.71
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 48310 0 0 0 48837 34 0 0 25 0 5 0 1132728226 1526370304 348244 4294967295 134512640 135237372 3215838752 3076522272 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1490596 CPUtime=461.14
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 84681 0 0 0 46073 41 0 0 25 0 5 0 1132728226 1526370304 348244 4294967295 134512640 135237372 3215838752 3068112048 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1490596 CPUtime=493.48
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 79781 0 0 0 49321 27 0 0 25 0 5 0 1132728226 1526370304 348244 4294967295 134512640 135237372 3215838752 3059783280 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1490596 CPUtime=581.13
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 102110 0 0 0 58040 73 0 0 25 0 5 0 1132728226 1526370304 348244 4294967295 134512640 135237372 3215838752 3051570976 134680367 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2038.66
Current children cumulated vsize (KiB) 1490596



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1000.01 s]
/proc/loadavg: 6.60 6.54 6.38 7/122 23534
/proc/meminfo: memFree=1616772/3368532 swapFree=1959840/1959888
[pid=19040] ppid=19038 vsize=1550992 CPUtime=2181.01
/proc/19040/stat : 19040 (ttsth-5-0) S 19038 19040 10612 0 -1 4202496 363547 0 0 0 217901 200 0 0 25 0 5 0 1132726805 1588215808 362543 4294967295 134512640 135237372 3215838752 3215838348 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/19040/statm: 387748 362543 95 177 0 387570 0
[pid=19040/tid=19063] ppid=19038 vsize=1550992 CPUtime=517.9
/proc/19040/task/19063/stat : 19063 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 50309 0 0 0 51756 34 0 0 25 0 5 0 1132728226 1588215808 362543 4294967295 134512640 135237372 3215838752 3076550176 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19040/tid=19064] ppid=19038 vsize=1550992 CPUtime=491.07
/proc/19040/task/19064/stat : 19064 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 87888 0 0 0 49063 44 0 0 25 0 5 0 1132728226 1588215808 362543 4294967295 134512640 135237372 3215838752 3068231116 134680506 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19040/tid=19065] ppid=19038 vsize=1550992 CPUtime=522.53
/proc/19040/task/19065/stat : 19065 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 82790 0 0 0 52225 28 0 0 25 0 5 0 1132728226 1588215808 362543 4294967295 134512640 135237372 3215838752 3059775376 134590140 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19040/tid=19066] ppid=19038 vsize=1550992 CPUtime=635.31
/proc/19040/task/19066/stat : 19066 (ttsth-5-0) R 19038 19040 10612 0 -1 4202560 108194 0 0 0 63448 83 0 0 25 0 5 0 1132728226 1588215808 362543 4294967295 134512640 135237372 3215838752 3051423456 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2181.01
Current children cumulated vsize (KiB) 1550992

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1000.13
CPU time (s): 2181.11
CPU user time (s): 2179.03
CPU system time (s): 2.08
CPU usage (%): 218.082
Max. virtual memory (cumulated for all children) (KiB): 1550992

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2179.03
system time used= 2.08
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 363552
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 759
involuntary context switches= 61934

runsolver used 0 second user time and 0 second system time

The end

Launcher Data 

Begin job on  at 2009-05-30 12:50:40
IDJOB=1780003
IDBENCH=70946
IDSOLVER=632
FILE ID=/1780003-1243680640
PBS_JOBID=
Free space on /tmp= 7979 MiB

SOLVER NAME= ttsth-5-0 2009-03-19 (4 threads)
BENCH NAME= SAT09/APPLICATIONS/bioinfo/q_query_3_l47_lambda.cnf
COMMAND LINE= HOME/ttsth-5-0 -num-threads MAXNBTHREAD BENCHNAME
RUNSOLVER COMMAND LINE= runsolver --timestamp  -w /tmp/evaluation-result-1780003-1243680640/watcher-1780003-1243680640 -o /tmp/evaluation-result-1780003-1243680640/solver-1780003-1243680640 -C 10000 -W 1000 -M 2047  HOME/ttsth-5-0 -num-threads 4 HOME/instance-1780003-1243680640.cnf

TIME LIMIT= 10000 seconds
MEMORY LIMIT= 2047 MiB
MAX NB THREAD= 4

MD5SUM BENCH= ed7689c65135a913d97feca3ab3874eb
RANDOM SEED= 1301377418

Linux (none) 2.6.21.5 #6 SMP Wed Jul 4 17:16:38 CEST 2007 i686 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.073
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3992.42
clflush size	: 64

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.073
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.12
clflush size	: 64

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.073
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.14
clflush size	: 64

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.073
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.14
clflush size	: 64


/proc/meminfo:
MemTotal:      3368532 kB
MemFree:       3067892 kB
Buffers:         93016 kB
Cached:          90676 kB
SwapCached:          0 kB
Active:         130808 kB
Inactive:        75240 kB
HighTotal:     2489216 kB
HighFree:      2376524 kB
LowTotal:       879316 kB
LowFree:        691368 kB
SwapTotal:     1959888 kB
SwapFree:      1959840 kB
Dirty:            4092 kB
Writeback:           0 kB
AnonPages:       22180 kB
Mapped:           8208 kB
Slab:            84308 kB
SReclaimable:    72072 kB
SUnreclaim:      12236 kB
PageTables:        844 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3644152 kB
Committed_AS:   123548 kB
VmallocTotal:   114680 kB
VmallocUsed:      1444 kB
VmallocChunk:   113072 kB

Free space on /tmp at the end= 7979 MiB
End job on  at 2009-05-30 13:07:22