Trace number 1780813

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth-5-0 2009-03-19 (4 threads)? (MO) 1627.92 838.259

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
ndhf_xits_12_UNSAT.cnf
MD5SUMecc4810d82e30597ea8451526243ec09
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2522
Number of clauses243459
Sum of the clauses size726523
Maximum clause length13
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 24164
Number of clauses of size 3239264
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 531

Solver Data 

0.35/0.44	c Command: HOME/ttsth-5-0 -num-threads 4 HOME/instance-1780813-1243683817.cnf 
0.35/0.44	c Using SATTIMEOUT of 10000
0.35/0.44	c Using SATRAM of 2047
0.35/0.44	c Size of subset node = 16
0.35/0.44	c Variables = 2522
0.35/0.44	c Clauses = 243459
0.35/0.44	c Literals = 726523
0.35/0.44	c Reduced size:
0.35/0.44	c Variables = 2522
0.35/0.44	c Clauses = 243459
0.35/0.44	c Literals = 726523
0.35/0.44	c a2
1.01/1.34	c a1
2.41/4.13	c Independent sub-problems = 1
2.41/4.13	c Sub-problem with 2522 variables and 243459 clauses
2.41/4.13	c using permutation method 4
2.41/4.13	c count = 1
2.41/4.13	c Ordering finished 2.310 seconds
2.41/4.13	c mapped 2.430 seconds
2.41/4.13	c permuted 2.430 seconds
2.41/4.13	c about to sort
2.51/4.31	c sorted
4.56/8.45	c clauses now 243459
4.56/8.45	c init finished 2.540 seconds
4.56/8.45	c tree built 4.550 seconds
4.56/8.45	c 
4.56/8.45	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
4.56/8.45	c ---------------------------------|----------------------------------|------------------------------------------------------
4.56/8.45	c ................................ | ..........                       |                        4.5           0.0             
4.56/8.45	c ................................ | ..........                     1 |                        4.5           0.2             
4.56/8.45	c ................................ | ..........                     2 |                        4.5           0.4             
4.56/8.45	c ................................ | ..........                     3 |                        4.5           0.7             
4.56/8.45	c ................................ | ..........                     4 |                        4.5           0.9             
4.56/8.45	c ................................ | ..........                     5 |                        4.5           1.1             
4.56/8.46	c ................................ | ..........                  1000 | 15                     4.5         219.8          246
200.08/112.99	c ................................ | ..........                722000 | 104832               200.3        3603.7     25614688
400.30/214.73	c ................................ | ..........               1152000 | 180967               400.3        2877.8     41604268
599.87/316.18	c ................................ | ..........               1627000 | 253952               600.0        2711.5     57071618
800.00/418.06	c ................................ | ..........               1798000 | 276097               800.1        2247.3     71001339
1000.58/520.94	c ................................ | *.........               1958000 | 300529              1000.7        1956.6     86388074
1201.37/624.44	c ................................ | *.........               2111000 | 320599              1201.4        1757.1     99914492
1400.09/723.52	c ................................ | *.........               2272000 | 343381              1400.1        1622.7    111017494
1600.15/824.34	c ................................ | *.........               2431000 | 365457              1600.2        1519.2    120065853

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:503) (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w /tmp/evaluation-result-1780813-1243683817/watcher-1780813-1243683817 -o /tmp/evaluation-result-1780813-1243683817/solver-1780813-1243683817 -C 10000 -W 1000 -M 2047 HOME/ttsth-5-0 -num-threads 4 HOME/instance-1780813-1243683817.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 10000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 10030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 2096128 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 2147328 KiB
Current StackSize limit: 8192 KiB


[startup+0 s]
/proc/loadavg: 5.59 5.49 5.62 6/113 19950
/proc/meminfo: memFree=2620880/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1144 CPUtime=0.01
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 189 0 0 0 1 0 0 0 18 0 1 0 1122519021 1171456 145 4294967295 134512640 135237372 3218405536 3218405008 134749343 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 286 145 75 177 0 108 0

[startup+0.105289 s]
/proc/loadavg: 5.59 5.49 5.62 6/113 19950
/proc/meminfo: memFree=2620880/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=2760 CPUtime=0.04
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 566 0 0 0 4 0 0 0 18 0 1 0 1122519021 2826240 522 4294967295 134512640 135237372 3218405536 3218404984 134732614 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 690 523 75 177 0 512 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 2760

[startup+0.215264 s]
/proc/loadavg: 5.59 5.49 5.62 6/113 19950
/proc/meminfo: memFree=2620880/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=7496 CPUtime=0.15
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 1756 0 0 0 15 0 0 0 18 0 1 0 1122519021 7675904 1712 4294967295 134512640 135237372 3218405536 3218404332 134639534 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 1874 1713 75 177 0 1696 0
Current children cumulated CPU time (s) 0.15
Current children cumulated vsize (KiB) 7496

[startup+0.315255 s]
/proc/loadavg: 5.59 5.49 5.62 6/113 19950
/proc/meminfo: memFree=2620880/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=11856 CPUtime=0.25
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 2851 0 0 0 23 2 0 0 18 0 1 0 1122519021 12140544 2807 4294967295 134512640 135237372 3218405536 3218404332 134639534 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 2964 2808 75 177 0 2786 0
Current children cumulated CPU time (s) 0.25
Current children cumulated vsize (KiB) 11856

[startup+0.715226 s]
/proc/loadavg: 5.59 5.49 5.62 6/113 19950
/proc/meminfo: memFree=2620880/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=32724 CPUtime=0.65
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 7652 0 0 0 57 8 0 0 18 0 1 0 1122519021 33509376 7607 4294967295 134512640 135237372 3218405536 3218403768 134519996 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 8181 7607 76 177 0 8003 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 32724

[startup+1.51516 s]
/proc/loadavg: 5.54 5.48 5.62 6/114 19951
/proc/meminfo: memFree=2590960/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=33676 CPUtime=1.11
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 7811 0 0 0 103 8 0 0 19 0 1 0 1122519021 34484224 7766 4294967295 134512640 135237372 3218405536 3218405040 134540092 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 8419 7766 76 177 0 8241 0
Current children cumulated CPU time (s) 1.11
Current children cumulated vsize (KiB) 33676

[startup+3.11504 s]
/proc/loadavg: 5.54 5.48 5.62 6/114 19951
/proc/meminfo: memFree=2593476/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=34932 CPUtime=1.91
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 11079 0 0 0 180 11 0 0 21 0 1 0 1122519021 35770368 8577 4294967295 134512640 135237372 3218405536 3218404896 134600884 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 8733 8577 77 177 0 8555 0
Current children cumulated CPU time (s) 1.91
Current children cumulated vsize (KiB) 34932
heavy processes:
  %CPU=84 pid=12341 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=85 pid=12739 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=85 pid=13124 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R
  %CPU=86 pid=13512 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R

[startup+6.31479 s]
/proc/loadavg: 5.54 5.48 5.62 6/114 19953
/proc/meminfo: memFree=2582008/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=43368 CPUtime=3.51
/proc/19950/stat : 19950 (ttsth-5-0) R 19948 19950 14986 0 -1 4202496 13357 0 0 0 340 11 0 0 25 0 1 0 1122519021 44408832 10617 4294967295 134512640 135237372 3218405536 3218352788 134732614 0 0 0 0 0 0 0 17 2 0 0 0
/proc/19950/statm: 10842 10617 88 177 0 10664 0
Current children cumulated CPU time (s) 3.51
Current children cumulated vsize (KiB) 43368
heavy processes:
  %CPU=84 pid=12341 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=85 pid=12739 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=85 pid=13124 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R
  %CPU=86 pid=13512 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R

[startup+12.7143 s]
/proc/loadavg: 5.77 5.53 5.64 9/118 19957
/proc/meminfo: memFree=2548756/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=109904 CPUtime=11.15
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 22053 0 0 0 1102 13 0 0 25 0 5 0 1122519021 112541696 19010 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 27476 19010 95 177 0 27298 0
[pid=19950/tid=19954] ppid=19948 vsize=109904 CPUtime=1.93
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 1921 0 0 0 193 0 0 0 25 0 5 0 1122519856 112541696 19010 4294967295 134512640 135237372 3218405536 3081676996 134550980 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=109904 CPUtime=1.71
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 1059 0 0 0 171 0 0 0 25 0 5 0 1122519856 112541696 19010 4294967295 134512640 135237372 3218405536 3073245568 134680257 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=109904 CPUtime=1.51
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 1067 0 0 0 150 1 0 0 25 0 5 0 1122519856 112541696 19010 4294967295 134512640 135237372 3218405536 3064836736 134562485 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=109904 CPUtime=1.44
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 851 0 0 0 144 0 0 0 25 0 5 0 1122519861 112541696 19010 4294967295 134512640 135237372 3218405536 3056413600 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.15
Current children cumulated vsize (KiB) 109904

[startup+25.5133 s]
/proc/loadavg: 6.35 5.66 5.68 9/118 19959
/proc/meminfo: memFree=2489588/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=168788 CPUtime=34.95
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 36335 0 0 0 3473 22 0 0 25 0 5 0 1122519021 172838912 33292 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 42197 33292 95 177 0 42019 0
[pid=19950/tid=19954] ppid=19948 vsize=168788 CPUtime=8.03
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 6480 0 0 0 801 2 0 0 25 0 5 0 1122519856 172838912 33292 4294967295 134512640 135237372 3218405536 3081673260 134549925 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=168788 CPUtime=7.61
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 3383 0 0 0 759 2 0 0 25 0 5 0 1122519856 172838912 33292 4294967295 134512640 135237372 3218405536 3073226400 134562278 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=168788 CPUtime=7.31
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 4509 0 0 0 728 3 0 0 25 0 5 0 1122519856 172838912 33292 4294967295 134512640 135237372 3218405536 3064826064 134562341 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=168788 CPUtime=7.44
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 4809 0 0 0 741 3 0 0 25 0 5 0 1122519861 172838912 33293 4294967295 134512640 135237372 3218405536 3056499520 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 34.95
Current children cumulated vsize (KiB) 168788

[startup+51.1113 s]
/proc/loadavg: 7.32 5.95 5.77 10/118 20142
/proc/meminfo: memFree=2379936/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=283648 CPUtime=79.52
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 63554 0 0 0 7921 31 0 0 25 0 5 0 1122519021 290455552 60511 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 70912 60511 95 177 0 70734 0
[pid=19950/tid=19954] ppid=19948 vsize=283648 CPUtime=19.63
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 16768 0 0 0 1958 5 0 0 25 0 5 0 1122519856 290455552 60511 4294967295 134512640 135237372 3218405536 3081680256 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=283648 CPUtime=17.21
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 8002 0 0 0 1717 4 0 0 25 0 5 0 1122519856 290455552 60511 4294967295 134512640 135237372 3218405536 3073250720 134562416 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=283648 CPUtime=17.67
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 9402 0 0 0 1761 6 0 0 25 0 5 0 1122519856 290455552 60511 4294967295 134512640 135237372 3218405536 3064863296 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=283648 CPUtime=20.45
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 12227 0 0 0 2041 4 0 0 25 0 5 0 1122519861 290455552 60511 4294967295 134512640 135237372 3218405536 3056462508 134550976 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 79.52
Current children cumulated vsize (KiB) 283648

[startup+102.307 s]
/proc/loadavg: 7.85 6.32 5.91 10/118 20331
/proc/meminfo: memFree=2191396/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=480140 CPUtime=180.11
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 110171 0 0 0 17954 57 0 0 25 0 5 0 1122519021 491663360 107128 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 120035 107128 95 177 0 119857 0
[pid=19950/tid=19954] ppid=19948 vsize=480140 CPUtime=44.33
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 31797 0 0 0 4423 10 0 0 25 0 5 0 1122519856 491663360 107128 4294967295 134512640 135237372 3218405536 3081681408 134562444 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=480140 CPUtime=41.62
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 18508 0 0 0 4147 15 0 0 25 0 5 0 1122519856 491663360 107128 4294967295 134512640 135237372 3218405536 3073253216 134562334 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=480140 CPUtime=44.25
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 18411 0 0 0 4413 12 0 0 25 0 5 0 1122519856 491663360 107128 4294967295 134512640 135237372 3218405536 3064838416 134561437 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=480140 CPUtime=45.35
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 24300 0 0 0 4527 8 0 0 25 0 5 0 1122519861 491663360 107128 4294967295 134512640 135237372 3218405536 3056441072 134680503 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 180.11
Current children cumulated vsize (KiB) 480140

[startup+162.303 s]
/proc/loadavg: 8.10 6.66 6.04 9/118 20518
/proc/meminfo: memFree=2026800/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=653748 CPUtime=297.31
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 151272 0 0 0 29657 74 0 0 25 0 5 0 1122519021 669437952 148229 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 163437 148229 95 177 0 163259 0
[pid=19950/tid=19954] ppid=19948 vsize=653748 CPUtime=73.69
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 44471 0 0 0 7354 15 0 0 25 0 5 0 1122519856 669437952 148229 4294967295 134512640 135237372 3218405536 3081683776 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=653748 CPUtime=70.44
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 29032 0 0 0 7027 17 0 0 25 0 5 0 1122519856 669437952 148229 4294967295 134512640 135237372 3218405536 3073248508 134562272 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=653748 CPUtime=74.17
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 24818 0 0 0 7401 16 0 0 25 0 5 0 1122519856 669437952 148229 4294967295 134512640 135237372 3218405536 3064885152 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=653748 CPUtime=74.45
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 35796 0 0 0 7431 14 0 0 25 0 5 0 1122519861 669437952 148229 4294967295 134512640 135237372 3218405536 3056484652 134551529 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 297.31
Current children cumulated vsize (KiB) 653748

[startup+222.308 s]
/proc/loadavg: 8.40 6.99 6.20 9/118 20709
/proc/meminfo: memFree=1892416/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=793804 CPUtime=414.7
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 184389 0 0 0 41362 108 0 0 25 0 5 0 1122519021 812855296 181346 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 198451 181346 95 177 0 198273 0
[pid=19950/tid=19954] ppid=19948 vsize=793804 CPUtime=103.43
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 55261 0 0 0 10323 20 0 0 25 0 5 0 1122519856 812855296 181346 4294967295 134512640 135237372 3218405536 3081666064 134587833 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=793804 CPUtime=99.61
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 36962 0 0 0 9937 24 0 0 25 0 5 0 1122519856 812855296 181346 4294967295 134512640 135237372 3218405536 3073252496 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=793804 CPUtime=103.05
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 30103 0 0 0 10286 19 0 0 25 0 5 0 1122519856 812855296 181346 4294967295 134512640 135237372 3218405536 3064851088 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=793804 CPUtime=104.05
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 44908 0 0 0 10372 33 0 0 25 0 5 0 1122519861 812855296 181346 4294967295 134512640 135237372 3218405536 3056452864 134593167 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 414.7
Current children cumulated vsize (KiB) 793804

[startup+282.313 s]
/proc/loadavg: 8.58 7.27 6.34 9/118 20897
/proc/meminfo: memFree=1741440/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=953520 CPUtime=533.15
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 222173 0 0 0 53189 126 0 0 25 0 5 0 1122519021 976404480 219130 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 238380 219130 95 177 0 238202 0

################
# More data... #
################

/proc/meminfo: memFree=1481160/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1225032 CPUtime=768.79
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 286645 0 0 0 76729 150 0 0 25 0 5 0 1122519021 1254432768 283602 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 306258 283602 95 177 0 306080 0
[pid=19950/tid=19954] ppid=19948 vsize=1225032 CPUtime=191.48
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 85947 0 0 0 19115 33 0 0 25 0 5 0 1122519856 1254432768 283602 4294967295 134512640 135237372 3218405536 3081681744 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1225032 CPUtime=186.61
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 56131 0 0 0 18635 26 0 0 25 0 5 0 1122519856 1254432768 283602 4294967295 134512640 135237372 3218405536 3073251296 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1225032 CPUtime=193.79
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 59082 0 0 0 19349 30 0 0 25 0 5 0 1122519856 1254432768 283602 4294967295 134512640 135237372 3218405536 3064863232 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1225032 CPUtime=192.35
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 68330 0 0 0 19186 49 0 0 25 0 5 0 1122519861 1254432768 283602 4294967295 134512640 135237372 3218405536 3056475840 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 768.79
Current children cumulated vsize (KiB) 1225032

[startup+462.309 s]
/proc/loadavg: 8.48 7.80 6.70 9/118 21463
/proc/meminfo: memFree=1341848/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1369596 CPUtime=887.96
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 320841 0 0 0 88626 170 0 0 25 0 5 0 1122519021 1402466304 317798 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 342399 317798 95 177 0 342221 0
[pid=19950/tid=19954] ppid=19948 vsize=1369596 CPUtime=221.39
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 95295 0 0 0 22101 38 0 0 25 0 5 0 1122519856 1402466304 317798 4294967295 134512640 135237372 3218405536 3081681952 134562344 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1369596 CPUtime=217.17
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 64130 0 0 0 21688 29 0 0 25 0 5 0 1122519856 1402466304 317798 4294967295 134512640 135237372 3218405536 3073252160 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1369596 CPUtime=222.89
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 66046 0 0 0 22254 35 0 0 25 0 5 0 1122519856 1402466304 317798 4294967295 134512640 135237372 3218405536 3064836928 134562320 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1369596 CPUtime=221.95
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 78215 0 0 0 22139 56 0 0 25 0 5 0 1122519861 1402466304 317798 4294967295 134512640 135237372 3218405536 3056487800 134519962 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 887.96
Current children cumulated vsize (KiB) 1369596

[startup+522.305 s]
/proc/loadavg: 8.69 7.97 6.83 9/118 21650
/proc/meminfo: memFree=1197368/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1521116 CPUtime=1003.38
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 357098 0 0 0 100148 190 0 0 25 0 5 0 1122519021 1557622784 354055 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 380279 354056 95 177 0 380101 0
[pid=19950/tid=19954] ppid=19948 vsize=1521116 CPUtime=250.47
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 101515 0 0 0 25007 40 0 0 25 0 5 0 1122519856 1557622784 354056 4294967295 134512640 135237372 3218405536 3081681760 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1521116 CPUtime=246.02
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 70271 0 0 0 24570 32 0 0 25 0 5 0 1122519856 1557622784 354056 4294967295 134512640 135237372 3218405536 3073266752 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1521116 CPUtime=251.48
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 71290 0 0 0 25108 40 0 0 25 0 5 0 1122519856 1557622784 354057 4294967295 134512640 135237372 3218405536 3064837216 134562384 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1521116 CPUtime=250.85
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 96868 0 0 0 25019 66 0 0 25 0 5 0 1122519861 1557622784 354057 4294967295 134512640 135237372 3218405536 3056461056 134562355 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1003.38
Current children cumulated vsize (KiB) 1521116

[startup+582.32 s]
/proc/loadavg: 8.54 8.05 6.92 11/119 21773
/proc/meminfo: memFree=1067932/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1655948 CPUtime=1120.65
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 388885 0 0 0 111858 207 0 0 25 0 5 0 1122519021 1695690752 385842 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 413987 385842 95 177 0 413809 0
[pid=19950/tid=19954] ppid=19948 vsize=1655948 CPUtime=280.09
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 109326 0 0 0 27966 43 0 0 25 0 5 0 1122519856 1695690752 385842 4294967295 134512640 135237372 3218405536 3081681104 134562360 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1655948 CPUtime=275.7
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 75309 0 0 0 27532 38 0 0 25 0 5 0 1122519856 1695690752 385842 4294967295 134512640 135237372 3218405536 3073252076 134636641 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1655948 CPUtime=280.56
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 75698 0 0 0 28016 40 0 0 25 0 5 0 1122519856 1695690752 385842 4294967295 134512640 135237372 3218405536 3064850448 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1655948 CPUtime=279.75
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 111397 0 0 0 27901 74 0 0 25 0 5 0 1122519861 1695690752 385842 4294967295 134512640 135237372 3218405536 3056489296 134591065 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1120.65
Current children cumulated vsize (KiB) 1655948

[startup+642.315 s]
/proc/loadavg: 8.64 8.16 7.03 12/119 22020
/proc/meminfo: memFree=952552/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1778680 CPUtime=1237.25
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 417946 0 0 0 123504 221 0 0 25 0 5 0 1122519021 1821368320 414903 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 444670 414903 95 177 0 444492 0
[pid=19950/tid=19954] ppid=19948 vsize=1778680 CPUtime=309.21
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 115757 0 0 0 30873 48 0 0 25 0 5 0 1122519856 1821368320 414903 4294967295 134512640 135237372 3218405536 3081673264 134562420 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1778680 CPUtime=305.01
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 79010 0 0 0 30462 39 0 0 25 0 5 0 1122519856 1821368320 414903 4294967295 134512640 135237372 3218405536 3073266224 134562328 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1778680 CPUtime=309.11
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 80438 0 0 0 30870 41 0 0 25 0 5 0 1122519856 1821368320 414903 4294967295 134512640 135237372 3218405536 3064878752 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1778680 CPUtime=309.36
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 125586 0 0 0 30855 81 0 0 25 0 5 0 1122519861 1821368320 414903 4294967295 134512640 135237372 3218405536 3056486332 134680502 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1237.25
Current children cumulated vsize (KiB) 1778680

[startup+702.31 s]
/proc/loadavg: 8.58 8.22 7.12 12/119 22183
/proc/meminfo: memFree=848932/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1884592 CPUtime=1358.13
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 442947 0 0 0 135578 235 0 0 25 0 5 0 1122519021 1929822208 439904 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 471148 439904 95 177 0 470970 0
[pid=19950/tid=19954] ppid=19948 vsize=1884592 CPUtime=338.7
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 122178 0 0 0 33819 51 0 0 25 0 5 0 1122519856 1929822208 439904 4294967295 134512640 135237372 3218405536 3081687200 134751534 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1884592 CPUtime=334.77
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 82735 0 0 0 33436 41 0 0 25 0 5 0 1122519856 1929822208 439904 4294967295 134512640 135237372 3218405536 3073250480 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1884592 CPUtime=341.86
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 84759 0 0 0 34140 46 0 0 25 0 5 0 1122519856 1929822208 439904 4294967295 134512640 135237372 3218405536 3064863392 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1884592 CPUtime=338.25
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 136120 0 0 0 33740 85 0 0 25 0 5 0 1122519861 1929822208 439904 4294967295 134512640 135237372 3218405536 3056516928 134680499 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1358.13
Current children cumulated vsize (KiB) 1884592

[startup+762.306 s]
/proc/loadavg: 8.49 8.25 7.20 10/118 22406
/proc/meminfo: memFree=750412/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=1987128 CPUtime=1477.24
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 467438 0 0 0 147476 248 0 0 25 0 5 0 1122519021 2034819072 464395 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 496782 464395 95 177 0 496604 0
[pid=19950/tid=19954] ppid=19948 vsize=1987128 CPUtime=368.03
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 128903 0 0 0 36749 54 0 0 25 0 5 0 1122519856 2034819072 464395 4294967295 134512640 135237372 3218405536 3081673520 134562430 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=1987128 CPUtime=364.51
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 86449 0 0 0 36407 44 0 0 25 0 5 0 1122519856 2034819072 464395 4294967295 134512640 135237372 3218405536 3073268944 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=1987128 CPUtime=371.99
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 89082 0 0 0 37152 47 0 0 25 0 5 0 1122519856 2034819072 464395 4294967295 134512640 135237372 3218405536 3064851024 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=1987128 CPUtime=368.15
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 145849 0 0 0 36724 91 0 0 25 0 5 0 1122519861 2034819072 464395 4294967295 134512640 135237372 3218405536 3056460096 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1477.24
Current children cumulated vsize (KiB) 1987128

[startup+822.311 s]
/proc/loadavg: 8.37 8.26 7.27 9/118 22593
/proc/meminfo: memFree=669048/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=2073796 CPUtime=1596.15
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 487709 0 0 0 159352 263 0 0 25 0 5 0 1122519021 2123567104 484666 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 518449 484666 95 177 0 518271 0
[pid=19950/tid=19954] ppid=19948 vsize=2073796 CPUtime=397.43
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 135429 0 0 0 39688 55 0 0 25 0 5 0 1122519856 2123567104 484666 4294967295 134512640 135237372 3218405536 3081685420 134562272 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=2073796 CPUtime=394.15
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 89750 0 0 0 39369 46 0 0 25 0 5 0 1122519856 2123567104 484666 4294967295 134512640 135237372 3218405536 3073269068 134636641 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=2073796 CPUtime=402.48
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 92761 0 0 0 40197 51 0 0 25 0 5 0 1122519856 2123567104 484666 4294967295 134512640 135237372 3218405536 3064850656 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=2073796 CPUtime=397.54
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 152614 0 0 0 39655 99 0 0 25 0 5 0 1122519861 2123567104 484666 4294967295 134512640 135237372 3218405536 3056449568 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1596.15
Current children cumulated vsize (KiB) 2073796



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+838.11 s]
/proc/loadavg: 8.37 8.27 7.28 10/118 22595
/proc/meminfo: memFree=647388/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=2096500 CPUtime=1627.77
/proc/19950/stat : 19950 (ttsth-5-0) S 19948 19950 14986 0 -1 4202496 492928 0 0 0 162510 267 0 0 25 0 5 0 1122519021 2146816000 489885 4294967295 134512640 135237372 3218405536 3218405132 134629544 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 524125 489885 95 177 0 523947 0
[pid=19950/tid=19954] ppid=19948 vsize=2096500 CPUtime=405.4
/proc/19950/task/19954/stat : 19954 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 136836 0 0 0 40485 55 0 0 25 0 5 0 1122519856 2146816000 489885 4294967295 134512640 135237372 3218405536 3081655952 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19955] ppid=19948 vsize=2096500 CPUtime=402.06
/proc/19950/task/19955/stat : 19955 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 91095 0 0 0 40158 48 0 0 25 0 5 0 1122519856 2146816000 489885 4294967295 134512640 135237372 3218405536 3073249808 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=19950/tid=19956] ppid=19948 vsize=2096500 CPUtime=410.31
/proc/19950/task/19956/stat : 19956 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 93748 0 0 0 40980 51 0 0 25 0 5 0 1122519856 2146816000 489885 4294967295 134512640 135237372 3218405536 3064825056 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=2096500 CPUtime=405.44
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4202560 154094 0 0 0 40443 101 0 0 25 0 5 0 1122519861 2146816000 489885 4294967295 134512640 135237372 3218405536 3056458960 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1627.77
Current children cumulated vsize (KiB) 2096500

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+838.23 s]
/proc/loadavg: 8.37 8.27 7.28 7/116 22596
/proc/meminfo: memFree=2210388/3368532 swapFree=1953000/1959888
[pid=19950] ppid=19948 vsize=0 CPUtime=1627.9
/proc/19950/stat : 19950 (ttsth-5-0) Z 19948 19950 14986 0 -1 4203524 492933 0 0 0 162514 276 0 0 21 0 2 0 1122519021 0 0 4294967295 0 0 0 0 0 0 0 0 0 4294967295 0 0 17 3 0 0 0
/proc/19950/statm: 0 0 0 0 0 0 0
[pid=19950/tid=19957] ppid=19948 vsize=0 CPUtime=405.54
/proc/19950/task/19957/stat : 19957 (ttsth-5-0) R 19948 19950 14986 0 -1 4203588 154094 0 0 0 40444 110 0 0 25 0 2 0 1122519861 0 0 4294967295 0 0 0 0 0 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1627.9
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 838.259
CPU time (s): 1627.92
CPU user time (s): 1625.14
CPU system time (s): 2.78
CPU usage (%): 194.203
Max. virtual memory (cumulated for all children) (KiB): 2096500

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1625.14
system time used= 2.78
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 492933
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 848
involuntary context switches= 43559

runsolver used 0 second user time and 0 second system time

The end

Launcher Data 

Begin job on tipi45 at 2009-05-30 13:43:37
IDJOB=1780813
IDBENCH=71068
IDSOLVER=632
FILE ID=tipi45/1780813-1243683817
PBS_JOBID=
Free space on /tmp= 15498 MiB

SOLVER NAME= ttsth-5-0 2009-03-19 (4 threads)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/ndhf_xits_12_UNSAT.cnf
COMMAND LINE= HOME/ttsth-5-0 -num-threads MAXNBTHREAD BENCHNAME
RUNSOLVER COMMAND LINE= runsolver --timestamp  -w /tmp/evaluation-result-1780813-1243683817/watcher-1780813-1243683817 -o /tmp/evaluation-result-1780813-1243683817/solver-1780813-1243683817 -C 10000 -W 1000 -M 2047  HOME/ttsth-5-0 -num-threads 4 HOME/instance-1780813-1243683817.cnf

TIME LIMIT= 10000 seconds
MEMORY LIMIT= 2047 MiB
MAX NB THREAD= 4

MD5SUM BENCH= ecc4810d82e30597ea8451526243ec09
RANDOM SEED= 140025108

Linux tipi45 2.6.21.5 #6 SMP Wed Jul 4 17:16:38 CEST 2007 i686 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3992.41
clflush size	: 64

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.10
clflush size	: 64

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.16
clflush size	: 64

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.14
clflush size	: 64


/proc/meminfo:
MemTotal:      3368532 kB
MemFree:       2620584 kB
Buffers:         34964 kB
Cached:         155504 kB
SwapCached:       1728 kB
Active:         580368 kB
Inactive:        88352 kB
HighTotal:     2489216 kB
HighFree:      1868556 kB
LowTotal:       879316 kB
LowFree:        752028 kB
SwapTotal:     1959888 kB
SwapFree:      1953000 kB
Dirty:            7180 kB
Writeback:           0 kB
AnonPages:      477912 kB
Mapped:           6600 kB
Slab:            65480 kB
SReclaimable:    54700 kB
SUnreclaim:      10780 kB
PageTables:       1124 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3644152 kB
Committed_AS:   565968 kB
VmallocTotal:   114680 kB
VmallocUsed:     17900 kB
VmallocChunk:    96244 kB

Free space on /tmp at the end= 15498 MiB
End job on tipi45 at 2009-05-30 13:57:38