Trace number 1779958

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth-5-0 2009-03-19 (4 threads)? (TO) 2058.75 1000.1

General information on the benchmark

NameAPPLICATIONS/bioinfo/
q_query_3_l43_lambda.cnf
MD5SUM040db958f11a89b7e62063379f36b96a
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark61.85
Satisfiable
(Un)Satisfiability was proved
Number of variables30771
Number of clauses153033
Sum of the clauses size440888
Maximum clause length9
Minimum clause length1
Number of clauses of size 14034
Number of clauses of size 238515
Number of clauses of size 389223
Number of clauses of size 414666
Number of clauses of size 56466
Number of clauses of size over 5129

Solver Data 

0.32/0.36	c Command: HOME/ttsth-5-0 -num-threads 4 HOME/instance-1779958-1243680050.cnf 
0.32/0.36	c Using SATTIMEOUT of 10000
0.32/0.36	c Using SATRAM of 2047
0.32/0.36	c Size of subset node = 16
0.32/0.36	c Variables = 30771
0.32/0.36	c Clauses = 153033
0.32/0.36	c Literals = 440888
0.32/0.36	c Reduced size:
0.32/0.36	c Variables = 30771
0.32/0.36	c Clauses = 137658
0.32/0.36	c Literals = 402281
0.32/0.36	c a2
0.62/0.70	c a1
9.22/9.29	c Independent sub-problems = 1
9.22/9.29	c Sub-problem with 25829 variables and 137658 clauses
9.22/9.29	c using permutation method 4
9.22/9.29	c count = 1
9.22/9.29	c Ordering finished 9.240 seconds
9.22/9.29	c mapped 9.300 seconds
9.22/9.29	c permuted 9.300 seconds
9.22/9.29	c about to sort
9.32/9.34	c sorted
11.42/11.46	c clauses now 137658
11.42/11.46	c init finished 9.520 seconds
11.42/11.46	c tree built 11.470 seconds
11.42/11.46	c 
11.42/11.46	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
11.42/11.46	c ---------------------------------|----------------------------------|------------------------------------------------------
11.42/11.46	c ................................ | ..........                       |                       11.5           0.0             
11.42/11.46	c ................................ | ..........                     1 |                       11.5           0.1             
11.42/11.46	c ****************................ | ..........                     2 |                       11.5           0.2            1
11.42/11.46	c ****************................ | ..........                     3 |                       11.5           0.3            1
11.42/11.46	c ****************................ | ..........                     4 |                       11.5           0.3            1
11.42/11.46	c ****************................ | ..........                     5 |                       11.5           0.4            1
11.42/11.46	c ****************................ | ..........                     6 |                       11.5           0.5            1
11.42/11.47	c ****************................ | ..........                  1000 | 2                     11.5          87.1          109
199.94/105.75	c ****************................ | ..........               2626000 | 131582               200.0       13128.7     13573576
399.89/196.87	c ****************................ | ..........               4149000 | 210528               400.0       10372.2     22495249
600.05/283.91	c ****************................ | ..........               5243000 | 267428               600.2        8735.0     29324946
799.83/383.80	c ****************................ | ..........               6307000 | 320693               800.0        7883.7     35187091
1000.25/483.94	c ****************................ | *.........               7349000 | 371495              1000.3        7346.7     40365266
1199.85/583.80	c ****************................ | *.........               8435000 | 423961              1200.0        7029.0     45507366
1399.90/670.68	c ****************................ | *.........               9334000 | 468107              1400.0        6667.0     49645235
1600.07/770.77	c ****************................ | *.........              10091000 | 504397              1600.2        6306.0     53229188
1800.09/870.72	c ****************................ | *.........              10900000 | 543927              1800.1        6055.3     57377321
1999.83/970.70	c ****************................ | **........              11775000 | 588950              2000.0        5887.4     62502069

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:503) (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w /tmp/evaluation-result-1779958-1243680050/watcher-1779958-1243680050 -o /tmp/evaluation-result-1779958-1243680050/solver-1779958-1243680050 -C 10000 -W 1000 -M 2047 HOME/ttsth-5-0 -num-threads 4 HOME/instance-1779958-1243680050.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 10000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 10030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 2096128 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 2147328 KiB
Current StackSize limit: 8192 KiB


[startup+0 s]
/proc/loadavg: 6.04 6.21 6.18 4/113 7538
/proc/meminfo: memFree=3117112/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1820 CPUtime=0.02
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 328 0 0 0 2 0 0 0 19 0 1 0 1132669032 1863680 284 4294967295 134512640 135237372 3213691936 3213690460 134639534 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 455 284 75 177 0 277 0

[startup+0.102801 s]
/proc/loadavg: 6.04 6.21 6.18 4/113 7538
/proc/meminfo: memFree=3117112/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=5496 CPUtime=0.11
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 1235 0 0 0 10 1 0 0 21 0 1 0 1132669032 5627904 1191 4294967295 134512640 135237372 3213691936 3213691384 134732614 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 1374 1191 75 177 0 1196 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 5496

[startup+0.212792 s]
/proc/loadavg: 6.04 6.21 6.18 4/113 7538
/proc/meminfo: memFree=3117112/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=10168 CPUtime=0.22
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 2364 0 0 0 20 2 0 0 22 0 1 0 1132669032 10412032 2320 4294967295 134512640 135237372 3213691936 3213691384 134732614 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 2542 2320 75 177 0 2364 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 10168

[startup+0.312783 s]
/proc/loadavg: 6.04 6.21 6.18 4/113 7538
/proc/meminfo: memFree=3117112/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=11612 CPUtime=0.32
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 2700 0 0 0 30 2 0 0 22 0 1 0 1132669032 11890688 2656 4294967295 134512640 135237372 3213691936 3213691628 134534520 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 2903 2656 75 177 0 2725 0
Current children cumulated CPU time (s) 0.32
Current children cumulated vsize (KiB) 11612

[startup+0.712749 s]
/proc/loadavg: 6.04 6.21 6.18 4/113 7538
/proc/meminfo: memFree=3117112/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=22528 CPUtime=0.72
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 5310 0 0 0 70 2 0 0 24 0 1 0 1132669032 23068672 5265 4294967295 134512640 135237372 3213691936 3213691520 134516664 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 5632 5265 76 177 0 5454 0
Current children cumulated CPU time (s) 0.72
Current children cumulated vsize (KiB) 22528

[startup+1.51268 s]
/proc/loadavg: 6.04 6.21 6.18 4/114 7543
/proc/meminfo: memFree=3093136/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=26156 CPUtime=1.52
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 6801 0 0 0 148 4 0 0 25 0 1 0 1132669032 26783744 6344 4294967295 134512640 135237372 3213691936 3213691264 134555272 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 6539 6344 77 177 0 6361 0
Current children cumulated CPU time (s) 1.52
Current children cumulated vsize (KiB) 26156

[startup+3.11254 s]
/proc/loadavg: 6.04 6.21 6.18 5/114 7543
/proc/meminfo: memFree=3091908/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=26448 CPUtime=3.12
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 6827 0 0 0 308 4 0 0 25 0 1 0 1132669032 27082752 6370 4294967295 134512640 135237372 3213691936 3213691296 134600696 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 6612 6370 77 177 0 6434 0
Current children cumulated CPU time (s) 3.12
Current children cumulated vsize (KiB) 26448

[startup+6.31227 s]
/proc/loadavg: 5.79 6.15 6.16 4/114 7547
/proc/meminfo: memFree=3091512/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=26580 CPUtime=6.32
/proc/7538/stat : 7538 (ttsth-5-0) R 7536 7538 32205 0 -1 4202496 6884 0 0 0 628 4 0 0 25 0 1 0 1132669032 27217920 6427 4294967295 134512640 135237372 3213691936 3213691296 134600718 0 0 0 0 0 0 0 17 0 0 0 0
/proc/7538/statm: 6645 6427 77 177 0 6467 0
Current children cumulated CPU time (s) 6.32
Current children cumulated vsize (KiB) 26580

[startup+12.7117 s]
/proc/loadavg: 5.57 6.10 6.14 7/118 7563
/proc/meminfo: memFree=2995240/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=156772 CPUtime=13.96
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 31995 0 0 0 1387 9 0 0 25 0 5 0 1132669032 160534528 31133 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 39193 31134 95 177 0 39015 0
[pid=7538/tid=7556] ppid=7536 vsize=156772 CPUtime=0.73
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 472 0 0 0 73 0 0 0 25 0 5 0 1132670178 160534528 31134 4294967295 134512640 135237372 3213691936 3081541552 134546026 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=156772 CPUtime=0.52
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 373 0 0 0 52 0 0 0 25 0 5 0 1132670178 160534528 31134 4294967295 134512640 135237372 3213691936 3073145200 134681718 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=156772 CPUtime=0.33
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 256 0 0 0 33 0 0 0 25 0 5 0 1132670178 160534528 31134 4294967295 134512640 135237372 3213691936 3064906268 134680502 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=156772 CPUtime=0.91
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 348 0 0 0 91 0 0 0 25 0 5 0 1132670178 160534528 31134 4294967295 134512640 135237372 3213691936 3056381520 134681633 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 13.96
Current children cumulated vsize (KiB) 156772

[startup+25.5106 s]
/proc/loadavg: 5.66 6.09 6.14 7/118 7580
/proc/meminfo: memFree=2963564/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=187324 CPUtime=39.56
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 39475 0 0 0 3943 13 0 0 25 0 5 0 1132669032 191819776 38613 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 46831 38613 95 177 0 46653 0
[pid=7538/tid=7556] ppid=7536 vsize=187324 CPUtime=7.14
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 3559 0 0 0 712 2 0 0 25 0 5 0 1132670178 191819776 38613 4294967295 134512640 135237372 3213691936 3081538608 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=187324 CPUtime=6.91
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 2002 0 0 0 690 1 0 0 25 0 5 0 1132670178 191819776 38613 4294967295 134512640 135237372 3213691936 3073167872 134591181 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=187324 CPUtime=6.63
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 1608 0 0 0 662 1 0 0 25 0 5 0 1132670178 191819776 38613 4294967295 134512640 135237372 3213691936 3064789584 134551736 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=187324 CPUtime=7.41
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 1759 0 0 0 741 0 0 0 25 0 5 0 1132670178 191819776 38613 4294967295 134512640 135237372 3213691936 3056382976 134562401 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39.56
Current children cumulated vsize (KiB) 187324

[startup+51.1084 s]
/proc/loadavg: 6.04 6.15 6.16 7/118 7807
/proc/meminfo: memFree=2896036/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=256300 CPUtime=90.75
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 56530 0 0 0 9057 18 0 0 25 0 5 0 1132669032 262451200 55668 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 64075 55668 95 177 0 63897 0
[pid=7538/tid=7556] ppid=7536 vsize=256300 CPUtime=20.04
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 14691 0 0 0 1998 6 0 0 25 0 5 0 1132670178 262451200 55668 4294967295 134512640 135237372 3213691936 3080876400 134562401 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=256300 CPUtime=19.61
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 5311 0 0 0 1959 2 0 0 25 0 5 0 1132670178 262451200 55668 4294967295 134512640 135237372 3213691936 3073182124 134562272 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=256300 CPUtime=19.62
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 2886 0 0 0 1961 1 0 0 25 0 5 0 1132670178 262451200 55668 4294967295 134512640 135237372 3213691936 3064777872 134680336 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=256300 CPUtime=20.01
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 3095 0 0 0 2001 0 0 0 25 0 5 0 1132670178 262451200 55668 4294967295 134512640 135237372 3213691936 3056376944 134562290 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 90.75
Current children cumulated vsize (KiB) 256300

[startup+102.314 s]
/proc/loadavg: 6.56 6.26 6.19 10/122 7989
/proc/meminfo: memFree=2788864/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=367228 CPUtime=193.14
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 83070 0 0 0 19282 32 0 0 25 0 5 0 1132669032 376041472 82208 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 91807 82208 95 177 0 91629 0
[pid=7538/tid=7556] ppid=7536 vsize=367228 CPUtime=45.64
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 31871 0 0 0 4553 11 0 0 25 0 5 0 1132670178 376041472 82210 4294967295 134512640 135237372 3213691936 3080979760 134546034 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=367228 CPUtime=45.22
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 8916 0 0 0 4516 6 0 0 25 0 5 0 1132670178 376041472 82210 4294967295 134512640 135237372 3213691936 3073160640 134562380 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=367228 CPUtime=45.3
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 5678 0 0 0 4526 4 0 0 25 0 5 0 1132670178 376041472 82210 4294967295 134512640 135237372 3213691936 3064780384 134561510 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=367228 CPUtime=45.51
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 6060 0 0 0 4549 2 0 0 25 0 5 0 1132670178 376041472 82210 4294967295 134512640 135237372 3213691936 3056403216 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 193.14
Current children cumulated vsize (KiB) 367228

[startup+162.309 s]
/proc/loadavg: 6.53 6.32 6.21 9/119 8203
/proc/meminfo: memFree=2699336/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=461884 CPUtime=313.88
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 105414 0 0 0 31347 41 0 0 25 0 5 0 1132669032 472969216 104552 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 115471 104552 95 177 0 115293 0
[pid=7538/tid=7556] ppid=7536 vsize=461884 CPUtime=75.64
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 46277 0 0 0 7546 18 0 0 25 0 5 0 1132670178 472969216 104552 4294967295 134512640 135237372 3213691936 3080932384 134589638 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=461884 CPUtime=75.22
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 11830 0 0 0 7516 6 0 0 25 0 5 0 1132670178 472969216 104552 4294967295 134512640 135237372 3213691936 3073180880 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=461884 CPUtime=75.26
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 8158 0 0 0 7521 5 0 0 25 0 5 0 1132670178 472969216 104552 4294967295 134512640 135237372 3213691936 3064770304 134562317 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=461884 CPUtime=76.3
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 8602 0 0 0 7627 3 0 0 25 0 5 0 1132670178 472969216 104552 4294967295 134512640 135237372 3213691936 3056402336 134562410 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 313.88
Current children cumulated vsize (KiB) 461884

[startup+222.314 s]
/proc/loadavg: 6.58 6.39 6.24 9/119 8472
/proc/meminfo: memFree=2598236/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=565356 CPUtime=463.52
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 130508 0 0 0 46295 57 0 0 25 0 5 0 1132669032 578924544 129646 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 141339 129646 95 177 0 141161 0
[pid=7538/tid=7556] ppid=7536 vsize=565356 CPUtime=105.84
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 61327 0 0 0 10559 25 0 0 25 0 5 0 1132670178 578924544 129646 4294967295 134512640 135237372 3213691936 3080982048 134749353 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=565356 CPUtime=105.03
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 14158 0 0 0 10495 8 0 0 25 0 5 0 1132670178 578924544 129646 4294967295 134512640 135237372 3213691936 3073180272 134562290 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=565356 CPUtime=105.11
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 9808 0 0 0 10501 10 0 0 25 0 5 0 1132670178 578924544 129646 4294967295 134512640 135237372 3213691936 3064777680 134588544 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=565356 CPUtime=136.07
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 14668 0 0 0 13602 5 0 0 25 0 5 0 1132670178 578924544 129646 4294967295 134512640 135237372 3213691936 3056376736 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 463.52
Current children cumulated vsize (KiB) 565356

[startup+282.318 s]
/proc/loadavg: 6.67 6.47 6.28 7/118 8715
/proc/meminfo: memFree=2536204/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=631292 CPUtime=597.05
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 146240 0 0 0 59641 64 0 0 25 0 5 0 1132669032 646443008 145378 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 157823 145378 95 177 0 157645 0
[pid=7538/tid=7556] ppid=7536 vsize=631292 CPUtime=135.86
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 69825 0 0 0 13558 28 0 0 25 0 5 0 1132670178 646443008 145378 4294967295 134512640 135237372 3213691936 3080916048 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=631292 CPUtime=135.02
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 16202 0 0 0 13493 9 0 0 25 0 5 0 1132670178 646443008 145378 4294967295 134512640 135237372 3213691936 3073153632 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=631292 CPUtime=135.21
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 11377 0 0 0 13510 11 0 0 25 0 5 0 1132670178 646443008 145378 4294967295 134512640 135237372 3213691936 3064771072 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=631292 CPUtime=179.49
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 18289 0 0 0 17942 7 0 0 25 0 5 0 1132670178 646443008 145378 4294967295 134512640 135237372 3213691936 3056394112 134592217 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 597.05
Current children cumulated vsize (KiB) 631292

################
# More data... #
################

[pid=7538/tid=7557] ppid=7536 vsize=832640 CPUtime=254.84
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 22696 0 0 0 25471 13 0 0 25 0 5 0 1132670178 852623360 194765 4294967295 134512640 135237372 3213691936 3073179024 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=832640 CPUtime=255.29
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 15908 0 0 0 25509 20 0 0 25 0 5 0 1132670178 852623360 194765 4294967295 134512640 135237372 3213691936 3064771136 134562370 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=832640 CPUtime=299.41
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 27243 0 0 0 29927 14 0 0 25 0 5 0 1132670178 852623360 194765 4294967295 134512640 135237372 3213691936 3056404928 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1077.05
Current children cumulated vsize (KiB) 832640

[startup+582.313 s]
/proc/loadavg: 6.35 6.47 6.34 7/118 10104
/proc/meminfo: memFree=2281980/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=893588 CPUtime=1197.07
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 209691 0 0 0 119601 106 0 0 25 0 5 0 1132669032 915034112 208829 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 223397 208829 95 177 0 223219 0
[pid=7538/tid=7556] ppid=7536 vsize=893588 CPUtime=286.07
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 108534 0 0 0 28561 46 0 0 25 0 5 0 1132670178 915034112 208829 4294967295 134512640 135237372 3213691936 3080966000 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=893588 CPUtime=284.82
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 23511 0 0 0 28469 13 0 0 25 0 5 0 1132670178 915034112 208829 4294967295 134512640 135237372 3213691936 3073147136 134680285 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=893588 CPUtime=285.41
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 17852 0 0 0 28520 21 0 0 25 0 5 0 1132670178 915034112 208829 4294967295 134512640 135237372 3213691936 3064771744 134562317 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=893588 CPUtime=329.3
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 29247 0 0 0 32913 17 0 0 25 0 5 0 1132670178 915034112 208829 4294967295 134512640 135237372 3213691936 3056381600 134749351 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1197.07
Current children cumulated vsize (KiB) 893588

[startup+642.307 s]
/proc/loadavg: 6.46 6.48 6.35 7/118 10373
/proc/meminfo: memFree=2238196/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=940484 CPUtime=1337.99
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 220717 0 0 0 133687 112 0 0 25 0 5 0 1132669032 963055616 219855 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 235121 219855 95 177 0 234943 0
[pid=7538/tid=7556] ppid=7536 vsize=940484 CPUtime=316.37
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 114756 0 0 0 31591 46 0 0 25 0 5 0 1132670178 963055616 219855 4294967295 134512640 135237372 3213691936 3080977024 134588158 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=940484 CPUtime=335.73
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 24664 0 0 0 33559 14 0 0 25 0 5 0 1132670178 963055616 219855 4294967295 134512640 135237372 3213691936 3073167872 134749353 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=940484 CPUtime=315.41
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 19301 0 0 0 31518 23 0 0 25 0 5 0 1132670178 963055616 219855 4294967295 134512640 135237372 3213691936 3064771808 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=940484 CPUtime=359.01
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 31449 0 0 0 35881 20 0 0 25 0 5 0 1132670178 963055616 219855 4294967295 134512640 135237372 3213691936 3056402048 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1337.99
Current children cumulated vsize (KiB) 940484

[startup+702.312 s]
/proc/loadavg: 6.45 6.48 6.36 8/118 10653
/proc/meminfo: memFree=2197080/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=983324 CPUtime=1463.29
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 230785 0 0 0 146211 118 0 0 25 0 5 0 1132669032 1006923776 229923 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 245831 229923 95 177 0 245653 0
[pid=7538/tid=7556] ppid=7536 vsize=983324 CPUtime=346.44
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 120025 0 0 0 34597 47 0 0 25 0 5 0 1132670178 1006923776 229923 4294967295 134512640 135237372 3213691936 3080887248 134562437 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=983324 CPUtime=371.02
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 26058 0 0 0 37086 16 0 0 25 0 5 0 1132670178 1006923776 229923 4294967295 134512640 135237372 3213691936 3073155920 134591069 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=983324 CPUtime=345.41
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 20653 0 0 0 34516 25 0 0 25 0 5 0 1132670178 1006923776 229923 4294967295 134512640 135237372 3213691936 3064770768 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=983324 CPUtime=388.95
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 33502 0 0 0 38874 21 0 0 25 0 5 0 1132670178 1006923776 229923 4294967295 134512640 135237372 3213691936 3056404432 134562407 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1463.29
Current children cumulated vsize (KiB) 983324

[startup+762.307 s]
/proc/loadavg: 6.54 6.51 6.37 7/118 10929
/proc/meminfo: memFree=2164128/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1018112 CPUtime=1583.27
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 238954 0 0 0 158205 122 0 0 25 0 5 0 1132669032 1042546688 238092 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 254528 238092 95 177 0 254350 0
[pid=7538/tid=7556] ppid=7536 vsize=1018112 CPUtime=376.44
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 124625 0 0 0 37594 50 0 0 25 0 5 0 1132670178 1042546688 238092 4294967295 134512640 135237372 3213691936 3080906240 134749353 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=1018112 CPUtime=401
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 26703 0 0 0 40083 17 0 0 25 0 5 0 1132670178 1042546688 238092 4294967295 134512640 135237372 3213691936 3073179152 134562416 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=1018112 CPUtime=375.41
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 21880 0 0 0 37516 25 0 0 25 0 5 0 1132670178 1042546688 238092 4294967295 134512640 135237372 3213691936 3064787472 134680346 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=1018112 CPUtime=418.95
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 35199 0 0 0 41874 21 0 0 25 0 5 0 1132670178 1042546688 238092 4294967295 134512640 135237372 3213691936 3056376752 134562399 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1583.27
Current children cumulated vsize (KiB) 1018112

[startup+822.312 s]
/proc/loadavg: 6.60 6.54 6.39 7/118 11206
/proc/meminfo: memFree=2126508/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1057776 CPUtime=1703.27
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 248459 0 0 0 170199 128 0 0 25 0 5 0 1132669032 1083162624 247597 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 264444 247597 95 177 0 264266 0
[pid=7538/tid=7556] ppid=7536 vsize=1057776 CPUtime=406.44
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 130373 0 0 0 40591 53 0 0 25 0 5 0 1132670178 1083162624 247597 4294967295 134512640 135237372 3213691936 3080925120 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=1057776 CPUtime=430.92
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 27791 0 0 0 43075 17 0 0 25 0 5 0 1132670178 1083162624 247597 4294967295 134512640 135237372 3213691936 3073179296 134562373 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=1057776 CPUtime=405.48
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 22787 0 0 0 40521 27 0 0 25 0 5 0 1132670178 1083162624 247597 4294967295 134512640 135237372 3213691936 3064793392 134562413 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=1057776 CPUtime=448.96
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 36961 0 0 0 44874 22 0 0 25 0 5 0 1132670178 1083162624 247597 4294967295 134512640 135237372 3213691936 3056403488 134562344 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1703.27
Current children cumulated vsize (KiB) 1057776

[startup+882.317 s]
/proc/loadavg: 6.50 6.53 6.39 8/118 11483
/proc/meminfo: memFree=2085804/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1100636 CPUtime=1823.29
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 258482 0 0 0 182198 131 0 0 25 0 5 0 1132669032 1127051264 257620 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 275159 257620 95 177 0 274981 0
[pid=7538/tid=7556] ppid=7536 vsize=1100636 CPUtime=436.5
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 136086 0 0 0 43597 53 0 0 25 0 5 0 1132670178 1127051264 257620 4294967295 134512640 135237372 3213691936 3080981792 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=1100636 CPUtime=461.02
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 28775 0 0 0 46085 17 0 0 25 0 5 0 1132670178 1127051264 257620 4294967295 134512640 135237372 3213691936 3073180144 134562448 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=1100636 CPUtime=435.39
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 24511 0 0 0 43510 29 0 0 25 0 5 0 1132670178 1127051264 257620 4294967295 134512640 135237372 3213691936 3064791968 134680346 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=1100636 CPUtime=478.91
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 38563 0 0 0 47868 23 0 0 25 0 5 0 1132670178 1127051264 257620 4294967295 134512640 135237372 3213691936 3056395200 134680737 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1823.29
Current children cumulated vsize (KiB) 1100636

[startup+942.312 s]
/proc/loadavg: 6.32 6.47 6.38 7/118 11760
/proc/meminfo: memFree=2048240/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1140648 CPUtime=1943.23
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 268099 0 0 0 194189 134 0 0 25 0 5 0 1132669032 1168023552 267237 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 285162 267237 95 177 0 284984 0
[pid=7538/tid=7556] ppid=7536 vsize=1140648 CPUtime=466.64
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 142256 0 0 0 46610 54 0 0 25 0 5 0 1132670178 1168023552 267237 4294967295 134512640 135237372 3213691936 3080899616 134680367 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=1140648 CPUtime=490.9
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 29414 0 0 0 49073 17 0 0 25 0 5 0 1132670178 1168023552 267237 4294967295 134512640 135237372 3213691936 3073178112 134562430 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=1140648 CPUtime=465.51
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 25914 0 0 0 46521 30 0 0 25 0 5 0 1132670178 1168023552 267237 4294967295 134512640 135237372 3213691936 3064792864 134562363 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=1140648 CPUtime=508.71
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 39968 0 0 0 50847 24 0 0 25 0 5 0 1132670178 1168023552 267237 4294967295 134512640 135237372 3213691936 3056387148 134680502 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1943.23
Current children cumulated vsize (KiB) 1140648



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1000.02 s]
/proc/loadavg: 6.63 6.53 6.40 8/118 12032
/proc/meminfo: memFree=1972864/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1216340 CPUtime=2058.65
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 286770 0 0 0 205721 144 0 0 25 0 5 0 1132669032 1245532160 285908 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 304085 285908 95 177 0 303907 0
[pid=7538/tid=7556] ppid=7536 vsize=1216340 CPUtime=495.54
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 157293 0 0 0 49494 60 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3080971488 134749337 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=1216340 CPUtime=519.81
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 30089 0 0 0 51963 18 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3073180640 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=1216340 CPUtime=494.31
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 27408 0 0 0 49398 33 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3064775584 134680257 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=1216340 CPUtime=537.52
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 41433 0 0 0 53728 24 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3056402044 134562323 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2058.65
Current children cumulated vsize (KiB) 1216340

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1000.02 s]
/proc/loadavg: 6.63 6.53 6.40 8/118 12032
/proc/meminfo: memFree=1972864/3368532 swapFree=1954888/1959888
[pid=7538] ppid=7536 vsize=1216340 CPUtime=2058.65
/proc/7538/stat : 7538 (ttsth-5-0) S 7536 7538 32205 0 -1 4202496 286770 0 0 0 205721 144 0 0 25 0 5 0 1132669032 1245532160 285908 4294967295 134512640 135237372 3213691936 3213691532 134629544 0 0 0 0 4294967295 0 0 17 0 0 0 0
/proc/7538/statm: 304085 285908 95 177 0 303907 0
[pid=7538/tid=7556] ppid=7536 vsize=1216340 CPUtime=495.54
/proc/7538/task/7556/stat : 7556 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 157293 0 0 0 49494 60 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3080971488 134749337 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=7538/tid=7557] ppid=7536 vsize=1216340 CPUtime=519.81
/proc/7538/task/7557/stat : 7557 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 30089 0 0 0 51963 18 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3073180640 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7558] ppid=7536 vsize=1216340 CPUtime=494.31
/proc/7538/task/7558/stat : 7558 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 27408 0 0 0 49398 33 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3064775584 134680257 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=7538/tid=7559] ppid=7536 vsize=1216340 CPUtime=537.52
/proc/7538/task/7559/stat : 7559 (ttsth-5-0) R 7536 7538 32205 0 -1 4202560 41433 0 0 0 53728 24 0 0 25 0 5 0 1132670178 1245532160 285908 4294967295 134512640 135237372 3213691936 3056402044 134562323 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2058.65
Current children cumulated vsize (KiB) 1216340

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1000.1
CPU time (s): 2058.75
CPU user time (s): 2057.25
CPU system time (s): 1.5
CPU usage (%): 205.855
Max. virtual memory (cumulated for all children) (KiB): 1216340

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2057.25
system time used= 1.5
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 286771
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 250
involuntary context switches= 40047

runsolver used 0 second user time and 0 second system time

The end

Launcher Data 

Begin job on  at 2009-05-30 12:40:50
IDJOB=1779958
IDBENCH=70943
IDSOLVER=632
FILE ID=/1779958-1243680050
PBS_JOBID=
Free space on /tmp= 15877 MiB

SOLVER NAME= ttsth-5-0 2009-03-19 (4 threads)
BENCH NAME= SAT09/APPLICATIONS/bioinfo/q_query_3_l43_lambda.cnf
COMMAND LINE= HOME/ttsth-5-0 -num-threads MAXNBTHREAD BENCHNAME
RUNSOLVER COMMAND LINE= runsolver --timestamp  -w /tmp/evaluation-result-1779958-1243680050/watcher-1779958-1243680050 -o /tmp/evaluation-result-1779958-1243680050/solver-1779958-1243680050 -C 10000 -W 1000 -M 2047  HOME/ttsth-5-0 -num-threads 4 HOME/instance-1779958-1243680050.cnf

TIME LIMIT= 10000 seconds
MEMORY LIMIT= 2047 MiB
MAX NB THREAD= 4

MD5SUM BENCH= 040db958f11a89b7e62063379f36b96a
RANDOM SEED= 1277253103

Linux (none) 2.6.21.5 #6 SMP Wed Jul 4 17:16:38 CEST 2007 i686 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3992.43
clflush size	: 64

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.11
clflush size	: 64

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.16
clflush size	: 64

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.15
clflush size	: 64


/proc/meminfo:
MemTotal:      3368532 kB
MemFree:       3117260 kB
Buffers:         53260 kB
Cached:          97772 kB
SwapCached:        720 kB
Active:         118872 kB
Inactive:        47820 kB
HighTotal:     2489216 kB
HighFree:      2371968 kB
LowTotal:       879316 kB
LowFree:        745292 kB
SwapTotal:     1959888 kB
SwapFree:      1954888 kB
Dirty:            3816 kB
Writeback:           0 kB
AnonPages:       15208 kB
Mapped:           6172 kB
Slab:            74588 kB
SReclaimable:    64964 kB
SUnreclaim:       9624 kB
PageTables:        740 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3644152 kB
Committed_AS:   117276 kB
VmallocTotal:   114680 kB
VmallocUsed:      1444 kB
VmallocChunk:   113072 kB

Free space on /tmp at the end= 15877 MiB
End job on  at 2009-05-30 12:57:32