Trace number 3386279

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 4112.23 1300.06

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
64/aes_64_1_keyfind_1.cnf
MD5SUM9e18e0ee8a56ea2c59fe8e682ff6c747
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark32.783
Satisfiable
(Un)Satisfiability was proved
Number of variables596
Number of clauses2780
Sum of the clauses size10460
Maximum clause length5
Minimum clause length1
Number of clauses of size 1132
Number of clauses of size 280
Number of clauses of size 31168
Number of clauses of size 4336
Number of clauses of size 51064
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386279-1305582132/watcher-3386279-1305582132 -o /tmp/evaluation-result-3386279-1305582132/solver-3386279-1305582132 -C 4800 -W 1300 -M 15500 SArTagnan HOME/instance-3386279-1305582132.cnf -threads=4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 2.00 2.29 3/506 30729
/proc/meminfo: memFree=16002240/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=49976 CPUtime=0.02 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 622 0 0 0 2 0 0 0 22 0 4 0 748443916 51175424 560 33554432000 4194304 4625164 140734708394816 18446744073709551615 46947540596652 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/30729/statm: 12494 560 303 106 0 7997 0
[pid=30729/tid=30730] ppid=30727 vsize=49976 CPUtime=0 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 0 0 0 0 0 0 0 0 23 0 4 0 748443917 51175424 560 33554432000 4194304 4625164 140734708394816 18446744073709551615 46947540596656 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=49976 CPUtime=0 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 0 0 0 0 0 0 0 0 23 0 4 0 748443917 51175424 560 33554432000 4194304 4625164 140734708394816 18446744073709551615 224284982257 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=49976 CPUtime=0.01 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 1 0 0 0 1 0 0 0 23 0 4 0 748443917 51175424 560 33554432000 4194304 4625164 140734708394816 18446744073709551615 46947540596658 0 0 4096 2 0 0 0 -1 4 0 0 0

[startup+0.120172 s]
/proc/loadavg: 2.00 2.00 2.29 3/506 30729
/proc/meminfo: memFree=16002240/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=182836 CPUtime=0.33 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 1602 0 0 0 33 0 0 0 23 0 4 0 748443916 187224064 1271 33554432000 4194304 4625164 140734708394816 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 45709 1271 344 106 0 41212 0
[pid=30729/tid=30730] ppid=30727 vsize=182836 CPUtime=0.05 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 123 0 0 0 5 0 0 0 24 0 4 0 748443917 187224064 1271 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351981 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=182836 CPUtime=0.05 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 205 0 0 0 5 0 0 0 24 0 4 0 748443917 187224064 1271 33554432000 4194304 4625164 140734708394816 18446744073709551615 4465206 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=182836 CPUtime=0.11 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 292 0 0 0 11 0 0 0 24 0 4 0 748443917 187224064 1271 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.33
Current children cumulated vsize (KiB) 182836

[startup+0.201157 s]
/proc/loadavg: 2.00 2.00 2.29 3/506 30729
/proc/meminfo: memFree=16002240/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=183864 CPUtime=0.65 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 2448 0 0 0 65 0 0 0 23 0 4 0 748443916 188276736 1599 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512084 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 45966 1599 344 106 0 41469 0
[pid=30729/tid=30730] ppid=30727 vsize=183864 CPUtime=0.13 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 340 0 0 0 13 0 0 0 25 0 4 0 748443917 188276736 1599 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352098 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=183864 CPUtime=0.13 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 448 0 0 0 13 0 0 0 18 0 4 0 748443917 188276736 1599 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=183864 CPUtime=0.18 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 577 0 0 0 18 0 0 0 25 0 4 0 748443917 188276736 1600 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 183864

[startup+0.301151 s]
/proc/loadavg: 2.00 2.00 2.29 3/506 30729
/proc/meminfo: memFree=16002240/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=250004 CPUtime=1.05 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3012 0 0 0 105 0 0 0 23 0 4 0 748443916 256004096 1894 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 62501 1894 344 106 0 58004 0
[pid=30729/tid=30730] ppid=30727 vsize=250004 CPUtime=0.23 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 504 0 0 0 23 0 0 0 25 0 4 0 748443917 256004096 1894 33554432000 4194304 4625164 140734708394816 18446744073709551615 224284583664 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=250004 CPUtime=0.23 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 510 0 0 0 23 0 0 0 18 0 4 0 748443917 256004096 1894 33554432000 4194304 4625164 140734708394816 18446744073709551615 4278468 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=250004 CPUtime=0.28 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 895 0 0 0 28 0 0 0 25 0 4 0 748443917 256004096 1894 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352776 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1.05
Current children cumulated vsize (KiB) 250004

[startup+0.701076 s]
/proc/loadavg: 2.00 2.00 2.29 3/506 30729
/proc/meminfo: memFree=16002240/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=252016 CPUtime=2.65 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 5855 0 0 0 264 1 0 0 25 0 4 0 748443916 258064384 3000 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512315 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 63004 3000 344 106 0 58507 0
[pid=30729/tid=30730] ppid=30727 vsize=252016 CPUtime=0.63 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 841 0 0 0 63 0 0 0 25 0 4 0 748443917 258064384 3000 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=252016 CPUtime=0.62 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 1170 0 0 0 62 0 0 0 18 0 4 0 748443917 258064384 3000 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=252016 CPUtime=0.68 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2457 0 0 0 68 0 0 0 25 0 4 0 748443917 258064384 3000 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352098 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2.65
Current children cumulated vsize (KiB) 252016

[startup+1.50092 s]
/proc/loadavg: 2.00 2.00 2.29 7/510 30733
/proc/meminfo: memFree=15985476/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=255140 CPUtime=5.83 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 9873 0 0 0 581 2 0 0 25 0 4 0 748443916 261263360 4875 33554432000 4194304 4625164 140734708394816 18446744073709551615 4511945 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 63785 4875 344 106 0 59288 0
[pid=30729/tid=30730] ppid=30727 vsize=255140 CPUtime=1.43 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 1230 0 0 0 143 0 0 0 25 0 4 0 748443917 261263360 4875 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=255140 CPUtime=1.43 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2094 0 0 0 142 1 0 0 19 0 4 0 748443917 261263360 4875 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=255140 CPUtime=1.48 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 4760 0 0 0 147 1 0 0 25 0 4 0 748443917 261263360 4875 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.83
Current children cumulated vsize (KiB) 255140

[startup+3.10063 s]
/proc/loadavg: 2.00 2.00 2.29 7/512 30737
/proc/meminfo: memFree=15975544/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=265120 CPUtime=12.22 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 20082 0 0 0 1218 4 0 0 25 0 4 0 748443916 271482880 7608 33554432000 4194304 4625164 140734708394816 18446744073709551615 4511953 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 66280 7608 345 106 0 61783 0
[pid=30729/tid=30730] ppid=30727 vsize=265120 CPUtime=3.03 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2820 0 0 0 303 0 0 0 25 0 4 0 748443917 271482880 7608 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=265120 CPUtime=3.02 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 4346 0 0 0 301 1 0 0 25 0 4 0 748443917 271482880 7608 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351989 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=265120 CPUtime=3.09 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 10408 0 0 0 307 2 0 0 25 0 4 0 748443917 271482880 7608 33554432000 4194304 4625164 140734708394816 18446744073709551615 4546721 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 12.22
Current children cumulated vsize (KiB) 265120

[startup+6.30109 s]
/proc/loadavg: 2.64 2.13 2.33 11/518 30746
/proc/meminfo: memFree=15881828/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=268044 CPUtime=24.37 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 26743 0 0 0 2431 6 0 0 25 0 4 0 748443916 274477056 10920 33554432000 4194304 4625164 140734708394816 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 67011 10920 361 106 0 62514 0
[pid=30729/tid=30730] ppid=30727 vsize=268044 CPUtime=5.63 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 4383 0 0 0 563 0 0 0 25 0 4 0 748443917 274477056 10920 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352299 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=268044 CPUtime=6.23 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 6530 0 0 0 621 2 0 0 25 0 4 0 748443917 274477056 10920 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351989 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=268044 CPUtime=6.28 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 12339 0 0 0 626 2 0 0 25 0 4 0 748443917 274477056 10920 33554432000 4194304 4625164 140734708394816 18446744073709551615 4547328 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.37
Current children cumulated vsize (KiB) 268044

[startup+12.7011 s]
/proc/loadavg: 2.91 2.20 2.35 8/512 30797
/proc/meminfo: memFree=15820552/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=284780 CPUtime=48.87 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 41754 0 0 0 4877 10 0 0 25 0 4 0 748443916 291614720 17769 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512017 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 71195 17769 446 106 0 66698 0
[pid=30729/tid=30730] ppid=30727 vsize=284780 CPUtime=11.05 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 7171 0 0 0 1105 0 0 0 25 0 4 0 748443917 291614720 17769 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=284780 CPUtime=12.63 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 10015 0 0 0 1259 4 0 0 25 0 4 0 748443917 291614720 17769 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=284780 CPUtime=12.68 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 19407 0 0 0 1264 4 0 0 25 0 4 0 748443917 291614720 17769 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 48.87
Current children cumulated vsize (KiB) 284780

[startup+25.5009 s]
/proc/loadavg: 4.48 2.58 2.47 11/518 30806
/proc/meminfo: memFree=14845644/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=303436 CPUtime=99.84 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 65703 0 0 0 9967 17 0 0 25 0 4 0 748443916 310718464 25249 33554432000 4194304 4625164 140734708394816 18446744073709551615 4511870 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 75859 25249 455 106 0 71362 0
[pid=30729/tid=30730] ppid=30727 vsize=303436 CPUtime=23.85 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 11757 0 0 0 2384 1 0 0 25 0 4 0 748443917 310718464 25249 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339559 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=303436 CPUtime=25.43 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 14944 0 0 0 2538 5 0 0 25 0 4 0 748443917 310718464 25249 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=303436 CPUtime=25.48 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 31360 0 0 0 2539 9 0 0 25 0 4 0 748443917 310718464 25249 33554432000 4194304 4625164 140734708394816 18446744073709551615 224284623026 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 99.84
Current children cumulated vsize (KiB) 303436

[startup+51.1006 s]
/proc/loadavg: 6.54 3.22 2.69 11/518 30806
/proc/meminfo: memFree=14706616/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=308660 CPUtime=197.2 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 101821 0 0 0 19695 25 0 0 25 0 4 0 748443916 316067840 36294 33554432000 4194304 4625164 140734708394816 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 77165 36294 562 106 0 72668 0
[pid=30729/tid=30730] ppid=30727 vsize=308660 CPUtime=49.45 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 18072 0 0 0 4942 3 0 0 25 0 4 0 748443917 316067840 36294 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=308660 CPUtime=51.02 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 17323 0 0 0 5097 5 0 0 25 0 4 0 748443917 316067840 36294 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=308660 CPUtime=46.47 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 54446 0 0 0 4633 14 0 0 25 0 4 0 748443917 316067840 36294 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 197.2
Current children cumulated vsize (KiB) 308660

[startup+102.301 s]
/proc/loadavg: 8.56 4.28 3.07 11/518 30808
/proc/meminfo: memFree=14448244/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=364504 CPUtime=401.16 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 418857 0 0 0 40041 75 0 0 25 0 4 0 748443916 373252096 58520 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 6 0 0 0
/proc/30729/statm: 91126 58520 988 106 0 86629 0
[pid=30729/tid=30730] ppid=30727 vsize=364504 CPUtime=100.68 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 26946 0 0 0 10063 5 0 0 25 0 4 0 748443917 373252096 58520 33554432000 4194304 4625164 140734708394816 18446744073709551615 4278655 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=364504 CPUtime=102.24 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 25601 0 0 0 10216 8 0 0 25 0 4 0 748443917 373252096 58521 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=364504 CPUtime=97.68 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 341124 0 0 0 9710 58 0 0 25 0 4 0 748443917 373252096 58522 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 401.16
Current children cumulated vsize (KiB) 364504

[startup+162.301 s]

################
# More data... #
################

[pid=30729] ppid=30727 vsize=808036 CPUtime=2485.62 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 2404982 0 0 0 248180 382 0 0 25 0 4 0 748443916 827428864 139533 33554432000 4194304 4625164 140734708394816 18446744073709551615 4511967 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 202009 139533 345 106 0 197512 0
[pid=30729/tid=30730] ppid=30727 vsize=808036 CPUtime=523.06 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 60853 0 0 0 52289 17 0 0 25 0 4 0 748443917 827428864 139533 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=808036 CPUtime=749.31 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 78865 0 0 0 74913 18 0 0 25 0 4 0 748443917 827428864 139533 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=808036 CPUtime=518.6 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2157300 0 0 0 51530 330 0 0 25 0 4 0 748443917 827428864 139533 33554432000 4194304 4625164 140734708394816 18446744073709551615 4552391 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2485.62
Current children cumulated vsize (KiB) 808036

[startup+822.301 s]
/proc/loadavg: 10.26 9.68 6.90 11/518 30828
/proc/meminfo: memFree=13377404/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=936200 CPUtime=2664.52 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 2690804 0 0 0 266023 429 0 0 25 0 4 0 748443916 958668800 148760 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234050 148760 348 106 0 229553 0
[pid=30729/tid=30730] ppid=30727 vsize=936200 CPUtime=564.13 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 80234 0 0 0 56394 19 0 0 25 0 4 0 748443917 958668800 148760 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=936200 CPUtime=797.69 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 79625 0 0 0 79750 19 0 0 25 0 4 0 748443917 958668800 148760 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=936200 CPUtime=564 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2405513 0 0 0 56029 371 0 0 25 0 4 0 748443917 958668800 148760 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2664.52
Current children cumulated vsize (KiB) 936200

[startup+882.32 s]
/proc/loadavg: 10.28 9.78 7.11 11/518 30830
/proc/meminfo: memFree=13316372/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=936200 CPUtime=2843.53 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 2869383 0 0 0 283894 459 0 0 25 0 4 0 748443916 958668800 153095 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512854 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234050 153095 345 106 0 229553 0
[pid=30729/tid=30730] ppid=30727 vsize=936200 CPUtime=601.05 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 80955 0 0 0 60085 20 0 0 25 0 4 0 748443917 958668800 153095 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352053 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=936200 CPUtime=852.65 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 81016 0 0 0 85244 21 0 0 25 0 4 0 748443917 958668800 153095 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352072 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=936200 CPUtime=595.42 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2580966 0 0 0 59143 399 0 0 25 0 4 0 748443917 958668800 153095 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2843.53
Current children cumulated vsize (KiB) 936200

[startup+942.301 s]
/proc/loadavg: 10.26 9.87 7.30 11/518 30830
/proc/meminfo: memFree=13241328/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=936200 CPUtime=3022.45 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3033626 0 0 0 301759 486 0 0 25 0 4 0 748443916 958668800 158703 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512029 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234050 158703 356 106 0 229553 0
[pid=30729/tid=30730] ppid=30727 vsize=936200 CPUtime=631.04 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 81725 0 0 0 63083 21 0 0 25 0 4 0 748443917 958668800 158703 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=936200 CPUtime=911.61 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 81806 0 0 0 91139 22 0 0 25 0 4 0 748443917 958668800 158703 33554432000 4194304 4625164 140734708394816 18446744073709551615 4339591 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=936200 CPUtime=625.4 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2741201 0 0 0 62117 423 0 0 25 0 4 0 748443917 958668800 158703 33554432000 4194304 4625164 140734708394816 18446744073709551615 4334485 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3022.45
Current children cumulated vsize (KiB) 936200

[startup+1002.3 s]
/proc/loadavg: 10.21 9.92 7.48 11/518 30832
/proc/meminfo: memFree=13207816/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=938248 CPUtime=3201.39 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3264835 0 0 0 319614 525 0 0 25 0 4 0 748443916 960765952 163886 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512312 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234562 163886 348 106 0 230065 0
[pid=30729/tid=30730] ppid=30727 vsize=938248 CPUtime=685.65 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 82733 0 0 0 68543 22 0 0 25 0 4 0 748443917 960765952 163886 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=938248 CPUtime=946.8 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 82246 0 0 0 94657 23 0 0 25 0 4 0 748443917 960765952 163886 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=938248 CPUtime=669.5 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 2969063 0 0 0 66491 459 0 0 25 0 4 0 748443917 960765952 163886 33554432000 4194304 4625164 140734708394816 18446744073709551615 4334728 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3201.39
Current children cumulated vsize (KiB) 938248

[startup+1062.3 s]
/proc/loadavg: 10.20 9.96 7.65 11/518 30834
/proc/meminfo: memFree=13147408/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=938248 CPUtime=3380.36 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3478696 0 0 0 337476 560 0 0 25 0 4 0 748443916 960765952 169440 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234562 169440 345 106 0 230065 0
[pid=30729/tid=30730] ppid=30727 vsize=938248 CPUtime=728.33 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 84101 0 0 0 72809 24 0 0 25 0 4 0 748443917 960765952 169440 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=938248 CPUtime=984.87 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 84172 0 0 0 98463 24 0 0 25 0 4 0 748443917 960765952 169440 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=938248 CPUtime=707.72 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 3178672 0 0 0 70281 491 0 0 25 0 4 0 748443917 960765952 169440 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3380.36
Current children cumulated vsize (KiB) 938248

[startup+1122.3 s]
/proc/loadavg: 10.19 10.00 7.80 11/518 30836
/proc/meminfo: memFree=13138316/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=938248 CPUtime=3559.3 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3639276 0 0 0 355342 588 0 0 25 0 4 0 748443916 960765952 173222 33554432000 4194304 4625164 140734708394816 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234562 173222 345 106 0 230065 0
[pid=30729/tid=30730] ppid=30727 vsize=938248 CPUtime=764.84 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 84919 0 0 0 76459 25 0 0 25 0 4 0 748443917 960765952 173222 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=938248 CPUtime=1037.82 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 85146 0 0 0 103757 25 0 0 25 0 4 0 748443917 960765952 173222 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=938248 CPUtime=737.19 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 3336493 0 0 0 73202 517 0 0 25 0 4 0 748443917 960765952 173222 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3559.3
Current children cumulated vsize (KiB) 938248

[startup+1182.3 s]
/proc/loadavg: 10.17 10.03 7.95 11/518 30845
/proc/meminfo: memFree=13106964/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=938760 CPUtime=3738.26 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3797511 0 0 0 373215 611 0 0 25 0 4 0 748443916 961290240 178866 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234690 178866 345 106 0 230193 0
[pid=30729/tid=30730] ppid=30727 vsize=938760 CPUtime=823.04 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 85969 0 0 0 82278 26 0 0 25 0 4 0 748443917 961290240 178866 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=938760 CPUtime=1069.46 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 86723 0 0 0 106920 26 0 0 25 0 4 0 748443917 961290240 178866 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=938760 CPUtime=766.32 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 3490318 0 0 0 76094 538 0 0 25 0 4 0 748443917 961290240 178866 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3738.26
Current children cumulated vsize (KiB) 938760

[startup+1242.3 s]
/proc/loadavg: 10.15 10.05 8.08 11/518 30847
/proc/meminfo: memFree=13070348/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=939016 CPUtime=3924.51 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 3998560 0 0 0 391808 643 0 0 25 0 4 0 748443916 961552384 184860 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512156 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 234754 184860 349 106 0 230257 0
[pid=30729/tid=30730] ppid=30727 vsize=939016 CPUtime=883.03 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 88417 0 0 0 88276 27 0 0 25 0 4 0 748443917 961552384 184860 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=939016 CPUtime=1099.05 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 87145 0 0 0 109878 27 0 0 25 0 4 0 748443917 961552384 184860 33554432000 4194304 4625164 140734708394816 18446744073709551615 4352032 0 0 4096 2 0 0 0 -1 2 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=939016 CPUtime=802.98 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 3686677 0 0 0 79732 566 0 0 25 0 4 0 748443917 961552384 184860 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3924.51
Current children cumulated vsize (KiB) 939016



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 10.14 10.07 8.21 11/518 30849
/proc/meminfo: memFree=13070064/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=985528 CPUtime=4112.18 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 4285791 0 0 0 410532 686 0 0 25 0 4 0 748443916 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512312 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 246382 191947 346 106 0 241885 0
[pid=30729/tid=30730] ppid=30727 vsize=985528 CPUtime=929.63 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 90377 0 0 0 92935 28 0 0 25 0 4 0 748443917 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=985528 CPUtime=1133.1 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 87713 0 0 0 113283 27 0 0 25 0 4 0 748443917 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351814 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=985528 CPUtime=852.27 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 3945676 0 0 0 84623 604 0 0 25 0 4 0 748443917 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4112.18
Current children cumulated vsize (KiB) 985528

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1300 s]
/proc/loadavg: 10.14 10.07 8.21 11/518 30849
/proc/meminfo: memFree=13070064/32951124 swapFree=45193380/67111528
[pid=30729] ppid=30727 vsize=985528 CPUtime=4112.18 cores=0,2,4,6
/proc/30729/stat : 30729 (SArTagnan) R 30727 30729 30701 0 -1 4202496 4285791 0 0 0 410532 686 0 0 25 0 4 0 748443916 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4512312 0 0 4096 2 18446744073709551615 0 0 17 0 0 0 0
/proc/30729/statm: 246382 191947 346 106 0 241885 0
[pid=30729/tid=30730] ppid=30727 vsize=985528 CPUtime=929.63 cores=0,2,4,6
/proc/30729/task/30730/stat : 30730 (SArTagnan) R 30727 30729 30701 0 -1 4202560 90377 0 0 0 92935 28 0 0 25 0 4 0 748443917 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=30729/tid=30731] ppid=30727 vsize=985528 CPUtime=1133.1 cores=0,2,4,6
/proc/30729/task/30731/stat : 30731 (SArTagnan) R 30727 30729 30701 0 -1 4202560 87713 0 0 0 113283 27 0 0 25 0 4 0 748443917 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351814 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=30729/tid=30732] ppid=30727 vsize=985528 CPUtime=852.27 cores=0,2,4,6
/proc/30729/task/30732/stat : 30732 (SArTagnan) R 30727 30729 30701 0 -1 4202560 3945676 0 0 0 84623 604 0 0 25 0 4 0 748443917 1009180672 191947 33554432000 4194304 4625164 140734708394816 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4112.18
Current children cumulated vsize (KiB) 985528

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1300.06
CPU time (s): 4112.23
CPU user time (s): 4105.34
CPU system time (s): 6.89695
CPU usage (%): 316.312
Max. virtual memory (cumulated for all children) (KiB): 1078172

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4105.34
system time used= 6.89695
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4285791
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 607
involuntary context switches= 29247

runsolver used 6.31604 second user time and 16.1175 second system time

The end

Launcher Data 

Begin job on node118 at 2011-05-16 23:42:12
IDJOB=3386279
IDBENCH=83420
IDSOLVER=1871
FILE ID=node118/3386279-1305582132
RUNJOBID= node118-1305582131-30719
PBS_JOBID= 13350301
Free space on /tmp= 73144 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/64/aes_64_1_keyfind_1.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386279-1305582132/watcher-3386279-1305582132 -o /tmp/evaluation-result-3386279-1305582132/solver-3386279-1305582132 -C 4800 -W 1300 -M 15500  SArTagnan HOME/instance-3386279-1305582132.cnf -threads=4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9e18e0ee8a56ea2c59fe8e682ff6c747
RANDOM SEED=1854898254

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16002520 kB
Buffers:        320564 kB
Cached:        4938952 kB
SwapCached:      22068 kB
Active:       13872624 kB
Inactive:      2737548 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16002520 kB
SwapTotal:    67111528 kB
SwapFree:     45193380 kB
Dirty:             424 kB
Writeback:           0 kB
AnonPages:    11341244 kB
Mapped:          26696 kB
Slab:           159928 kB
PageTables:     109108 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 162641032 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73044 MiB
End job on node118 at 2011-05-17 00:03:55