Trace number 3386125

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 4141.16 1300.07

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n160-s73223945-unsat.cnf
MD5SUMebbfc0ffb6aa4fcba0d5ce3964018543
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables161
Number of clauses332
Sum of the clauses size996
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3332
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386125-1305580403/watcher-3386125-1305580403 -o /tmp/evaluation-result-3386125-1305580403/solver-3386125-1305580403 -C 4800 -W 1300 -M 15500 SArTagnan HOME/instance-3386125-1305580403.cnf -threads=4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.07 10.14 10.09 8/515 10435
/proc/meminfo: memFree=13585544/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=49212 CPUtime=0 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 450 0 0 0 0 0 0 0 25 0 4 0 669638068 50393088 388 33554432000 4194304 4625164 140736011414336 18446744073709551615 47111345674160 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/10435/statm: 12303 388 300 106 0 7806 0
[pid=10435/tid=10436] ppid=10433 vsize=49212 CPUtime=0.01 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1 0 0 0 1 0 0 0 25 0 4 0 669638068 50393088 388 33554432000 4194304 4625164 140736011414336 18446744073709551615 47111345674162 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=49212 CPUtime=0 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 669638068 50393088 388 33554432000 4194304 4625164 140736011414336 18446744073709551615 47111345674162 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=49212 CPUtime=0 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 4 0 669638068 50393088 388 33554432000 4194304 4625164 140736011414336 18446744073709551615 212687731697 0 0 4096 2 0 0 0 -1 1 0 0 0

[startup+0.169859 s]
/proc/loadavg: 10.07 10.14 10.09 8/515 10435
/proc/meminfo: memFree=13585544/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=182320 CPUtime=0.47 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 2051 0 0 0 47 0 0 0 25 0 4 0 669638068 186695680 1543 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511970 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/10435/statm: 45580 1543 343 106 0 41083 0
[pid=10435/tid=10436] ppid=10433 vsize=182320 CPUtime=0.16 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 693 0 0 0 16 0 0 0 18 0 4 0 669638068 186695680 1544 33554432000 4194304 4625164 140736011414336 18446744073709551615 4352047 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=182320 CPUtime=0 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 4 0 669638068 186695680 1544 33554432000 4194304 4625164 140736011414336 18446744073709551615 4335165 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=182320 CPUtime=0.14 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 485 0 0 0 14 0 0 0 25 0 4 0 669638068 186695680 1544 33554432000 4194304 4625164 140736011414336 18446744073709551615 212687333638 0 0 4096 2 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 0.47
Current children cumulated vsize (KiB) 182320

[startup+0.200854 s]
/proc/loadavg: 10.07 10.14 10.09 8/515 10435
/proc/meminfo: memFree=13585544/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=182320 CPUtime=0.56 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 2235 0 0 0 56 0 0 0 25 0 4 0 669638068 186695680 1727 33554432000 4194304 4625164 140736011414336 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/10435/statm: 45580 1727 343 106 0 41083 0
[pid=10435/tid=10436] ppid=10433 vsize=182320 CPUtime=0.19 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 739 0 0 0 19 0 0 0 18 0 4 0 669638068 186695680 1727 33554432000 4194304 4625164 140736011414336 18446744073709551615 4352760 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=182320 CPUtime=0 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 4 0 669638068 186695680 1727 33554432000 4194304 4625164 140736011414336 18446744073709551615 4335165 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=182320 CPUtime=0.17 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 600 0 0 0 17 0 0 0 25 0 4 0 669638068 186695680 1727 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 0.56
Current children cumulated vsize (KiB) 182320

[startup+0.300836 s]
/proc/loadavg: 10.07 10.14 10.09 8/515 10435
/proc/meminfo: memFree=13585544/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=182860 CPUtime=0.86 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 2726 0 0 0 86 0 0 0 25 0 4 0 669638068 187248640 2104 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511811 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/10435/statm: 45715 2104 343 106 0 41218 0
[pid=10435/tid=10436] ppid=10433 vsize=182860 CPUtime=0.29 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 894 0 0 0 29 0 0 0 18 0 4 0 669638068 187248640 2104 33554432000 4194304 4625164 140736011414336 18446744073709551615 4352855 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=182860 CPUtime=0 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 4 0 669638068 187248640 2104 33554432000 4194304 4625164 140736011414336 18446744073709551615 4334488 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=182860 CPUtime=0.27 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 716 0 0 0 27 0 0 0 25 0 4 0 669638068 187248640 2104 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 0.86
Current children cumulated vsize (KiB) 182860

[startup+0.700765 s]
/proc/loadavg: 10.07 10.14 10.09 8/515 10435
/proc/meminfo: memFree=13585544/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=184848 CPUtime=2.07 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 5307 0 0 0 206 1 0 0 25 0 4 0 669638068 189284352 3572 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511870 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/10435/statm: 46212 3573 343 106 0 41715 0
[pid=10435/tid=10436] ppid=10433 vsize=184848 CPUtime=0.69 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1974 0 0 0 69 0 0 0 18 0 4 0 669638068 189284352 3573 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=184848 CPUtime=0 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 4 0 669638068 189284352 3573 33554432000 4194304 4625164 140736011414336 18446744073709551615 4334488 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=184848 CPUtime=0.67 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 1794 0 0 0 67 0 0 0 25 0 4 0 669638068 189284352 3573 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552425 0 0 4096 2 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2.07
Current children cumulated vsize (KiB) 184848

[startup+1.50062 s]
/proc/loadavg: 10.07 10.14 10.09 11/519 10439
/proc/meminfo: memFree=13549452/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=188016 CPUtime=4.44 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 9600 0 0 0 443 1 0 0 25 0 4 0 669638068 192528384 5919 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 1 0 0 0
/proc/10435/statm: 47004 5919 345 106 0 42507 0
[pid=10435/tid=10436] ppid=10433 vsize=188016 CPUtime=1.19 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 2896 0 0 0 119 0 0 0 19 0 4 0 669638068 192528384 5919 33554432000 4194304 4625164 140736011414336 18446744073709551615 4464328 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=188016 CPUtime=0.29 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 759 0 0 0 29 0 0 0 25 0 4 0 669638068 192528384 5919 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351798 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=188016 CPUtime=1.45 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 3657 0 0 0 145 0 0 0 25 0 4 0 669638068 192528384 5919 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552452 0 0 4096 2 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4.44
Current children cumulated vsize (KiB) 188016

[startup+3.10033 s]
/proc/loadavg: 10.06 10.14 10.09 11/519 10439
/proc/meminfo: memFree=13525684/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=197128 CPUtime=9.22 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 14984 0 0 0 920 2 0 0 25 0 4 0 669638068 201859072 9654 33554432000 4194304 4625164 140736011414336 18446744073709551615 4392981 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 49282 9654 347 106 0 44785 0
[pid=10435/tid=10436] ppid=10433 vsize=197128 CPUtime=2.1 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 4240 0 0 0 210 0 0 0 21 0 4 0 669638068 201859072 9654 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=197128 CPUtime=1.66 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 2882 0 0 0 166 0 0 0 25 0 4 0 669638068 201859072 9654 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351798 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=197128 CPUtime=2.99 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 4728 0 0 0 298 1 0 0 25 0 4 0 669638068 201859072 9654 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 9.22
Current children cumulated vsize (KiB) 197128

[startup+6.30075 s]
/proc/loadavg: 10.06 10.14 10.09 11/519 10439
/proc/meminfo: memFree=13485340/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=271456 CPUtime=18.75 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 21904 0 0 0 1870 5 0 0 25 0 4 0 669638068 277970944 14695 33554432000 4194304 4625164 140736011414336 18446744073709551615 4338085 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 67864 14695 392 106 0 63367 0
[pid=10435/tid=10436] ppid=10433 vsize=271456 CPUtime=3.8 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 6330 0 0 0 379 1 0 0 25 0 4 0 669638068 277970944 14695 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351731 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=271456 CPUtime=4.27 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 5497 0 0 0 426 1 0 0 18 0 4 0 669638068 277970944 14695 33554432000 4194304 4625164 140736011414336 18446744073709551615 4339643 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=271456 CPUtime=6.19 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 5855 0 0 0 618 1 0 0 18 0 4 0 669638068 277970944 14695 33554432000 4194304 4625164 140736011414336 18446744073709551615 4547603 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 18.75
Current children cumulated vsize (KiB) 271456

[startup+12.7006 s]
/proc/loadavg: 10.21 10.16 10.10 11/519 10439
/proc/meminfo: memFree=13412084/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=289040 CPUtime=39.24 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 32276 0 0 0 3917 7 0 0 25 0 4 0 669638068 295976960 22041 33554432000 4194304 4625164 140736011414336 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 72260 22041 437 106 0 67763 0
[pid=10435/tid=10436] ppid=10433 vsize=289040 CPUtime=8.27 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 9926 0 0 0 825 2 0 0 25 0 4 0 669638068 295976960 22041 33554432000 4194304 4625164 140736011414336 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=289040 CPUtime=7.49 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 7949 0 0 0 748 1 0 0 25 0 4 0 669638068 295976960 22041 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=289040 CPUtime=12.58 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 7241 0 0 0 1257 1 0 0 25 0 4 0 669638068 295976960 22041 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39.24
Current children cumulated vsize (KiB) 289040

[startup+25.5013 s]
/proc/loadavg: 10.18 10.16 10.10 11/519 10440
/proc/meminfo: memFree=13337328/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=306368 CPUtime=78.14 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 91858 0 0 0 7794 20 0 0 25 0 4 0 669638068 313720832 31014 33554432000 4194304 4625164 140736011414336 18446744073709551615 4329151 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 76592 31014 349 106 0 72095 0
[pid=10435/tid=10436] ppid=10433 vsize=306368 CPUtime=19.85 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 16329 0 0 0 1979 6 0 0 25 0 4 0 669638068 313720832 31014 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=306368 CPUtime=14.55 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 12728 0 0 0 1451 4 0 0 25 0 4 0 669638068 313720832 31014 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=306368 CPUtime=25.38 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 52057 0 0 0 2530 8 0 0 25 0 4 0 669638068 313720832 31015 33554432000 4194304 4625164 140736011414336 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 78.14
Current children cumulated vsize (KiB) 306368

[startup+51.1006 s]
/proc/loadavg: 10.12 10.14 10.09 11/519 10440
/proc/meminfo: memFree=13226192/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=337512 CPUtime=154.5 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 172599 0 0 0 15412 38 0 0 25 0 4 0 669638068 345612288 46293 33554432000 4194304 4625164 140736011414336 18446744073709551615 4512156 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 84378 46293 563 106 0 79881 0
[pid=10435/tid=10436] ppid=10433 vsize=337512 CPUtime=45.01 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 22520 0 0 0 4494 7 0 0 25 0 4 0 669638068 345612288 46293 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=337512 CPUtime=28.45 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 25747 0 0 0 2836 9 0 0 25 0 4 0 669638068 345612288 46293 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351981 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=337512 CPUtime=38.43 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 108590 0 0 0 3825 18 0 0 25 0 4 0 669638068 345612288 46293 33554432000 4194304 4625164 140736011414336 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 154.5
Current children cumulated vsize (KiB) 337512

[startup+102.301 s]
/proc/loadavg: 10.14 10.15 10.09 11/519 10442
/proc/meminfo: memFree=13067584/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=401296 CPUtime=307.24 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 320456 0 0 0 30662 62 0 0 25 0 4 0 669638068 410927104 67207 33554432000 4194304 4625164 140736011414336 18446744073709551615 4512818 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 100324 67207 727 106 0 95827 0
[pid=10435/tid=10436] ppid=10433 vsize=401296 CPUtime=95.34 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 43292 0 0 0 9523 11 0 0 25 0 4 0 669638068 410927104 67209 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=401296 CPUtime=54.17 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 28655 0 0 0 5407 10 0 0 25 0 4 0 669638068 410927104 67209 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=401296 CPUtime=63.93 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 225912 0 0 0 6357 36 0 0 25 0 4 0 669638068 410927104 67209 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 307.24
Current children cumulated vsize (KiB) 401296

[startup+162.3 s]

################
# More data... #
################

[pid=10435] ppid=10433 vsize=967336 CPUtime=2507.95 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 842218 0 0 0 250563 232 0 0 25 0 4 0 669638068 990552064 192783 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/10435/statm: 241834 192783 694 106 0 237337 0
[pid=10435/tid=10436] ppid=10433 vsize=967336 CPUtime=727.72 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 120928 0 0 0 72711 61 0 0 25 0 4 0 669638068 990552064 192783 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=967336 CPUtime=599 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 106752 0 0 0 59835 65 0 0 25 0 4 0 669638068 990552064 192783 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=967336 CPUtime=440.1 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 529753 0 0 0 43922 88 0 0 25 0 4 0 669638068 990552064 192783 33554432000 4194304 4625164 140736011414336 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2507.95
Current children cumulated vsize (KiB) 967336

[startup+822.301 s]
/proc/loadavg: 10.07 10.11 10.09 11/519 10461
/proc/meminfo: memFree=12065916/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=967336 CPUtime=2716.32 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 847643 0 0 0 271393 239 0 0 25 0 4 0 669638068 990552064 198083 33554432000 4194304 4625164 140736011414336 18446744073709551615 4519319 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/10435/statm: 241834 198083 683 106 0 237337 0
[pid=10435/tid=10436] ppid=10433 vsize=967336 CPUtime=787.17 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 123655 0 0 0 78653 64 0 0 25 0 4 0 669638068 990552064 198083 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=967336 CPUtime=628.94 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 108095 0 0 0 62827 67 0 0 25 0 4 0 669638068 990552064 198083 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=967336 CPUtime=500.08 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 529941 0 0 0 49920 88 0 0 25 0 4 0 669638068 990552064 198083 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552315 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2716.32
Current children cumulated vsize (KiB) 967336

[startup+882.32 s]
/proc/loadavg: 10.02 10.08 10.08 11/519 10463
/proc/meminfo: memFree=12025476/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=967336 CPUtime=2895.33 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 852752 0 0 0 289288 245 0 0 25 0 4 0 669638068 990552064 203061 33554432000 4194304 4625164 140736011414336 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 241834 203061 673 106 0 237337 0
[pid=10435/tid=10436] ppid=10433 vsize=967336 CPUtime=821.95 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 125466 0 0 0 82128 67 0 0 25 0 4 0 669638068 990552064 203061 33554432000 4194304 4625164 140736011414336 18446744073709551615 4352728 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=967336 CPUtime=658.99 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 109842 0 0 0 65829 70 0 0 25 0 4 0 669638068 990552064 203061 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=967336 CPUtime=560.1 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 530435 0 0 0 55921 89 0 0 25 0 4 0 669638068 990552064 203061 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2895.33
Current children cumulated vsize (KiB) 967336

[startup+942.3 s]
/proc/loadavg: 10.13 10.10 10.09 11/519 10464
/proc/meminfo: memFree=11967552/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1036864 CPUtime=3074.23 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 877905 0 0 0 307171 252 0 0 25 0 4 0 669638068 1061748736 214386 33554432000 4194304 4625164 140736011414336 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/10435/statm: 259216 214386 661 106 0 254719 0
[pid=10435/tid=10436] ppid=10433 vsize=1036864 CPUtime=852.16 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 131983 0 0 0 85147 69 0 0 25 0 4 0 669638068 1061748736 214386 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1036864 CPUtime=701.78 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 112086 0 0 0 70106 72 0 0 25 0 4 0 669638068 1061748736 214386 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1036864 CPUtime=620.08 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 530527 0 0 0 61919 89 0 0 25 0 4 0 669638068 1061748736 214386 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3074.23
Current children cumulated vsize (KiB) 1036864

[startup+1002.3 s]
/proc/loadavg: 10.15 10.11 10.09 11/519 10466
/proc/meminfo: memFree=11941508/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1102400 CPUtime=3253.19 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 917592 0 0 0 325055 264 0 0 25 0 4 0 669638068 1128857600 217009 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/10435/statm: 275600 217009 606 106 0 271103 0
[pid=10435/tid=10436] ppid=10433 vsize=1102400 CPUtime=885.4 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 132372 0 0 0 88471 69 0 0 25 0 4 0 669638068 1128857600 217009 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1102400 CPUtime=731.78 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 113210 0 0 0 73104 74 0 0 25 0 4 0 669638068 1128857600 217009 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1102400 CPUtime=680.08 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 530678 0 0 0 67919 89 0 0 25 0 4 0 669638068 1128857600 217009 33554432000 4194304 4625164 140736011414336 18446744073709551615 4552369 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3253.19
Current children cumulated vsize (KiB) 1102400

[startup+1062.3 s]
/proc/loadavg: 10.30 10.15 10.10 11/519 10467
/proc/meminfo: memFree=11915088/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1102400 CPUtime=3432.17 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 992394 0 0 0 342935 282 0 0 25 0 4 0 669638068 1128857600 219894 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/10435/statm: 275600 219894 596 106 0 271103 0
[pid=10435/tid=10436] ppid=10433 vsize=1102400 CPUtime=945.4 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 132905 0 0 0 94471 69 0 0 25 0 4 0 669638068 1128857600 219894 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1102400 CPUtime=767.88 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 114537 0 0 0 76711 77 0 0 25 0 4 0 669638068 1128857600 219894 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1102400 CPUtime=733.36 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 531058 0 0 0 73247 89 0 0 25 0 4 0 669638068 1128857600 219894 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351981 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3432.17
Current children cumulated vsize (KiB) 1102400

[startup+1122.3 s]
/proc/loadavg: 10.29 10.18 10.11 11/519 10469
/proc/meminfo: memFree=11847124/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1223128 CPUtime=3611.13 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 1185263 0 0 0 360792 321 0 0 25 0 4 0 669638068 1252483072 233313 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/10435/statm: 305782 233313 627 106 0 301285 0
[pid=10435/tid=10436] ppid=10433 vsize=1223128 CPUtime=994.11 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 166299 0 0 0 99337 74 0 0 25 0 4 0 669638068 1252483072 233313 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1223128 CPUtime=797.64 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 115836 0 0 0 79685 79 0 0 25 0 4 0 669638068 1252483072 233313 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351795 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1223128 CPUtime=793.37 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 537872 0 0 0 79247 90 0 0 25 0 4 0 669638068 1252483072 233313 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3611.13
Current children cumulated vsize (KiB) 1223128

[startup+1182.3 s]
/proc/loadavg: 10.21 10.18 10.11 11/519 10471
/proc/meminfo: memFree=11803716/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1223128 CPUtime=3790.01 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 1296984 0 0 0 378657 344 0 0 25 0 4 0 669638068 1252483072 237458 33554432000 4194304 4625164 140736011414336 18446744073709551615 4392981 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 305782 237458 614 106 0 301285 0
[pid=10435/tid=10436] ppid=10433 vsize=1223128 CPUtime=1034.53 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 167663 0 0 0 103377 76 0 0 25 0 4 0 669638068 1252483072 237458 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1223128 CPUtime=846.72 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 117630 0 0 0 84589 83 0 0 25 0 4 0 669638068 1252483072 237458 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1223128 CPUtime=853.37 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 538382 0 0 0 85246 91 0 0 25 0 4 0 669638068 1252483072 237458 33554432000 4194304 4625164 140736011414336 18446744073709551615 4548757 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3790.01
Current children cumulated vsize (KiB) 1223128

[startup+1242.3 s]
/proc/loadavg: 10.12 10.15 10.10 11/519 10472
/proc/meminfo: memFree=11738104/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1318476 CPUtime=3968.94 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 1441587 0 0 0 396517 377 0 0 25 0 4 0 669638068 1350119424 250051 33554432000 4194304 4625164 140736011414336 18446744073709551615 4392981 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 329619 250051 605 106 0 325122 0
[pid=10435/tid=10436] ppid=10433 vsize=1318476 CPUtime=1067.46 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 169035 0 0 0 106667 79 0 0 25 0 4 0 669638068 1350119424 250051 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1318476 CPUtime=903.3 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 151796 0 0 0 90240 90 0 0 25 0 4 0 669638068 1350119424 250051 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351777 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1318476 CPUtime=913.37 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 538464 0 0 0 91246 91 0 0 25 0 4 0 669638068 1350119424 250051 33554432000 4194304 4625164 140736011414336 18446744073709551615 4547291 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3968.94
Current children cumulated vsize (KiB) 1318476



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 10.05 10.13 10.09 7/515 10475
/proc/meminfo: memFree=12565832/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1318476 CPUtime=4141.07 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 1553382 0 0 0 413708 399 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 329619 254254 513 106 0 325122 0
[pid=10435/tid=10436] ppid=10433 vsize=1318476 CPUtime=1096.33 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 170105 0 0 0 109552 81 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1318476 CPUtime=960.02 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 154045 0 0 0 95911 91 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1318476 CPUtime=971.09 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 538600 0 0 0 97018 91 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4141.07
Current children cumulated vsize (KiB) 1318476

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1300 s]
/proc/loadavg: 10.05 10.13 10.09 7/515 10475
/proc/meminfo: memFree=12565832/32951124 swapFree=67111348/67111528
[pid=10435] ppid=10433 vsize=1318476 CPUtime=4141.07 cores=1,3,5,7
/proc/10435/stat : 10435 (SArTagnan) R 10433 10435 10092 0 -1 4202496 1553382 0 0 0 413708 399 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/10435/statm: 329619 254254 513 106 0 325122 0
[pid=10435/tid=10436] ppid=10433 vsize=1318476 CPUtime=1096.33 cores=1,3,5,7
/proc/10435/task/10436/stat : 10436 (SArTagnan) R 10433 10435 10092 0 -1 4202560 170105 0 0 0 109552 81 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=10435/tid=10437] ppid=10433 vsize=1318476 CPUtime=960.02 cores=1,3,5,7
/proc/10435/task/10437/stat : 10437 (SArTagnan) R 10433 10435 10092 0 -1 4202560 154045 0 0 0 95911 91 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=10435/tid=10438] ppid=10433 vsize=1318476 CPUtime=971.09 cores=1,3,5,7
/proc/10435/task/10438/stat : 10438 (SArTagnan) R 10433 10435 10092 0 -1 4202560 538600 0 0 0 97018 91 0 0 25 0 4 0 669638068 1350119424 254254 33554432000 4194304 4625164 140736011414336 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4141.07
Current children cumulated vsize (KiB) 1318476

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1300.07
CPU time (s): 4141.16
CPU user time (s): 4137.12
CPU system time (s): 4.03739
CPU usage (%): 318.534
Max. virtual memory (cumulated for all children) (KiB): 1363704

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4137.12
system time used= 4.03739
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1553382
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1054
involuntary context switches= 42591

runsolver used 6.21005 second user time and 16.2645 second system time

The end

Launcher Data 

Begin job on node139 at 2011-05-16 23:13:23
IDJOB=3386125
IDBENCH=83144
IDSOLVER=1871
FILE ID=node139/3386125-1305580403
RUNJOBID= node139-1305576495-10111
PBS_JOBID= 13350204
Free space on /tmp= 73276 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n160-s73223945-unsat.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386125-1305580403/watcher-3386125-1305580403 -o /tmp/evaluation-result-3386125-1305580403/solver-3386125-1305580403 -C 4800 -W 1300 -M 15500  SArTagnan HOME/instance-3386125-1305580403.cnf -threads=4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ebbfc0ffb6aa4fcba0d5ce3964018543
RANDOM SEED=1735063726

node139.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5321.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      13585576 kB
Buffers:        342248 kB
Cached:        3338084 kB
SwapCached:          0 kB
Active:       16824516 kB
Inactive:      2193180 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      13585576 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:             804 kB
Writeback:           0 kB
AnonPages:    15337516 kB
Mapped:          39148 kB
Slab:           208940 kB
PageTables:      70140 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 140761120 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73276 MiB
End job on node139 at 2011-05-16 23:35:05