Trace number 3439361

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SArTagnan 2011-05-15 (fixed)? (TO) 40000.7 5011.42

General information on the benchmark

NameSAT03/handmade/stanion/hwb/
hwb-n28-01-S136611085.shuffled-as.sat03-1627.cnf
MD5SUM17624c574c7f9ee0eff4cc4a13e42e12
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark668.579
Satisfiable
(Un)Satisfiability was proved
Number of variables184
Number of clauses896
Sum of the clauses size3042
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 214
Number of clauses of size 3514
Number of clauses of size 4368
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3439361-1306961204/watcher-3439361-1306961204 -o /tmp/evaluation-result-3439361-1306961204/solver-3439361-1306961204 -C 40000 -W 5100 -M 31000 SArTagnan HOME/instance-3439361-1306961204.cnf -threads=8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 7.49 4.63 2/514 11231
/proc/meminfo: memFree=21941552/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=287204 CPUtime=0.02 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 680 0 0 0 2 0 0 0 24 0 8 0 788535913 294096896 613 33554432000 4194304 4625164 140737334671088 18446744073709551615 228095556078 0 0 4096 2 18446744073709551615 0 0 17 2 0 0 0
/proc/11231/statm: 71801 613 348 106 0 67304 0
[pid=11231/tid=11232] ppid=11229 vsize=287368 CPUtime=0.01 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 38 0 0 0 1 0 0 0 25 0 8 0 788535913 294264832 738 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=287368 CPUtime=0 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 30 0 0 0 0 0 0 0 18 0 8 0 788535913 294264832 738 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=287368 CPUtime=0 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 8 0 788535913 294264832 738 33554432000 4194304 4625164 140737334671088 18446744073709551615 4559864 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=287368 CPUtime=0.01 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 30 0 0 0 1 0 0 0 25 0 8 0 788535913 294264832 739 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=287368 CPUtime=0 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 56 0 0 0 0 0 0 0 17 0 8 0 788535913 294264832 739 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348941 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=287368 CPUtime=0 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 2 0 0 0 0 0 0 0 25 0 8 0 788535913 294264832 740 33554432000 4194304 4625164 140737334671088 18446744073709551615 228095556127 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=287368 CPUtime=0 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 30 0 0 0 0 0 0 0 24 0 8 0 788535913 294264832 740 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339447 0 0 4096 2 0 0 0 -1 2 0 0 0

[startup+0.132202 s]
/proc/loadavg: 8.04 7.49 4.63 2/514 11231
/proc/meminfo: memFree=21941552/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=366864 CPUtime=6.42 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 13216 0 0 0 641 1 0 0 19 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4511896 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 91716 7559 361 106 0 87219 0
[pid=11231/tid=11232] ppid=11229 vsize=366864 CPUtime=0.8 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1339 0 0 0 80 0 0 0 19 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=366864 CPUtime=0.8 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1315 0 0 0 80 0 0 0 18 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=366864 CPUtime=0.8 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 3833 0 0 0 80 0 0 0 18 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4353037 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=366864 CPUtime=0.8 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1328 0 0 0 80 0 0 0 18 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=366864 CPUtime=0.8 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1195 0 0 0 80 0 0 0 19 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4437699 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=366864 CPUtime=0.78 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1729 0 0 0 78 0 0 0 25 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 228111434779 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=366864 CPUtime=0.8 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1201 0 0 0 80 0 0 0 25 0 8 0 788535913 375668736 7559 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351798 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 6.42
Current children cumulated vsize (KiB) 366864

[startup+0.915559 s]
/proc/loadavg: 8.04 7.49 4.63 2/514 11231
/proc/meminfo: memFree=21941552/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=367384 CPUtime=7.22 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 13931 0 0 0 721 1 0 0 19 0 8 0 788535913 376201216 8243 33554432000 4194304 4625164 140737334671088 18446744073709551615 4511866 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 91846 8243 361 106 0 87349 0
[pid=11231/tid=11232] ppid=11229 vsize=367384 CPUtime=0.9 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1448 0 0 0 90 0 0 0 19 0 8 0 788535913 376201216 8243 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351818 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=367384 CPUtime=0.9 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1336 0 0 0 90 0 0 0 19 0 8 0 788535913 376201216 8243 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351734 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=367384 CPUtime=0.9 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4123 0 0 0 90 0 0 0 18 0 8 0 788535913 376201216 8243 33554432000 4194304 4625164 140737334671088 18446744073709551615 4552399 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=367384 CPUtime=0.9 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1342 0 0 0 90 0 0 0 19 0 8 0 788535913 376201216 8244 33554432000 4194304 4625164 140737334671088 18446744073709551615 4340391 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=367384 CPUtime=0.9 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1285 0 0 0 90 0 0 0 19 0 8 0 788535913 376201216 8244 33554432000 4194304 4625164 140737334671088 18446744073709551615 4349019 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=367384 CPUtime=0.88 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1792 0 0 0 88 0 0 0 25 0 8 0 788535913 376201216 8244 33554432000 4194304 4625164 140737334671088 18446744073709551615 4288653 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=367384 CPUtime=0.9 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1316 0 0 0 90 0 0 0 25 0 8 0 788535913 376201216 8244 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352072 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 7.22
Current children cumulated vsize (KiB) 367384

[startup+1.01554 s]
/proc/loadavg: 8.04 7.49 4.63 2/514 11231
/proc/meminfo: memFree=21941552/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=368440 CPUtime=8.02 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 15214 0 0 0 800 2 0 0 19 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4519388 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 92110 9051 352 106 0 87613 0
[pid=11231/tid=11232] ppid=11229 vsize=368440 CPUtime=1 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1916 0 0 0 100 0 0 0 19 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351818 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=368440 CPUtime=1 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1429 0 0 0 100 0 0 0 19 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352208 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=368440 CPUtime=1 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4618 0 0 0 100 0 0 0 19 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=368440 CPUtime=1 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1360 0 0 0 100 0 0 0 19 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352280 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=368440 CPUtime=1 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1309 0 0 0 100 0 0 0 19 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348870 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=368440 CPUtime=0.98 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1831 0 0 0 98 0 0 0 25 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=368440 CPUtime=1 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1324 0 0 0 100 0 0 0 25 0 8 0 788535913 377282560 9051 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351818 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.02
Current children cumulated vsize (KiB) 368440

[startup+1.10321 s]
/proc/loadavg: 8.04 7.49 4.63 2/514 11231
/proc/meminfo: memFree=21941552/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=368760 CPUtime=8.82 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 15592 0 0 0 880 2 0 0 19 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4512818 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 92190 9420 352 106 0 87693 0
[pid=11231/tid=11232] ppid=11229 vsize=368760 CPUtime=1.1 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1956 0 0 0 110 0 0 0 19 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=368760 CPUtime=1.1 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1437 0 0 0 110 0 0 0 19 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=368760 CPUtime=1.1 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4783 0 0 0 110 0 0 0 19 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=368760 CPUtime=1.1 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1402 0 0 0 110 0 0 0 19 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4353037 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=368760 CPUtime=1.1 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1322 0 0 0 110 0 0 0 19 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4349320 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=368760 CPUtime=1.08 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1865 0 0 0 108 0 0 0 25 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=368760 CPUtime=1.1 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1383 0 0 0 110 0 0 0 25 0 8 0 788535913 377610240 9420 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352105 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.82
Current children cumulated vsize (KiB) 368760

[startup+1.50046 s]
/proc/loadavg: 8.04 7.49 4.63 2/514 11231
/proc/meminfo: memFree=21941552/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=374376 CPUtime=11.9 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 20216 0 0 0 1188 2 0 0 20 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 93594 10983 352 106 0 89097 0
[pid=11231/tid=11232] ppid=11229 vsize=374376 CPUtime=1.49 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 2044 0 0 0 149 0 0 0 20 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=374376 CPUtime=1.48 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 2030 0 0 0 148 0 0 0 20 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=374376 CPUtime=1.48 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 6410 0 0 0 148 0 0 0 19 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339612 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=374376 CPUtime=1.48 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1919 0 0 0 148 0 0 0 19 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351839 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=374376 CPUtime=1.48 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1857 0 0 0 148 0 0 0 19 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348858 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=374376 CPUtime=1.46 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 2574 0 0 0 146 0 0 0 25 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=374376 CPUtime=1.48 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 1884 0 0 0 148 0 0 0 25 0 8 0 788535913 383361024 10983 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352021 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.9
Current children cumulated vsize (KiB) 374376

[startup+3.10116 s]
/proc/loadavg: 8.04 7.50 4.65 9/522 11239
/proc/meminfo: memFree=21876024/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=388448 CPUtime=24.67 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 35267 0 0 0 2462 5 0 0 25 0 8 0 788535913 397770752 17335 33554432000 4194304 4625164 140737334671088 18446744073709551615 4512057 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 97112 17335 352 106 0 92615 0
[pid=11231/tid=11232] ppid=11229 vsize=388448 CPUtime=3.09 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 3382 0 0 0 309 0 0 0 25 0 8 0 788535913 397770752 17335 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351818 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=388448 CPUtime=3.08 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 3027 0 0 0 308 0 0 0 25 0 8 0 788535913 397770752 17335 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=388448 CPUtime=3.08 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 13542 0 0 0 307 1 0 0 25 0 8 0 788535913 397770752 17335 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=388448 CPUtime=3.08 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 3131 0 0 0 308 0 0 0 25 0 8 0 788535913 397770752 17335 33554432000 4194304 4625164 140737334671088 18446744073709551615 4464855 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=388448 CPUtime=3.08 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 2879 0 0 0 308 0 0 0 25 0 8 0 788535913 397770752 17335 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348817 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=388448 CPUtime=3.06 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4102 0 0 0 306 0 0 0 25 0 8 0 788535913 397770752 17336 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=388448 CPUtime=3.08 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 3149 0 0 0 308 0 0 0 25 0 8 0 788535913 397770752 17336 33554432000 4194304 4625164 140737334671088 18446744073709551615 4464266 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.67
Current children cumulated vsize (KiB) 388448

[startup+6.31957 s]
/proc/loadavg: 8.04 7.50 4.65 9/522 11240
/proc/meminfo: memFree=21836220/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=401796 CPUtime=50.36 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 60245 0 0 0 5026 10 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 7 0 0 0
/proc/11231/statm: 100449 27015 352 106 0 95952 0
[pid=11231/tid=11232] ppid=11229 vsize=401796 CPUtime=6.3 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 6578 0 0 0 630 0 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=401796 CPUtime=6.3 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4557 0 0 0 630 0 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 3 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=401796 CPUtime=6.3 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 25100 0 0 0 627 3 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4552336 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=401796 CPUtime=6.31 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4968 0 0 0 630 1 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=401796 CPUtime=6.3 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 4839 0 0 0 629 1 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348858 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=401796 CPUtime=6.28 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 6308 0 0 0 628 0 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=401796 CPUtime=6.3 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 5058 0 0 0 630 0 0 0 25 0 8 0 788535913 411439104 27015 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 2 0 0 0

################
# More data... #
################

[pid=11231/tid=11234] ppid=11229 vsize=4459700 CPUtime=4662.36 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 15146917 0 0 0 463788 2448 0 0 25 0 8 0 788535913 4566732800 879871 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339680 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4459700 CPUtime=4661.07 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 195639 0 0 0 466066 41 0 0 25 0 8 0 788535913 4566732800 879871 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352728 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4459700 CPUtime=4662.1 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 92754 0 0 0 466177 33 0 0 25 0 8 0 788535913 4566732800 879871 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348864 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4459700 CPUtime=4662.37 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 373093 0 0 0 466153 84 0 0 25 0 8 0 788535913 4566732800 879871 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4459700 CPUtime=4661.63 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234582 0 0 0 466114 49 0 0 25 0 8 0 788535913 4566732800 879871 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37215.3
Current children cumulated vsize (KiB) 4459700

[startup+4722.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/522 11386
/proc/meminfo: memFree=18401144/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=4459700 CPUtime=37694.2 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 17823847 0 0 0 3766405 3014 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4512794 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 1114925 883412 588 106 0 1110428 0
[pid=11231/tid=11232] ppid=11229 vsize=4459700 CPUtime=4722.29 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 212177 0 0 0 472184 45 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352728 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=4459700 CPUtime=4711.63 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 213327 0 0 0 471117 46 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=4459700 CPUtime=4722.36 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 15350022 0 0 0 469754 2482 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4459700 CPUtime=4721.05 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 196135 0 0 0 472064 41 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351827 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4459700 CPUtime=4722.1 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 93140 0 0 0 472177 33 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4349019 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4459700 CPUtime=4722.37 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 373203 0 0 0 472153 84 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4459700 CPUtime=4721.52 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234716 0 0 0 472103 49 0 0 25 0 8 0 788535913 4566732800 883412 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351827 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37694.2
Current children cumulated vsize (KiB) 4459700

[startup+4782.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/522 11388
/proc/meminfo: memFree=18408036/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=4459700 CPUtime=38173.1 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 17991087 0 0 0 3814269 3043 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4512009 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 1114925 883326 542 106 0 1110428 0
[pid=11231/tid=11232] ppid=11229 vsize=4459700 CPUtime=4782.29 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 212272 0 0 0 478184 45 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=4459700 CPUtime=4770.59 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 213328 0 0 0 477013 46 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=4459700 CPUtime=4782.36 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 15516502 0 0 0 475726 2510 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4552315 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4459700 CPUtime=4781.04 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 196246 0 0 0 478063 41 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4459700 CPUtime=4782.09 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 93209 0 0 0 478176 33 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348903 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4459700 CPUtime=4782.38 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 373293 0 0 0 478154 84 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4459700 CPUtime=4781.49 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234716 0 0 0 478100 49 0 0 25 0 8 0 788535913 4566732800 883326 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351792 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38173.1
Current children cumulated vsize (KiB) 4459700

[startup+4842.3 s]
/proc/loadavg: 8.13 8.08 8.01 9/522 11389
/proc/meminfo: memFree=18393600/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=4461668 CPUtime=38652 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 18184895 0 0 0 3862132 3071 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4511904 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 1115417 881493 390 106 0 1110920 0
[pid=11231/tid=11232] ppid=11229 vsize=4461668 CPUtime=4842.29 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 212361 0 0 0 484184 45 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=4461668 CPUtime=4829.55 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 213328 0 0 0 482909 46 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=4461668 CPUtime=4842.36 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 15709399 0 0 0 481698 2538 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4547280 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4461668 CPUtime=4841.02 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 196465 0 0 0 484061 41 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351824 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4461668 CPUtime=4842.09 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 93215 0 0 0 484176 33 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4349075 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4461668 CPUtime=4842.38 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 373681 0 0 0 484154 84 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4461668 CPUtime=4841.47 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234716 0 0 0 484098 49 0 0 25 0 8 0 788535913 4568748032 881493 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351897 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38652
Current children cumulated vsize (KiB) 4461668

[startup+4902.3 s]
/proc/loadavg: 8.10 8.08 8.01 9/522 11391
/proc/meminfo: memFree=18390736/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=4527204 CPUtime=39131 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 18376045 0 0 0 3910001 3102 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 1131801 885790 554 106 0 1127304 0
[pid=11231/tid=11232] ppid=11229 vsize=4527204 CPUtime=4902.31 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 212453 0 0 0 490186 45 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351764 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=4527204 CPUtime=4888.51 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 213328 0 0 0 488805 46 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=4527204 CPUtime=4902.39 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 15899439 0 0 0 487671 2568 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4527204 CPUtime=4901.03 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 196576 0 0 0 490061 42 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4527204 CPUtime=4902.1 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 93256 0 0 0 490177 33 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348826 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4527204 CPUtime=4902.4 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 373736 0 0 0 490156 84 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4527204 CPUtime=4901.43 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234716 0 0 0 490094 49 0 0 25 0 8 0 788535913 4635856896 885790 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39131
Current children cumulated vsize (KiB) 4527204

[startup+4962.3 s]
/proc/loadavg: 8.03 8.06 8.00 9/522 11393
/proc/meminfo: memFree=18381408/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=4461668 CPUtime=39609.9 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 18632078 0 0 0 3957844 3143 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4519334 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 1115417 884270 390 106 0 1110920 0
[pid=11231/tid=11232] ppid=11229 vsize=4461668 CPUtime=4962.29 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 212568 0 0 0 496184 45 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351979 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=4461668 CPUtime=4947.47 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 213328 0 0 0 494701 46 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4352739 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=4461668 CPUtime=4962.37 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 16153631 0 0 0 493628 2609 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4547427 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4461668 CPUtime=4961 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 196680 0 0 0 496058 42 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351860 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4461668 CPUtime=4962.08 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 93266 0 0 0 496175 33 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4349075 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4461668 CPUtime=4962.37 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 375233 0 0 0 496153 84 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351863 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4461668 CPUtime=4961.41 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234716 0 0 0 496092 49 0 0 25 0 8 0 788535913 4568748032 884270 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339591 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39609.9
Current children cumulated vsize (KiB) 4461668



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.2 s]
/proc/loadavg: 8.01 8.05 8.00 10/522 11394
/proc/meminfo: memFree=18391040/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=4462272 CPUtime=40000.5 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) R 11229 11231 11150 0 -1 4202496 18747496 0 0 0 3996882 3164 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4511930 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 1115568 885446 390 106 0 1111071 0
[pid=11231/tid=11232] ppid=11229 vsize=4462272 CPUtime=5011.23 cores=0-7
/proc/11231/task/11232/stat : 11232 (SArTagnan) R 11229 11231 11150 0 -1 4202560 212635 0 0 0 501078 45 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 6 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=4462272 CPUtime=4995.53 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4202560 213328 0 0 0 499507 46 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351804 0 0 4096 2 0 0 0 -1 7 0 0 0
[pid=11231/tid=11234] ppid=11229 vsize=4462272 CPUtime=5011.31 cores=0-7
/proc/11231/task/11234/stat : 11234 (SArTagnan) R 11229 11231 11150 0 -1 4202560 16267955 0 0 0 498503 2628 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4552383 0 0 4096 2 0 0 0 -1 5 0 0 0
[pid=11231/tid=11235] ppid=11229 vsize=4462272 CPUtime=5009.93 cores=0-7
/proc/11231/task/11235/stat : 11235 (SArTagnan) R 11229 11231 11150 0 -1 4202560 197345 0 0 0 500951 42 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4339536 0 0 4096 2 0 0 0 -1 4 0 0 0
[pid=11231/tid=11236] ppid=11229 vsize=4462272 CPUtime=5011.02 cores=0-7
/proc/11231/task/11236/stat : 11236 (SArTagnan) R 11229 11231 11150 0 -1 4202560 93268 0 0 0 501068 34 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4348858 0 0 4096 2 0 0 0 -1 0 0 0 0
[pid=11231/tid=11237] ppid=11229 vsize=4462272 CPUtime=5011.32 cores=0-7
/proc/11231/task/11237/stat : 11237 (SArTagnan) R 11229 11231 11150 0 -1 4202560 375290 0 0 0 501047 85 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351781 0 0 4096 2 0 0 0 -1 1 0 0 0
[pid=11231/tid=11238] ppid=11229 vsize=4462272 CPUtime=5010.35 cores=0-7
/proc/11231/task/11238/stat : 11238 (SArTagnan) R 11229 11231 11150 0 -1 4202560 234716 0 0 0 500986 49 0 0 25 0 8 0 788535913 4569366528 885446 33554432000 4194304 4625164 140737334671088 18446744073709551615 4351830 0 0 4096 2 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 4462272

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011.4 s]
/proc/loadavg: 8.01 8.05 8.00 10/522 11394
/proc/meminfo: memFree=18391040/32951124 swapFree=58800704/67111528
[pid=11231] ppid=11229 vsize=0 CPUtime=40000.7 cores=0-7
/proc/11231/stat : 11231 (SArTagnan) Z 11229 11231 11150 0 -1 4203548 18747496 0 0 0 3996892 3179 0 0 25 0 2 0 788535913 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 17 3 0 0 0
/proc/11231/statm: 0 0 0 0 0 0 0
[pid=11231/tid=11233] ppid=11229 vsize=0 CPUtime=4995.7 cores=0-7
/proc/11231/task/11233/stat : 11233 (SArTagnan) R 11229 11231 11150 0 -1 4203588 213328 0 0 0 499508 62 0 0 25 0 2 0 788535913 0 0 33554432000 0 0 0 0 0 0 0 4096 2 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5011.42
CPU time (s): 40000.7
CPU user time (s): 39968.9
CPU system time (s): 31.8102
CPU usage (%): 798.192
Max. virtual memory (cumulated for all children) (KiB): 4632988

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39968.9
system time used= 31.8102
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18747496
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3007
involuntary context switches= 169029

runsolver used 23.1205 second user time and 64.0913 second system time

The end

Launcher Data 

Begin job on node142 at 2011-06-01 22:46:44
IDJOB=3439361
IDBENCH=84030
IDSOLVER=1936
FILE ID=node142/3439361-1306961204
RUNJOBID= node142-1306960405-11164
PBS_JOBID= 13497609
Free space on /tmp= 73848 MiB

SOLVER NAME= SArTagnan 2011-05-15 (fixed)
BENCH NAME= SAT03/handmade/stanion/hwb/hwb-n28-01-S136611085.shuffled-as.sat03-1627.cnf
COMMAND LINE= SArTagnan BENCHNAME -threads=NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3439361-1306961204/watcher-3439361-1306961204 -o /tmp/evaluation-result-3439361-1306961204/solver-3439361-1306961204 -C 40000 -W 5100 -M 31000  SArTagnan HOME/instance-3439361-1306961204.cnf -threads=8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 17624c574c7f9ee0eff4cc4a13e42e12
RANDOM SEED=467888127

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21941940 kB
Buffers:        144736 kB
Cached:         199584 kB
SwapCached:      11420 kB
Active:        9115024 kB
Inactive:      1640672 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21941940 kB
SwapTotal:    67111528 kB
SwapFree:     58800704 kB
Dirty:             392 kB
Writeback:           0 kB
AnonPages:    10409184 kB
Mapped:          31268 kB
Slab:           106180 kB
PageTables:      74760 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 145376080 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73848 MiB
End job on node142 at 2011-06-02 00:10:18