Trace number 3372450

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 157765 5101.95

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
transport-transport-city-sequential-35nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.050-SAT.cnf
MD5SUM48f03e5fbe31fc85b9a464480fc8aa5a
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark5148.61
Satisfiable
(Un)Satisfiability was proved
Number of variables1575206
Number of clauses8464620
Sum of the clauses size20118920
Maximum clause length982
Minimum clause length1
Number of clauses of size 1720
Number of clauses of size 28393300
Number of clauses of size 3400
Number of clauses of size 41600
Number of clauses of size 54000
Number of clauses of size over 564600

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372450-1305621302/watcher-3372450-1305621302 -o /tmp/evaluation-result-3372450-1305621302/solver-3372450-1305621302 -C 160000 -W 5100 -M 266338304 plingeling -t 32 HOME/instance-3372450-1305621302.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 19.80 29.15 30.95 1/413 24352
/proc/meminfo: memFree=255358632/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=36168 CPUtime=0.02 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 7094 1250 0 0 1 1 0 0 22 0 1 0 673617286 37036032 7026 268435456000 4194304 4417452 140733738525984 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 9042 7041 141 55 0 6928 0

[startup+0.111934 s]
/proc/loadavg: 19.80 29.15 30.95 1/413 24352
/proc/meminfo: memFree=255358632/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=54140 CPUtime=0.09 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 11607 1250 0 0 8 1 0 0 22 0 1 0 673617286 55439360 11538 268435456000 4194304 4417452 140733738525984 18446744073709551615 4264940 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 13535 11538 144 55 0 11421 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 54140

[startup+0.2009 s]
/proc/loadavg: 19.80 29.15 30.95 1/413 24352
/proc/meminfo: memFree=255358632/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=64496 CPUtime=0.19 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 14197 1250 0 0 17 2 0 0 23 0 1 0 673617286 66043904 14128 268435456000 4194304 4417452 140733738525984 18446744073709551615 4264934 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 16124 14128 145 55 0 14010 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 64496

[startup+0.3009 s]
/proc/loadavg: 19.80 29.15 30.95 1/413 24352
/proc/meminfo: memFree=255358632/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=64496 CPUtime=0.29 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 14197 1250 0 0 27 2 0 0 23 0 1 0 673617286 66043904 14128 268435456000 4194304 4417452 140733738525984 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 16124 14128 145 55 0 14010 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 64496

[startup+0.700909 s]
/proc/loadavg: 19.80 29.15 30.95 1/413 24352
/proc/meminfo: memFree=255358632/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=113668 CPUtime=0.69 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 26506 1250 0 0 64 5 0 0 25 0 1 0 673617286 116396032 26437 268435456000 4194304 4417452 140733738525984 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 28417 26437 145 55 0 26303 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 113668

[startup+1.50094 s]
/proc/loadavg: 19.80 29.15 30.95 2/414 24358
/proc/meminfo: memFree=255237716/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=203316 CPUtime=1.48 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 48919 1250 0 0 139 9 0 0 25 0 1 0 673617286 208195584 48850 268435456000 4194304 4417452 140733738525984 18446744073709551615 4264909 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 50829 48850 146 55 0 48715 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 203316

[startup+3.101 s]
/proc/loadavg: 18.30 28.68 30.79 2/414 24358
/proc/meminfo: memFree=255137320/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=386996 CPUtime=3.09 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 94843 1250 0 0 289 20 0 0 25 0 1 0 673617286 396283904 94770 268435456000 4194304 4417452 140733738525984 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 96749 94770 146 55 0 94635 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 386996

[startup+6.30111 s]
/proc/loadavg: 18.30 28.68 30.79 2/414 24358
/proc/meminfo: memFree=254920712/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=667052 CPUtime=6.29 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 164000 1250 0 0 595 34 0 0 25 0 1 0 673617286 683061248 163908 268435456000 4194304 4417452 140733738525984 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 166763 163926 146 55 0 164649 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 667052

[startup+12.7003 s]
/proc/loadavg: 15.71 27.78 30.48 2/414 24358
/proc/meminfo: memFree=254506036/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=1008200 CPUtime=12.68 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 250163 1250 0 0 1217 51 0 0 25 0 1 0 673617286 1032396800 250071 268435456000 4194304 4417452 140733738525984 18446744073709551615 4264909 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 252050 250071 146 55 0 249936 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 1008200

[startup+25.5008 s]
/proc/loadavg: 13.45 26.90 30.16 2/414 24358
/proc/meminfo: memFree=253466120/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=1900628 CPUtime=25.49 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 473270 1250 0 0 2454 95 0 0 25 0 1 0 673617286 1946243072 473178 268435456000 4194304 4417452 140733738525984 18446744073709551615 4259618 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 475157 473178 146 55 0 473043 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 1900628

[startup+51.1007 s]
/proc/loadavg: 9.20 24.82 29.38 2/414 24360
/proc/meminfo: memFree=251679256/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=3685472 CPUtime=51.08 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 919481 1250 0 0 4925 183 0 0 25 0 1 0 673617286 3773923328 919389 268435456000 4194304 4417452 140733738525984 18446744073709551615 4259780 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 921368 919389 146 55 0 919254 0
Current children cumulated CPU time (s) 51.08
Current children cumulated vsize (KiB) 3685472

[startup+102.301 s]
/proc/loadavg: 4.27 20.81 27.75 2/414 24360
/proc/meminfo: memFree=248098376/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=7260272 CPUtime=102.3 cores=0-31
/proc/24352/stat : 24352 (plingeling) R 24350 24352 24326 0 -1 4202496 1813181 1250 0 0 9875 355 0 0 25 0 1 0 673617286 7434518528 1813089 268435456000 4194304 4417452 140733738525984 18446744073709551615 4320663 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/24352/statm: 1815068 1813089 146 55 0 1812954 0
Current children cumulated CPU time (s) 102.3
Current children cumulated vsize (KiB) 7260272

[startup+162.301 s]
/proc/loadavg: 6.97 18.22 26.40 33/446 24394
/proc/meminfo: memFree=244359768/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=11419808 CPUtime=383.85 cores=0-31
/proc/24352/stat : 24352 (plingeling) S 24350 24352 24326 0 -1 4202496 3676102 1250 0 0 36157 2228 0 0 18 0 33 0 673617286 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 27 0 0 0
/proc/24352/statm: 2854952 2745622 167 55 0 2852838 0
[pid=24352/tid=24363] ppid=24350 vsize=11419808 CPUtime=7.35 cores=0-31
/proc/24352/task/24363/stat : 24363 (plingeling) R 24350 24352 24326 0 -1 4202560 40657 1250 0 0 701 34 0 0 25 0 33 0 673632772 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4287486 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=24352/tid=24364] ppid=24350 vsize=11419808 CPUtime=7.23 cores=0-31
/proc/24352/task/24364/stat : 24364 (plingeling) R 24350 24352 24326 0 -1 4202560 40649 1250 0 0 638 85 0 0 25 0 33 0 673632772 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4281863 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24352/tid=24365] ppid=24350 vsize=11419808 CPUtime=7.31 cores=0-31
/proc/24352/task/24365/stat : 24365 (plingeling) R 24350 24352 24326 0 -1 4202560 40648 1250 0 0 678 53 0 0 25 0 33 0 673632772 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4287492 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=24352/tid=24366] ppid=24350 vsize=11419808 CPUtime=7.36 cores=0-31
/proc/24352/task/24366/stat : 24366 (plingeling) R 24350 24352 24326 0 -1 4202560 30455 1250 0 0 703 33 0 0 25 0 33 0 673632773 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232755 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=24352/tid=24367] ppid=24350 vsize=11419808 CPUtime=7.31 cores=0-31
/proc/24352/task/24367/stat : 24367 (plingeling) R 24350 24352 24326 0 -1 4202560 31689 1250 0 0 705 26 0 0 25 0 33 0 673632773 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284484 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24352/tid=24368] ppid=24350 vsize=11419808 CPUtime=7.21 cores=0-31
/proc/24352/task/24368/stat : 24368 (plingeling) R 24350 24352 24326 0 -1 4202560 40660 1250 0 0 706 15 0 0 25 0 33 0 673632774 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284626 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=24352/tid=24369] ppid=24350 vsize=11419808 CPUtime=7.34 cores=0-31
/proc/24352/task/24369/stat : 24369 (plingeling) R 24350 24352 24326 0 -1 4202560 40659 1250 0 0 696 38 0 0 25 0 33 0 673632774 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284626 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24352/tid=24370] ppid=24350 vsize=11419808 CPUtime=7.23 cores=0-31
/proc/24352/task/24370/stat : 24370 (plingeling) R 24350 24352 24326 0 -1 4202560 40648 1250 0 0 634 89 0 0 25 0 33 0 673632774 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284484 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=24352/tid=24371] ppid=24350 vsize=11419808 CPUtime=7.23 cores=0-31
/proc/24352/task/24371/stat : 24371 (plingeling) R 24350 24352 24326 0 -1 4202560 40648 1250 0 0 635 88 0 0 25 0 33 0 673632775 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284784 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24352/tid=24372] ppid=24350 vsize=11419808 CPUtime=7.27 cores=0-31
/proc/24352/task/24372/stat : 24372 (plingeling) R 24350 24352 24326 0 -1 4202560 40656 1250 0 0 670 57 0 0 25 0 33 0 673632775 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284781 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24352/tid=24373] ppid=24350 vsize=11419808 CPUtime=7.34 cores=0-31
/proc/24352/task/24373/stat : 24373 (plingeling) R 24350 24352 24326 0 -1 4202560 40646 1250 0 0 694 40 0 0 25 0 33 0 673632775 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284626 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=24352/tid=24374] ppid=24350 vsize=11419808 CPUtime=7.16 cores=0-31
/proc/24352/task/24374/stat : 24374 (plingeling) R 24350 24352 24326 0 -1 4202560 40648 1250 0 0 646 70 0 0 25 0 33 0 673632776 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284341 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=24352/tid=24375] ppid=24350 vsize=11419808 CPUtime=7.29 cores=0-31
/proc/24352/task/24375/stat : 24375 (plingeling) R 24350 24352 24326 0 -1 4202560 40624 1250 0 0 695 34 0 0 25 0 33 0 673632776 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284484 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=24352/tid=24376] ppid=24350 vsize=11419808 CPUtime=7.23 cores=0-31
/proc/24352/task/24376/stat : 24376 (plingeling) R 24350 24352 24326 0 -1 4202560 40655 1250 0 0 695 28 0 0 25 0 33 0 673632777 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284484 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=24352/tid=24377] ppid=24350 vsize=11419808 CPUtime=7.28 cores=0-31
/proc/24352/task/24377/stat : 24377 (plingeling) R 24350 24352 24326 0 -1 4202560 40657 1250 0 0 685 43 0 0 25 0 33 0 673632777 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284514 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=24352/tid=24378] ppid=24350 vsize=11419808 CPUtime=7.17 cores=0-31
/proc/24352/task/24378/stat : 24378 (plingeling) R 24350 24352 24326 0 -1 4202560 40647 1250 0 0 634 83 0 0 25 0 33 0 673632777 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4288704 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=24352/tid=24379] ppid=24350 vsize=11419808 CPUtime=7.25 cores=0-31
/proc/24352/task/24379/stat : 24379 (plingeling) R 24350 24352 24326 0 -1 4202560 40646 1250 0 0 664 61 0 0 25 0 33 0 673632778 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4288259 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=24352/tid=24380] ppid=24350 vsize=11419808 CPUtime=7.27 cores=0-31
/proc/24352/task/24380/stat : 24380 (plingeling) R 24350 24352 24326 0 -1 4202560 40658 1250 0 0 683 44 0 0 25 0 33 0 673632778 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4286414 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=24352/tid=24381] ppid=24350 vsize=11419808 CPUtime=7.17 cores=0-31
/proc/24352/task/24381/stat : 24381 (plingeling) R 24350 24352 24326 0 -1 4202560 40657 1250 0 0 633 84 0 0 25 0 33 0 673632779 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4287492 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=24352/tid=24382] ppid=24350 vsize=11419808 CPUtime=7.21 cores=0-31
/proc/24352/task/24382/stat : 24382 (plingeling) R 24350 24352 24326 0 -1 4202560 40656 1250 0 0 654 67 0 0 25 0 33 0 673632779 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284289 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=24352/tid=24383] ppid=24350 vsize=11419808 CPUtime=7.07 cores=0-31
/proc/24352/task/24383/stat : 24383 (plingeling) R 24350 24352 24326 0 -1 4202560 40646 1250 0 0 665 42 0 0 25 0 33 0 673632779 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284484 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=24352/tid=24384] ppid=24350 vsize=11419808 CPUtime=7.12 cores=0-31
/proc/24352/task/24384/stat : 24384 (plingeling) R 24350 24352 24326 0 -1 4202560 32035 1250 0 0 677 35 0 0 25 0 33 0 673632780 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284784 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=24352/tid=24385] ppid=24350 vsize=11419808 CPUtime=7.03 cores=0-31
/proc/24352/task/24385/stat : 24385 (plingeling) R 24350 24352 24326 0 -1 4202560 40657 1250 0 0 636 67 0 0 25 0 33 0 673632790 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4287187 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=24352/tid=24386] ppid=24350 vsize=11419808 CPUtime=7 cores=0-31
/proc/24352/task/24386/stat : 24386 (plingeling) R 24350 24352 24326 0 -1 4202560 40648 1250 0 0 631 69 0 0 25 0 33 0 673632798 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284784 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=24352/tid=24387] ppid=24350 vsize=11419808 CPUtime=6.89 cores=0-31
/proc/24352/task/24387/stat : 24387 (plingeling) R 24350 24352 24326 0 -1 4202560 40646 1250 0 0 633 56 0 0 25 0 33 0 673632799 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284514 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24352/tid=24388] ppid=24350 vsize=11419808 CPUtime=6.98 cores=0-31
/proc/24352/task/24388/stat : 24388 (plingeling) R 24350 24352 24326 0 -1 4202560 40646 1250 0 0 630 68 0 0 25 0 33 0 673632799 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4285565 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=24352/tid=24389] ppid=24350 vsize=11419808 CPUtime=6.97 cores=0-31
/proc/24352/task/24389/stat : 24389 (plingeling) R 24350 24352 24326 0 -1 4202560 40647 1250 0 0 637 60 0 0 25 0 33 0 673632800 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4287486 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24352/tid=24390] ppid=24350 vsize=11419808 CPUtime=6.99 cores=0-31
/proc/24352/task/24390/stat : 24390 (plingeling) R 24350 24352 24326 0 -1 4202560 40655 1250 0 0 635 64 0 0 25 0 33 0 673632800 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4286915 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=24352/tid=24391] ppid=24350 vsize=11419808 CPUtime=6.94 cores=0-31
/proc/24352/task/24391/stat : 24391 (plingeling) R 24350 24352 24326 0 -1 4202560 40656 1250 0 0 633 61 0 0 25 0 33 0 673632801 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284484 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24352/tid=24392] ppid=24350 vsize=11419808 CPUtime=6.96 cores=0-31
/proc/24352/task/24392/stat : 24392 (plingeling) R 24350 24352 24326 0 -1 4202560 40657 1250 0 0 651 45 0 0 25 0 33 0 673632801 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4286467 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=24352/tid=24393] ppid=24350 vsize=11419808 CPUtime=6.77 cores=0-31
/proc/24352/task/24393/stat : 24393 (plingeling) R 24350 24352 24326 0 -1 4202560 40656 1250 0 0 615 62 0 0 25 0 33 0 673632807 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4286308 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=24352/tid=24394] ppid=24350 vsize=11419808 CPUtime=6.79 cores=0-31
/proc/24352/task/24394/stat : 24394 (plingeling) R 24350 24352 24326 0 -1 4202560 40647 1250 0 0 644 35 0 0 25 0 33 0 673632807 11693883392 2745622 268435456000 4194304 4417452 140733738525984 18446744073709551615 4284514 0 0 4096 17442 0 0 0 -1 20 0 0 0
Current children cumulated CPU time (s) 383.85
Current children cumulated vsize (KiB) 11419808


################
# More data... #
################

/proc/24352/task/24368/stat : 24368 (plingeling) R 24350 24352 24326 0 -1 4202560 1357259 1250 0 0 488457 2053 0 0 25 0 33 0 673632774 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4357260 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24352/tid=24369] ppid=24350 vsize=33440488 CPUtime=4915.9 cores=0-31
/proc/24352/task/24369/stat : 24369 (plingeling) R 24350 24352 24326 0 -1 4202560 1568820 1250 0 0 489320 2270 0 0 25 0 33 0 673632774 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364456 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=24352/tid=24370] ppid=24350 vsize=33440488 CPUtime=4906.21 cores=0-31
/proc/24352/task/24370/stat : 24370 (plingeling) R 24350 24352 24326 0 -1 4202560 1142734 1250 0 0 489131 1490 0 0 25 0 33 0 673632774 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=24352/tid=24371] ppid=24350 vsize=33440488 CPUtime=4897.13 cores=0-31
/proc/24352/task/24371/stat : 24371 (plingeling) R 24350 24352 24326 0 -1 4202560 1069477 1250 0 0 488250 1463 0 0 25 0 33 0 673632775 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=24352/tid=24372] ppid=24350 vsize=33440488 CPUtime=4911.8 cores=0-31
/proc/24352/task/24372/stat : 24372 (plingeling) R 24350 24352 24326 0 -1 4202560 754665 1250 0 0 490289 891 0 0 25 0 33 0 673632775 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4237157 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=24352/tid=24373] ppid=24350 vsize=33440488 CPUtime=4913.72 cores=0-31
/proc/24352/task/24373/stat : 24373 (plingeling) R 24350 24352 24326 0 -1 4202560 1508690 1250 0 0 489053 2319 0 0 25 0 33 0 673632775 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=24352/tid=24374] ppid=24350 vsize=33440488 CPUtime=4906.01 cores=0-31
/proc/24352/task/24374/stat : 24374 (plingeling) R 24350 24352 24326 0 -1 4202560 866971 1250 0 0 489562 1039 0 0 25 0 33 0 673632776 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236797 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=24352/tid=24375] ppid=24350 vsize=33440488 CPUtime=4910.04 cores=0-31
/proc/24352/task/24375/stat : 24375 (plingeling) R 24350 24352 24326 0 -1 4202560 1072689 1250 0 0 489561 1443 0 0 25 0 33 0 673632776 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=24352/tid=24376] ppid=24350 vsize=33440488 CPUtime=4910.51 cores=0-31
/proc/24352/task/24376/stat : 24376 (plingeling) R 24350 24352 24326 0 -1 4202560 1033815 1250 0 0 489432 1619 0 0 25 0 33 0 673632777 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=24352/tid=24377] ppid=24350 vsize=33440488 CPUtime=4914.02 cores=0-31
/proc/24352/task/24377/stat : 24377 (plingeling) R 24350 24352 24326 0 -1 4202560 1195445 1250 0 0 489732 1670 0 0 25 0 33 0 673632777 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4281694 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=24352/tid=24378] ppid=24350 vsize=33440488 CPUtime=4903.74 cores=0-31
/proc/24352/task/24378/stat : 24378 (plingeling) R 24350 24352 24326 0 -1 4202560 1471923 1250 0 0 488197 2177 0 0 25 0 33 0 673632777 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24352/tid=24379] ppid=24350 vsize=33440488 CPUtime=4910.98 cores=0-31
/proc/24352/task/24379/stat : 24379 (plingeling) R 24350 24352 24326 0 -1 4202560 1237856 1250 0 0 489545 1553 0 0 25 0 33 0 673632778 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4264909 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24352/tid=24380] ppid=24350 vsize=33440488 CPUtime=4908.01 cores=0-31
/proc/24352/task/24380/stat : 24380 (plingeling) R 24350 24352 24326 0 -1 4202560 993988 1250 0 0 489707 1094 0 0 25 0 33 0 673632778 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=24352/tid=24381] ppid=24350 vsize=33440488 CPUtime=4889.23 cores=0-31
/proc/24352/task/24381/stat : 24381 (plingeling) R 24350 24352 24326 0 -1 4202560 1307529 1250 0 0 487324 1599 0 0 25 0 33 0 673632779 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24352/tid=24382] ppid=24350 vsize=33440488 CPUtime=4911.54 cores=0-31
/proc/24352/task/24382/stat : 24382 (plingeling) R 24350 24352 24326 0 -1 4202560 1513644 1250 0 0 488807 2347 0 0 25 0 33 0 673632779 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24352/tid=24383] ppid=24350 vsize=33440488 CPUtime=4907.26 cores=0-31
/proc/24352/task/24383/stat : 24383 (plingeling) R 24350 24352 24326 0 -1 4202560 784308 1250 0 0 489703 1023 0 0 25 0 33 0 673632779 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=24352/tid=24384] ppid=24350 vsize=33440488 CPUtime=4911.53 cores=0-31
/proc/24352/task/24384/stat : 24384 (plingeling) R 24350 24352 24326 0 -1 4202560 746125 1250 0 0 490245 908 0 0 25 0 33 0 673632780 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=24352/tid=24385] ppid=24350 vsize=33440488 CPUtime=4906.63 cores=0-31
/proc/24352/task/24385/stat : 24385 (plingeling) R 24350 24352 24326 0 -1 4202560 1711740 1250 0 0 488180 2483 0 0 25 0 33 0 673632790 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4237371 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=24352/tid=24386] ppid=24350 vsize=33440488 CPUtime=4908.38 cores=0-31
/proc/24352/task/24386/stat : 24386 (plingeling) R 24350 24352 24326 0 -1 4202560 1466821 1250 0 0 488272 2566 0 0 25 0 33 0 673632798 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=24352/tid=24387] ppid=24350 vsize=33440488 CPUtime=4908.48 cores=0-31
/proc/24352/task/24387/stat : 24387 (plingeling) R 24350 24352 24326 0 -1 4202560 2073787 1250 0 0 487995 2853 0 0 25 0 33 0 673632799 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232801 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=24352/tid=24388] ppid=24350 vsize=33440488 CPUtime=4903.4 cores=0-31
/proc/24352/task/24388/stat : 24388 (plingeling) R 24350 24352 24326 0 -1 4202560 1020040 1250 0 0 488881 1459 0 0 25 0 33 0 673632799 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4237202 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24352/tid=24389] ppid=24350 vsize=33440488 CPUtime=4907.51 cores=0-31
/proc/24352/task/24389/stat : 24389 (plingeling) R 24350 24352 24326 0 -1 4202560 996101 1250 0 0 489429 1322 0 0 25 0 33 0 673632800 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=24352/tid=24390] ppid=24350 vsize=33440488 CPUtime=4906.75 cores=0-31
/proc/24352/task/24390/stat : 24390 (plingeling) R 24350 24352 24326 0 -1 4202560 1261060 1250 0 0 488722 1953 0 0 25 0 33 0 673632800 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232705 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=24352/tid=24391] ppid=24350 vsize=33440488 CPUtime=4909.71 cores=0-31
/proc/24352/task/24391/stat : 24391 (plingeling) R 24350 24352 24326 0 -1 4202560 1539182 1250 0 0 488716 2255 0 0 25 0 33 0 673632801 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232801 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=24352/tid=24392] ppid=24350 vsize=33440488 CPUtime=4914.14 cores=0-31
/proc/24352/task/24392/stat : 24392 (plingeling) R 24350 24352 24326 0 -1 4202560 933025 1250 0 0 490377 1037 0 0 25 0 33 0 673632801 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=24352/tid=24393] ppid=24350 vsize=33440488 CPUtime=4908.4 cores=0-31
/proc/24352/task/24393/stat : 24393 (plingeling) R 24350 24352 24326 0 -1 4202560 1313743 1250 0 0 489062 1778 0 0 25 0 33 0 673632807 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24352/tid=24394] ppid=24350 vsize=33440488 CPUtime=4901.79 cores=0-31
/proc/24352/task/24394/stat : 24394 (plingeling) R 24350 24352 24326 0 -1 4202560 1785255 1250 0 0 487485 2694 0 0 25 0 33 0 673632807 34243059712 8130960 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232666 0 0 4096 17442 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 157197
Current children cumulated vsize (KiB) 33440488



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 32.11 32.09 32.01 33/446 24538
/proc/meminfo: memFree=222908764/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=33308180 CPUtime=157763 cores=0-31
/proc/24352/stat : 24352 (plingeling) S 24350 24352 24326 0 -1 4202496 44468133 1250 0 0 15716845 59414 0 0 18 0 33 0 673617286 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 27 0 0 0
/proc/24352/statm: 8327045 8097883 175 55 0 8324931 0
[pid=24352/tid=24363] ppid=24350 vsize=33308180 CPUtime=4930.4 cores=0-31
/proc/24352/task/24363/stat : 24363 (plingeling) R 24350 24352 24326 0 -1 4202560 975477 1250 0 0 491983 1057 0 0 25 0 33 0 673632772 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4263136 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24352/tid=24364] ppid=24350 vsize=33308180 CPUtime=4919.55 cores=0-31
/proc/24352/task/24364/stat : 24364 (plingeling) R 24350 24352 24326 0 -1 4202560 1143509 1250 0 0 490459 1496 0 0 25 0 33 0 673632772 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4250651 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=24352/tid=24365] ppid=24350 vsize=33308180 CPUtime=4931.48 cores=0-31
/proc/24352/task/24365/stat : 24365 (plingeling) R 24350 24352 24326 0 -1 4202560 1690198 1250 0 0 490471 2677 0 0 25 0 33 0 673632772 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=24352/tid=24366] ppid=24350 vsize=33308180 CPUtime=4916.16 cores=0-31
/proc/24352/task/24366/stat : 24366 (plingeling) R 24350 24352 24326 0 -1 4202560 2650630 1250 0 0 488481 3135 0 0 25 0 33 0 673632773 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=24352/tid=24367] ppid=24350 vsize=33308180 CPUtime=4924.57 cores=0-31
/proc/24352/task/24367/stat : 24367 (plingeling) R 24350 24352 24326 0 -1 4202560 1759312 1250 0 0 489443 3014 0 0 25 0 33 0 673632773 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=24352/tid=24368] ppid=24350 vsize=33308180 CPUtime=4922.82 cores=0-31
/proc/24352/task/24368/stat : 24368 (plingeling) R 24350 24352 24326 0 -1 4202560 1357259 1250 0 0 490229 2053 0 0 25 0 33 0 673632774 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=24352/tid=24369] ppid=24350 vsize=33308180 CPUtime=4933.62 cores=0-31
/proc/24352/task/24369/stat : 24369 (plingeling) R 24350 24352 24326 0 -1 4202560 1568820 1250 0 0 491092 2270 0 0 25 0 33 0 673632774 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=24352/tid=24370] ppid=24350 vsize=33308180 CPUtime=4923.92 cores=0-31
/proc/24352/task/24370/stat : 24370 (plingeling) R 24350 24352 24326 0 -1 4202560 1146476 1250 0 0 490891 1501 0 0 25 0 33 0 673632774 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=24352/tid=24371] ppid=24350 vsize=33308180 CPUtime=4914.85 cores=0-31
/proc/24352/task/24371/stat : 24371 (plingeling) R 24350 24352 24326 0 -1 4202560 1069477 1250 0 0 490022 1463 0 0 25 0 33 0 673632775 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4356472 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=24352/tid=24372] ppid=24350 vsize=33308180 CPUtime=4929.35 cores=0-31
/proc/24352/task/24372/stat : 24372 (plingeling) R 24350 24352 24326 0 -1 4202560 762068 1250 0 0 492038 897 0 0 25 0 33 0 673632775 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=24352/tid=24373] ppid=24350 vsize=33308180 CPUtime=4931.44 cores=0-31
/proc/24352/task/24373/stat : 24373 (plingeling) R 24350 24352 24326 0 -1 4202560 1508690 1250 0 0 490825 2319 0 0 25 0 33 0 673632775 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4253491 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=24352/tid=24374] ppid=24350 vsize=33308180 CPUtime=4923.73 cores=0-31
/proc/24352/task/24374/stat : 24374 (plingeling) R 24350 24352 24326 0 -1 4202560 866971 1250 0 0 491334 1039 0 0 25 0 33 0 673632776 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=24352/tid=24375] ppid=24350 vsize=33308180 CPUtime=4927.76 cores=0-31
/proc/24352/task/24375/stat : 24375 (plingeling) R 24350 24352 24326 0 -1 4202560 1072689 1250 0 0 491332 1444 0 0 25 0 33 0 673632776 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232801 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=24352/tid=24376] ppid=24350 vsize=33308180 CPUtime=4928.23 cores=0-31
/proc/24352/task/24376/stat : 24376 (plingeling) R 24350 24352 24326 0 -1 4202560 1043091 1250 0 0 491178 1645 0 0 25 0 33 0 673632777 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=24352/tid=24377] ppid=24350 vsize=33308180 CPUtime=4931.74 cores=0-31
/proc/24352/task/24377/stat : 24377 (plingeling) R 24350 24352 24326 0 -1 4202560 1195445 1250 0 0 491504 1670 0 0 25 0 33 0 673632777 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=24352/tid=24378] ppid=24350 vsize=33308180 CPUtime=4921.42 cores=0-31
/proc/24352/task/24378/stat : 24378 (plingeling) R 24350 24352 24326 0 -1 4202560 1472399 1250 0 0 489963 2179 0 0 25 0 33 0 673632777 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4320983 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24352/tid=24379] ppid=24350 vsize=33308180 CPUtime=4928.7 cores=0-31
/proc/24352/task/24379/stat : 24379 (plingeling) R 24350 24352 24326 0 -1 4202560 1237856 1250 0 0 491317 1553 0 0 25 0 33 0 673632778 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4355830 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24352/tid=24380] ppid=24350 vsize=33308180 CPUtime=4925.73 cores=0-31
/proc/24352/task/24380/stat : 24380 (plingeling) R 24350 24352 24326 0 -1 4202560 993988 1250 0 0 491479 1094 0 0 25 0 33 0 673632778 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232808 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=24352/tid=24381] ppid=24350 vsize=33308180 CPUtime=4906.62 cores=0-31
/proc/24352/task/24381/stat : 24381 (plingeling) R 24350 24352 24326 0 -1 4202560 1330461 1250 0 0 489013 1649 0 0 25 0 33 0 673632779 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=24352/tid=24382] ppid=24350 vsize=33308180 CPUtime=4929.26 cores=0-31
/proc/24352/task/24382/stat : 24382 (plingeling) R 24350 24352 24326 0 -1 4202560 1513644 1250 0 0 490579 2347 0 0 25 0 33 0 673632779 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4365538 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24352/tid=24383] ppid=24350 vsize=33308180 CPUtime=4924.98 cores=0-31
/proc/24352/task/24383/stat : 24383 (plingeling) R 24350 24352 24326 0 -1 4202560 801537 1250 0 0 491471 1027 0 0 25 0 33 0 673632779 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4358116 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=24352/tid=24384] ppid=24350 vsize=33308180 CPUtime=4929.23 cores=0-31
/proc/24352/task/24384/stat : 24384 (plingeling) R 24350 24352 24326 0 -1 4202560 751411 1250 0 0 492001 922 0 0 25 0 33 0 673632780 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236603 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=24352/tid=24385] ppid=24350 vsize=33308180 CPUtime=4924.35 cores=0-31
/proc/24352/task/24385/stat : 24385 (plingeling) R 24350 24352 24326 0 -1 4202560 1711740 1250 0 0 489952 2483 0 0 25 0 33 0 673632790 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=24352/tid=24386] ppid=24350 vsize=33308180 CPUtime=4926.1 cores=0-31
/proc/24352/task/24386/stat : 24386 (plingeling) R 24350 24352 24326 0 -1 4202560 1474316 1250 0 0 490038 2572 0 0 25 0 33 0 673632798 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236623 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=24352/tid=24387] ppid=24350 vsize=33308180 CPUtime=4926.2 cores=0-31
/proc/24352/task/24387/stat : 24387 (plingeling) R 24350 24352 24326 0 -1 4202560 2073787 1250 0 0 489767 2853 0 0 25 0 33 0 673632799 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=24352/tid=24388] ppid=24350 vsize=33308180 CPUtime=4920.91 cores=0-31
/proc/24352/task/24388/stat : 24388 (plingeling) R 24350 24352 24326 0 -1 4202560 1025128 1250 0 0 490617 1474 0 0 25 0 33 0 673632799 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=24352/tid=24389] ppid=24350 vsize=33308180 CPUtime=4925.22 cores=0-31
/proc/24352/task/24389/stat : 24389 (plingeling) R 24350 24352 24326 0 -1 4202560 1016999 1250 0 0 491184 1338 0 0 25 0 33 0 673632800 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=24352/tid=24390] ppid=24350 vsize=33308180 CPUtime=4924.47 cores=0-31
/proc/24352/task/24390/stat : 24390 (plingeling) R 24350 24352 24326 0 -1 4202560 1272691 1250 0 0 490473 1974 0 0 25 0 33 0 673632800 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=24352/tid=24391] ppid=24350 vsize=33308180 CPUtime=4927.43 cores=0-31
/proc/24352/task/24391/stat : 24391 (plingeling) R 24350 24352 24326 0 -1 4202560 1539182 1250 0 0 490488 2255 0 0 25 0 33 0 673632801 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=24352/tid=24392] ppid=24350 vsize=33308180 CPUtime=4931.86 cores=0-31
/proc/24352/task/24392/stat : 24392 (plingeling) R 24350 24352 24326 0 -1 4202560 933862 1250 0 0 492149 1037 0 0 25 0 33 0 673632801 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=24352/tid=24393] ppid=24350 vsize=33308180 CPUtime=4925.88 cores=0-31
/proc/24352/task/24393/stat : 24393 (plingeling) R 24350 24352 24326 0 -1 4202560 1320722 1250 0 0 490795 1793 0 0 25 0 33 0 673632807 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=24352/tid=24394] ppid=24350 vsize=33308180 CPUtime=4919.51 cores=0-31
/proc/24352/task/24394/stat : 24394 (plingeling) R 24350 24352 24326 0 -1 4202560 1785255 1250 0 0 489257 2694 0 0 25 0 33 0 673632807 34107576320 8097883 268435456000 4194304 4417452 140733738525984 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 157763
Current children cumulated vsize (KiB) 33308180

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.7 s]
/proc/loadavg: 32.11 32.09 32.01 2/416 24539
/proc/meminfo: memFree=231491936/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=0 CPUtime=157764 cores=0-31
/proc/24352/stat : 24352 (plingeling) Z 24350 24352 24326 0 -1 4203548 44468133 1250 0 0 15716867 59483 0 0 18 0 2 0 673617286 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 27 0 0 0
/proc/24352/statm: 0 0 0 0 0 0 0
[pid=24352/tid=24369] ppid=24350 vsize=0 CPUtime=4934.32 cores=0-31
/proc/24352/task/24369/stat : 24369 (plingeling) R 24350 24352 24326 0 -1 4203588 1568820 1250 0 0 491093 2339 0 0 25 0 2 0 673632774 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 157764
Current children cumulated vsize (KiB) 0

[startup+5101.5 s]
/proc/loadavg: 32.11 32.09 32.01 2/416 24539
/proc/meminfo: memFree=231491936/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=0 CPUtime=157764 cores=0-31
/proc/24352/stat : 24352 (plingeling) Z 24350 24352 24326 0 -1 4203548 44468133 1250 0 0 15716867 59561 0 0 18 0 2 0 673617286 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 27 0 0 0
/proc/24352/statm: 0 0 0 0 0 0 0
[pid=24352/tid=24369] ppid=24350 vsize=0 CPUtime=4935.1 cores=0-31
/proc/24352/task/24369/stat : 24369 (plingeling) R 24350 24352 24326 0 -1 4203588 1568820 1250 0 0 491093 2417 0 0 25 0 2 0 673632774 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 157764
Current children cumulated vsize (KiB) 0

[startup+5101.92 s]
/proc/loadavg: 29.62 31.57 31.85 2/416 24539
/proc/meminfo: memFree=252658472/264257624 swapFree=67111528/67111528
[pid=24352] ppid=24350 vsize=0 CPUtime=157765 cores=0-31
/proc/24352/stat : 24352 (plingeling) Z 24350 24352 24326 0 -1 4203548 44468133 1250 0 0 15716867 59603 0 0 18 0 2 0 673617286 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 27 0 0 0
/proc/24352/statm: 0 0 0 0 0 0 0
[pid=24352/tid=24369] ppid=24350 vsize=0 CPUtime=4935.52 cores=0-31
/proc/24352/task/24369/stat : 24369 (plingeling) R 24350 24352 24326 0 -1 4203588 1568820 1250 0 0 491093 2459 0 0 25 0 2 0 673632774 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 157765
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5101.95
CPU time (s): 157765
CPU user time (s): 157169
CPU system time (s): 596.064
CPU usage (%): 3092.25
Max. virtual memory (cumulated for all children) (KiB): 33956488

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 157169
system time used= 596.064
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 44469383
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17471
involuntary context switches= 405457

runsolver used 26.531 second user time and 118.501 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-17 10:35:09
IDJOB=3372450
IDBENCH=82388
IDSOLVER=1852
FILE ID=node200/3372450-1305621302
RUNJOBID= node200-1305621301-24340
PBS_JOBID= 13285566
Free space on /tmp= 61884 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/transport-transport-city-sequential-35nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.050-SAT.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372450-1305621302/watcher-3372450-1305621302 -o /tmp/evaluation-result-3372450-1305621302/solver-3372450-1305621302 -C 160000 -W 5100 -M 266338304  plingeling -t 32 HOME/instance-3372450-1305621302.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 48f03e5fbe31fc85b9a464480fc8aa5a
RANDOM SEED=1285658473

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255359168 kB
Buffers:        448652 kB
Cached:        8048948 kB
SwapCached:          0 kB
Active:        7565864 kB
Inactive:      1000564 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255359168 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:          171528 kB
Writeback:           0 kB
AnonPages:       68736 kB
Mapped:          14480 kB
Slab:           253340 kB
PageTables:       4376 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   188148 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61884 MiB
End job on node200 at 2011-05-17 12:00:11