Trace number 3372438

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 160002 5028.85

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.050-NOTKNOWN.cnf
MD5SUM6ed645315c9ffb8177453b955fbe1737
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark36639.2
Satisfiable
(Un)Satisfiability was proved
Number of variables109684
Number of clauses1767096
Sum of the clauses size4034296
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 21705200
Number of clauses of size 34800
Number of clauses of size 422200
Number of clauses of size 59600
Number of clauses of size over 524600

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372438-1305607495/watcher-3372438-1305607495 -o /tmp/evaluation-result-3372438-1305607495/solver-3372438-1305607495 -C 160000 -W 5100 -M 266338304 plingeling -t 32 HOME/instance-3372438-1305607495.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 29.57 31.61 31.87 1/413 23590
/proc/meminfo: memFree=255502032/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=17368 CPUtime=0.02 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 2420 1250 0 0 2 0 0 0 22 0 1 0 672236067 17784832 2350 268435456000 4194304 4417452 140733252056464 18446744073709551615 258650955760 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 4342 2350 141 55 0 2228 0

[startup+0.114255 s]
/proc/loadavg: 29.57 31.61 31.87 1/413 23590
/proc/meminfo: memFree=255502032/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=24812 CPUtime=0.09 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 4292 1250 0 0 9 0 0 0 22 0 1 0 672236067 25407488 4222 268435456000 4194304 4417452 140733252056464 18446744073709551615 258650955760 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 6203 4222 144 55 0 4089 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 24812

[startup+0.201233 s]
/proc/loadavg: 29.57 31.61 31.87 1/413 23590
/proc/meminfo: memFree=255502032/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=34800 CPUtime=0.18 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 6791 1250 0 0 17 1 0 0 22 0 1 0 672236067 35635200 6721 268435456000 4194304 4417452 140733252056464 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 8700 6721 146 55 0 6586 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 34800

[startup+0.301228 s]
/proc/loadavg: 29.57 31.61 31.87 1/413 23590
/proc/meminfo: memFree=255502032/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=34800 CPUtime=0.28 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 6791 1250 0 0 27 1 0 0 23 0 1 0 672236067 35635200 6721 268435456000 4194304 4417452 140733252056464 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 8700 6721 146 55 0 6586 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 34800

[startup+0.701273 s]
/proc/loadavg: 29.57 31.61 31.87 1/413 23590
/proc/meminfo: memFree=255502032/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=59504 CPUtime=0.69 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 12951 1250 0 0 66 3 0 0 25 0 1 0 672236067 60932096 12881 268435456000 4194304 4417452 140733252056464 18446744073709551615 4264909 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 14876 12881 146 55 0 12762 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 59504

[startup+1.5003 s]
/proc/loadavg: 29.57 31.61 31.87 2/414 23596
/proc/meminfo: memFree=255445572/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=116396 CPUtime=1.49 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 27166 1250 0 0 143 6 0 0 25 0 1 0 672236067 119189504 27096 268435456000 4194304 4417452 140733252056464 18446744073709551615 4321587 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 29099 27096 146 55 0 26985 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 116396

[startup+3.1004 s]
/proc/loadavg: 29.57 31.61 31.87 2/414 23596
/proc/meminfo: memFree=255363576/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=199076 CPUtime=3.08 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 47884 1250 0 0 297 11 0 0 25 0 1 0 672236067 203853824 47790 268435456000 4194304 4417452 140733252056464 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 49769 47790 146 55 0 47655 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 199076

[startup+6.30048 s]
/proc/loadavg: 27.28 31.10 31.70 2/414 23596
/proc/meminfo: memFree=255161748/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=478600 CPUtime=6.28 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 117797 1250 0 0 602 26 0 0 25 0 1 0 672236067 490086400 117671 268435456000 4194304 4417452 140733252056464 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 119650 117671 146 55 0 117536 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 478600

[startup+12.7007 s]
/proc/loadavg: 25.18 30.60 31.54 2/414 23596
/proc/meminfo: memFree=254586060/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=924360 CPUtime=12.68 cores=0-31
/proc/23590/stat : 23590 (plingeling) R 23588 23590 23564 0 -1 4202496 229237 1250 0 0 1220 48 0 0 25 0 1 0 672236067 946544640 229111 268435456000 4194304 4417452 140733252056464 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23590/statm: 231090 229111 146 55 0 228976 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 924360

[startup+25.502 s]
/proc/loadavg: 22.30 29.67 31.21 33/446 23628
/proc/meminfo: memFree=253982300/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=1945728 CPUtime=235.1 cores=0-31
/proc/23590/stat : 23590 (plingeling) S 23588 23590 23564 0 -1 4202496 706970 1250 0 0 23292 218 0 0 19 0 33 0 672236067 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 486432 370595 175 55 0 484318 0
[pid=23590/tid=23597] ppid=23588 vsize=1945728 CPUtime=6.87 cores=0-31
/proc/23590/task/23597/stat : 23597 (plingeling) R 23588 23590 23564 0 -1 4202560 11322 1250 0 0 684 3 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4235981 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23590/tid=23598] ppid=23588 vsize=1945728 CPUtime=6.88 cores=0-31
/proc/23590/task/23598/stat : 23598 (plingeling) R 23588 23590 23564 0 -1 4202560 7099 1250 0 0 684 4 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4330612 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23590/tid=23599] ppid=23588 vsize=1945728 CPUtime=6.72 cores=0-31
/proc/23590/task/23599/stat : 23599 (plingeling) R 23588 23590 23564 0 -1 4202560 12141 1250 0 0 668 4 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4258627 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23590/tid=23600] ppid=23588 vsize=1945728 CPUtime=6.72 cores=0-31
/proc/23590/task/23600/stat : 23600 (plingeling) R 23588 23590 23564 0 -1 4202560 1714 1250 0 0 671 1 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4365135 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23590/tid=23601] ppid=23588 vsize=1945728 CPUtime=6.85 cores=0-31
/proc/23590/task/23601/stat : 23601 (plingeling) R 23588 23590 23564 0 -1 4202560 13888 1250 0 0 681 4 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23590/tid=23602] ppid=23588 vsize=1945728 CPUtime=6.87 cores=0-31
/proc/23590/task/23602/stat : 23602 (plingeling) R 23588 23590 23564 0 -1 4202560 15343 1250 0 0 683 4 0 0 20 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4330445 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23590/tid=23603] ppid=23588 vsize=1945728 CPUtime=6.81 cores=0-31
/proc/23590/task/23603/stat : 23603 (plingeling) R 23588 23590 23564 0 -1 4202560 10848 1250 0 0 677 4 0 0 23 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4258188 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23590/tid=23604] ppid=23588 vsize=1945728 CPUtime=6.86 cores=0-31
/proc/23590/task/23604/stat : 23604 (plingeling) R 23588 23590 23564 0 -1 4202560 8069 1250 0 0 684 2 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331733 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23590/tid=23605] ppid=23588 vsize=1945728 CPUtime=6.77 cores=0-31
/proc/23590/task/23605/stat : 23605 (plingeling) R 23588 23590 23564 0 -1 4202560 12380 1250 0 0 672 5 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4253440 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23590/tid=23606] ppid=23588 vsize=1945728 CPUtime=6.88 cores=0-31
/proc/23590/task/23606/stat : 23606 (plingeling) R 23588 23590 23564 0 -1 4202560 13204 1250 0 0 683 5 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4211504 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23590/tid=23607] ppid=23588 vsize=1945728 CPUtime=6.65 cores=0-31
/proc/23590/task/23607/stat : 23607 (plingeling) R 23588 23590 23564 0 -1 4202560 10223 1250 0 0 662 3 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236255 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23590/tid=23608] ppid=23588 vsize=1945728 CPUtime=6.87 cores=0-31
/proc/23590/task/23608/stat : 23608 (plingeling) R 23588 23590 23564 0 -1 4202560 6383 1250 0 0 684 3 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4251958 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23590/tid=23609] ppid=23588 vsize=1945728 CPUtime=6.75 cores=0-31
/proc/23590/task/23609/stat : 23609 (plingeling) R 23588 23590 23564 0 -1 4202560 8445 1250 0 0 672 3 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4253830 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23590/tid=23610] ppid=23588 vsize=1945728 CPUtime=6.76 cores=0-31
/proc/23590/task/23610/stat : 23610 (plingeling) R 23588 23590 23564 0 -1 4202560 6081 1250 0 0 673 3 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4254123 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23590/tid=23611] ppid=23588 vsize=1945728 CPUtime=6.69 cores=0-31
/proc/23590/task/23611/stat : 23611 (plingeling) R 23588 23590 23564 0 -1 4202560 12192 1250 0 0 665 4 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236461 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23590/tid=23612] ppid=23588 vsize=1945728 CPUtime=6.64 cores=0-31
/proc/23590/task/23612/stat : 23612 (plingeling) R 23588 23590 23564 0 -1 4202560 8823 1250 0 0 661 3 0 0 25 0 33 0 672237919 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23590/tid=23613] ppid=23588 vsize=1945728 CPUtime=6.79 cores=0-31
/proc/23590/task/23613/stat : 23613 (plingeling) R 23588 23590 23564 0 -1 4202560 14909 1250 0 0 675 4 0 0 21 0 33 0 672237920 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=23590/tid=23614] ppid=23588 vsize=1945728 CPUtime=6.78 cores=0-31
/proc/23590/task/23614/stat : 23614 (plingeling) R 23588 23590 23564 0 -1 4202560 13351 1250 0 0 674 4 0 0 25 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4330515 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23590/tid=23615] ppid=23588 vsize=1945728 CPUtime=6.67 cores=0-31
/proc/23590/task/23615/stat : 23615 (plingeling) R 23588 23590 23564 0 -1 4202560 6080 1250 0 0 666 1 0 0 22 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236008 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23590/tid=23616] ppid=23588 vsize=1945728 CPUtime=6.83 cores=0-31
/proc/23590/task/23616/stat : 23616 (plingeling) R 23588 23590 23564 0 -1 4202560 7379 1250 0 0 680 3 0 0 25 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331729 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23590/tid=23617] ppid=23588 vsize=1945728 CPUtime=6.56 cores=0-31
/proc/23590/task/23617/stat : 23617 (plingeling) R 23588 23590 23564 0 -1 4202560 15195 1250 0 0 652 4 0 0 20 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4330289 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=1945728 CPUtime=6.74 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4202560 6439 1250 0 0 672 2 0 0 25 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331715 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23590/tid=23619] ppid=23588 vsize=1945728 CPUtime=6.73 cores=0-31
/proc/23590/task/23619/stat : 23619 (plingeling) R 23588 23590 23564 0 -1 4202560 8881 1250 0 0 671 2 0 0 25 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4335976 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23590/tid=23620] ppid=23588 vsize=1945728 CPUtime=6.72 cores=0-31
/proc/23590/task/23620/stat : 23620 (plingeling) R 23588 23590 23564 0 -1 4202560 14118 1250 0 0 667 5 0 0 25 0 33 0 672237921 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4330612 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23590/tid=23621] ppid=23588 vsize=1945728 CPUtime=6.76 cores=0-31
/proc/23590/task/23621/stat : 23621 (plingeling) R 23588 23590 23564 0 -1 4202560 9446 1250 0 0 672 4 0 0 25 0 33 0 672237922 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4233516 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23590/tid=23622] ppid=23588 vsize=1945728 CPUtime=6.73 cores=0-31
/proc/23590/task/23622/stat : 23622 (plingeling) R 23588 23590 23564 0 -1 4202560 11456 1250 0 0 670 3 0 0 25 0 33 0 672237922 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331822 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23590/tid=23623] ppid=23588 vsize=1945728 CPUtime=6.68 cores=0-31
/proc/23590/task/23623/stat : 23623 (plingeling) R 23588 23590 23564 0 -1 4202560 12229 1250 0 0 659 9 0 0 22 0 33 0 672237922 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236355 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23590/tid=23624] ppid=23588 vsize=1945728 CPUtime=6.74 cores=0-31
/proc/23590/task/23624/stat : 23624 (plingeling) R 23588 23590 23564 0 -1 4202560 8964 1250 0 0 671 3 0 0 25 0 33 0 672237922 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23590/tid=23625] ppid=23588 vsize=1945728 CPUtime=6.77 cores=0-31
/proc/23590/task/23625/stat : 23625 (plingeling) R 23588 23590 23564 0 -1 4202560 6119 1250 0 0 674 3 0 0 25 0 33 0 672237922 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236296 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23590/tid=23626] ppid=23588 vsize=1945728 CPUtime=6.72 cores=0-31
/proc/23590/task/23626/stat : 23626 (plingeling) R 23588 23590 23564 0 -1 4202560 13478 1250 0 0 668 4 0 0 21 0 33 0 672237923 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23590/tid=23627] ppid=23588 vsize=1945728 CPUtime=6.72 cores=0-31
/proc/23590/task/23627/stat : 23627 (plingeling) R 23588 23590 23564 0 -1 4202560 12238 1250 0 0 660 12 0 0 23 0 33 0 672237923 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4235999 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23590/tid=23628] ppid=23588 vsize=1945728 CPUtime=6.71 cores=0-31
/proc/23590/task/23628/stat : 23628 (plingeling) R 23588 23590 23564 0 -1 4202560 11802 1250 0 0 667 4 0 0 25 0 33 0 672237923 1992425472 370595 268435456000 4194304 4417452 140733252056464 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 235.1
Current children cumulated vsize (KiB) 1945728

[startup+51.1019 s]
/proc/loadavg: 25.61 29.85 31.23 33/446 23630
/proc/meminfo: memFree=253946580/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=2010604 CPUtime=1053.37 cores=0-31
/proc/23590/stat : 23590 (plingeling) S 23588 23590 23564 0 -1 4202496 730287 1250 0 0 105113 224 0 0 19 0 33 0 672236067 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 502651 387747 175 55 0 500537 0
[pid=23590/tid=23597] ppid=23588 vsize=2010604 CPUtime=32.47 cores=0-31
/proc/23590/task/23597/stat : 23597 (plingeling) R 23588 23590 23564 0 -1 4202560 12570 1250 0 0 3243 4 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4264942 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23590/tid=23598] ppid=23588 vsize=2010604 CPUtime=32.47 cores=0-31
/proc/23590/task/23598/stat : 23598 (plingeling) R 23588 23590 23564 0 -1 4202560 7223 1250 0 0 3243 4 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23590/tid=23599] ppid=23588 vsize=2010604 CPUtime=32.31 cores=0-31
/proc/23590/task/23599/stat : 23599 (plingeling) R 23588 23590 23564 0 -1 4202560 13024 1250 0 0 3227 4 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23590/tid=23600] ppid=23588 vsize=2010604 CPUtime=32.32 cores=0-31
/proc/23590/task/23600/stat : 23600 (plingeling) R 23588 23590 23564 0 -1 4202560 6273 1250 0 0 3230 2 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23590/tid=23601] ppid=23588 vsize=2010604 CPUtime=32.44 cores=0-31
/proc/23590/task/23601/stat : 23601 (plingeling) R 23588 23590 23564 0 -1 4202560 14273 1250 0 0 3240 4 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4250505 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23590/tid=23602] ppid=23588 vsize=2010604 CPUtime=32.46 cores=0-31
/proc/23590/task/23602/stat : 23602 (plingeling) R 23588 23590 23564 0 -1 4202560 15943 1250 0 0 3242 4 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236852 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23590/tid=23603] ppid=23588 vsize=2010604 CPUtime=32.41 cores=0-31
/proc/23590/task/23603/stat : 23603 (plingeling) R 23588 23590 23564 0 -1 4202560 11246 1250 0 0 3237 4 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23590/tid=23604] ppid=23588 vsize=2010604 CPUtime=32.46 cores=0-31
/proc/23590/task/23604/stat : 23604 (plingeling) R 23588 23590 23564 0 -1 4202560 8497 1250 0 0 3243 3 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4232584 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23590/tid=23605] ppid=23588 vsize=2010604 CPUtime=32.36 cores=0-31
/proc/23590/task/23605/stat : 23605 (plingeling) R 23588 23590 23564 0 -1 4202560 13491 1250 0 0 3230 6 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23590/tid=23606] ppid=23588 vsize=2010604 CPUtime=32.46 cores=0-31
/proc/23590/task/23606/stat : 23606 (plingeling) R 23588 23590 23564 0 -1 4202560 15150 1250 0 0 3241 5 0 0 25 0 33 0 672237919 2058858496 387747 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23590/tid=23607] ppid=23588 vsize=2010604 CPUtime=32.25 cores=0-31

################
# More data... #
################

[pid=23590/tid=23608] ppid=23588 vsize=18438160 CPUtime=4991.18 cores=0-31
/proc/23590/task/23608/stat : 23608 (plingeling) R 23588 23590 23564 0 -1 4202560 690146 1250 0 0 498570 548 0 0 25 0 33 0 672237919 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23590/tid=23609] ppid=23588 vsize=18438160 CPUtime=4999.2 cores=0-31
/proc/23590/task/23609/stat : 23609 (plingeling) R 23588 23590 23564 0 -1 4202560 628589 1250 0 0 499515 405 0 0 25 0 33 0 672237919 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23590/tid=23610] ppid=23588 vsize=18438160 CPUtime=4991.26 cores=0-31
/proc/23590/task/23610/stat : 23610 (plingeling) R 23588 23590 23564 0 -1 4202560 1174750 1250 0 0 497213 1913 0 0 25 0 33 0 672237919 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357554 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23590/tid=23611] ppid=23588 vsize=18438160 CPUtime=4998.64 cores=0-31
/proc/23590/task/23611/stat : 23611 (plingeling) R 23588 23590 23564 0 -1 4202560 511850 1250 0 0 499549 315 0 0 25 0 33 0 672237919 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23590/tid=23612] ppid=23588 vsize=18438160 CPUtime=4992.31 cores=0-31
/proc/23590/task/23612/stat : 23612 (plingeling) R 23588 23590 23564 0 -1 4202560 388566 1250 0 0 498926 305 0 0 25 0 33 0 672237919 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4355835 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23590/tid=23613] ppid=23588 vsize=18438160 CPUtime=4996.82 cores=0-31
/proc/23590/task/23613/stat : 23613 (plingeling) R 23588 23590 23564 0 -1 4202560 340716 1250 0 0 499510 172 0 0 25 0 33 0 672237920 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4321595 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23590/tid=23614] ppid=23588 vsize=18438160 CPUtime=4991.04 cores=0-31
/proc/23590/task/23614/stat : 23614 (plingeling) R 23588 23590 23564 0 -1 4202560 616590 1250 0 0 498523 581 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4320867 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23590/tid=23615] ppid=23588 vsize=18438160 CPUtime=4996.15 cores=0-31
/proc/23590/task/23615/stat : 23615 (plingeling) R 23588 23590 23564 0 -1 4202560 655063 1250 0 0 499033 582 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23590/tid=23616] ppid=23588 vsize=18438160 CPUtime=4998.57 cores=0-31
/proc/23590/task/23616/stat : 23616 (plingeling) R 23588 23590 23564 0 -1 4202560 583888 1250 0 0 499433 424 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23590/tid=23617] ppid=23588 vsize=18438160 CPUtime=4986.1 cores=0-31
/proc/23590/task/23617/stat : 23617 (plingeling) R 23588 23590 23564 0 -1 4202560 531216 1250 0 0 498215 395 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=18438160 CPUtime=4991.29 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4202560 943581 1250 0 0 498036 1093 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357248 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23590/tid=23619] ppid=23588 vsize=18438160 CPUtime=4979.97 cores=0-31
/proc/23590/task/23619/stat : 23619 (plingeling) R 23588 23590 23564 0 -1 4202560 556170 1250 0 0 497613 384 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23590/tid=23620] ppid=23588 vsize=18438160 CPUtime=4999.3 cores=0-31
/proc/23590/task/23620/stat : 23620 (plingeling) R 23588 23590 23564 0 -1 4202560 653622 1250 0 0 499290 640 0 0 25 0 33 0 672237921 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4366523 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23590/tid=23621] ppid=23588 vsize=18438160 CPUtime=4994.07 cores=0-31
/proc/23590/task/23621/stat : 23621 (plingeling) R 23588 23590 23564 0 -1 4202560 802279 1250 0 0 498382 1025 0 0 25 0 33 0 672237922 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23590/tid=23622] ppid=23588 vsize=18438160 CPUtime=4997.24 cores=0-31
/proc/23590/task/23622/stat : 23622 (plingeling) R 23588 23590 23564 0 -1 4202560 461561 1250 0 0 499207 517 0 0 25 0 33 0 672237922 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23590/tid=23623] ppid=23588 vsize=18438160 CPUtime=4994.76 cores=0-31
/proc/23590/task/23623/stat : 23623 (plingeling) R 23588 23590 23564 0 -1 4202560 503482 1250 0 0 499089 387 0 0 25 0 33 0 672237922 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4312581 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23590/tid=23624] ppid=23588 vsize=18438160 CPUtime=4999.32 cores=0-31
/proc/23590/task/23624/stat : 23624 (plingeling) R 23588 23590 23564 0 -1 4202560 1128409 1250 0 0 499250 682 0 0 25 0 33 0 672237922 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23590/tid=23625] ppid=23588 vsize=18438160 CPUtime=4992.02 cores=0-31
/proc/23590/task/23625/stat : 23625 (plingeling) R 23588 23590 23564 0 -1 4202560 557282 1250 0 0 498762 440 0 0 25 0 33 0 672237922 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4250569 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23590/tid=23626] ppid=23588 vsize=18438160 CPUtime=4993.58 cores=0-31
/proc/23590/task/23626/stat : 23626 (plingeling) R 23588 23590 23564 0 -1 4202560 1628429 1250 0 0 498102 1256 0 0 25 0 33 0 672237923 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364780 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23590/tid=23627] ppid=23588 vsize=18438160 CPUtime=4999.93 cores=0-31
/proc/23590/task/23627/stat : 23627 (plingeling) R 23588 23590 23564 0 -1 4202560 1021438 1250 0 0 498561 1432 0 0 25 0 33 0 672237923 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23590/tid=23628] ppid=23588 vsize=18438160 CPUtime=4984.08 cores=0-31
/proc/23590/task/23628/stat : 23628 (plingeling) R 23588 23590 23564 0 -1 4202560 825597 1250 0 0 497539 869 0 0 25 0 33 0 672237923 18880675840 4485931 268435456000 4194304 4417452 140733252056464 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 25 0 0 0
Current children cumulated CPU time (s) 159831
Current children cumulated vsize (KiB) 18438160



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5027.6 s]
/proc/loadavg: 32.03 32.06 32.02 33/446 23783
/proc/meminfo: memFree=237503952/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=18463380 CPUtime=160001 cores=0-31
/proc/23590/stat : 23590 (plingeling) S 23588 23590 23564 0 -1 4202496 24709693 1250 0 0 15976167 23914 0 0 19 0 33 0 672236067 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 4615845 4492236 175 55 0 4613731 0
[pid=23590/tid=23597] ppid=23588 vsize=18463380 CPUtime=4994.85 cores=0-31
/proc/23590/task/23597/stat : 23597 (plingeling) R 23588 23590 23564 0 -1 4202560 593373 1250 0 0 499025 460 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23590/tid=23598] ppid=23588 vsize=18463380 CPUtime=5003.14 cores=0-31
/proc/23590/task/23598/stat : 23598 (plingeling) R 23588 23590 23564 0 -1 4202560 587360 1250 0 0 499869 445 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23590/tid=23599] ppid=23588 vsize=18463380 CPUtime=4998.39 cores=0-31
/proc/23590/task/23599/stat : 23599 (plingeling) R 23588 23590 23564 0 -1 4202560 810754 1250 0 0 499104 735 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23590/tid=23600] ppid=23588 vsize=18463380 CPUtime=5004.75 cores=0-31
/proc/23590/task/23600/stat : 23600 (plingeling) R 23588 23590 23564 0 -1 4202560 1700067 1250 0 0 497739 2736 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23590/tid=23601] ppid=23588 vsize=18463380 CPUtime=4999.25 cores=0-31
/proc/23590/task/23601/stat : 23601 (plingeling) R 23588 23590 23564 0 -1 4202560 648800 1250 0 0 499412 513 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4250765 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23590/tid=23602] ppid=23588 vsize=18463380 CPUtime=5001.58 cores=0-31
/proc/23590/task/23602/stat : 23602 (plingeling) R 23588 23590 23564 0 -1 4202560 1063844 1250 0 0 499006 1152 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23590/tid=23603] ppid=23588 vsize=18463380 CPUtime=4992.24 cores=0-31
/proc/23590/task/23603/stat : 23603 (plingeling) R 23588 23590 23564 0 -1 4202560 588166 1250 0 0 498648 576 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23590/tid=23604] ppid=23588 vsize=18463380 CPUtime=5001.9 cores=0-31
/proc/23590/task/23604/stat : 23604 (plingeling) R 23588 23590 23564 0 -1 4202560 733307 1250 0 0 499414 776 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357554 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23590/tid=23605] ppid=23588 vsize=18463380 CPUtime=5002.02 cores=0-31
/proc/23590/task/23605/stat : 23605 (plingeling) R 23588 23590 23564 0 -1 4202560 758621 1250 0 0 499638 564 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357544 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23590/tid=23606] ppid=23588 vsize=18463380 CPUtime=5004.56 cores=0-31
/proc/23590/task/23606/stat : 23606 (plingeling) R 23588 23590 23564 0 -1 4202560 932932 1250 0 0 499451 1005 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23590/tid=23607] ppid=23588 vsize=18463380 CPUtime=5001 cores=0-31
/proc/23590/task/23607/stat : 23607 (plingeling) R 23588 23590 23564 0 -1 4202560 710919 1250 0 0 499606 494 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357297 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23590/tid=23608] ppid=23588 vsize=18463380 CPUtime=4996.5 cores=0-31
/proc/23590/task/23608/stat : 23608 (plingeling) R 23588 23590 23564 0 -1 4202560 690146 1250 0 0 499102 548 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357714 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23590/tid=23609] ppid=23588 vsize=18463380 CPUtime=5004.52 cores=0-31
/proc/23590/task/23609/stat : 23609 (plingeling) R 23588 23590 23564 0 -1 4202560 628589 1250 0 0 500047 405 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357554 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23590/tid=23610] ppid=23588 vsize=18463380 CPUtime=4996.57 cores=0-31
/proc/23590/task/23610/stat : 23610 (plingeling) R 23588 23590 23564 0 -1 4202560 1174750 1250 0 0 497744 1913 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23590/tid=23611] ppid=23588 vsize=18463380 CPUtime=5003.96 cores=0-31
/proc/23590/task/23611/stat : 23611 (plingeling) R 23588 23590 23564 0 -1 4202560 511850 1250 0 0 500081 315 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23590/tid=23612] ppid=23588 vsize=18463380 CPUtime=4997.63 cores=0-31
/proc/23590/task/23612/stat : 23612 (plingeling) R 23588 23590 23564 0 -1 4202560 389673 1250 0 0 499458 305 0 0 25 0 33 0 672237919 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23590/tid=23613] ppid=23588 vsize=18463380 CPUtime=5001.97 cores=0-31
/proc/23590/task/23613/stat : 23613 (plingeling) R 23588 23590 23564 0 -1 4202560 341204 1250 0 0 500025 172 0 0 25 0 33 0 672237920 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4346497 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23590/tid=23614] ppid=23588 vsize=18463380 CPUtime=4996.36 cores=0-31
/proc/23590/task/23614/stat : 23614 (plingeling) R 23588 23590 23564 0 -1 4202560 616590 1250 0 0 499055 581 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23590/tid=23615] ppid=23588 vsize=18463380 CPUtime=5001.47 cores=0-31
/proc/23590/task/23615/stat : 23615 (plingeling) R 23588 23590 23564 0 -1 4202560 655063 1250 0 0 499565 582 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23590/tid=23616] ppid=23588 vsize=18463380 CPUtime=5003.89 cores=0-31
/proc/23590/task/23616/stat : 23616 (plingeling) R 23588 23590 23564 0 -1 4202560 583888 1250 0 0 499965 424 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4358025 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23590/tid=23617] ppid=23588 vsize=18463380 CPUtime=4991.42 cores=0-31
/proc/23590/task/23617/stat : 23617 (plingeling) R 23588 23590 23564 0 -1 4202560 531216 1250 0 0 498747 395 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364711 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=18463380 CPUtime=4996.61 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4202560 943581 1250 0 0 498568 1093 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23590/tid=23619] ppid=23588 vsize=18463380 CPUtime=4985.29 cores=0-31
/proc/23590/task/23619/stat : 23619 (plingeling) R 23588 23590 23564 0 -1 4202560 556170 1250 0 0 498145 384 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23590/tid=23620] ppid=23588 vsize=18463380 CPUtime=5004.62 cores=0-31
/proc/23590/task/23620/stat : 23620 (plingeling) R 23588 23590 23564 0 -1 4202560 653622 1250 0 0 499822 640 0 0 25 0 33 0 672237921 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357213 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23590/tid=23621] ppid=23588 vsize=18463380 CPUtime=4999.39 cores=0-31
/proc/23590/task/23621/stat : 23621 (plingeling) R 23588 23590 23564 0 -1 4202560 802279 1250 0 0 498914 1025 0 0 25 0 33 0 672237922 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357436 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23590/tid=23622] ppid=23588 vsize=18463380 CPUtime=5002.56 cores=0-31
/proc/23590/task/23622/stat : 23622 (plingeling) R 23588 23590 23564 0 -1 4202560 461561 1250 0 0 499739 517 0 0 25 0 33 0 672237922 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4237290 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23590/tid=23623] ppid=23588 vsize=18463380 CPUtime=5000.08 cores=0-31
/proc/23590/task/23623/stat : 23623 (plingeling) R 23588 23590 23564 0 -1 4202560 503482 1250 0 0 499621 387 0 0 25 0 33 0 672237922 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4346458 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23590/tid=23624] ppid=23588 vsize=18463380 CPUtime=5004.64 cores=0-31
/proc/23590/task/23624/stat : 23624 (plingeling) R 23588 23590 23564 0 -1 4202560 1128409 1250 0 0 499782 682 0 0 25 0 33 0 672237922 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23590/tid=23625] ppid=23588 vsize=18463380 CPUtime=4997.33 cores=0-31
/proc/23590/task/23625/stat : 23625 (plingeling) R 23588 23590 23564 0 -1 4202560 557282 1250 0 0 499293 440 0 0 25 0 33 0 672237922 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23590/tid=23626] ppid=23588 vsize=18463380 CPUtime=4998.9 cores=0-31
/proc/23590/task/23626/stat : 23626 (plingeling) R 23588 23590 23564 0 -1 4202560 1628429 1250 0 0 498634 1256 0 0 25 0 33 0 672237923 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4355852 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23590/tid=23627] ppid=23588 vsize=18463380 CPUtime=5005.25 cores=0-31
/proc/23590/task/23627/stat : 23627 (plingeling) R 23588 23590 23564 0 -1 4202560 1021438 1250 0 0 499093 1432 0 0 25 0 33 0 672237923 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23590/tid=23628] ppid=23588 vsize=18463380 CPUtime=4989.4 cores=0-31
/proc/23590/task/23628/stat : 23628 (plingeling) R 23588 23590 23564 0 -1 4202560 825597 1250 0 0 498071 869 0 0 25 0 33 0 672237923 18906501120 4492236 268435456000 4194304 4417452 140733252056464 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 25 0 0 0
Current children cumulated CPU time (s) 160001
Current children cumulated vsize (KiB) 18463380

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5027.9 s]
/proc/loadavg: 32.03 32.06 32.02 33/446 23783
/proc/meminfo: memFree=237503952/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=0 CPUtime=160001 cores=0-31
/proc/23590/stat : 23590 (plingeling) Z 23588 23590 23564 0 -1 4203548 24709693 1250 0 0 15976189 23941 0 0 18 0 2 0 672236067 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 0 0 0 0 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=0 CPUtime=4996.9 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4203588 943581 1250 0 0 498569 1121 0 0 25 0 2 0 672237921 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 160001
Current children cumulated vsize (KiB) 0

[startup+5028.3 s]
/proc/loadavg: 32.03 32.06 32.02 2/416 23784
/proc/meminfo: memFree=243321360/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=0 CPUtime=160002 cores=0-31
/proc/23590/stat : 23590 (plingeling) Z 23588 23590 23564 0 -1 4203548 24709693 1250 0 0 15976189 23981 0 0 18 0 2 0 672236067 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 0 0 0 0 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=0 CPUtime=4997.3 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4203588 943581 1250 0 0 498569 1161 0 0 25 0 2 0 672237921 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 0

[startup+5028.7 s]
/proc/loadavg: 32.03 32.06 32.02 2/416 23784
/proc/meminfo: memFree=243321360/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=0 CPUtime=160002 cores=0-31
/proc/23590/stat : 23590 (plingeling) Z 23588 23590 23564 0 -1 4203548 24709693 1250 0 0 15976189 24021 0 0 18 0 2 0 672236067 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 0 0 0 0 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=0 CPUtime=4997.7 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4203588 943581 1250 0 0 498569 1201 0 0 25 0 2 0 672237921 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 0

[startup+5028.8 s]
/proc/loadavg: 32.03 32.06 32.02 2/416 23784
/proc/meminfo: memFree=243321360/264257624 swapFree=67111528/67111528
[pid=23590] ppid=23588 vsize=0 CPUtime=160002 cores=0-31
/proc/23590/stat : 23590 (plingeling) Z 23588 23590 23564 0 -1 4203548 24709693 1250 0 0 15976189 24031 0 0 18 0 2 0 672236067 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/23590/statm: 0 0 0 0 0 0 0
[pid=23590/tid=23618] ppid=23588 vsize=0 CPUtime=4997.8 cores=0-31
/proc/23590/task/23618/stat : 23618 (plingeling) R 23588 23590 23564 0 -1 4203588 943581 1250 0 0 498569 1211 0 0 25 0 2 0 672237921 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5028.85
CPU time (s): 160002
CPU user time (s): 159762
CPU system time (s): 240.372
CPU usage (%): 3181.68
Max. virtual memory (cumulated for all children) (KiB): 19306092

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159762
system time used= 240.372
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 24710943
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 15910
involuntary context switches= 401577

runsolver used 27.7828 second user time and 114.211 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-17 06:44:57
IDJOB=3372438
IDBENCH=82379
IDSOLVER=1852
FILE ID=node200/3372438-1305607495
RUNJOBID= node200-1305607495-23578
PBS_JOBID= 13285562
Free space on /tmp= 62020 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.050-NOTKNOWN.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372438-1305607495/watcher-3372438-1305607495 -o /tmp/evaluation-result-3372438-1305607495/solver-3372438-1305607495 -C 160000 -W 5100 -M 266338304  plingeling -t 32 HOME/instance-3372438-1305607495.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 6ed645315c9ffb8177453b955fbe1737
RANDOM SEED=2040815588

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255502180 kB
Buffers:        448184 kB
Cached:        7905820 kB
SwapCached:          0 kB
Active:        7424196 kB
Inactive:       998412 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255502180 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           30432 kB
Writeback:           0 kB
AnonPages:       68876 kB
Mapped:          14480 kB
Slab:           251956 kB
PageTables:       4444 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   186992 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62020 MiB
End job on node200 at 2011-05-17 08:08:46