Trace number 3372462

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 160003 5039.63

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.080-SAT.cnf
MD5SUM1ca09a2b911e29894c6603aee8d3e672
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark26196.5
Satisfiable
(Un)Satisfiability was proved
Number of variables175084
Number of clauses2826936
Sum of the clauses size6454456
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22728320
Number of clauses of size 37680
Number of clauses of size 435520
Number of clauses of size 515360
Number of clauses of size over 539360

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372462-1305641720/watcher-3372462-1305641720 -o /tmp/evaluation-result-3372462-1305641720/solver-3372462-1305641720 -C 160000 -W 5100 -M 266338304 plingeling -t 32 HOME/instance-3372462-1305641720.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 22.98 29.95 31.32 1/413 25313
/proc/meminfo: memFree=255475844/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=18124 CPUtime=0.02 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 2609 1250 0 0 2 0 0 0 21 0 1 0 675658676 18558976 2539 268435456000 4194304 4417452 140734233912528 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 4531 2539 141 55 0 2417 0

[startup+0.107834 s]
/proc/loadavg: 22.98 29.95 31.32 1/413 25313
/proc/meminfo: memFree=255475844/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=25568 CPUtime=0.09 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 4481 1250 0 0 8 1 0 0 21 0 1 0 675658676 26181632 4411 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236649 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 6392 4411 144 55 0 4278 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 25568

[startup+0.200829 s]
/proc/loadavg: 22.98 29.95 31.32 1/413 25313
/proc/meminfo: memFree=255475844/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=35556 CPUtime=0.18 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 6980 1250 0 0 17 1 0 0 22 0 1 0 675658676 36409344 6910 268435456000 4194304 4417452 140734233912528 18446744073709551615 258650955760 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 8889 6910 146 55 0 6775 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 35556

[startup+0.300833 s]
/proc/loadavg: 22.98 29.95 31.32 1/413 25313
/proc/meminfo: memFree=255475844/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=35556 CPUtime=0.28 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 6980 1250 0 0 27 1 0 0 22 0 1 0 675658676 36409344 6910 268435456000 4194304 4417452 140734233912528 18446744073709551615 4227779 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 8889 6910 146 55 0 6775 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 35556

[startup+0.700852 s]
/proc/loadavg: 22.98 29.95 31.32 1/413 25313
/proc/meminfo: memFree=255475844/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=60552 CPUtime=0.69 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 13221 1250 0 0 66 3 0 0 24 0 1 0 675658676 62005248 13151 268435456000 4194304 4417452 140734233912528 18446744073709551615 258650574384 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 15138 13151 146 55 0 13024 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 60552

[startup+1.50087 s]
/proc/loadavg: 22.98 29.95 31.32 2/414 25319
/proc/meminfo: memFree=255389444/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=117888 CPUtime=1.49 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 27531 1250 0 0 143 6 0 0 25 0 1 0 675658676 120717312 27461 268435456000 4194304 4417452 140734233912528 18446744073709551615 4231373 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 29472 27461 146 55 0 27358 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 117888

[startup+3.10097 s]
/proc/loadavg: 22.98 29.95 31.32 2/414 25319
/proc/meminfo: memFree=255336868/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=210712 CPUtime=3.08 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 50614 1250 0 0 298 10 0 0 25 0 1 0 675658676 215769088 50520 268435456000 4194304 4417452 140734233912528 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 52678 50537 146 55 0 50564 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 210712

[startup+6.30104 s]
/proc/loadavg: 21.22 29.47 31.15 2/414 25319
/proc/meminfo: memFree=255064024/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=479356 CPUtime=6.28 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 117986 1250 0 0 603 25 0 0 25 0 1 0 675658676 490860544 117860 268435456000 4194304 4417452 140734233912528 18446744073709551615 4322393 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 119839 117860 146 55 0 117725 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 479356

[startup+12.7013 s]
/proc/loadavg: 19.60 28.99 30.99 2/414 25319
/proc/meminfo: memFree=254560168/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=925116 CPUtime=12.68 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 229426 1250 0 0 1221 47 0 0 25 0 1 0 675658676 947318784 229300 268435456000 4194304 4417452 140734233912528 18446744073709551615 4264940 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 231279 229300 146 55 0 229165 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 925116

[startup+25.5007 s]
/proc/loadavg: 15.48 27.62 30.51 2/414 25319
/proc/meminfo: memFree=253674904/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=1812892 CPUtime=25.48 cores=0-31
/proc/25313/stat : 25313 (plingeling) R 25311 25313 25287 0 -1 4202496 451387 1250 0 0 2454 94 0 0 25 0 1 0 675658676 1856401408 451244 268435456000 4194304 4417452 140734233912528 18446744073709551615 4259124 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/25313/statm: 453223 451244 146 55 0 451109 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 1812892

[startup+51.1025 s]
/proc/loadavg: 18.73 27.35 30.34 33/446 25353
/proc/meminfo: memFree=253650720/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=2282964 CPUtime=619.26 cores=0-31
/proc/25313/stat : 25313 (plingeling) S 25311 25313 25287 0 -1 4202496 1112604 1250 0 0 60978 948 0 0 25 0 33 0 675658676 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 30 0 0 0
/proc/25313/statm: 570741 449306 175 55 0 568627 0
[pid=25313/tid=25322] ppid=25311 vsize=2282964 CPUtime=18.29 cores=0-31
/proc/25313/task/25322/stat : 25322 (plingeling) R 25311 25313 25287 0 -1 4202560 12169 1250 0 0 1814 15 0 0 25 0 33 0 675661795 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25313/tid=25323] ppid=25311 vsize=2282964 CPUtime=18.57 cores=0-31
/proc/25313/task/25323/stat : 25323 (plingeling) R 25311 25313 25287 0 -1 4202560 22075 1250 0 0 1825 32 0 0 25 0 33 0 675661795 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=25313/tid=25324] ppid=25311 vsize=2282964 CPUtime=19.06 cores=0-31
/proc/25313/task/25324/stat : 25324 (plingeling) R 25311 25313 25287 0 -1 4202560 18662 1250 0 0 1894 12 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236797 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=25313/tid=25325] ppid=25311 vsize=2282964 CPUtime=18.89 cores=0-31
/proc/25313/task/25325/stat : 25325 (plingeling) R 25311 25313 25287 0 -1 4202560 2250 1250 0 0 1887 2 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25313/tid=25326] ppid=25311 vsize=2282964 CPUtime=18.59 cores=0-31
/proc/25313/task/25326/stat : 25326 (plingeling) R 25311 25313 25287 0 -1 4202560 24578 1250 0 0 1827 32 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366528 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25313/tid=25327] ppid=25311 vsize=2282964 CPUtime=18.4 cores=0-31
/proc/25313/task/25327/stat : 25327 (plingeling) R 25311 25313 25287 0 -1 4202560 23454 1250 0 0 1820 20 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=25313/tid=25328] ppid=25311 vsize=2282964 CPUtime=17.85 cores=0-31
/proc/25313/task/25328/stat : 25328 (plingeling) R 25311 25313 25287 0 -1 4202560 27807 1250 0 0 1753 32 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=25313/tid=25329] ppid=25311 vsize=2282964 CPUtime=18.68 cores=0-31
/proc/25313/task/25329/stat : 25329 (plingeling) R 25311 25313 25287 0 -1 4202560 20317 1250 0 0 1838 30 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=25313/tid=25330] ppid=25311 vsize=2282964 CPUtime=18.54 cores=0-31
/proc/25313/task/25330/stat : 25330 (plingeling) R 25311 25313 25287 0 -1 4202560 23349 1250 0 0 1829 25 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=25313/tid=25331] ppid=25311 vsize=2282964 CPUtime=19.08 cores=0-31
/proc/25313/task/25331/stat : 25331 (plingeling) R 25311 25313 25287 0 -1 4202560 21446 1250 0 0 1881 27 0 0 25 0 33 0 675661796 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=25313/tid=25332] ppid=25311 vsize=2282964 CPUtime=18.55 cores=0-31
/proc/25313/task/25332/stat : 25332 (plingeling) R 25311 25313 25287 0 -1 4202560 22864 1250 0 0 1818 37 0 0 25 0 33 0 675661797 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=25313/tid=25333] ppid=25311 vsize=2282964 CPUtime=18.43 cores=0-31
/proc/25313/task/25333/stat : 25333 (plingeling) R 25311 25313 25287 0 -1 4202560 12145 1250 0 0 1836 7 0 0 25 0 33 0 675661797 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=25313/tid=25334] ppid=25311 vsize=2282964 CPUtime=18.48 cores=0-31
/proc/25313/task/25334/stat : 25334 (plingeling) R 25311 25313 25287 0 -1 4202560 22199 1250 0 0 1818 30 0 0 25 0 33 0 675661797 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232612 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25313/tid=25335] ppid=25311 vsize=2282964 CPUtime=18.54 cores=0-31
/proc/25313/task/25335/stat : 25335 (plingeling) R 25311 25313 25287 0 -1 4202560 24274 1250 0 0 1823 31 0 0 25 0 33 0 675661797 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25313/tid=25336] ppid=25311 vsize=2282964 CPUtime=18.32 cores=0-31
/proc/25313/task/25336/stat : 25336 (plingeling) R 25311 25313 25287 0 -1 4202560 22445 1250 0 0 1800 32 0 0 25 0 33 0 675661797 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232595 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=25313/tid=25337] ppid=25311 vsize=2282964 CPUtime=18.08 cores=0-31
/proc/25313/task/25337/stat : 25337 (plingeling) R 25311 25313 25287 0 -1 4202560 23098 1250 0 0 1780 28 0 0 25 0 33 0 675661797 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=25313/tid=25338] ppid=25311 vsize=2282964 CPUtime=18.21 cores=0-31
/proc/25313/task/25338/stat : 25338 (plingeling) R 25311 25313 25287 0 -1 4202560 23783 1250 0 0 1789 32 0 0 25 0 33 0 675661798 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=25313/tid=25339] ppid=25311 vsize=2282964 CPUtime=18.09 cores=0-31
/proc/25313/task/25339/stat : 25339 (plingeling) R 25311 25313 25287 0 -1 4202560 25712 1250 0 0 1776 33 0 0 25 0 33 0 675661798 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=25313/tid=25340] ppid=25311 vsize=2282964 CPUtime=18.32 cores=0-31
/proc/25313/task/25340/stat : 25340 (plingeling) R 25311 25313 25287 0 -1 4202560 15708 1250 0 0 1811 21 0 0 25 0 33 0 675661798 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=25313/tid=25341] ppid=25311 vsize=2282964 CPUtime=18.28 cores=0-31
/proc/25313/task/25341/stat : 25341 (plingeling) R 25311 25313 25287 0 -1 4202560 12103 1250 0 0 1809 19 0 0 25 0 33 0 675661799 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250633 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=25313/tid=25342] ppid=25311 vsize=2282964 CPUtime=18.1 cores=0-31
/proc/25313/task/25342/stat : 25342 (plingeling) R 25311 25313 25287 0 -1 4202560 25491 1250 0 0 1775 35 0 0 25 0 33 0 675661799 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4365579 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=25313/tid=25343] ppid=25311 vsize=2282964 CPUtime=18.37 cores=0-31
/proc/25313/task/25343/stat : 25343 (plingeling) R 25311 25313 25287 0 -1 4202560 21967 1250 0 0 1810 27 0 0 25 0 33 0 675661799 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=25313/tid=25344] ppid=25311 vsize=2282964 CPUtime=18.62 cores=0-31
/proc/25313/task/25344/stat : 25344 (plingeling) R 25311 25313 25287 0 -1 4202560 20921 1250 0 0 1834 28 0 0 25 0 33 0 675661800 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=25313/tid=25345] ppid=25311 vsize=2282964 CPUtime=18.26 cores=0-31
/proc/25313/task/25345/stat : 25345 (plingeling) R 25311 25313 25287 0 -1 4202560 25484 1250 0 0 1808 18 0 0 25 0 33 0 675661800 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=25313/tid=25346] ppid=25311 vsize=2282964 CPUtime=18.25 cores=0-31
/proc/25313/task/25346/stat : 25346 (plingeling) R 25311 25313 25287 0 -1 4202560 13112 1250 0 0 1811 14 0 0 25 0 33 0 675661800 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=25313/tid=25347] ppid=25311 vsize=2282964 CPUtime=18.21 cores=0-31
/proc/25313/task/25347/stat : 25347 (plingeling) R 25311 25313 25287 0 -1 4202560 22930 1250 0 0 1783 38 0 0 25 0 33 0 675661800 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25313/tid=25348] ppid=25311 vsize=2282964 CPUtime=17.56 cores=0-31
/proc/25313/task/25348/stat : 25348 (plingeling) R 25311 25313 25287 0 -1 4202560 23421 1250 0 0 1716 40 0 0 25 0 33 0 675661801 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=25313/tid=25349] ppid=25311 vsize=2282964 CPUtime=18.32 cores=0-31
/proc/25313/task/25349/stat : 25349 (plingeling) R 25311 25313 25287 0 -1 4202560 23263 1250 0 0 1792 40 0 0 25 0 33 0 675661801 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357554 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=25313/tid=25350] ppid=25311 vsize=2282964 CPUtime=17.75 cores=0-31
/proc/25313/task/25350/stat : 25350 (plingeling) R 25311 25313 25287 0 -1 4202560 21990 1250 0 0 1759 16 0 0 25 0 33 0 675661801 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4354070 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=25313/tid=25351] ppid=25311 vsize=2282964 CPUtime=18.03 cores=0-31
/proc/25313/task/25351/stat : 25351 (plingeling) R 25311 25313 25287 0 -1 4202560 12068 1250 0 0 1794 9 0 0 25 0 33 0 675661801 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4237002 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25313/tid=25352] ppid=25311 vsize=2282964 CPUtime=18.48 cores=0-31
/proc/25313/task/25352/stat : 25352 (plingeling) R 25311 25313 25287 0 -1 4202560 24774 1250 0 0 1812 36 0 0 25 0 33 0 675661801 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25313/tid=25353] ppid=25311 vsize=2282964 CPUtime=18.6 cores=0-31
/proc/25313/task/25353/stat : 25353 (plingeling) R 25311 25313 25287 0 -1 4202560 25320 1250 0 0 1823 37 0 0 25 0 33 0 675661802 2337755136 449306 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 16 0 0 0
Current children cumulated CPU time (s) 619.26
Current children cumulated vsize (KiB) 2282964

[startup+102.302 s]
/proc/loadavg: 26.24 28.06 30.42 33/446 25353
/proc/meminfo: memFree=253620796/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=2344444 CPUtime=2256.48 cores=0-31
/proc/25313/stat : 25313 (plingeling) S 25311 25313 25287 0 -1 4202496 1128643 1250 0 0 224694 954 0 0 25 0 33 0 675658676 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 30 0 0 0
/proc/25313/statm: 586111 464677 175 55 0 583997 0
[pid=25313/tid=25322] ppid=25311 vsize=2344444 CPUtime=69.5 cores=0-31
/proc/25313/task/25322/stat : 25322 (plingeling) R 25311 25313 25287 0 -1 4202560 13234 1250 0 0 6935 15 0 0 25 0 33 0 675661795 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25313/tid=25323] ppid=25311 vsize=2344444 CPUtime=69.77 cores=0-31
/proc/25313/task/25323/stat : 25323 (plingeling) R 25311 25313 25287 0 -1 4202560 22389 1250 0 0 6945 32 0 0 25 0 33 0 675661795 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=25313/tid=25324] ppid=25311 vsize=2344444 CPUtime=70.27 cores=0-31
/proc/25313/task/25324/stat : 25324 (plingeling) R 25311 25313 25287 0 -1 4202560 18966 1250 0 0 7015 12 0 0 25 0 33 0 675661796 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=25313/tid=25325] ppid=25311 vsize=2344444 CPUtime=70.08 cores=0-31
/proc/25313/task/25325/stat : 25325 (plingeling) R 25311 25313 25287 0 -1 4202560 4161 1250 0 0 7005 3 0 0 25 0 33 0 675661796 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 4353830 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25313/tid=25326] ppid=25311 vsize=2344444 CPUtime=69.79 cores=0-31
/proc/25313/task/25326/stat : 25326 (plingeling) R 25311 25313 25287 0 -1 4202560 25086 1250 0 0 6947 32 0 0 25 0 33 0 675661796 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25313/tid=25327] ppid=25311 vsize=2344444 CPUtime=69.61 cores=0-31
/proc/25313/task/25327/stat : 25327 (plingeling) R 25311 25313 25287 0 -1 4202560 23804 1250 0 0 6941 20 0 0 25 0 33 0 675661796 2400710656 464677 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 8 0 0 0

################
# More data... #
################

/proc/25313/task/25327/stat : 25327 (plingeling) R 25311 25313 25287 0 -1 4202560 260394 1250 0 0 497340 175 0 0 25 0 33 0 675661796 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=25313/tid=25328] ppid=25311 vsize=10813688 CPUtime=4985.77 cores=0-31
/proc/25313/task/25328/stat : 25328 (plingeling) R 25311 25313 25287 0 -1 4202560 823479 1250 0 0 497934 643 0 0 25 0 33 0 675661796 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=25313/tid=25329] ppid=25311 vsize=10813688 CPUtime=4983.29 cores=0-31
/proc/25313/task/25329/stat : 25329 (plingeling) R 25311 25313 25287 0 -1 4202560 321402 1250 0 0 498049 280 0 0 25 0 33 0 675661796 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=25313/tid=25330] ppid=25311 vsize=10813688 CPUtime=4978.4 cores=0-31
/proc/25313/task/25330/stat : 25330 (plingeling) R 25311 25313 25287 0 -1 4202560 507536 1250 0 0 497427 413 0 0 25 0 33 0 675661796 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25313/tid=25331] ppid=25311 vsize=10813688 CPUtime=4984.62 cores=0-31
/proc/25313/task/25331/stat : 25331 (plingeling) R 25311 25313 25287 0 -1 4202560 283176 1250 0 0 498206 256 0 0 25 0 33 0 675661796 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25313/tid=25332] ppid=25311 vsize=10813688 CPUtime=4979.76 cores=0-31
/proc/25313/task/25332/stat : 25332 (plingeling) R 25311 25313 25287 0 -1 4202560 360241 1250 0 0 497670 306 0 0 25 0 33 0 675661797 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4358082 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=25313/tid=25333] ppid=25311 vsize=10813688 CPUtime=4986.94 cores=0-31
/proc/25313/task/25333/stat : 25333 (plingeling) R 25311 25313 25287 0 -1 4202560 463821 1250 0 0 498307 387 0 0 25 0 33 0 675661797 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366517 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25313/tid=25334] ppid=25311 vsize=10813688 CPUtime=4982.19 cores=0-31
/proc/25313/task/25334/stat : 25334 (plingeling) R 25311 25313 25287 0 -1 4202560 370583 1250 0 0 497919 300 0 0 25 0 33 0 675661797 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=25313/tid=25335] ppid=25311 vsize=10813688 CPUtime=4985.98 cores=0-31
/proc/25313/task/25335/stat : 25335 (plingeling) R 25311 25313 25287 0 -1 4202560 427373 1250 0 0 498254 344 0 0 25 0 33 0 675661797 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25313/tid=25336] ppid=25311 vsize=10813688 CPUtime=4985.17 cores=0-31
/proc/25313/task/25336/stat : 25336 (plingeling) R 25311 25313 25287 0 -1 4202560 451123 1250 0 0 498209 308 0 0 25 0 33 0 675661797 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=25313/tid=25337] ppid=25311 vsize=10813688 CPUtime=4986.53 cores=0-31
/proc/25313/task/25337/stat : 25337 (plingeling) R 25311 25313 25287 0 -1 4202560 777286 1250 0 0 498048 605 0 0 25 0 33 0 675661797 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=25313/tid=25338] ppid=25311 vsize=10813688 CPUtime=4981.46 cores=0-31
/proc/25313/task/25338/stat : 25338 (plingeling) R 25311 25313 25287 0 -1 4202560 684894 1250 0 0 497747 399 0 0 25 0 33 0 675661798 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4358106 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=25313/tid=25339] ppid=25311 vsize=10813688 CPUtime=4974.43 cores=0-31
/proc/25313/task/25339/stat : 25339 (plingeling) R 25311 25313 25287 0 -1 4202560 439859 1250 0 0 497017 426 0 0 25 0 33 0 675661798 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366500 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=25313/tid=25340] ppid=25311 vsize=10813688 CPUtime=4987.19 cores=0-31
/proc/25313/task/25340/stat : 25340 (plingeling) R 25311 25313 25287 0 -1 4202560 523790 1250 0 0 498375 344 0 0 25 0 33 0 675661798 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=25313/tid=25341] ppid=25311 vsize=10813688 CPUtime=4980.59 cores=0-31
/proc/25313/task/25341/stat : 25341 (plingeling) R 25311 25313 25287 0 -1 4202560 361155 1250 0 0 497864 195 0 0 25 0 33 0 675661799 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4356867 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=25313/tid=25342] ppid=25311 vsize=10813688 CPUtime=4986.54 cores=0-31
/proc/25313/task/25342/stat : 25342 (plingeling) R 25311 25313 25287 0 -1 4202560 489127 1250 0 0 498304 350 0 0 25 0 33 0 675661799 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=25313/tid=25343] ppid=25311 vsize=10813688 CPUtime=4980.61 cores=0-31
/proc/25313/task/25343/stat : 25343 (plingeling) R 25311 25313 25287 0 -1 4202560 604322 1250 0 0 497549 512 0 0 25 0 33 0 675661799 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25313/tid=25344] ppid=25311 vsize=10813688 CPUtime=4984.89 cores=0-31
/proc/25313/task/25344/stat : 25344 (plingeling) R 25311 25313 25287 0 -1 4202560 551610 1250 0 0 498162 327 0 0 25 0 33 0 675661800 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=25313/tid=25345] ppid=25311 vsize=10813688 CPUtime=4985.71 cores=0-31
/proc/25313/task/25345/stat : 25345 (plingeling) R 25311 25313 25287 0 -1 4202560 514373 1250 0 0 498198 373 0 0 25 0 33 0 675661800 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=25313/tid=25346] ppid=25311 vsize=10813688 CPUtime=4985.02 cores=0-31
/proc/25313/task/25346/stat : 25346 (plingeling) R 25311 25313 25287 0 -1 4202560 1605382 1250 0 0 497679 823 0 0 25 0 33 0 675661800 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236544 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=25313/tid=25347] ppid=25311 vsize=10813688 CPUtime=4982.65 cores=0-31
/proc/25313/task/25347/stat : 25347 (plingeling) R 25311 25313 25287 0 -1 4202560 217220 1250 0 0 498144 121 0 0 25 0 33 0 675661800 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25313/tid=25348] ppid=25311 vsize=10813688 CPUtime=4977 cores=0-31
/proc/25313/task/25348/stat : 25348 (plingeling) R 25311 25313 25287 0 -1 4202560 437883 1250 0 0 497408 292 0 0 25 0 33 0 675661801 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=25313/tid=25349] ppid=25311 vsize=10813688 CPUtime=4985.82 cores=0-31
/proc/25313/task/25349/stat : 25349 (plingeling) R 25311 25313 25287 0 -1 4202560 612834 1250 0 0 497932 650 0 0 25 0 33 0 675661801 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=25313/tid=25350] ppid=25311 vsize=10813688 CPUtime=4981.5 cores=0-31
/proc/25313/task/25350/stat : 25350 (plingeling) R 25311 25313 25287 0 -1 4202560 288016 1250 0 0 497981 169 0 0 25 0 33 0 675661801 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4346372 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25313/tid=25351] ppid=25311 vsize=10813688 CPUtime=4974.61 cores=0-31
/proc/25313/task/25351/stat : 25351 (plingeling) R 25311 25313 25287 0 -1 4202560 133815 1250 0 0 497359 102 0 0 25 0 33 0 675661801 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=25313/tid=25352] ppid=25311 vsize=10813688 CPUtime=4987.12 cores=0-31
/proc/25313/task/25352/stat : 25352 (plingeling) R 25311 25313 25287 0 -1 4202560 505474 1250 0 0 498140 572 0 0 25 0 33 0 675661801 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4358265 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=25313/tid=25353] ppid=25311 vsize=10813688 CPUtime=4982.75 cores=0-31
/proc/25313/task/25353/stat : 25353 (plingeling) R 25311 25313 25287 0 -1 4202560 640439 1250 0 0 497631 644 0 0 25 0 33 0 675661802 11073216512 2581402 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 159474
Current children cumulated vsize (KiB) 10813688



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5038.8 s]
/proc/loadavg: 32.03 32.05 32.01 33/446 25500
/proc/meminfo: memFree=245068120/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=10883108 CPUtime=160002 cores=0-31
/proc/25313/stat : 25313 (plingeling) S 25311 25313 25287 0 -1 4202496 18008185 1250 0 0 15986345 13882 0 0 25 0 33 0 675658676 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 30 0 0 0
/proc/25313/statm: 2720777 2598757 175 55 0 2718663 0
[pid=25313/tid=25322] ppid=25311 vsize=10883108 CPUtime=4996.47 cores=0-31
/proc/25313/task/25322/stat : 25322 (plingeling) R 25311 25313 25287 0 -1 4202560 530100 1250 0 0 499159 488 0 0 25 0 33 0 675661795 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250977 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=25313/tid=25323] ppid=25311 vsize=10883108 CPUtime=5004.61 cores=0-31
/proc/25313/task/25323/stat : 25323 (plingeling) R 25311 25313 25287 0 -1 4202560 302446 1250 0 0 500316 145 0 0 25 0 33 0 675661795 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=25313/tid=25324] ppid=25311 vsize=10883108 CPUtime=5004.21 cores=0-31
/proc/25313/task/25324/stat : 25324 (plingeling) R 25311 25313 25287 0 -1 4202560 502849 1250 0 0 499897 524 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=25313/tid=25325] ppid=25311 vsize=10883108 CPUtime=4998.49 cores=0-31
/proc/25313/task/25325/stat : 25325 (plingeling) R 25311 25313 25287 0 -1 4202560 1986693 1250 0 0 498107 1742 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=25313/tid=25326] ppid=25311 vsize=10883108 CPUtime=4989.39 cores=0-31
/proc/25313/task/25326/stat : 25326 (plingeling) R 25311 25313 25287 0 -1 4202560 547991 1250 0 0 498407 532 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357930 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=25313/tid=25327] ppid=25311 vsize=10883108 CPUtime=4991.67 cores=0-31
/proc/25313/task/25327/stat : 25327 (plingeling) R 25311 25313 25287 0 -1 4202560 260394 1250 0 0 498992 175 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364669 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=25313/tid=25328] ppid=25311 vsize=10883108 CPUtime=5002.28 cores=0-31
/proc/25313/task/25328/stat : 25328 (plingeling) R 25311 25313 25287 0 -1 4202560 823479 1250 0 0 499585 643 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232640 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=25313/tid=25329] ppid=25311 vsize=10883108 CPUtime=4999.82 cores=0-31
/proc/25313/task/25329/stat : 25329 (plingeling) R 25311 25313 25287 0 -1 4202560 323724 1250 0 0 499695 287 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=25313/tid=25330] ppid=25311 vsize=10883108 CPUtime=4994.92 cores=0-31
/proc/25313/task/25330/stat : 25330 (plingeling) R 25311 25313 25287 0 -1 4202560 507536 1250 0 0 499079 413 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25313/tid=25331] ppid=25311 vsize=10883108 CPUtime=5001.14 cores=0-31
/proc/25313/task/25331/stat : 25331 (plingeling) R 25311 25313 25287 0 -1 4202560 283176 1250 0 0 499858 256 0 0 25 0 33 0 675661796 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25313/tid=25332] ppid=25311 vsize=10883108 CPUtime=4996.28 cores=0-31
/proc/25313/task/25332/stat : 25332 (plingeling) R 25311 25313 25287 0 -1 4202560 360241 1250 0 0 499322 306 0 0 25 0 33 0 675661797 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=25313/tid=25333] ppid=25311 vsize=10883108 CPUtime=5003.46 cores=0-31
/proc/25313/task/25333/stat : 25333 (plingeling) R 25311 25313 25287 0 -1 4202560 463821 1250 0 0 499959 387 0 0 25 0 33 0 675661797 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4346677 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25313/tid=25334] ppid=25311 vsize=10883108 CPUtime=4998.7 cores=0-31
/proc/25313/task/25334/stat : 25334 (plingeling) R 25311 25313 25287 0 -1 4202560 371551 1250 0 0 499570 300 0 0 25 0 33 0 675661797 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4358118 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=25313/tid=25335] ppid=25311 vsize=10883108 CPUtime=5002.5 cores=0-31
/proc/25313/task/25335/stat : 25335 (plingeling) R 25311 25313 25287 0 -1 4202560 427373 1250 0 0 499906 344 0 0 25 0 33 0 675661797 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4353568 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25313/tid=25336] ppid=25311 vsize=10883108 CPUtime=5001.69 cores=0-31
/proc/25313/task/25336/stat : 25336 (plingeling) R 25311 25313 25287 0 -1 4202560 451123 1250 0 0 499861 308 0 0 25 0 33 0 675661797 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4321693 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=25313/tid=25337] ppid=25311 vsize=10883108 CPUtime=5003.04 cores=0-31
/proc/25313/task/25337/stat : 25337 (plingeling) R 25311 25313 25287 0 -1 4202560 777286 1250 0 0 499699 605 0 0 25 0 33 0 675661797 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=25313/tid=25338] ppid=25311 vsize=10883108 CPUtime=4997.98 cores=0-31
/proc/25313/task/25338/stat : 25338 (plingeling) R 25311 25313 25287 0 -1 4202560 684894 1250 0 0 499399 399 0 0 25 0 33 0 675661798 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=25313/tid=25339] ppid=25311 vsize=10883108 CPUtime=4990.95 cores=0-31
/proc/25313/task/25339/stat : 25339 (plingeling) R 25311 25313 25287 0 -1 4202560 442867 1250 0 0 498668 427 0 0 25 0 33 0 675661798 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=25313/tid=25340] ppid=25311 vsize=10883108 CPUtime=5003.71 cores=0-31
/proc/25313/task/25340/stat : 25340 (plingeling) R 25311 25313 25287 0 -1 4202560 523790 1250 0 0 500027 344 0 0 25 0 33 0 675661798 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=25313/tid=25341] ppid=25311 vsize=10883108 CPUtime=4997.11 cores=0-31
/proc/25313/task/25341/stat : 25341 (plingeling) R 25311 25313 25287 0 -1 4202560 361155 1250 0 0 499516 195 0 0 25 0 33 0 675661799 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=25313/tid=25342] ppid=25311 vsize=10883108 CPUtime=5003.06 cores=0-31
/proc/25313/task/25342/stat : 25342 (plingeling) R 25311 25313 25287 0 -1 4202560 492175 1250 0 0 499955 351 0 0 25 0 33 0 675661799 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=25313/tid=25343] ppid=25311 vsize=10883108 CPUtime=4997.12 cores=0-31
/proc/25313/task/25343/stat : 25343 (plingeling) R 25311 25313 25287 0 -1 4202560 609188 1250 0 0 499199 513 0 0 25 0 33 0 675661799 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25313/tid=25344] ppid=25311 vsize=10883108 CPUtime=5001.41 cores=0-31
/proc/25313/task/25344/stat : 25344 (plingeling) R 25311 25313 25287 0 -1 4202560 554650 1250 0 0 499813 328 0 0 25 0 33 0 675661800 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357907 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=25313/tid=25345] ppid=25311 vsize=10883108 CPUtime=5002.22 cores=0-31
/proc/25313/task/25345/stat : 25345 (plingeling) R 25311 25313 25287 0 -1 4202560 516723 1250 0 0 499849 373 0 0 25 0 33 0 675661800 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4236623 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=25313/tid=25346] ppid=25311 vsize=10883108 CPUtime=5001.49 cores=0-31
/proc/25313/task/25346/stat : 25346 (plingeling) R 25311 25313 25287 0 -1 4202560 1606459 1250 0 0 499325 824 0 0 25 0 33 0 675661800 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4355830 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=25313/tid=25347] ppid=25311 vsize=10883108 CPUtime=4999.17 cores=0-31
/proc/25313/task/25347/stat : 25347 (plingeling) R 25311 25313 25287 0 -1 4202560 217220 1250 0 0 499796 121 0 0 25 0 33 0 675661800 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4357970 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25313/tid=25348] ppid=25311 vsize=10883108 CPUtime=4993.52 cores=0-31
/proc/25313/task/25348/stat : 25348 (plingeling) R 25311 25313 25287 0 -1 4202560 440726 1250 0 0 499059 293 0 0 25 0 33 0 675661801 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=25313/tid=25349] ppid=25311 vsize=10883108 CPUtime=5002.34 cores=0-31
/proc/25313/task/25349/stat : 25349 (plingeling) R 25311 25313 25287 0 -1 4202560 615793 1250 0 0 499575 659 0 0 25 0 33 0 675661801 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=25313/tid=25350] ppid=25311 vsize=10883108 CPUtime=4998.01 cores=0-31
/proc/25313/task/25350/stat : 25350 (plingeling) R 25311 25313 25287 0 -1 4202560 291609 1250 0 0 499632 169 0 0 25 0 33 0 675661801 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4354228 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25313/tid=25351] ppid=25311 vsize=10883108 CPUtime=4991.12 cores=0-31
/proc/25313/task/25351/stat : 25351 (plingeling) R 25311 25313 25287 0 -1 4202560 133815 1250 0 0 499010 102 0 0 25 0 33 0 675661801 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=25313/tid=25352] ppid=25311 vsize=10883108 CPUtime=5003.63 cores=0-31
/proc/25313/task/25352/stat : 25352 (plingeling) R 25311 25313 25287 0 -1 4202560 505474 1250 0 0 499791 572 0 0 25 0 33 0 675661801 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4365184 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=25313/tid=25353] ppid=25311 vsize=10883108 CPUtime=4999.27 cores=0-31
/proc/25313/task/25353/stat : 25353 (plingeling) R 25311 25313 25287 0 -1 4202560 640439 1250 0 0 499283 644 0 0 25 0 33 0 675661802 11144302592 2598757 268435456000 4194304 4417452 140734233912528 18446744073709551615 4353598 0 0 4096 17442 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 10883108

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5039.1 s]
/proc/loadavg: 32.03 32.05 32.01 2/416 25501
/proc/meminfo: memFree=248190308/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=0 CPUtime=160003 cores=0-31
/proc/25313/stat : 25313 (plingeling) Z 25311 25313 25287 0 -1 4203548 18008185 1250 0 0 15986367 13911 0 0 18 0 2 0 675658676 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 30 0 0 0
/proc/25313/statm: 0 0 0 0 0 0 0
[pid=25313/tid=25341] ppid=25311 vsize=0 CPUtime=4997.4 cores=0-31
/proc/25313/task/25341/stat : 25341 (plingeling) R 25311 25313 25287 0 -1 4203588 361155 1250 0 0 499516 224 0 0 25 0 2 0 675661799 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 11 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 0

[startup+5039.5 s]
/proc/loadavg: 32.03 32.05 32.01 2/416 25501
/proc/meminfo: memFree=248190308/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=0 CPUtime=160003 cores=0-31
/proc/25313/stat : 25313 (plingeling) Z 25311 25313 25287 0 -1 4203548 18008185 1250 0 0 15986367 13949 0 0 18 0 2 0 675658676 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 30 0 0 0
/proc/25313/statm: 0 0 0 0 0 0 0
[pid=25313/tid=25341] ppid=25311 vsize=0 CPUtime=4997.78 cores=0-31
/proc/25313/task/25341/stat : 25341 (plingeling) R 25311 25313 25287 0 -1 4203588 361155 1250 0 0 499516 262 0 0 25 0 2 0 675661799 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 11 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 0

[startup+5039.6 s]
/proc/loadavg: 32.03 32.05 32.01 2/416 25501
/proc/meminfo: memFree=248190308/264257624 swapFree=67111528/67111528
[pid=25313] ppid=25311 vsize=0 CPUtime=160003 cores=0-31
/proc/25313/stat : 25313 (plingeling) Z 25311 25313 25287 0 -1 4203548 18008185 1250 0 0 15986367 13959 0 0 18 0 2 0 675658676 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 30 0 0 0
/proc/25313/statm: 0 0 0 0 0 0 0
[pid=25313/tid=25341] ppid=25311 vsize=0 CPUtime=4997.88 cores=0-31
/proc/25313/task/25341/stat : 25341 (plingeling) R 25311 25313 25287 0 -1 4203588 361155 1250 0 0 499516 272 0 0 25 0 2 0 675661799 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 11 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5039.63
CPU time (s): 160003
CPU user time (s): 159864
CPU system time (s): 139.633
CPU usage (%): 3174.9
Max. virtual memory (cumulated for all children) (KiB): 12859032

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159864
system time used= 139.633
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18009435
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 15241
involuntary context switches= 418926

runsolver used 26.7559 second user time and 115.706 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-17 16:15:23
IDJOB=3372462
IDBENCH=82394
IDSOLVER=1852
FILE ID=node200/3372462-1305641720
RUNJOBID= node200-1305641720-25301
PBS_JOBID= 13354306
Free space on /tmp= 62000 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.080-SAT.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372462-1305641720/watcher-3372462-1305641720 -o /tmp/evaluation-result-3372462-1305641720/solver-3372462-1305641720 -C 160000 -W 5100 -M 266338304  plingeling -t 32 HOME/instance-3372462-1305641720.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 1ca09a2b911e29894c6603aee8d3e672
RANDOM SEED=2141007493

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255476000 kB
Buffers:        448932 kB
Cached:        7928960 kB
SwapCached:          0 kB
Active:        7446292 kB
Inactive:      1000276 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255476000 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           50832 kB
Writeback:           0 kB
AnonPages:       68880 kB
Mapped:          14480 kB
Slab:           252780 kB
PageTables:       4344 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   187352 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62000 MiB
End job on node200 at 2011-05-17 17:39:24