Trace number 3372432

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 158596 5101.98

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
transport-transport-city-sequential-35nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.040-NOTKNOWN.cnf
MD5SUMce0d180356398dfbcfd2bbbb1cb3f9e5
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark20626.8
Satisfiable
(Un)Satisfiability was proved
Number of variables1260306
Number of clauses6771840
Sum of the clauses size16095280
Maximum clause length982
Minimum clause length1
Number of clauses of size 1720
Number of clauses of size 26714640
Number of clauses of size 3320
Number of clauses of size 41280
Number of clauses of size 53200
Number of clauses of size over 551680

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372432-1305599430/watcher-3372432-1305599430 -o /tmp/evaluation-result-3372432-1305599430/solver-3372432-1305599430 -C 160000 -W 5100 -M 266338304 plingeling -t 32 HOME/instance-3372432-1305599430.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 12.01 18.49 26.17 3/413 23177
/proc/meminfo: memFree=255385632/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=36600 CPUtime=0.02 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 7186 1240 0 0 1 1 0 0 22 0 1 0 671429935 37478400 7117 268435456000 4194304 4417452 140737042072768 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 9150 7131 141 55 0 7036 0

[startup+0.0992229 s]
/proc/loadavg: 12.01 18.49 26.17 3/413 23177
/proc/meminfo: memFree=255385632/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=50444 CPUtime=0.08 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 10684 1240 0 0 7 1 0 0 22 0 1 0 671429935 51654656 10614 268435456000 4194304 4417452 140737042072768 18446744073709551615 4209661 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 12611 10614 144 55 0 10497 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 50444

[startup+0.10118 s]
/proc/loadavg: 12.01 18.49 26.17 3/413 23177
/proc/meminfo: memFree=255385632/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=50444 CPUtime=0.08 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 10684 1240 0 0 7 1 0 0 22 0 1 0 671429935 51654656 10614 268435456000 4194304 4417452 140737042072768 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 12611 10614 144 55 0 10497 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 50444

[startup+0.301187 s]
/proc/loadavg: 12.01 18.49 26.17 3/413 23177
/proc/meminfo: memFree=255385632/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=62852 CPUtime=0.29 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 13779 1240 0 0 27 2 0 0 23 0 1 0 671429935 64360448 13709 268435456000 4194304 4417452 140737042072768 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 15745 13729 145 55 0 13631 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 62852

[startup+0.701206 s]
/proc/loadavg: 12.01 18.49 26.17 3/413 23177
/proc/meminfo: memFree=255385632/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=80784 CPUtime=0.69 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 18254 1240 0 0 66 3 0 0 25 0 1 0 671429935 82722816 18184 268435456000 4194304 4417452 140737042072768 18446744073709551615 4264916 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 20196 18184 145 55 0 18082 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 80784

[startup+1.50122 s]
/proc/loadavg: 12.01 18.49 26.17 2/414 23183
/proc/meminfo: memFree=255274840/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=199620 CPUtime=1.48 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 47996 1240 0 0 139 9 0 0 25 0 1 0 671429935 204410880 47926 268435456000 4194304 4417452 140737042072768 18446744073709551615 4231667 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 49905 47926 146 55 0 47791 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 199620

[startup+3.10032 s]
/proc/loadavg: 12.01 18.49 26.17 2/414 23183
/proc/meminfo: memFree=255191264/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=377144 CPUtime=3.08 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 92335 1240 0 0 290 18 0 0 25 0 1 0 671429935 386195456 92261 268435456000 4194304 4417452 140737042072768 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 94286 92278 146 55 0 92172 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 377144

[startup+6.30042 s]
/proc/loadavg: 11.21 18.22 26.04 2/414 23183
/proc/meminfo: memFree=254954604/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=521316 CPUtime=6.29 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 128389 1240 0 0 603 26 0 0 25 0 1 0 671429935 533827584 128316 268435456000 4194304 4417452 140737042072768 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 130329 128333 146 55 0 128215 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 521316

[startup+12.7006 s]
/proc/loadavg: 10.39 17.93 25.90 2/414 23185
/proc/meminfo: memFree=254540544/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=1004504 CPUtime=12.68 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 249240 1240 0 0 1219 49 0 0 25 0 1 0 671429935 1028612096 249147 268435456000 4194304 4417452 140737042072768 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 251126 249147 146 55 0 249012 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 1004504

[startup+25.5011 s]
/proc/loadavg: 8.31 17.10 25.50 2/414 23185
/proc/meminfo: memFree=253502324/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=1896932 CPUtime=25.48 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 472347 1240 0 0 2456 92 0 0 25 0 1 0 671429935 1942458368 472254 268435456000 4194304 4417452 140737042072768 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/23177/statm: 474233 472254 146 55 0 472119 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 1896932

[startup+51.101 s]
/proc/loadavg: 5.89 15.83 24.85 2/414 23185
/proc/meminfo: memFree=251715104/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=3681776 CPUtime=51.08 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 918558 1240 0 0 4928 180 0 0 25 0 1 0 671429935 3770138624 918465 268435456000 4194304 4417452 140737042072768 18446744073709551615 4259674 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/23177/statm: 920444 918465 146 55 0 918330 0
Current children cumulated CPU time (s) 51.08
Current children cumulated vsize (KiB) 3681776

[startup+102.319 s]
/proc/loadavg: 3.12 13.54 23.60 2/414 23187
/proc/meminfo: memFree=248650796/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=6908416 CPUtime=102.28 cores=0-31
/proc/23177/stat : 23177 (plingeling) R 23175 23177 23080 0 -1 4202496 1717903 1240 0 0 9900 328 0 0 25 0 1 0 671429935 7074217984 1717810 268435456000 4194304 4417452 140737042072768 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/23177/statm: 1727104 1717822 146 55 0 1724990 0
Current children cumulated CPU time (s) 102.28
Current children cumulated vsize (KiB) 6908416

[startup+162.302 s]
/proc/loadavg: 14.97 14.57 23.29 33/446 23221
/proc/meminfo: memFree=246327424/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=9650000 CPUtime=1012.88 cores=0-31
/proc/23177/stat : 23177 (plingeling) S 23175 23177 23080 0 -1 4202496 7205768 1240 0 0 93040 8248 0 0 18 0 33 0 671429935 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 10 0 0 0
/proc/23177/statm: 2412500 2252144 174 55 0 2410386 0
[pid=23177/tid=23188] ppid=23175 vsize=9650000 CPUtime=26.95 cores=0-31
/proc/23177/task/23188/stat : 23188 (plingeling) R 23175 23177 23080 0 -1 4202560 182089 1240 0 0 2414 281 0 0 25 0 33 0 671442658 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23177/tid=23189] ppid=23175 vsize=9650000 CPUtime=26.24 cores=0-31
/proc/23177/task/23189/stat : 23189 (plingeling) R 23175 23177 23080 0 -1 4202560 182327 1240 0 0 2371 253 0 0 25 0 33 0 671442658 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331850 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23177/tid=23190] ppid=23175 vsize=9650000 CPUtime=27.85 cores=0-31
/proc/23177/task/23190/stat : 23190 (plingeling) R 23175 23177 23080 0 -1 4202560 158146 1240 0 0 2550 235 0 0 25 0 33 0 671442658 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4210366 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23177/tid=23191] ppid=23175 vsize=9650000 CPUtime=30.29 cores=0-31
/proc/23177/task/23191/stat : 23191 (plingeling) R 23175 23177 23080 0 -1 4202560 24920 1240 0 0 3005 24 0 0 25 0 33 0 671442658 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364461 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23177/tid=23192] ppid=23175 vsize=9650000 CPUtime=27.94 cores=0-31
/proc/23177/task/23192/stat : 23192 (plingeling) R 23175 23177 23080 0 -1 4202560 128503 1240 0 0 2553 241 0 0 25 0 33 0 671442658 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23177/tid=23193] ppid=23175 vsize=9650000 CPUtime=28.52 cores=0-31
/proc/23177/task/23193/stat : 23193 (plingeling) R 23175 23177 23080 0 -1 4202560 130509 1240 0 0 2612 240 0 0 25 0 33 0 671442659 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=23177/tid=23194] ppid=23175 vsize=9650000 CPUtime=27.67 cores=0-31
/proc/23177/task/23194/stat : 23194 (plingeling) R 23175 23177 23080 0 -1 4202560 152407 1240 0 0 2587 180 0 0 25 0 33 0 671442660 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23177/tid=23195] ppid=23175 vsize=9650000 CPUtime=28.1 cores=0-31
/proc/23177/task/23195/stat : 23195 (plingeling) R 23175 23177 23080 0 -1 4202560 181736 1240 0 0 2557 253 0 0 25 0 33 0 671442661 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23177/tid=23196] ppid=23175 vsize=9650000 CPUtime=28.45 cores=0-31
/proc/23177/task/23196/stat : 23196 (plingeling) R 23175 23177 23080 0 -1 4202560 145028 1240 0 0 2575 270 0 0 25 0 33 0 671442661 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23177/tid=23197] ppid=23175 vsize=9650000 CPUtime=27.87 cores=0-31
/proc/23177/task/23197/stat : 23197 (plingeling) R 23175 23177 23080 0 -1 4202560 150173 1240 0 0 2554 233 0 0 25 0 33 0 671442664 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23177/tid=23198] ppid=23175 vsize=9650000 CPUtime=28.23 cores=0-31
/proc/23177/task/23198/stat : 23198 (plingeling) R 23175 23177 23080 0 -1 4202560 157013 1240 0 0 2610 213 0 0 25 0 33 0 671442664 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4254855 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23177/tid=23199] ppid=23175 vsize=9650000 CPUtime=27.94 cores=0-31
/proc/23177/task/23199/stat : 23199 (plingeling) R 23175 23177 23080 0 -1 4202560 147078 1240 0 0 2553 241 0 0 25 0 33 0 671442664 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23177/tid=23200] ppid=23175 vsize=9650000 CPUtime=28.53 cores=0-31
/proc/23177/task/23200/stat : 23200 (plingeling) R 23175 23177 23080 0 -1 4202560 150855 1240 0 0 2562 291 0 0 25 0 33 0 671442665 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23177/tid=23201] ppid=23175 vsize=9650000 CPUtime=27.25 cores=0-31
/proc/23177/task/23201/stat : 23201 (plingeling) R 23175 23177 23080 0 -1 4202560 166389 1240 0 0 2438 287 0 0 25 0 33 0 671442665 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23177/tid=23202] ppid=23175 vsize=9650000 CPUtime=27.65 cores=0-31
/proc/23177/task/23202/stat : 23202 (plingeling) R 23175 23177 23080 0 -1 4202560 147971 1240 0 0 2545 220 0 0 25 0 33 0 671442666 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4253528 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23177/tid=23203] ppid=23175 vsize=9650000 CPUtime=27.49 cores=0-31
/proc/23177/task/23203/stat : 23203 (plingeling) R 23175 23177 23080 0 -1 4202560 138472 1240 0 0 2514 235 0 0 25 0 33 0 671442667 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23177/tid=23204] ppid=23175 vsize=9650000 CPUtime=27.26 cores=0-31
/proc/23177/task/23204/stat : 23204 (plingeling) R 23175 23177 23080 0 -1 4202560 153793 1240 0 0 2399 327 0 0 25 0 33 0 671442675 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4330387 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23177/tid=23205] ppid=23175 vsize=9650000 CPUtime=26.32 cores=0-31
/proc/23177/task/23205/stat : 23205 (plingeling) R 23175 23177 23080 0 -1 4202560 160647 1240 0 0 2365 267 0 0 25 0 33 0 671442676 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23177/tid=23206] ppid=23175 vsize=9650000 CPUtime=26.75 cores=0-31
/proc/23177/task/23206/stat : 23206 (plingeling) R 23175 23177 23080 0 -1 4202560 154564 1240 0 0 2425 250 0 0 25 0 33 0 671442677 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23177/tid=23207] ppid=23175 vsize=9650000 CPUtime=26.56 cores=0-31
/proc/23177/task/23207/stat : 23207 (plingeling) R 23175 23177 23080 0 -1 4202560 123445 1240 0 0 2419 237 0 0 25 0 33 0 671442680 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23177/tid=23208] ppid=23175 vsize=9650000 CPUtime=27.44 cores=0-31
/proc/23177/task/23208/stat : 23208 (plingeling) R 23175 23177 23080 0 -1 4202560 161001 1240 0 0 2510 234 0 0 25 0 33 0 671442680 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331802 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23177/tid=23209] ppid=23175 vsize=9650000 CPUtime=28.03 cores=0-31
/proc/23177/task/23209/stat : 23209 (plingeling) R 23175 23177 23080 0 -1 4202560 167457 1240 0 0 2535 268 0 0 25 0 33 0 671442681 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331782 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23177/tid=23210] ppid=23175 vsize=9650000 CPUtime=27.36 cores=0-31
/proc/23177/task/23210/stat : 23210 (plingeling) R 23175 23177 23080 0 -1 4202560 132936 1240 0 0 2570 166 0 0 25 0 33 0 671442681 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4253542 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23177/tid=23211] ppid=23175 vsize=9650000 CPUtime=26.8 cores=0-31
/proc/23177/task/23211/stat : 23211 (plingeling) R 23175 23177 23080 0 -1 4202560 148738 1240 0 0 2437 243 0 0 25 0 33 0 671442682 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331850 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23177/tid=23212] ppid=23175 vsize=9650000 CPUtime=26.5 cores=0-31
/proc/23177/task/23212/stat : 23212 (plingeling) R 23175 23177 23080 0 -1 4202560 151737 1240 0 0 2388 262 0 0 25 0 33 0 671442682 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23177/tid=23213] ppid=23175 vsize=9650000 CPUtime=27.61 cores=0-31
/proc/23177/task/23213/stat : 23213 (plingeling) R 23175 23177 23080 0 -1 4202560 166742 1240 0 0 2488 273 0 0 25 0 33 0 671442683 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23177/tid=23214] ppid=23175 vsize=9650000 CPUtime=27.72 cores=0-31
/proc/23177/task/23214/stat : 23214 (plingeling) R 23175 23177 23080 0 -1 4202560 168254 1240 0 0 2493 279 0 0 25 0 33 0 671442684 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4233447 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=9650000 CPUtime=28.62 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4202560 151460 1240 0 0 2605 257 0 0 25 0 33 0 671442684 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23177/tid=23216] ppid=23175 vsize=9650000 CPUtime=26.58 cores=0-31
/proc/23177/task/23216/stat : 23216 (plingeling) R 23175 23177 23080 0 -1 4202560 142779 1240 0 0 2378 280 0 0 25 0 33 0 671442685 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23177/tid=23217] ppid=23175 vsize=9650000 CPUtime=27.07 cores=0-31
/proc/23177/task/23217/stat : 23217 (plingeling) R 23175 23177 23080 0 -1 4202560 163855 1240 0 0 2423 284 0 0 25 0 33 0 671442686 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23177/tid=23218] ppid=23175 vsize=9650000 CPUtime=28.99 cores=0-31
/proc/23177/task/23218/stat : 23218 (plingeling) R 23175 23177 23080 0 -1 4202560 150186 1240 0 0 2680 219 0 0 25 0 33 0 671442687 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4253513 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23177/tid=23219] ppid=23175 vsize=9650000 CPUtime=28.8 cores=0-31
/proc/23177/task/23219/stat : 23219 (plingeling) R 23175 23177 23080 0 -1 4202560 162438 1240 0 0 2647 233 0 0 25 0 33 0 671442688 9881600000 2252144 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1012.88
Current children cumulated vsize (KiB) 9650000


################
# More data... #
################

[pid=23177/tid=23199] ppid=23175 vsize=34783148 CPUtime=4938.39 cores=0-31
/proc/23177/task/23199/stat : 23199 (plingeling) R 23175 23177 23080 0 -1 4202560 1345291 1240 0 0 491809 2030 0 0 25 0 33 0 671442664 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236784 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23177/tid=23200] ppid=23175 vsize=34783148 CPUtime=4934.56 cores=0-31
/proc/23177/task/23200/stat : 23200 (plingeling) R 23175 23177 23080 0 -1 4202560 1095423 1240 0 0 492077 1379 0 0 25 0 33 0 671442665 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23177/tid=23201] ppid=23175 vsize=34783148 CPUtime=4933.82 cores=0-31
/proc/23177/task/23201/stat : 23201 (plingeling) R 23175 23177 23080 0 -1 4202560 1166645 1240 0 0 491609 1773 0 0 25 0 33 0 671442665 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23177/tid=23202] ppid=23175 vsize=34783148 CPUtime=4939.24 cores=0-31
/proc/23177/task/23202/stat : 23202 (plingeling) R 23175 23177 23080 0 -1 4202560 1577891 1240 0 0 491380 2544 0 0 25 0 33 0 671442666 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4265832 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23177/tid=23203] ppid=23175 vsize=34783148 CPUtime=4935.72 cores=0-31
/proc/23177/task/23203/stat : 23203 (plingeling) R 23175 23177 23080 0 -1 4202560 1450677 1240 0 0 491273 2299 0 0 25 0 33 0 671442667 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4263109 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23177/tid=23204] ppid=23175 vsize=34783148 CPUtime=4928.92 cores=0-31
/proc/23177/task/23204/stat : 23204 (plingeling) R 23175 23177 23080 0 -1 4202560 1374943 1240 0 0 491097 1795 0 0 25 0 33 0 671442675 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23177/tid=23205] ppid=23175 vsize=34783148 CPUtime=4935.52 cores=0-31
/proc/23177/task/23205/stat : 23205 (plingeling) R 23175 23177 23080 0 -1 4202560 1103099 1240 0 0 492000 1552 0 0 25 0 33 0 671442676 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23177/tid=23206] ppid=23175 vsize=34783148 CPUtime=4934.97 cores=0-31
/proc/23177/task/23206/stat : 23206 (plingeling) R 23175 23177 23080 0 -1 4202560 1081181 1240 0 0 492012 1485 0 0 25 0 33 0 671442677 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23177/tid=23207] ppid=23175 vsize=34783148 CPUtime=4930.7 cores=0-31
/proc/23177/task/23207/stat : 23207 (plingeling) R 23175 23177 23080 0 -1 4202560 1227535 1240 0 0 491441 1629 0 0 25 0 33 0 671442680 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232595 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23177/tid=23208] ppid=23175 vsize=34783148 CPUtime=4929.93 cores=0-31
/proc/23177/task/23208/stat : 23208 (plingeling) R 23175 23177 23080 0 -1 4202560 1174807 1240 0 0 491278 1715 0 0 25 0 33 0 671442680 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4252015 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23177/tid=23209] ppid=23175 vsize=34783148 CPUtime=4937.94 cores=0-31
/proc/23177/task/23209/stat : 23209 (plingeling) R 23175 23177 23080 0 -1 4202560 1014667 1240 0 0 492777 1017 0 0 25 0 33 0 671442681 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232664 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23177/tid=23210] ppid=23175 vsize=34783148 CPUtime=4938.74 cores=0-31
/proc/23177/task/23210/stat : 23210 (plingeling) R 23175 23177 23080 0 -1 4202560 1565155 1240 0 0 491452 2422 0 0 25 0 33 0 671442681 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232685 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23177/tid=23211] ppid=23175 vsize=34783148 CPUtime=4936.66 cores=0-31
/proc/23177/task/23211/stat : 23211 (plingeling) R 23175 23177 23080 0 -1 4202560 1249544 1240 0 0 491770 1896 0 0 25 0 33 0 671442682 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23177/tid=23212] ppid=23175 vsize=34783148 CPUtime=4937.1 cores=0-31
/proc/23177/task/23212/stat : 23212 (plingeling) R 23175 23177 23080 0 -1 4202560 919815 1240 0 0 492599 1111 0 0 25 0 33 0 671442682 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23177/tid=23213] ppid=23175 vsize=34783148 CPUtime=4930.14 cores=0-31
/proc/23177/task/23213/stat : 23213 (plingeling) R 23175 23177 23080 0 -1 4202560 1230940 1240 0 0 491229 1785 0 0 25 0 33 0 671442683 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23177/tid=23214] ppid=23175 vsize=34783148 CPUtime=4929.53 cores=0-31
/proc/23177/task/23214/stat : 23214 (plingeling) R 23175 23177 23080 0 -1 4202560 1496256 1240 0 0 490433 2520 0 0 25 0 33 0 671442684 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=34783148 CPUtime=4938.82 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4202560 1612709 1240 0 0 491701 2181 0 0 25 0 33 0 671442684 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23177/tid=23216] ppid=23175 vsize=34783148 CPUtime=4937.73 cores=0-31
/proc/23177/task/23216/stat : 23216 (plingeling) R 23175 23177 23080 0 -1 4202560 1132530 1240 0 0 491893 1880 0 0 25 0 33 0 671442685 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232755 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23177/tid=23217] ppid=23175 vsize=34783148 CPUtime=4938.45 cores=0-31
/proc/23177/task/23217/stat : 23217 (plingeling) R 23175 23177 23080 0 -1 4202560 805733 1240 0 0 492937 908 0 0 25 0 33 0 671442686 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23177/tid=23218] ppid=23175 vsize=34783148 CPUtime=4915.71 cores=0-31
/proc/23177/task/23218/stat : 23218 (plingeling) R 23175 23177 23080 0 -1 4202560 1445070 1240 0 0 489529 2042 0 0 25 0 33 0 671442687 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4353568 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23177/tid=23219] ppid=23175 vsize=34783148 CPUtime=4935.12 cores=0-31
/proc/23177/task/23219/stat : 23219 (plingeling) R 23175 23177 23080 0 -1 4202560 1267422 1240 0 0 491916 1596 0 0 25 0 33 0 671442688 35617943552 8483883 268435456000 4194304 4417452 140737042072768 18446744073709551615 4366659 0 0 4096 17442 0 0 0 -1 19 0 0 0
Current children cumulated CPU time (s) 158028
Current children cumulated vsize (KiB) 34783148



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 32.00 32.04 32.00 33/446 23363
/proc/meminfo: memFree=221955304/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=34227172 CPUtime=158594 cores=0-31
/proc/23177/stat : 23177 (plingeling) S 23175 23177 23080 0 -1 4202496 45043229 1240 0 0 15799000 60356 0 0 18 0 33 0 671429935 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 10 0 0 0
/proc/23177/statm: 8556793 8344889 175 55 0 8554679 0
[pid=23177/tid=23188] ppid=23175 vsize=34227172 CPUtime=4955.63 cores=0-31
/proc/23177/task/23188/stat : 23188 (plingeling) R 23175 23177 23080 0 -1 4202560 1221885 1240 0 0 493804 1759 0 0 25 0 33 0 671442658 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=23177/tid=23189] ppid=23175 vsize=34227172 CPUtime=4954.27 cores=0-31
/proc/23177/task/23189/stat : 23189 (plingeling) R 23175 23177 23080 0 -1 4202560 1188809 1240 0 0 493915 1512 0 0 25 0 33 0 671442658 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4353598 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23177/tid=23190] ppid=23175 vsize=34227172 CPUtime=4956.57 cores=0-31
/proc/23177/task/23190/stat : 23190 (plingeling) R 23175 23177 23080 0 -1 4202560 1199054 1240 0 0 494114 1543 0 0 25 0 33 0 671442658 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23177/tid=23191] ppid=23175 vsize=34227172 CPUtime=4928.68 cores=0-31
/proc/23177/task/23191/stat : 23191 (plingeling) R 23175 23177 23080 0 -1 4202560 3094309 1240 0 0 488995 3873 0 0 25 0 33 0 671442658 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23177/tid=23192] ppid=23175 vsize=34227172 CPUtime=4953.74 cores=0-31
/proc/23177/task/23192/stat : 23192 (plingeling) R 23175 23177 23080 0 -1 4202560 1433801 1240 0 0 493365 2009 0 0 25 0 33 0 671442658 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23177/tid=23193] ppid=23175 vsize=34227172 CPUtime=4957.24 cores=0-31
/proc/23177/task/23193/stat : 23193 (plingeling) R 23175 23177 23080 0 -1 4202560 1147037 1240 0 0 494033 1691 0 0 25 0 33 0 671442659 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=23177/tid=23194] ppid=23175 vsize=34227172 CPUtime=4956.19 cores=0-31
/proc/23177/task/23194/stat : 23194 (plingeling) R 23175 23177 23080 0 -1 4202560 1557780 1240 0 0 493672 1947 0 0 25 0 33 0 671442660 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23177/tid=23195] ppid=23175 vsize=34227172 CPUtime=4957.94 cores=0-31
/proc/23177/task/23195/stat : 23195 (plingeling) R 23175 23177 23080 0 -1 4202560 1468344 1240 0 0 493704 2090 0 0 25 0 33 0 671442661 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=23177/tid=23196] ppid=23175 vsize=34227172 CPUtime=4947.52 cores=0-31
/proc/23177/task/23196/stat : 23196 (plingeling) R 23175 23177 23080 0 -1 4202560 1326462 1240 0 0 492681 2071 0 0 25 0 33 0 671442661 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23177/tid=23197] ppid=23175 vsize=34227172 CPUtime=4952.36 cores=0-31
/proc/23177/task/23197/stat : 23197 (plingeling) R 23175 23177 23080 0 -1 4202560 1230875 1240 0 0 493702 1534 0 0 25 0 33 0 671442664 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23177/tid=23198] ppid=23175 vsize=34227172 CPUtime=4956.37 cores=0-31
/proc/23177/task/23198/stat : 23198 (plingeling) R 23175 23177 23080 0 -1 4202560 1258282 1240 0 0 493798 1839 0 0 25 0 33 0 671442664 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23177/tid=23199] ppid=23175 vsize=34227172 CPUtime=4956.11 cores=0-31
/proc/23177/task/23199/stat : 23199 (plingeling) R 23175 23177 23080 0 -1 4202560 1345291 1240 0 0 493581 2030 0 0 25 0 33 0 671442664 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23177/tid=23200] ppid=23175 vsize=34227172 CPUtime=4952.28 cores=0-31
/proc/23177/task/23200/stat : 23200 (plingeling) R 23175 23177 23080 0 -1 4202560 1095423 1240 0 0 493849 1379 0 0 25 0 33 0 671442665 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23177/tid=23201] ppid=23175 vsize=34227172 CPUtime=4951.54 cores=0-31
/proc/23177/task/23201/stat : 23201 (plingeling) R 23175 23177 23080 0 -1 4202560 1166645 1240 0 0 493381 1773 0 0 25 0 33 0 671442665 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236772 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23177/tid=23202] ppid=23175 vsize=34227172 CPUtime=4956.84 cores=0-31
/proc/23177/task/23202/stat : 23202 (plingeling) R 23175 23177 23080 0 -1 4202560 1721548 1240 0 0 492747 2937 0 0 25 0 33 0 671442666 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232580 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23177/tid=23203] ppid=23175 vsize=34227172 CPUtime=4953.44 cores=0-31
/proc/23177/task/23203/stat : 23203 (plingeling) R 23175 23177 23080 0 -1 4202560 1450677 1240 0 0 493045 2299 0 0 25 0 33 0 671442667 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23177/tid=23204] ppid=23175 vsize=34227172 CPUtime=4946.58 cores=0-31
/proc/23177/task/23204/stat : 23204 (plingeling) R 23175 23177 23080 0 -1 4202560 1389617 1240 0 0 492833 1825 0 0 25 0 33 0 671442675 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23177/tid=23205] ppid=23175 vsize=34227172 CPUtime=4953.23 cores=0-31
/proc/23177/task/23205/stat : 23205 (plingeling) R 23175 23177 23080 0 -1 4202560 1103099 1240 0 0 493771 1552 0 0 25 0 33 0 671442676 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23177/tid=23206] ppid=23175 vsize=34227172 CPUtime=4952.69 cores=0-31
/proc/23177/task/23206/stat : 23206 (plingeling) R 23175 23177 23080 0 -1 4202560 1081181 1240 0 0 493784 1485 0 0 25 0 33 0 671442677 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4331796 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23177/tid=23207] ppid=23175 vsize=34227172 CPUtime=4948.42 cores=0-31
/proc/23177/task/23207/stat : 23207 (plingeling) R 23175 23177 23080 0 -1 4202560 1227535 1240 0 0 493213 1629 0 0 25 0 33 0 671442680 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4365538 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23177/tid=23208] ppid=23175 vsize=34227172 CPUtime=4947.65 cores=0-31
/proc/23177/task/23208/stat : 23208 (plingeling) R 23175 23177 23080 0 -1 4202560 1174807 1240 0 0 493050 1715 0 0 25 0 33 0 671442680 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4302954 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23177/tid=23209] ppid=23175 vsize=34227172 CPUtime=4955.66 cores=0-31
/proc/23177/task/23209/stat : 23209 (plingeling) R 23175 23177 23080 0 -1 4202560 1014667 1240 0 0 494549 1017 0 0 25 0 33 0 671442681 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23177/tid=23210] ppid=23175 vsize=34227172 CPUtime=4956.46 cores=0-31
/proc/23177/task/23210/stat : 23210 (plingeling) R 23175 23177 23080 0 -1 4202560 1565155 1240 0 0 493224 2422 0 0 25 0 33 0 671442681 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=23177/tid=23211] ppid=23175 vsize=34227172 CPUtime=4954.38 cores=0-31
/proc/23177/task/23211/stat : 23211 (plingeling) R 23175 23177 23080 0 -1 4202560 1257981 1240 0 0 493526 1912 0 0 25 0 33 0 671442682 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23177/tid=23212] ppid=23175 vsize=34227172 CPUtime=4954.82 cores=0-31
/proc/23177/task/23212/stat : 23212 (plingeling) R 23175 23177 23080 0 -1 4202560 919815 1240 0 0 494371 1111 0 0 25 0 33 0 671442682 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232787 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23177/tid=23213] ppid=23175 vsize=34227172 CPUtime=4947.86 cores=0-31
/proc/23177/task/23213/stat : 23213 (plingeling) R 23175 23177 23080 0 -1 4202560 1230940 1240 0 0 493001 1785 0 0 25 0 33 0 671442683 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=23177/tid=23214] ppid=23175 vsize=34227172 CPUtime=4947.25 cores=0-31
/proc/23177/task/23214/stat : 23214 (plingeling) R 23175 23177 23080 0 -1 4202560 1502013 1240 0 0 492189 2536 0 0 25 0 33 0 671442684 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=34227172 CPUtime=4956.54 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4202560 1612709 1240 0 0 493473 2181 0 0 25 0 33 0 671442684 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232702 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23177/tid=23216] ppid=23175 vsize=34227172 CPUtime=4955.45 cores=0-31
/proc/23177/task/23216/stat : 23216 (plingeling) R 23175 23177 23080 0 -1 4202560 1132530 1240 0 0 493665 1880 0 0 25 0 33 0 671442685 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23177/tid=23217] ppid=23175 vsize=34227172 CPUtime=4956.09 cores=0-31
/proc/23177/task/23217/stat : 23217 (plingeling) R 23175 23177 23080 0 -1 4202560 810346 1240 0 0 494700 909 0 0 25 0 33 0 671442686 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23177/tid=23218] ppid=23175 vsize=34227172 CPUtime=4933.41 cores=0-31
/proc/23177/task/23218/stat : 23218 (plingeling) R 23175 23177 23080 0 -1 4202560 1445070 1240 0 0 491299 2042 0 0 25 0 33 0 671442687 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23177/tid=23219] ppid=23175 vsize=34227172 CPUtime=4952.84 cores=0-31
/proc/23177/task/23219/stat : 23219 (plingeling) R 23175 23177 23080 0 -1 4202560 1267422 1240 0 0 493688 1596 0 0 25 0 33 0 671442688 35048624128 8344889 268435456000 4194304 4417452 140737042072768 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 19 0 0 0
Current children cumulated CPU time (s) 158594
Current children cumulated vsize (KiB) 34227172

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 32.00 32.04 32.00 2/416 23364
/proc/meminfo: memFree=230017776/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=0 CPUtime=158594 cores=0-31
/proc/23177/stat : 23177 (plingeling) Z 23175 23177 23080 0 -1 4203548 45043229 1240 0 0 15799022 60417 0 0 18 0 2 0 671429935 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 10 0 0 0
/proc/23177/statm: 0 0 0 0 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=0 CPUtime=4957.15 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4203588 1612709 1240 0 0 493474 2241 0 0 25 0 2 0 671442684 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 10 0 0 0
Current children cumulated CPU time (s) 158594
Current children cumulated vsize (KiB) 0

[startup+5101.4 s]
/proc/loadavg: 32.00 32.04 32.00 2/416 23364
/proc/meminfo: memFree=230017776/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=0 CPUtime=158595 cores=0-31
/proc/23177/stat : 23177 (plingeling) Z 23175 23177 23080 0 -1 4203548 45043229 1240 0 0 15799022 60493 0 0 18 0 2 0 671429935 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 10 0 0 0
/proc/23177/statm: 0 0 0 0 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=0 CPUtime=4957.92 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4203588 1612709 1240 0 0 493474 2318 0 0 25 0 2 0 671442684 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 10 0 0 0
Current children cumulated CPU time (s) 158595
Current children cumulated vsize (KiB) 0

[startup+5101.83 s]
/proc/loadavg: 32.00 32.04 32.00 2/416 23364
/proc/meminfo: memFree=250958484/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=0 CPUtime=158596 cores=0-31
/proc/23177/stat : 23177 (plingeling) Z 23175 23177 23080 0 -1 4203548 45043229 1240 0 0 15799022 60536 0 0 18 0 2 0 671429935 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 10 0 0 0
/proc/23177/statm: 0 0 0 0 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=0 CPUtime=4958.35 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4203588 1612709 1240 0 0 493474 2361 0 0 25 0 2 0 671442684 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 10 0 0 0
Current children cumulated CPU time (s) 158596
Current children cumulated vsize (KiB) 0

[startup+5101.9 s]
/proc/loadavg: 32.00 32.04 32.00 2/416 23364
/proc/meminfo: memFree=250958484/264257624 swapFree=67111528/67111528
[pid=23177] ppid=23175 vsize=0 CPUtime=158596 cores=0-31
/proc/23177/stat : 23177 (plingeling) Z 23175 23177 23080 0 -1 4203548 45043229 1240 0 0 15799022 60543 0 0 18 0 2 0 671429935 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 10 0 0 0
/proc/23177/statm: 0 0 0 0 0 0 0
[pid=23177/tid=23215] ppid=23175 vsize=0 CPUtime=4958.42 cores=0-31
/proc/23177/task/23215/stat : 23215 (plingeling) R 23175 23177 23080 0 -1 4203588 1612709 1240 0 0 493474 2368 0 0 25 0 2 0 671442684 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 10 0 0 0
Current children cumulated CPU time (s) 158596
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5101.98
CPU time (s): 158596
CPU user time (s): 157990
CPU system time (s): 605.518
CPU usage (%): 3108.52
Max. virtual memory (cumulated for all children) (KiB): 36102908

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 157990
system time used= 605.518
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 45044469
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17351
involuntary context switches= 409979

runsolver used 27.2729 second user time and 116.114 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-17 04:30:35
IDJOB=3372432
IDBENCH=82374
IDSOLVER=1852
FILE ID=node200/3372432-1305599430
RUNJOBID= node200-1305599205-23094
PBS_JOBID= 13285560
Free space on /tmp= 61920 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/transport-transport-city-sequential-35nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.040-NOTKNOWN.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372432-1305599430/watcher-3372432-1305599430 -o /tmp/evaluation-result-3372432-1305599430/solver-3372432-1305599430 -C 160000 -W 5100 -M 266338304  plingeling -t 32 HOME/instance-3372432-1305599430.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= ce0d180356398dfbcfd2bbbb1cb3f9e5
RANDOM SEED=104442429

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255385796 kB
Buffers:        448000 kB
Cached:        8009756 kB
SwapCached:          0 kB
Active:        7530712 kB
Inactive:       998560 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255385796 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:          133940 kB
Writeback:           0 kB
AnonPages:       71964 kB
Mapped:          14536 kB
Slab:           254836 kB
PageTables:       4436 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   193544 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61920 MiB
End job on node200 at 2011-05-17 05:55:37