Trace number 3372486

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 159276 5102.11

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
transport-transport-three-cities-sequential-14nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.030-NOTKNOWN.cnf
MD5SUMc6d3480970bb2819448dd0a800a3cf1c
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark98266.1
Satisfiable
(Un)Satisfiability was proved
Number of variables1134832
Number of clauses6175026
Sum of the clauses size14647926
Maximum clause length1178
Minimum clause length1
Number of clauses of size 1846
Number of clauses of size 26124260
Number of clauses of size 3960
Number of clauses of size 41200
Number of clauses of size 51200
Number of clauses of size over 546560

Solver Data


Verifier Data

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372486-1305668190/watcher-3372486-1305668190 -o /tmp/evaluation-result-3372486-1305668190/solver-3372486-1305668190 -C 160000 -W 5100 -M 266338304 plingeling -t 32 HOME/instance-3372486-1305668190.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 18.57 25.19 25.91 2/413 26730
/proc/meminfo: memFree=255399416/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=24440 CPUtime=0.02 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 4177 1241 0 0 1 1 0 0 21 0 1 0 678305883 25026560 4108 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649791 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 6110 4118 141 55 0 3996 0

[startup+0.0763911 s]
/proc/loadavg: 18.57 25.19 25.91 2/413 26730
/proc/meminfo: memFree=255399416/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=54644 CPUtime=0.06 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 11672 1241 0 0 3 3 0 0 21 0 1 0 678305883 55955456 11603 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 13661 11616 142 55 0 11547 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 54644

[startup+0.10041 s]
/proc/loadavg: 18.57 25.19 25.91 2/413 26730
/proc/meminfo: memFree=255399416/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=60324 CPUtime=0.08 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 13135 1241 0 0 5 3 0 0 21 0 1 0 678305883 61771776 13066 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236986 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 15081 13066 142 55 0 12967 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 60324

[startup+0.300417 s]
/proc/loadavg: 18.57 25.19 25.91 2/413 26730
/proc/meminfo: memFree=255399416/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=72816 CPUtime=0.29 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 16274 1241 0 0 25 4 0 0 22 0 1 0 678305883 74563584 16205 268435456000 4194304 4417452 140733366487440 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 18204 16205 142 55 0 16090 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 72816

[startup+0.700413 s]
/proc/loadavg: 18.57 25.19 25.91 2/413 26730
/proc/meminfo: memFree=255399416/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=95704 CPUtime=0.69 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 21980 1241 0 0 63 6 0 0 23 0 1 0 678305883 98000896 21911 268435456000 4194304 4417452 140733366487440 18446744073709551615 4231335 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 23926 21911 142 55 0 21812 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 95704

[startup+1.50043 s]
/proc/loadavg: 18.57 25.19 25.91 2/414 26736
/proc/meminfo: memFree=255313592/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=137244 CPUtime=1.49 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 32369 1241 0 0 141 8 0 0 25 0 1 0 678305883 140537856 32300 268435456000 4194304 4417452 140733366487440 18446744073709551615 4235997 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 34311 32300 146 55 0 32197 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 137244

[startup+3.1005 s]
/proc/loadavg: 18.57 25.19 25.91 2/414 26736
/proc/meminfo: memFree=255173128/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=255964 CPUtime=3.09 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 62089 1241 0 0 295 14 0 0 25 0 1 0 678305883 262107136 62012 268435456000 4194304 4417452 140733366487440 18446744073709551615 4264909 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 63991 62012 146 55 0 61877 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 255964

[startup+6.30059 s]
/proc/loadavg: 17.16 24.79 25.78 2/414 26736
/proc/meminfo: memFree=255026548/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=489028 CPUtime=6.28 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 120355 1241 0 0 603 25 0 0 25 0 1 0 678305883 500764672 120278 268435456000 4194304 4417452 140733366487440 18446744073709551615 4227787 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 122257 120278 146 55 0 120143 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 489028

[startup+12.7008 s]
/proc/loadavg: 15.87 24.39 25.64 2/414 26736
/proc/meminfo: memFree=254564720/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=975412 CPUtime=12.69 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 241991 1241 0 0 1218 51 0 0 25 0 1 0 678305883 998821888 241874 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236549 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 243853 241874 146 55 0 241739 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 975412

[startup+25.5013 s]
/proc/loadavg: 12.57 23.25 25.25 2/414 26736
/proc/meminfo: memFree=253513596/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=1895544 CPUtime=25.48 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 472024 1241 0 0 2452 96 0 0 25 0 1 0 678305883 1941037056 471907 268435456000 4194304 4417452 140733366487440 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 473886 471907 146 55 0 471772 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 1895544

[startup+51.1012 s]
/proc/loadavg: 8.62 21.46 24.60 2/414 26738
/proc/meminfo: memFree=251725684/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=3680900 CPUtime=51.08 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 918363 1241 0 0 4928 180 0 0 25 0 1 0 678305883 3769241600 918246 268435456000 4194304 4417452 140733366487440 18446744073709551615 4259140 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 920225 918246 146 55 0 918111 0
Current children cumulated CPU time (s) 51.08
Current children cumulated vsize (KiB) 3680900

[startup+102.334 s]
/proc/loadavg: 4.31 18.30 23.36 2/414 26738
/proc/meminfo: memFree=248145348/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=7254672 CPUtime=102.3 cores=0-31
/proc/26730/stat : 26730 (plingeling) R 26728 26730 26704 0 -1 4202496 1811806 1241 0 0 9878 352 0 0 25 0 1 0 678305883 7428784128 1811689 268435456000 4194304 4417452 140733366487440 18446744073709551615 4322280 0 0 4096 17442 0 0 0 17 26 0 0 0
/proc/26730/statm: 1813668 1811689 146 55 0 1811554 0
Current children cumulated CPU time (s) 102.3
Current children cumulated vsize (KiB) 7254672

[startup+162.302 s]
/proc/loadavg: 20.96 20.40 23.75 31/446 26772
/proc/meminfo: memFree=246384156/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=10096116 CPUtime=1775.44 cores=0-31
/proc/26730/stat : 26730 (plingeling) S 26728 26730 26704 0 -1 4202496 5188219 1241 0 0 172652 4892 0 0 18 0 33 0 678305883 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 2524029 2258868 175 55 0 2521915 0
[pid=26730/tid=26739] ppid=26728 vsize=10096116 CPUtime=52.14 cores=0-31
/proc/26730/task/26739/stat : 26739 (plingeling) R 26728 26730 26704 0 -1 4202560 103032 1241 0 0 5083 131 0 0 25 0 33 0 678316571 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4252349 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=26730/tid=26740] ppid=26728 vsize=10096116 CPUtime=51.39 cores=0-31
/proc/26730/task/26740/stat : 26740 (plingeling) R 26728 26730 26704 0 -1 4202560 116180 1241 0 0 4977 162 0 0 25 0 33 0 678316571 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236017 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=26730/tid=26741] ppid=26728 vsize=10096116 CPUtime=52.32 cores=0-31
/proc/26730/task/26741/stat : 26741 (plingeling) D 26728 26730 26704 0 -1 4202560 82079 1241 0 0 5159 73 0 0 25 0 33 0 678316571 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650999159 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=26730/tid=26742] ppid=26728 vsize=10096116 CPUtime=54.05 cores=0-31
/proc/26730/task/26742/stat : 26742 (plingeling) R 26728 26730 26704 0 -1 4202560 45635 1241 0 0 5304 101 0 0 25 0 33 0 678316572 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4353504 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26730/tid=26743] ppid=26728 vsize=10096116 CPUtime=52.28 cores=0-31
/proc/26730/task/26743/stat : 26743 (plingeling) D 26728 26730 26704 0 -1 4202560 96683 1241 0 0 5070 158 0 0 20 0 33 0 678316572 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=26730/tid=26744] ppid=26728 vsize=10096116 CPUtime=51.64 cores=0-31
/proc/26730/task/26744/stat : 26744 (plingeling) R 26728 26730 26704 0 -1 4202560 90330 1241 0 0 5051 113 0 0 25 0 33 0 678316573 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4318898 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=26730/tid=26745] ppid=26728 vsize=10096116 CPUtime=52.18 cores=0-31
/proc/26730/task/26745/stat : 26745 (plingeling) D 26728 26730 26704 0 -1 4202560 141496 1241 0 0 5078 140 0 0 19 0 33 0 678316574 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649923 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=26730/tid=26746] ppid=26728 vsize=10096116 CPUtime=51.78 cores=0-31
/proc/26730/task/26746/stat : 26746 (plingeling) R 26728 26730 26704 0 -1 4202560 120901 1241 0 0 5050 128 0 0 25 0 33 0 678316574 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236443 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=26730/tid=26747] ppid=26728 vsize=10096116 CPUtime=51.18 cores=0-31
/proc/26730/task/26747/stat : 26747 (plingeling) R 26728 26730 26704 0 -1 4202560 92983 1241 0 0 4938 180 0 0 18 0 33 0 678316575 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4253167 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=26730/tid=26748] ppid=26728 vsize=10096116 CPUtime=53.61 cores=0-31
/proc/26730/task/26748/stat : 26748 (plingeling) R 26728 26730 26704 0 -1 4202560 78971 1241 0 0 5220 141 0 0 25 0 33 0 678316576 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4258206 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=26730/tid=26749] ppid=26728 vsize=10096116 CPUtime=52.6 cores=0-31
/proc/26730/task/26749/stat : 26749 (plingeling) D 26728 26730 26704 0 -1 4202560 132037 1241 0 0 5153 107 0 0 19 0 33 0 678316576 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649923 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=26730/tid=26750] ppid=26728 vsize=10096116 CPUtime=52.71 cores=0-31
/proc/26730/task/26750/stat : 26750 (plingeling) R 26728 26730 26704 0 -1 4202560 117441 1241 0 0 5155 116 0 0 25 0 33 0 678316577 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4252361 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26730/tid=26751] ppid=26728 vsize=10096116 CPUtime=52.08 cores=0-31
/proc/26730/task/26751/stat : 26751 (plingeling) R 26728 26730 26704 0 -1 4202560 121487 1241 0 0 5063 145 0 0 25 0 33 0 678316578 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4252176 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=26730/tid=26752] ppid=26728 vsize=10096116 CPUtime=51.78 cores=0-31
/proc/26730/task/26752/stat : 26752 (plingeling) R 26728 26730 26704 0 -1 4202560 119718 1241 0 0 5019 159 0 0 25 0 33 0 678316579 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4253083 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26730/tid=26753] ppid=26728 vsize=10096116 CPUtime=53.24 cores=0-31
/proc/26730/task/26753/stat : 26753 (plingeling) D 26728 26730 26704 0 -1 4202560 78669 1241 0 0 5186 138 0 0 22 0 33 0 678316580 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26730/tid=26754] ppid=26728 vsize=10096116 CPUtime=50.92 cores=0-31
/proc/26730/task/26754/stat : 26754 (plingeling) R 26728 26730 26704 0 -1 4202560 99439 1241 0 0 4943 149 0 0 25 0 33 0 678316581 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4272008 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=26730/tid=26755] ppid=26728 vsize=10096116 CPUtime=50.89 cores=0-31
/proc/26730/task/26755/stat : 26755 (plingeling) R 26728 26730 26704 0 -1 4202560 91148 1241 0 0 4990 99 0 0 25 0 33 0 678316582 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4253219 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=26730/tid=26756] ppid=26728 vsize=10096116 CPUtime=52.17 cores=0-31
/proc/26730/task/26756/stat : 26756 (plingeling) R 26728 26730 26704 0 -1 4202560 107440 1241 0 0 5052 165 0 0 25 0 33 0 678316583 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4252343 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26730/tid=26757] ppid=26728 vsize=10096116 CPUtime=52.22 cores=0-31
/proc/26730/task/26757/stat : 26757 (plingeling) R 26728 26730 26704 0 -1 4202560 104300 1241 0 0 5089 133 0 0 25 0 33 0 678316583 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4248893 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=26730/tid=26758] ppid=26728 vsize=10096116 CPUtime=52.02 cores=0-31
/proc/26730/task/26758/stat : 26758 (plingeling) R 26728 26730 26704 0 -1 4202560 113268 1241 0 0 5025 177 0 0 25 0 33 0 678316584 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4273707 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26730/tid=26759] ppid=26728 vsize=10096116 CPUtime=52.72 cores=0-31
/proc/26730/task/26759/stat : 26759 (plingeling) R 26728 26730 26704 0 -1 4202560 114926 1241 0 0 5110 162 0 0 25 0 33 0 678316585 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4271657 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=26730/tid=26760] ppid=26728 vsize=10096116 CPUtime=53.21 cores=0-31
/proc/26730/task/26760/stat : 26760 (plingeling) R 26728 26730 26704 0 -1 4202560 99805 1241 0 0 5169 152 0 0 25 0 33 0 678316586 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4233447 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=26730/tid=26761] ppid=26728 vsize=10096116 CPUtime=51.69 cores=0-31
/proc/26730/task/26761/stat : 26761 (plingeling) R 26728 26730 26704 0 -1 4202560 137540 1241 0 0 5035 134 0 0 25 0 33 0 678316588 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4287187 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26730/tid=26762] ppid=26728 vsize=10096116 CPUtime=51.95 cores=0-31
/proc/26730/task/26762/stat : 26762 (plingeling) R 26728 26730 26704 0 -1 4202560 136168 1241 0 0 5013 182 0 0 25 0 33 0 678316588 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4252431 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=10096116 CPUtime=51.64 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4202560 129402 1241 0 0 4956 208 0 0 25 0 33 0 678316589 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=26730/tid=26764] ppid=26728 vsize=10096116 CPUtime=52.05 cores=0-31
/proc/26730/task/26764/stat : 26764 (plingeling) R 26728 26730 26704 0 -1 4202560 92567 1241 0 0 5070 135 0 0 25 0 33 0 678316591 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4319240 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=26730/tid=26765] ppid=26728 vsize=10096116 CPUtime=51.81 cores=0-31
/proc/26730/task/26765/stat : 26765 (plingeling) D 26728 26730 26704 0 -1 4202560 121220 1241 0 0 5068 113 0 0 18 0 33 0 678316592 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=26730/tid=26766] ppid=26728 vsize=10096116 CPUtime=51.6 cores=0-31
/proc/26730/task/26766/stat : 26766 (plingeling) R 26728 26730 26704 0 -1 4202560 79252 1241 0 0 5036 124 0 0 18 0 33 0 678316593 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4272705 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26730/tid=26767] ppid=26728 vsize=10096116 CPUtime=52.56 cores=0-31
/proc/26730/task/26767/stat : 26767 (plingeling) R 26728 26730 26704 0 -1 4202560 95845 1241 0 0 5110 146 0 0 25 0 33 0 678316594 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4252117 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26730/tid=26768] ppid=26728 vsize=10096116 CPUtime=52.49 cores=0-31
/proc/26730/task/26768/stat : 26768 (plingeling) R 26728 26730 26704 0 -1 4202560 111989 1241 0 0 5089 160 0 0 25 0 33 0 678316595 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4250017 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26730/tid=26769] ppid=26728 vsize=10096116 CPUtime=52.86 cores=0-31
/proc/26730/task/26769/stat : 26769 (plingeling) R 26728 26730 26704 0 -1 4202560 108773 1241 0 0 5167 119 0 0 25 0 33 0 678316596 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 258651013786 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=26730/tid=26770] ppid=26728 vsize=10096116 CPUtime=50.48 cores=0-31
/proc/26730/task/26770/stat : 26770 (plingeling) R 26728 26730 26704 0 -1 4202560 95651 1241 0 0 4877 171 0 0 25 0 33 0 678316597 10338422784 2258868 268435456000 4194304 4417452 140733366487440 18446744073709551615 4253264 0 0 4096 17442 0 0 0 -1 10 0 0 0
Current children cumulated CPU time (s) 1775.44
Current children cumulated vsize (KiB) 10096116


################
# More data... #
################

/proc/26730/task/26757/stat : 26757 (plingeling) R 26728 26730 26704 0 -1 4202560 1566851 1241 0 0 493926 2043 0 0 25 0 33 0 678316583 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4353557 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26730/tid=26758] ppid=26728 vsize=37589460 CPUtime=4956.53 cores=0-31
/proc/26730/task/26758/stat : 26758 (plingeling) R 26728 26730 26704 0 -1 4202560 1527363 1241 0 0 493285 2368 0 0 25 0 33 0 678316584 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26730/tid=26759] ppid=26728 vsize=37589460 CPUtime=4965.45 cores=0-31
/proc/26730/task/26759/stat : 26759 (plingeling) R 26728 26730 26704 0 -1 4202560 1038748 1241 0 0 495094 1451 0 0 25 0 33 0 678316585 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=26730/tid=26760] ppid=26728 vsize=37589460 CPUtime=4960.17 cores=0-31
/proc/26730/task/26760/stat : 26760 (plingeling) R 26728 26730 26704 0 -1 4202560 1030577 1241 0 0 494557 1460 0 0 25 0 33 0 678316586 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4254433 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=26730/tid=26761] ppid=26728 vsize=37589460 CPUtime=4955.73 cores=0-31
/proc/26730/task/26761/stat : 26761 (plingeling) R 26728 26730 26704 0 -1 4202560 1795109 1241 0 0 492632 2941 0 0 25 0 33 0 678316588 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=26730/tid=26762] ppid=26728 vsize=37589460 CPUtime=4958.73 cores=0-31
/proc/26730/task/26762/stat : 26762 (plingeling) R 26728 26730 26704 0 -1 4202560 941404 1241 0 0 494614 1259 0 0 25 0 33 0 678316588 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4357842 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=37589460 CPUtime=4945.02 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4202560 1325083 1241 0 0 492576 1926 0 0 25 0 33 0 678316589 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4256243 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26730/tid=26764] ppid=26728 vsize=37589460 CPUtime=4942.23 cores=0-31
/proc/26730/task/26764/stat : 26764 (plingeling) R 26728 26730 26704 0 -1 4202560 1329120 1241 0 0 491895 2328 0 0 25 0 33 0 678316591 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4365579 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=26730/tid=26765] ppid=26728 vsize=37589460 CPUtime=4960.65 cores=0-31
/proc/26730/task/26765/stat : 26765 (plingeling) R 26728 26730 26704 0 -1 4202560 1608492 1241 0 0 493889 2176 0 0 25 0 33 0 678316592 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26730/tid=26766] ppid=26728 vsize=37589460 CPUtime=4960.18 cores=0-31
/proc/26730/task/26766/stat : 26766 (plingeling) R 26728 26730 26704 0 -1 4202560 891246 1241 0 0 494808 1210 0 0 25 0 33 0 678316593 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26730/tid=26767] ppid=26728 vsize=37589460 CPUtime=4958.07 cores=0-31
/proc/26730/task/26767/stat : 26767 (plingeling) R 26728 26730 26704 0 -1 4202560 1490476 1241 0 0 493350 2457 0 0 25 0 33 0 678316594 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26730/tid=26768] ppid=26728 vsize=37589460 CPUtime=4960.63 cores=0-31
/proc/26730/task/26768/stat : 26768 (plingeling) R 26728 26730 26704 0 -1 4202560 1115151 1241 0 0 494518 1545 0 0 25 0 33 0 678316595 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4353830 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26730/tid=26769] ppid=26728 vsize=37589460 CPUtime=4958.23 cores=0-31
/proc/26730/task/26769/stat : 26769 (plingeling) R 26728 26730 26704 0 -1 4202560 1189703 1241 0 0 494058 1765 0 0 25 0 33 0 678316596 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26730/tid=26770] ppid=26728 vsize=37589460 CPUtime=4961.21 cores=0-31
/proc/26730/task/26770/stat : 26770 (plingeling) R 26728 26730 26704 0 -1 4202560 1291729 1241 0 0 494259 1862 0 0 25 0 33 0 678316597 38491607040 9166731 268435456000 4194304 4417452 140733366487440 18446744073709551615 258651013786 0 0 4096 17442 0 0 0 -1 10 0 0 0
Current children cumulated CPU time (s) 158714
Current children cumulated vsize (KiB) 37589460



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 32.21 32.14 32.04 32/446 26925
/proc/meminfo: memFree=218706532/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=37522248 CPUtime=159274 cores=0-31
/proc/26730/stat : 26730 (plingeling) S 26728 26730 26704 0 -1 4202496 46373122 1241 0 0 15859892 67499 0 0 18 0 33 0 678305883 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 9380562 9157542 175 55 0 9378448 0
[pid=26730/tid=26739] ppid=26728 vsize=37522248 CPUtime=4980.8 cores=0-31
/proc/26730/task/26739/stat : 26739 (plingeling) R 26728 26730 26704 0 -1 4202560 1284710 1241 0 0 496615 1465 0 0 25 0 33 0 678316571 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4302930 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=26730/tid=26740] ppid=26728 vsize=37522248 CPUtime=4975.94 cores=0-31
/proc/26730/task/26740/stat : 26740 (plingeling) R 26728 26730 26704 0 -1 4202560 1284873 1241 0 0 495541 2053 0 0 25 0 33 0 678316571 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4366528 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=26730/tid=26741] ppid=26728 vsize=37522248 CPUtime=4977.08 cores=0-31
/proc/26730/task/26741/stat : 26741 (plingeling) R 26728 26730 26704 0 -1 4202560 1292430 1241 0 0 495931 1777 0 0 25 0 33 0 678316571 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4365545 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26730/tid=26742] ppid=26728 vsize=37522248 CPUtime=4963.18 cores=0-31
/proc/26730/task/26742/stat : 26742 (plingeling) R 26728 26730 26704 0 -1 4202560 2570989 1241 0 0 492422 3896 0 0 25 0 33 0 678316572 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4251034 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=26730/tid=26743] ppid=26728 vsize=37522248 CPUtime=4973.14 cores=0-31
/proc/26730/task/26743/stat : 26743 (plingeling) R 26728 26730 26704 0 -1 4202560 1116549 1241 0 0 495677 1637 0 0 25 0 33 0 678316572 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26730/tid=26744] ppid=26728 vsize=37522248 CPUtime=4974.03 cores=0-31
/proc/26730/task/26744/stat : 26744 (plingeling) D 26728 26730 26704 0 -1 4202560 1077917 1241 0 0 495742 1661 0 0 18 0 33 0 678316573 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 258651013786 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26730/tid=26745] ppid=26728 vsize=37522248 CPUtime=4980.24 cores=0-31
/proc/26730/task/26745/stat : 26745 (plingeling) R 26728 26730 26704 0 -1 4202560 1342110 1241 0 0 496179 1845 0 0 25 0 33 0 678316574 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=26730/tid=26746] ppid=26728 vsize=37522248 CPUtime=4968.8 cores=0-31
/proc/26730/task/26746/stat : 26746 (plingeling) R 26728 26730 26704 0 -1 4202560 1995807 1241 0 0 493534 3346 0 0 25 0 33 0 678316574 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4232598 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=26730/tid=26747] ppid=26728 vsize=37522248 CPUtime=4974.96 cores=0-31
/proc/26730/task/26747/stat : 26747 (plingeling) R 26728 26730 26704 0 -1 4202560 1878617 1241 0 0 494173 3323 0 0 25 0 33 0 678316575 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=26730/tid=26748] ppid=26728 vsize=37522248 CPUtime=4968.1 cores=0-31
/proc/26730/task/26748/stat : 26748 (plingeling) R 26728 26730 26704 0 -1 4202560 977969 1241 0 0 495356 1454 0 0 25 0 33 0 678316576 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=26730/tid=26749] ppid=26728 vsize=37522248 CPUtime=4978.71 cores=0-31
/proc/26730/task/26749/stat : 26749 (plingeling) R 26728 26730 26704 0 -1 4202560 1628340 1241 0 0 495406 2465 0 0 25 0 33 0 678316576 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4357554 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=26730/tid=26750] ppid=26728 vsize=37522248 CPUtime=4968.65 cores=0-31
/proc/26730/task/26750/stat : 26750 (plingeling) R 26728 26730 26704 0 -1 4202560 1756110 1241 0 0 493650 3215 0 0 25 0 33 0 678316577 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=26730/tid=26751] ppid=26728 vsize=37522248 CPUtime=4975.61 cores=0-31
/proc/26730/task/26751/stat : 26751 (plingeling) R 26728 26730 26704 0 -1 4202560 1369188 1241 0 0 495535 2026 0 0 25 0 33 0 678316578 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4255615 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=26730/tid=26752] ppid=26728 vsize=37522248 CPUtime=4966.37 cores=0-31
/proc/26730/task/26752/stat : 26752 (plingeling) R 26728 26730 26704 0 -1 4202560 871305 1241 0 0 495598 1039 0 0 25 0 33 0 678316579 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=26730/tid=26753] ppid=26728 vsize=37522248 CPUtime=4969.36 cores=0-31
/proc/26730/task/26753/stat : 26753 (plingeling) R 26728 26730 26704 0 -1 4202560 1726948 1241 0 0 494068 2868 0 0 25 0 33 0 678316580 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=26730/tid=26754] ppid=26728 vsize=37522248 CPUtime=4973.19 cores=0-31
/proc/26730/task/26754/stat : 26754 (plingeling) R 26728 26730 26704 0 -1 4202560 1377978 1241 0 0 495173 2146 0 0 25 0 33 0 678316581 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26730/tid=26755] ppid=26728 vsize=37522248 CPUtime=4976.15 cores=0-31
/proc/26730/task/26755/stat : 26755 (plingeling) R 26728 26730 26704 0 -1 4202560 1586469 1241 0 0 495362 2253 0 0 25 0 33 0 678316582 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26730/tid=26756] ppid=26728 vsize=37522248 CPUtime=4973.48 cores=0-31
/proc/26730/task/26756/stat : 26756 (plingeling) R 26728 26730 26704 0 -1 4202560 1008382 1241 0 0 495997 1351 0 0 25 0 33 0 678316583 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236603 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=26730/tid=26757] ppid=26728 vsize=37522248 CPUtime=4977.2 cores=0-31
/proc/26730/task/26757/stat : 26757 (plingeling) R 26728 26730 26704 0 -1 4202560 1583236 1241 0 0 495672 2048 0 0 25 0 33 0 678316583 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4255615 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26730/tid=26758] ppid=26728 vsize=37522248 CPUtime=4974.25 cores=0-31
/proc/26730/task/26758/stat : 26758 (plingeling) R 26728 26730 26704 0 -1 4202560 1527363 1241 0 0 495057 2368 0 0 25 0 33 0 678316584 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26730/tid=26759] ppid=26728 vsize=37522248 CPUtime=4983.17 cores=0-31
/proc/26730/task/26759/stat : 26759 (plingeling) R 26728 26730 26704 0 -1 4202560 1038748 1241 0 0 496866 1451 0 0 25 0 33 0 678316585 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=26730/tid=26760] ppid=26728 vsize=37522248 CPUtime=4977.89 cores=0-31
/proc/26730/task/26760/stat : 26760 (plingeling) R 26728 26730 26704 0 -1 4202560 1030577 1241 0 0 496329 1460 0 0 25 0 33 0 678316586 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4249186 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=26730/tid=26761] ppid=26728 vsize=37522248 CPUtime=4973.32 cores=0-31
/proc/26730/task/26761/stat : 26761 (plingeling) R 26728 26730 26704 0 -1 4202560 1807243 1241 0 0 494358 2974 0 0 25 0 33 0 678316588 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=26730/tid=26762] ppid=26728 vsize=37522248 CPUtime=4976.45 cores=0-31
/proc/26730/task/26762/stat : 26762 (plingeling) R 26728 26730 26704 0 -1 4202560 941404 1241 0 0 496386 1259 0 0 25 0 33 0 678316588 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=37522248 CPUtime=4961.68 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4202560 1505990 1241 0 0 493877 2291 0 0 23 0 33 0 678316589 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 258651013786 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26730/tid=26764] ppid=26728 vsize=37522248 CPUtime=4959.95 cores=0-31
/proc/26730/task/26764/stat : 26764 (plingeling) R 26728 26730 26704 0 -1 4202560 1339962 1241 0 0 493665 2330 0 0 25 0 33 0 678316591 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4353598 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=26730/tid=26765] ppid=26728 vsize=37522248 CPUtime=4978.37 cores=0-31
/proc/26730/task/26765/stat : 26765 (plingeling) R 26728 26730 26704 0 -1 4202560 1608492 1241 0 0 495661 2176 0 0 25 0 33 0 678316592 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26730/tid=26766] ppid=26728 vsize=37522248 CPUtime=4977.9 cores=0-31
/proc/26730/task/26766/stat : 26766 (plingeling) R 26728 26730 26704 0 -1 4202560 891246 1241 0 0 496580 1210 0 0 25 0 33 0 678316593 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26730/tid=26767] ppid=26728 vsize=37522248 CPUtime=4975.79 cores=0-31
/proc/26730/task/26767/stat : 26767 (plingeling) R 26728 26730 26704 0 -1 4202560 1490476 1241 0 0 495122 2457 0 0 25 0 33 0 678316594 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26730/tid=26768] ppid=26728 vsize=37522248 CPUtime=4978.35 cores=0-31
/proc/26730/task/26768/stat : 26768 (plingeling) R 26728 26730 26704 0 -1 4202560 1115151 1241 0 0 496290 1545 0 0 25 0 33 0 678316595 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4236544 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26730/tid=26769] ppid=26728 vsize=37522248 CPUtime=4975.95 cores=0-31
/proc/26730/task/26769/stat : 26769 (plingeling) R 26728 26730 26704 0 -1 4202560 1189703 1241 0 0 495830 1765 0 0 25 0 33 0 678316596 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26730/tid=26770] ppid=26728 vsize=37522248 CPUtime=4978.66 cores=0-31
/proc/26730/task/26770/stat : 26770 (plingeling) R 26728 26730 26704 0 -1 4202560 1344996 1241 0 0 495893 1973 0 0 25 0 33 0 678316597 38422781952 9157542 268435456000 4194304 4417452 140733366487440 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 12 0 0 0
Current children cumulated CPU time (s) 159274
Current children cumulated vsize (KiB) 37522248

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -26730

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 32.21 32.14 32.04 2/416 26926
/proc/meminfo: memFree=226585348/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=0 CPUtime=159275 cores=0-31
/proc/26730/stat : 26730 (plingeling) Z 26728 26730 26704 0 -1 4203548 46373122 1241 0 0 15859913 67562 0 0 18 0 2 0 678305883 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 0 0 0 0 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=0 CPUtime=4962.29 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4203588 1505990 1241 0 0 493877 2352 0 0 25 0 2 0 678316589 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 159275
Current children cumulated vsize (KiB) 0

[startup+5101.4 s]
/proc/loadavg: 32.21 32.14 32.04 2/416 26926
/proc/meminfo: memFree=226585348/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=0 CPUtime=159276 cores=0-31
/proc/26730/stat : 26730 (plingeling) Z 26728 26730 26704 0 -1 4203548 46373122 1241 0 0 15859913 67639 0 0 18 0 2 0 678305883 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 0 0 0 0 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=0 CPUtime=4963.06 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4203588 1505990 1241 0 0 493877 2429 0 0 25 0 2 0 678316589 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 159276
Current children cumulated vsize (KiB) 0

[startup+5101.83 s]
/proc/loadavg: 32.21 32.14 32.04 2/416 26926
/proc/meminfo: memFree=247716864/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=0 CPUtime=159276 cores=0-31
/proc/26730/stat : 26730 (plingeling) Z 26728 26730 26704 0 -1 4203548 46373122 1241 0 0 15859913 67682 0 0 18 0 2 0 678305883 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 0 0 0 0 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=0 CPUtime=4963.49 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4203588 1505990 1241 0 0 493877 2472 0 0 25 0 2 0 678316589 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 159276
Current children cumulated vsize (KiB) 0

[startup+5102 s]
/proc/loadavg: 32.21 32.14 32.04 2/416 26926
/proc/meminfo: memFree=247716864/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=0 CPUtime=159276 cores=0-31
/proc/26730/stat : 26730 (plingeling) Z 26728 26730 26704 0 -1 4203548 46373122 1241 0 0 15859913 67699 0 0 18 0 2 0 678305883 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 0 0 0 0 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=0 CPUtime=4963.66 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4203588 1505990 1241 0 0 493877 2489 0 0 25 0 2 0 678316589 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 159276
Current children cumulated vsize (KiB) 0

[startup+5102.1 s]
/proc/loadavg: 32.21 32.14 32.04 2/416 26926
/proc/meminfo: memFree=247716864/264257624 swapFree=67111528/67111528
[pid=26730] ppid=26728 vsize=0 CPUtime=159276 cores=0-31
/proc/26730/stat : 26730 (plingeling) Z 26728 26730 26704 0 -1 4203548 46373122 1241 0 0 15859913 67709 0 0 18 0 2 0 678305883 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 26 0 0 0
/proc/26730/statm: 0 0 0 0 0 0 0
[pid=26730/tid=26763] ppid=26728 vsize=0 CPUtime=4963.76 cores=0-31
/proc/26730/task/26763/stat : 26763 (plingeling) R 26728 26730 26704 0 -1 4203588 1505990 1241 0 0 493877 2499 0 0 25 0 2 0 678316589 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 159276
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5102.11
CPU time (s): 159276
CPU user time (s): 158599
CPU system time (s): 677.106
CPU usage (%): 3121.77
Max. virtual memory (cumulated for all children) (KiB): 37787648

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 158599
system time used= 677.106
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 46374363
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17508
involuntary context switches= 401301

runsolver used 26.275 second user time and 118.571 second system time

The end

Launcher Data

Begin job on node200 at 2011-05-17 23:36:35
IDJOB=3372486
IDBENCH=82413
IDSOLVER=1852
FILE ID=node200/3372486-1305668190
RUNJOBID= node200-1305668189-26718
PBS_JOBID= 13357372
Free space on /tmp= 61932 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/transport-transport-three-cities-sequential-14nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.030-NOTKNOWN.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372486-1305668190/watcher-3372486-1305668190 -o /tmp/evaluation-result-3372486-1305668190/solver-3372486-1305668190 -C 160000 -W 5100 -M 266338304  plingeling -t 32 HOME/instance-3372486-1305668190.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= c6d3480970bb2819448dd0a800a3cf1c
RANDOM SEED=965169004

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255399704 kB
Buffers:        450072 kB
Cached:        8001376 kB
SwapCached:          0 kB
Active:        7517904 kB
Inactive:      1002484 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255399704 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:          120804 kB
Writeback:           0 kB
AnonPages:       68692 kB
Mapped:          14424 kB
Slab:           254876 kB
PageTables:       4324 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   190300 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61932 MiB
End job on node200 at 2011-05-18 01:01:37