Trace number 3372837

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 160005 5017.2

General information on the benchmark

NameSAT03/industrial/li/li-exam/
li-exam-61.shuffled-as.sat03-366.cnf
MD5SUM3be662940f394fa876137a905247d755
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables28147
Number of clauses108436
Sum of the clauses size373110
Maximum clause length166
Minimum clause length1
Number of clauses of size 114
Number of clauses of size 218786
Number of clauses of size 351588
Number of clauses of size 437440
Number of clauses of size 53
Number of clauses of size over 5605

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372837-1306070457/watcher-3372837-1306070457 -o /tmp/evaluation-result-3372837-1306070457/solver-3372837-1306070457 -C 160000 -W 5100 -M 266338304 plingeling -t 32 HOME/instance-3372837-1306070457.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 29.51 31.51 31.83 1/413 26211
/proc/meminfo: memFree=255493480/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=26880 CPUtime=0.02 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 4775 1245 0 0 1 1 0 0 22 0 1 0 718532143 27525120 4706 268435456000 4194304 4417452 140734594555792 18446744073709551615 258650652667 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 6720 4715 145 55 0 4606 0

[startup+0.0824869 s]
/proc/loadavg: 29.51 31.51 31.83 1/413 26211
/proc/meminfo: memFree=255493480/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=34100 CPUtime=0.07 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 6609 1245 0 0 6 1 0 0 22 0 1 0 718532143 34918400 6540 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232617 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 8525 6540 145 55 0 6411 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 34100

[startup+0.100443 s]
/proc/loadavg: 29.51 31.51 31.83 1/413 26211
/proc/meminfo: memFree=255493480/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=34100 CPUtime=0.08 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 6609 1245 0 0 7 1 0 0 22 0 1 0 718532143 34918400 6540 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232651 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 8525 6540 145 55 0 6411 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 34100

[startup+0.300452 s]
/proc/loadavg: 29.51 31.51 31.83 1/413 26211
/proc/meminfo: memFree=255493480/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=58568 CPUtime=0.28 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 12717 1245 0 0 26 2 0 0 23 0 1 0 718532143 59973632 12648 268435456000 4194304 4417452 140734594555792 18446744073709551615 4231335 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 14642 12648 146 55 0 12528 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 58568

[startup+0.700466 s]
/proc/loadavg: 29.51 31.51 31.83 1/413 26211
/proc/meminfo: memFree=255493480/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=97592 CPUtime=0.68 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 22490 1245 0 0 63 5 0 0 25 0 1 0 718532143 99934208 22420 268435456000 4194304 4417452 140734594555792 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 24398 22420 146 55 0 22284 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 97592

[startup+1.5005 s]
/proc/loadavg: 29.51 31.51 31.83 2/414 26217
/proc/meminfo: memFree=255394720/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=127348 CPUtime=1.48 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 29929 1245 0 0 141 7 0 0 25 0 1 0 718532143 130404352 29859 268435456000 4194304 4417452 140734594555792 18446744073709551615 4231321 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 31837 29859 146 55 0 29723 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 127348

[startup+3.10054 s]
/proc/loadavg: 29.51 31.51 31.83 2/414 26217
/proc/meminfo: memFree=255364960/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=169740 CPUtime=3.09 cores=0-31
/proc/26211/stat : 26211 (plingeling) R 26209 26211 26185 0 -1 4202496 40623 1245 0 0 299 10 0 0 25 0 1 0 718532143 173813760 40457 268435456000 4194304 4417452 140734594555792 18446744073709551615 4274358 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/26211/statm: 42435 40457 146 55 0 40321 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 169740

[startup+6.30054 s]
/proc/loadavg: 27.23 31.01 31.67 2/414 26217
/proc/meminfo: memFree=255251916/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=905660 CPUtime=30.82 cores=0-31
/proc/26211/stat : 26211 (plingeling) S 26209 26211 26185 0 -1 4202496 94455 1245 0 0 3037 45 0 0 19 0 33 0 718532143 927395840 69799 268435456000 4194304 4417452 140734594555792 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 226415 69799 170 55 0 224301 0
Current children cumulated CPU time (s) 30.82
Current children cumulated vsize (KiB) 905660

[startup+12.7017 s]
/proc/loadavg: 27.45 30.99 31.66 33/446 26251
/proc/meminfo: memFree=255019236/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=1132976 CPUtime=233 cores=0-31
/proc/26211/stat : 26211 (plingeling) S 26209 26211 26185 0 -1 4202496 166948 1245 0 0 23175 125 0 0 19 0 33 0 718532143 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 283244 113462 175 55 0 281130 0
[pid=26211/tid=26218] ppid=26209 vsize=1132976 CPUtime=7.1 cores=0-31
/proc/26211/task/26218/stat : 26218 (plingeling) R 26209 26211 26185 0 -1 4202560 2194 1245 0 0 708 2 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252507 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=26211/tid=26219] ppid=26209 vsize=1132976 CPUtime=7.12 cores=0-31
/proc/26211/task/26219/stat : 26219 (plingeling) R 26209 26211 26185 0 -1 4202560 2937 1245 0 0 709 3 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252252 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26211/tid=26220] ppid=26209 vsize=1132976 CPUtime=7.14 cores=0-31
/proc/26211/task/26220/stat : 26220 (plingeling) R 26209 26211 26185 0 -1 4202560 6286 1245 0 0 711 3 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4377903 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=26211/tid=26221] ppid=26209 vsize=1132976 CPUtime=7.17 cores=0-31
/proc/26211/task/26221/stat : 26221 (plingeling) R 26209 26211 26185 0 -1 4202560 336 1245 0 0 717 0 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366005 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26211/tid=26222] ppid=26209 vsize=1132976 CPUtime=7.13 cores=0-31
/proc/26211/task/26222/stat : 26222 (plingeling) R 26209 26211 26185 0 -1 4202560 2388 1245 0 0 711 2 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4273436 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26211/tid=26223] ppid=26209 vsize=1132976 CPUtime=7.09 cores=0-31
/proc/26211/task/26223/stat : 26223 (plingeling) R 26209 26211 26185 0 -1 4202560 2354 1245 0 0 706 3 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4320176 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=26211/tid=26224] ppid=26209 vsize=1132976 CPUtime=7.09 cores=0-31
/proc/26211/task/26224/stat : 26224 (plingeling) R 26209 26211 26185 0 -1 4202560 3283 1245 0 0 705 4 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4258229 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=26211/tid=26225] ppid=26209 vsize=1132976 CPUtime=7.09 cores=0-31
/proc/26211/task/26225/stat : 26225 (plingeling) R 26209 26211 26185 0 -1 4202560 3190 1245 0 0 706 3 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4253264 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26211/tid=26226] ppid=26209 vsize=1132976 CPUtime=7.13 cores=0-31
/proc/26211/task/26226/stat : 26226 (plingeling) R 26209 26211 26185 0 -1 4202560 3997 1245 0 0 709 4 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252336 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=26211/tid=26227] ppid=26209 vsize=1132976 CPUtime=7.1 cores=0-31
/proc/26211/task/26227/stat : 26227 (plingeling) R 26209 26211 26185 0 -1 4202560 4783 1245 0 0 705 5 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4272829 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26211/tid=26228] ppid=26209 vsize=1132976 CPUtime=7.14 cores=0-31
/proc/26211/task/26228/stat : 26228 (plingeling) R 26209 26211 26185 0 -1 4202560 2824 1245 0 0 713 1 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4249991 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=1132976 CPUtime=7.13 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4202560 2572 1245 0 0 710 3 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252757 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26211/tid=26230] ppid=26209 vsize=1132976 CPUtime=6.99 cores=0-31
/proc/26211/task/26230/stat : 26230 (plingeling) R 26209 26211 26185 0 -1 4202560 4025 1245 0 0 698 1 0 0 25 0 33 0 718532691 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252163 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=26211/tid=26231] ppid=26209 vsize=1132976 CPUtime=7.11 cores=0-31
/proc/26211/task/26231/stat : 26231 (plingeling) R 26209 26211 26185 0 -1 4202560 3131 1245 0 0 708 3 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4257391 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26211/tid=26232] ppid=26209 vsize=1132976 CPUtime=7.07 cores=0-31
/proc/26211/task/26232/stat : 26232 (plingeling) R 26209 26211 26185 0 -1 4202560 4446 1245 0 0 704 3 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4377773 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26211/tid=26233] ppid=26209 vsize=1132976 CPUtime=7.11 cores=0-31
/proc/26211/task/26233/stat : 26233 (plingeling) R 26209 26211 26185 0 -1 4202560 2630 1245 0 0 708 3 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4272604 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=26211/tid=26234] ppid=26209 vsize=1132976 CPUtime=7.11 cores=0-31
/proc/26211/task/26234/stat : 26234 (plingeling) R 26209 26211 26185 0 -1 4202560 3024 1245 0 0 707 4 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4253296 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=26211/tid=26235] ppid=26209 vsize=1132976 CPUtime=7.08 cores=0-31
/proc/26211/task/26235/stat : 26235 (plingeling) R 26209 26211 26185 0 -1 4202560 3874 1245 0 0 705 3 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4253280 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=26211/tid=26236] ppid=26209 vsize=1132976 CPUtime=7.12 cores=0-31
/proc/26211/task/26236/stat : 26236 (plingeling) R 26209 26211 26185 0 -1 4202560 3997 1245 0 0 711 1 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4377742 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26211/tid=26237] ppid=26209 vsize=1132976 CPUtime=7.1 cores=0-31
/proc/26211/task/26237/stat : 26237 (plingeling) R 26209 26211 26185 0 -1 4202560 2704 1245 0 0 705 5 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4273098 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=26211/tid=26238] ppid=26209 vsize=1132976 CPUtime=7.12 cores=0-31
/proc/26211/task/26238/stat : 26238 (plingeling) R 26209 26211 26185 0 -1 4202560 4963 1245 0 0 708 4 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252377 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=26211/tid=26239] ppid=26209 vsize=1132976 CPUtime=7.15 cores=0-31
/proc/26211/task/26239/stat : 26239 (plingeling) R 26209 26211 26185 0 -1 4202560 3512 1245 0 0 713 2 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252419 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=26211/tid=26240] ppid=26209 vsize=1132976 CPUtime=7.14 cores=0-31
/proc/26211/task/26240/stat : 26240 (plingeling) R 26209 26211 26185 0 -1 4202560 2729 1245 0 0 713 1 0 0 25 0 33 0 718532692 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4258389 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=26211/tid=26241] ppid=26209 vsize=1132976 CPUtime=7.06 cores=0-31
/proc/26211/task/26241/stat : 26241 (plingeling) R 26209 26211 26185 0 -1 4202560 4624 1245 0 0 701 5 0 0 25 0 33 0 718532693 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252276 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=26211/tid=26242] ppid=26209 vsize=1132976 CPUtime=7.13 cores=0-31
/proc/26211/task/26242/stat : 26242 (plingeling) R 26209 26211 26185 0 -1 4202560 2410 1245 0 0 708 5 0 0 25 0 33 0 718532693 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252507 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26211/tid=26243] ppid=26209 vsize=1132976 CPUtime=7.06 cores=0-31
/proc/26211/task/26243/stat : 26243 (plingeling) R 26209 26211 26185 0 -1 4202560 4099 1245 0 0 702 4 0 0 25 0 33 0 718532694 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252349 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26211/tid=26244] ppid=26209 vsize=1132976 CPUtime=7.08 cores=0-31
/proc/26211/task/26244/stat : 26244 (plingeling) R 26209 26211 26185 0 -1 4202560 3934 1245 0 0 704 4 0 0 25 0 33 0 718532694 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4376574 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26211/tid=26245] ppid=26209 vsize=1132976 CPUtime=7.08 cores=0-31
/proc/26211/task/26245/stat : 26245 (plingeling) R 26209 26211 26185 0 -1 4202560 2334 1245 0 0 708 0 0 0 25 0 33 0 718532694 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4272361 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=26211/tid=26246] ppid=26209 vsize=1132976 CPUtime=7.1 cores=0-31
/proc/26211/task/26246/stat : 26246 (plingeling) R 26209 26211 26185 0 -1 4202560 3423 1245 0 0 706 4 0 0 25 0 33 0 718532694 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252336 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=26211/tid=26247] ppid=26209 vsize=1132976 CPUtime=7.03 cores=0-31
/proc/26211/task/26247/stat : 26247 (plingeling) R 26209 26211 26185 0 -1 4202560 3390 1245 0 0 701 2 0 0 25 0 33 0 718532694 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4252336 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=26211/tid=26248] ppid=26209 vsize=1132976 CPUtime=7.12 cores=0-31
/proc/26211/task/26248/stat : 26248 (plingeling) R 26209 26211 26185 0 -1 4202560 2172 1245 0 0 709 3 0 0 25 0 33 0 718532694 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4258161 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=26211/tid=26249] ppid=26209 vsize=1132976 CPUtime=7.01 cores=0-31
/proc/26211/task/26249/stat : 26249 (plingeling) R 26209 26211 26185 0 -1 4202560 3240 1245 0 0 699 2 0 0 25 0 33 0 718532696 1160167424 113462 268435456000 4194304 4417452 140734594555792 18446744073709551615 4271639 0 0 4096 17442 0 0 0 -1 22 0 0 0
Current children cumulated CPU time (s) 233
Current children cumulated vsize (KiB) 1132976

[startup+25.5022 s]
/proc/loadavg: 28.53 31.05 31.67 33/446 26251
/proc/meminfo: memFree=254992204/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=1173464 CPUtime=642.13 cores=0-31
/proc/26211/stat : 26211 (plingeling) S 26209 26211 26185 0 -1 4202496 178614 1245 0 0 64065 148 0 0 19 0 33 0 718532143 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 293366 122406 175 55 0 291252 0
[pid=26211/tid=26218] ppid=26209 vsize=1173464 CPUtime=19.9 cores=0-31
/proc/26211/task/26218/stat : 26218 (plingeling) R 26209 26211 26185 0 -1 4202560 2194 1245 0 0 1988 2 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=26211/tid=26219] ppid=26209 vsize=1173464 CPUtime=19.91 cores=0-31
/proc/26211/task/26219/stat : 26219 (plingeling) R 26209 26211 26185 0 -1 4202560 3343 1245 0 0 1987 4 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366014 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26211/tid=26220] ppid=26209 vsize=1173464 CPUtime=19.92 cores=0-31
/proc/26211/task/26220/stat : 26220 (plingeling) R 26209 26211 26185 0 -1 4202560 6746 1245 0 0 1988 4 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364637 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=26211/tid=26221] ppid=26209 vsize=1173464 CPUtime=19.97 cores=0-31
/proc/26211/task/26221/stat : 26221 (plingeling) R 26209 26211 26185 0 -1 4202560 1360 1245 0 0 1997 0 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364657 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26211/tid=26222] ppid=26209 vsize=1173464 CPUtime=19.93 cores=0-31
/proc/26211/task/26222/stat : 26222 (plingeling) R 26209 26211 26185 0 -1 4202560 2795 1245 0 0 1990 3 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26211/tid=26223] ppid=26209 vsize=1173464 CPUtime=19.89 cores=0-31
/proc/26211/task/26223/stat : 26223 (plingeling) R 26209 26211 26185 0 -1 4202560 2761 1245 0 0 1986 3 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232644 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=26211/tid=26224] ppid=26209 vsize=1173464 CPUtime=19.89 cores=0-31
/proc/26211/task/26224/stat : 26224 (plingeling) R 26209 26211 26185 0 -1 4202560 3710 1245 0 0 1984 5 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232770 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=26211/tid=26225] ppid=26209 vsize=1173464 CPUtime=19.87 cores=0-31
/proc/26211/task/26225/stat : 26225 (plingeling) R 26209 26211 26185 0 -1 4202560 3596 1245 0 0 1983 4 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26211/tid=26226] ppid=26209 vsize=1173464 CPUtime=19.93 cores=0-31
/proc/26211/task/26226/stat : 26226 (plingeling) R 26209 26211 26185 0 -1 4202560 4403 1245 0 0 1988 5 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232464 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=26211/tid=26227] ppid=26209 vsize=1173464 CPUtime=19.9 cores=0-31
/proc/26211/task/26227/stat : 26227 (plingeling) R 26209 26211 26185 0 -1 4202560 5189 1245 0 0 1984 6 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232523 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26211/tid=26228] ppid=26209 vsize=1173464 CPUtime=19.94 cores=0-31
/proc/26211/task/26228/stat : 26228 (plingeling) R 26209 26211 26185 0 -1 4202560 3251 1245 0 0 1991 3 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4265695 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=1173464 CPUtime=19.93 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4202560 2978 1245 0 0 1990 3 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26211/tid=26230] ppid=26209 vsize=1173464 CPUtime=19.78 cores=0-31
/proc/26211/task/26230/stat : 26230 (plingeling) R 26209 26211 26185 0 -1 4202560 4431 1245 0 0 1976 2 0 0 25 0 33 0 718532691 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4365628 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=26211/tid=26231] ppid=26209 vsize=1173464 CPUtime=19.9 cores=0-31
/proc/26211/task/26231/stat : 26231 (plingeling) R 26209 26211 26185 0 -1 4202560 3537 1245 0 0 1986 4 0 0 25 0 33 0 718532692 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4265437 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26211/tid=26232] ppid=26209 vsize=1173464 CPUtime=19.86 cores=0-31
/proc/26211/task/26232/stat : 26232 (plingeling) R 26209 26211 26185 0 -1 4202560 4873 1245 0 0 1982 4 0 0 25 0 33 0 718532692 1201627136 122406 268435456000 4194304 4417452 140734594555792 18446744073709551615 4265465 0 0 4096 17442 0 0 0 -1 4 0 0 0

################
# More data... #
################

[pid=26211/tid=26229] ppid=26209 vsize=35469408 CPUtime=4945.27 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4202560 1087125 1245 0 0 493310 1217 0 0 25 0 33 0 718532691 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=26211/tid=26230] ppid=26209 vsize=35469408 CPUtime=4952.01 cores=0-31
/proc/26211/task/26230/stat : 26230 (plingeling) R 26209 26211 26185 0 -1 4202560 905533 1245 0 0 494051 1150 0 0 25 0 33 0 718532691 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26211/tid=26231] ppid=26209 vsize=35469408 CPUtime=4952.17 cores=0-31
/proc/26211/task/26231/stat : 26231 (plingeling) R 26209 26211 26185 0 -1 4202560 1071175 1245 0 0 493704 1513 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26211/tid=26232] ppid=26209 vsize=35469408 CPUtime=4944.06 cores=0-31
/proc/26211/task/26232/stat : 26232 (plingeling) R 26209 26211 26185 0 -1 4202560 1187124 1245 0 0 492840 1566 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26211/tid=26233] ppid=26209 vsize=35469408 CPUtime=4947.41 cores=0-31
/proc/26211/task/26233/stat : 26233 (plingeling) R 26209 26211 26185 0 -1 4202560 778010 1245 0 0 493988 753 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=26211/tid=26234] ppid=26209 vsize=35469408 CPUtime=4940.78 cores=0-31
/proc/26211/task/26234/stat : 26234 (plingeling) R 26209 26211 26185 0 -1 4202560 872750 1245 0 0 492993 1085 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=26211/tid=26235] ppid=26209 vsize=35469408 CPUtime=4944.99 cores=0-31
/proc/26211/task/26235/stat : 26235 (plingeling) R 26209 26211 26185 0 -1 4202560 899653 1245 0 0 493182 1317 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26211/tid=26236] ppid=26209 vsize=35469408 CPUtime=4936.77 cores=0-31
/proc/26211/task/26236/stat : 26236 (plingeling) R 26209 26211 26185 0 -1 4202560 846095 1245 0 0 492173 1504 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=26211/tid=26237] ppid=26209 vsize=35469408 CPUtime=4939.97 cores=0-31
/proc/26211/task/26237/stat : 26237 (plingeling) R 26209 26211 26185 0 -1 4202560 843820 1245 0 0 492697 1300 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=26211/tid=26238] ppid=26209 vsize=35469408 CPUtime=4944.56 cores=0-31
/proc/26211/task/26238/stat : 26238 (plingeling) R 26209 26211 26185 0 -1 4202560 1197198 1245 0 0 492949 1507 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=26211/tid=26239] ppid=26209 vsize=35469408 CPUtime=4947.94 cores=0-31
/proc/26211/task/26239/stat : 26239 (plingeling) R 26209 26211 26185 0 -1 4202560 920958 1245 0 0 493639 1155 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=26211/tid=26240] ppid=26209 vsize=35469408 CPUtime=4943.58 cores=0-31
/proc/26211/task/26240/stat : 26240 (plingeling) R 26209 26211 26185 0 -1 4202560 1139532 1245 0 0 492793 1565 0 0 25 0 33 0 718532692 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26211/tid=26241] ppid=26209 vsize=35469408 CPUtime=4950.37 cores=0-31
/proc/26211/task/26241/stat : 26241 (plingeling) R 26209 26211 26185 0 -1 4202560 1232829 1245 0 0 493503 1534 0 0 25 0 33 0 718532693 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=26211/tid=26242] ppid=26209 vsize=35469408 CPUtime=4949.84 cores=0-31
/proc/26211/task/26242/stat : 26242 (plingeling) R 26209 26211 26185 0 -1 4202560 876851 1245 0 0 494082 902 0 0 25 0 33 0 718532693 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4236559 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=26211/tid=26243] ppid=26209 vsize=35469408 CPUtime=4952.62 cores=0-31
/proc/26211/task/26243/stat : 26243 (plingeling) R 26209 26211 26185 0 -1 4202560 1175379 1245 0 0 493619 1643 0 0 25 0 33 0 718532694 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26211/tid=26244] ppid=26209 vsize=35469408 CPUtime=4951.76 cores=0-31
/proc/26211/task/26244/stat : 26244 (plingeling) R 26209 26211 26185 0 -1 4202560 938343 1245 0 0 493949 1227 0 0 25 0 33 0 718532694 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=26211/tid=26245] ppid=26209 vsize=35469408 CPUtime=4949.79 cores=0-31
/proc/26211/task/26245/stat : 26245 (plingeling) R 26209 26211 26185 0 -1 4202560 884049 1245 0 0 493788 1191 0 0 25 0 33 0 718532694 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26211/tid=26246] ppid=26209 vsize=35469408 CPUtime=4951.6 cores=0-31
/proc/26211/task/26246/stat : 26246 (plingeling) R 26209 26211 26185 0 -1 4202560 787171 1245 0 0 494132 1028 0 0 25 0 33 0 718532694 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26211/tid=26247] ppid=26209 vsize=35469408 CPUtime=4944.47 cores=0-31
/proc/26211/task/26247/stat : 26247 (plingeling) R 26209 26211 26185 0 -1 4202560 831196 1245 0 0 493343 1104 0 0 25 0 33 0 718532694 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26211/tid=26248] ppid=26209 vsize=35469408 CPUtime=4943.82 cores=0-31
/proc/26211/task/26248/stat : 26248 (plingeling) R 26209 26211 26185 0 -1 4202560 1123664 1245 0 0 492883 1499 0 0 25 0 33 0 718532694 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366661 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26211/tid=26249] ppid=26209 vsize=35469408 CPUtime=4950.36 cores=0-31
/proc/26211/task/26249/stat : 26249 (plingeling) R 26209 26211 26185 0 -1 4202560 834555 1245 0 0 494159 877 0 0 25 0 33 0 718532696 36320673792 8678206 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 16 0 0 0
Current children cumulated CPU time (s) 158312
Current children cumulated vsize (KiB) 35469408



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5015.2 s]
/proc/loadavg: 32.09 32.09 32.09 33/446 26398
/proc/meminfo: memFree=220378956/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=35775644 CPUtime=160003 cores=0-31
/proc/26211/stat : 26211 (plingeling) S 26209 26211 26185 0 -1 4202496 32494397 1245 0 0 15956118 44153 0 0 19 0 33 0 718532143 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 8943911 8764067 175 55 0 8941797 0
[pid=26211/tid=26218] ppid=26209 vsize=35775644 CPUtime=5005.79 cores=0-31
/proc/26211/task/26218/stat : 26218 (plingeling) R 26209 26211 26185 0 -1 4202560 917670 1245 0 0 499501 1078 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4234640 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26211/tid=26219] ppid=26209 vsize=35775644 CPUtime=4999.39 cores=0-31
/proc/26211/task/26219/stat : 26219 (plingeling) R 26209 26211 26185 0 -1 4202560 1042778 1245 0 0 498378 1561 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=26211/tid=26220] ppid=26209 vsize=35775644 CPUtime=5000.84 cores=0-31
/proc/26211/task/26220/stat : 26220 (plingeling) R 26209 26211 26185 0 -1 4202560 1195707 1245 0 0 498394 1690 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=26211/tid=26221] ppid=26209 vsize=35775644 CPUtime=4987.94 cores=0-31
/proc/26211/task/26221/stat : 26221 (plingeling) R 26209 26211 26185 0 -1 4202560 1387772 1245 0 0 496091 2703 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=26211/tid=26222] ppid=26209 vsize=35775644 CPUtime=5004.36 cores=0-31
/proc/26211/task/26222/stat : 26222 (plingeling) R 26209 26211 26185 0 -1 4202560 1042679 1245 0 0 499098 1338 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=26211/tid=26223] ppid=26209 vsize=35775644 CPUtime=5003.94 cores=0-31
/proc/26211/task/26223/stat : 26223 (plingeling) R 26209 26211 26185 0 -1 4202560 975017 1245 0 0 498888 1506 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=26211/tid=26224] ppid=26209 vsize=35775644 CPUtime=5000.66 cores=0-31
/proc/26211/task/26224/stat : 26224 (plingeling) R 26209 26211 26185 0 -1 4202560 1085727 1245 0 0 498432 1634 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364724 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=26211/tid=26225] ppid=26209 vsize=35775644 CPUtime=5004.39 cores=0-31
/proc/26211/task/26225/stat : 26225 (plingeling) R 26209 26211 26185 0 -1 4202560 1036615 1245 0 0 498941 1498 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26211/tid=26226] ppid=26209 vsize=35775644 CPUtime=5002.61 cores=0-31
/proc/26211/task/26226/stat : 26226 (plingeling) R 26209 26211 26185 0 -1 4202560 913146 1245 0 0 499029 1232 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=26211/tid=26227] ppid=26209 vsize=35775644 CPUtime=5000.06 cores=0-31
/proc/26211/task/26227/stat : 26227 (plingeling) R 26209 26211 26185 0 -1 4202560 941210 1245 0 0 499005 1001 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=26211/tid=26228] ppid=26209 vsize=35775644 CPUtime=4994.03 cores=0-31
/proc/26211/task/26228/stat : 26228 (plingeling) R 26209 26211 26185 0 -1 4202560 1179503 1245 0 0 497686 1717 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=35775644 CPUtime=4998.18 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4202560 1103509 1245 0 0 498556 1262 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=26211/tid=26230] ppid=26209 vsize=35775644 CPUtime=5004.92 cores=0-31
/proc/26211/task/26230/stat : 26230 (plingeling) R 26209 26211 26185 0 -1 4202560 905533 1245 0 0 499342 1150 0 0 25 0 33 0 718532691 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=26211/tid=26231] ppid=26209 vsize=35775644 CPUtime=5005.08 cores=0-31
/proc/26211/task/26231/stat : 26231 (plingeling) R 26209 26211 26185 0 -1 4202560 1071175 1245 0 0 498995 1513 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=26211/tid=26232] ppid=26209 vsize=35775644 CPUtime=4996.97 cores=0-31
/proc/26211/task/26232/stat : 26232 (plingeling) R 26209 26211 26185 0 -1 4202560 1187124 1245 0 0 498131 1566 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26211/tid=26233] ppid=26209 vsize=35775644 CPUtime=5000.32 cores=0-31
/proc/26211/task/26233/stat : 26233 (plingeling) R 26209 26211 26185 0 -1 4202560 778010 1245 0 0 499279 753 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=26211/tid=26234] ppid=26209 vsize=35775644 CPUtime=4993.69 cores=0-31
/proc/26211/task/26234/stat : 26234 (plingeling) R 26209 26211 26185 0 -1 4202560 889134 1245 0 0 498281 1088 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=26211/tid=26235] ppid=26209 vsize=35775644 CPUtime=4997.9 cores=0-31
/proc/26211/task/26235/stat : 26235 (plingeling) R 26209 26211 26185 0 -1 4202560 899653 1245 0 0 498473 1317 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=26211/tid=26236] ppid=26209 vsize=35775644 CPUtime=4989.67 cores=0-31
/proc/26211/task/26236/stat : 26236 (plingeling) R 26209 26211 26185 0 -1 4202560 846095 1245 0 0 497463 1504 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=26211/tid=26237] ppid=26209 vsize=35775644 CPUtime=4992.89 cores=0-31
/proc/26211/task/26237/stat : 26237 (plingeling) R 26209 26211 26185 0 -1 4202560 843820 1245 0 0 497989 1300 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=26211/tid=26238] ppid=26209 vsize=35775644 CPUtime=4995.96 cores=0-31
/proc/26211/task/26238/stat : 26238 (plingeling) R 26209 26211 26185 0 -1 4202560 1197198 1245 0 0 498089 1507 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=26211/tid=26239] ppid=26209 vsize=35775644 CPUtime=5000.85 cores=0-31
/proc/26211/task/26239/stat : 26239 (plingeling) R 26209 26211 26185 0 -1 4202560 920958 1245 0 0 498930 1155 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4358082 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=26211/tid=26240] ppid=26209 vsize=35775644 CPUtime=4996.49 cores=0-31
/proc/26211/task/26240/stat : 26240 (plingeling) R 26209 26211 26185 0 -1 4202560 1139532 1245 0 0 498084 1565 0 0 25 0 33 0 718532692 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4232685 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26211/tid=26241] ppid=26209 vsize=35775644 CPUtime=5003.28 cores=0-31
/proc/26211/task/26241/stat : 26241 (plingeling) R 26209 26211 26185 0 -1 4202560 1232829 1245 0 0 498794 1534 0 0 25 0 33 0 718532693 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=26211/tid=26242] ppid=26209 vsize=35775644 CPUtime=5002.37 cores=0-31
/proc/26211/task/26242/stat : 26242 (plingeling) R 26209 26211 26185 0 -1 4202560 876862 1245 0 0 499335 902 0 0 25 0 33 0 718532693 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=26211/tid=26243] ppid=26209 vsize=35775644 CPUtime=5005.42 cores=0-31
/proc/26211/task/26243/stat : 26243 (plingeling) R 26209 26211 26185 0 -1 4202560 1180499 1245 0 0 498898 1644 0 0 25 0 33 0 718532694 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=26211/tid=26244] ppid=26209 vsize=35775644 CPUtime=5004.33 cores=0-31
/proc/26211/task/26244/stat : 26244 (plingeling) R 26209 26211 26185 0 -1 4202560 1065141 1245 0 0 498998 1435 0 0 25 0 33 0 718532694 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=26211/tid=26245] ppid=26209 vsize=35775644 CPUtime=5002.7 cores=0-31
/proc/26211/task/26245/stat : 26245 (plingeling) R 26209 26211 26185 0 -1 4202560 1000008 1245 0 0 498812 1458 0 0 25 0 33 0 718532694 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=26211/tid=26246] ppid=26209 vsize=35775644 CPUtime=5004.51 cores=0-31
/proc/26211/task/26246/stat : 26246 (plingeling) R 26209 26211 26185 0 -1 4202560 787171 1245 0 0 499423 1028 0 0 25 0 33 0 718532694 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=26211/tid=26247] ppid=26209 vsize=35775644 CPUtime=4997.38 cores=0-31
/proc/26211/task/26247/stat : 26247 (plingeling) R 26209 26211 26185 0 -1 4202560 832960 1245 0 0 498632 1106 0 0 25 0 33 0 718532694 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=26211/tid=26248] ppid=26209 vsize=35775644 CPUtime=4996.73 cores=0-31
/proc/26211/task/26248/stat : 26248 (plingeling) R 26209 26211 26185 0 -1 4202560 1123664 1245 0 0 498174 1499 0 0 25 0 33 0 718532694 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=26211/tid=26249] ppid=26209 vsize=35775644 CPUtime=5003.27 cores=0-31
/proc/26211/task/26249/stat : 26249 (plingeling) R 26209 26211 26185 0 -1 4202560 834555 1245 0 0 499450 877 0 0 25 0 33 0 718532696 36634259456 8764067 268435456000 4194304 4417452 140734594555792 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 16 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 35775644

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5015.8 s]
/proc/loadavg: 32.09 32.09 32.09 33/446 26398
/proc/meminfo: memFree=220378956/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=0 CPUtime=160004 cores=0-31
/proc/26211/stat : 26211 (plingeling) Z 26209 26211 26185 0 -1 4203548 32494397 1245 0 0 15956140 44210 0 0 18 0 2 0 718532143 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 0 0 0 0 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=0 CPUtime=4998.76 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4203588 1103509 1245 0 0 498557 1319 0 0 25 0 2 0 718532691 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 0

[startup+5016.6 s]
/proc/loadavg: 32.09 32.09 32.09 2/416 26399
/proc/meminfo: memFree=230376072/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=0 CPUtime=160004 cores=0-31
/proc/26211/stat : 26211 (plingeling) Z 26209 26211 26185 0 -1 4203548 32494397 1245 0 0 15956140 44290 0 0 18 0 2 0 718532143 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 0 0 0 0 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=0 CPUtime=4999.56 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4203588 1103509 1245 0 0 498557 1399 0 0 25 0 2 0 718532691 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 0

[startup+5017 s]
/proc/loadavg: 32.09 32.09 32.09 2/416 26399
/proc/meminfo: memFree=251806000/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=0 CPUtime=160005 cores=0-31
/proc/26211/stat : 26211 (plingeling) Z 26209 26211 26185 0 -1 4203548 32494397 1245 0 0 15956140 44332 0 0 18 0 2 0 718532143 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 0 0 0 0 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=0 CPUtime=4999.97 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4203588 1103509 1245 0 0 498557 1440 0 0 25 0 2 0 718532691 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

[startup+5017.2 s]
/proc/loadavg: 32.09 32.09 32.09 2/416 26399
/proc/meminfo: memFree=251806000/264257624 swapFree=67111528/67111528
[pid=26211] ppid=26209 vsize=0 CPUtime=160005 cores=0-31
/proc/26211/stat : 26211 (plingeling) Z 26209 26211 26185 0 -1 4203548 32494397 1245 0 0 15956140 44350 0 0 18 0 2 0 718532143 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 24 0 0 0
/proc/26211/statm: 0 0 0 0 0 0 0
[pid=26211/tid=26229] ppid=26209 vsize=0 CPUtime=5000.16 cores=0-31
/proc/26211/task/26229/stat : 26229 (plingeling) R 26209 26211 26185 0 -1 4203588 1103509 1245 0 0 498557 1459 0 0 25 0 2 0 718532691 0 0 268435456000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5017.2
CPU time (s): 160005
CPU user time (s): 159561
CPU system time (s): 443.516
CPU usage (%): 3189.13
Max. virtual memory (cumulated for all children) (KiB): 36034336

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159561
system time used= 443.516
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 32495642
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7388
involuntary context switches= 405986

runsolver used 29.1256 second user time and 112.459 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-22 15:20:57
IDJOB=3372837
IDBENCH=84223
IDSOLVER=1852
FILE ID=node200/3372837-1306070457
RUNJOBID= node200-1306070457-26199
PBS_JOBID= 13387814
Free space on /tmp= 62044 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT03/industrial/li/li-exam/li-exam-61.shuffled-as.sat03-366.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372837-1306070457/watcher-3372837-1306070457 -o /tmp/evaluation-result-3372837-1306070457/solver-3372837-1306070457 -C 160000 -W 5100 -M 266338304  plingeling -t 32 HOME/instance-3372837-1306070457.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 3be662940f394fa876137a905247d755
RANDOM SEED=304027457

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255493628 kB
Buffers:        455336 kB
Cached:        7901872 kB
SwapCached:          0 kB
Active:        7452820 kB
Inactive:       972980 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255493628 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            2912 kB
Writeback:           0 kB
AnonPages:       70168 kB
Mapped:          13484 kB
Slab:           253048 kB
PageTables:       3996 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   177324 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62044 MiB
End job on node200 at 2011-05-22 16:44:35