Trace number 3347573

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160002 5007.35

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_10_keyfind_1.cnf
MD5SUMf35f3aee72237c1bf3d673feb4883756
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables8080
Number of clauses96704
Sum of the clauses size539496
Maximum clause length8
Minimum clause length1
Number of clauses of size 1336
Number of clauses of size 21152
Number of clauses of size 315040
Number of clauses of size 42944
Number of clauses of size 518432
Number of clauses of size over 558800

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347573-1305357136/watcher-3347573-1305357136 -o /tmp/evaluation-result-3347573-1305357136/solver-3347573-1305357136 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347573-1305357136.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.28 31.07 31.73 1/413 29147
/proc/meminfo: memFree=255607436/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 21 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0

[startup+0.106089 s]
/proc/loadavg: 27.28 31.07 31.73 1/413 29147
/proc/meminfo: memFree=255607436/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 21 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.201098 s]
/proc/loadavg: 27.28 31.07 31.73 1/413 29147
/proc/meminfo: memFree=255607436/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 20 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.30111 s]
/proc/loadavg: 27.28 31.07 31.73 1/413 29147
/proc/meminfo: memFree=255607436/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 20 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.701075 s]
/proc/loadavg: 27.28 31.07 31.73 1/413 29147
/proc/meminfo: memFree=255607436/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 20 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50114 s]
/proc/loadavg: 27.28 31.07 31.73 6/419 29158
/proc/meminfo: memFree=255405516/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
[pid=29148] ppid=29147 vsize=47992 CPUtime=1.48 cores=0
/proc/29148/stat : 29148 (cryptominisat) R 29147 29147 29121 0 -1 4202496 9481 0 0 0 146 2 0 0 20 0 1 0 647200026 49143808 7312 268435456000 4194304 4587354 140733212970528 18446744073709551615 4238003 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29148/statm: 11998 7312 330 96 0 8560 0
[pid=29149] ppid=29147 vsize=28688 CPUtime=1.48 cores=1
/proc/29149/stat : 29149 (clasp) R 29147 29147 29121 0 -1 4202496 5338 0 0 0 146 2 0 0 20 0 1 0 647200026 29376512 4488 268435456000 4194304 4785740 140737138674304 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29149/statm: 7172 4488 339 145 0 4217 0
[pid=29150] ppid=29147 vsize=462164 CPUtime=1.48 cores=2
/proc/29150/stat : 29150 (TNM) R 29147 29147 29121 0 -1 4202496 3220 0 0 0 147 1 0 0 20 0 1 0 647200026 473255936 3117 268435456000 134512640 134534564 4289915312 18446744073709551615 134524556 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29150/statm: 115541 3117 91 6 0 114206 0
[pid=29151] ppid=29147 vsize=33280 CPUtime=1.48 cores=3
/proc/29151/stat : 29151 (march_hi) R 29147 29147 29121 0 -1 4202496 17669 0 0 0 144 4 0 0 20 0 1 0 647200026 34078720 5862 268435456000 4194304 4331988 140734958892672 18446744073709551615 258650616632 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29151/statm: 8320 5862 151 34 0 6762 0
[pid=29152] ppid=29147 vsize=135184 CPUtime=1.47 cores=4-31
/proc/29152/stat : 29152 (plingeling) R 29147 29147 29121 0 -1 4202496 32564 1248 0 0 139 8 0 0 21 0 1 0 647200026 138428416 32448 268435456000 4194304 4356188 140736043052320 18446744073709551615 4212999 0 0 4096 17442 0 0 0 17 8 0 0 0
/proc/29152/statm: 33796 32448 129 40 0 32340 0
Current children cumulated CPU time (s) 7.39
Current children cumulated vsize (KiB) 725632

[startup+3.10213 s]
/proc/loadavg: 27.28 31.07 31.73 33/447 29186
/proc/meminfo: memFree=255156748/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
[pid=29148] ppid=29147 vsize=1443536 CPUtime=3.08 cores=0
/proc/29148/stat : 29148 (cryptominisat) R 29147 29147 29121 0 -1 4202496 12634 0 0 0 305 3 0 0 25 0 1 0 647200026 1478180864 9051 268435456000 4194304 4587354 140733212970528 18446744073709551615 258650652594 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29148/statm: 360884 9051 336 96 0 357446 0
[pid=29149] ppid=29147 vsize=30904 CPUtime=3.08 cores=1
/proc/29149/stat : 29149 (clasp) R 29147 29147 29121 0 -1 4202496 5890 0 0 0 306 2 0 0 25 0 1 0 647200026 31645696 5040 268435456000 4194304 4785740 140737138674304 18446744073709551615 4553071 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29149/statm: 7726 5040 339 145 0 4771 0
[pid=29150] ppid=29147 vsize=466392 CPUtime=3.08 cores=2
/proc/29150/stat : 29150 (TNM) R 29147 29147 29121 0 -1 4202496 4003 0 0 0 307 1 0 0 25 0 1 0 647200026 477585408 3900 268435456000 134512640 134534564 4289915312 18446744073709551615 3901829 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29150/statm: 116598 3900 95 6 0 115263 0
[pid=29151] ppid=29147 vsize=87720 CPUtime=3.09 cores=3
/proc/29151/stat : 29151 (march_hi) R 29147 29147 29121 0 -1 4202496 48158 0 0 0 298 11 0 0 25 0 1 0 647200026 89825280 14220 268435456000 4194304 4331988 140734958892672 18446744073709551615 4210951 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29151/statm: 21930 14220 180 34 0 20372 0
[pid=29152] ppid=29147 vsize=930480 CPUtime=35.77 cores=4-31
/proc/29152/stat : 29152 (plingeling) S 29147 29147 29121 0 -1 4202496 101665 1248 0 0 3432 145 0 0 18 0 29 0 647200026 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 22 0 0 0
/proc/29152/statm: 232620 78353 158 40 0 231164 0
[pid=29152/tid=29159] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29159/stat : 29159 (plingeling) R 29147 29147 29121 0 -1 4202560 1571 1248 0 0 117 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4233408 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=29152/tid=29160] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29160/stat : 29160 (plingeling) R 29147 29147 29121 0 -1 4202560 1491 1248 0 0 117 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4231350 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29152/tid=29161] ppid=29147 vsize=930480 CPUtime=1.22 cores=4-31
/proc/29152/task/29161/stat : 29161 (plingeling) R 29147 29147 29121 0 -1 4202560 1452 1248 0 0 118 4 0 0 25 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4231367 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=29152/tid=29162] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29162/stat : 29162 (plingeling) R 29147 29147 29121 0 -1 4202560 1333 1248 0 0 114 5 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4334437 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=29152/tid=29163] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29163/stat : 29163 (plingeling) R 29147 29147 29121 0 -1 4202560 1428 1248 0 0 115 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4300680 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29152/tid=29164] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29164/stat : 29164 (plingeling) R 29147 29147 29121 0 -1 4202560 1368 1248 0 0 115 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4301736 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=29152/tid=29165] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29165/stat : 29165 (plingeling) R 29147 29147 29121 0 -1 4202560 1421 1248 0 0 115 4 0 0 25 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4217742 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29152/tid=29166] ppid=29147 vsize=930480 CPUtime=1.17 cores=4-31
/proc/29152/task/29166/stat : 29166 (plingeling) R 29147 29147 29121 0 -1 4202560 1457 1248 0 0 113 4 0 0 25 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4232971 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=29152/tid=29167] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29167/stat : 29167 (plingeling) R 29147 29147 29121 0 -1 4202560 1449 1248 0 0 117 4 0 0 25 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4216149 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=29152/tid=29168] ppid=29147 vsize=930480 CPUtime=1.23 cores=4-31
/proc/29152/task/29168/stat : 29168 (plingeling) R 29147 29147 29121 0 -1 4202560 1473 1248 0 0 119 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4232894 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=29152/tid=29169] ppid=29147 vsize=930480 CPUtime=1.2 cores=4-31
/proc/29152/task/29169/stat : 29169 (plingeling) R 29147 29147 29121 0 -1 4202560 1400 1248 0 0 116 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4231370 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=29152/tid=29170] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29170/stat : 29170 (plingeling) R 29147 29147 29121 0 -1 4202560 1735 1248 0 0 113 6 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=29152/tid=29171] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29171/stat : 29171 (plingeling) R 29147 29147 29121 0 -1 4202560 1457 1248 0 0 117 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4231350 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29152/tid=29172] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29172/stat : 29172 (plingeling) R 29147 29147 29121 0 -1 4202560 1364 1248 0 0 115 4 0 0 19 0 29 0 647200209 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4233312 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=29152/tid=29173] ppid=29147 vsize=930480 CPUtime=1.18 cores=4-31
/proc/29152/task/29173/stat : 29173 (plingeling) R 29147 29147 29121 0 -1 4202560 1305 1248 0 0 114 4 0 0 22 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4303196 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=29152/tid=29174] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29174/stat : 29174 (plingeling) R 29147 29147 29121 0 -1 4202560 1391 1248 0 0 117 4 0 0 19 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4233150 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=29152/tid=29175] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29175/stat : 29175 (plingeling) R 29147 29147 29121 0 -1 4202560 1508 1248 0 0 118 3 0 0 19 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4216142 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=29152/tid=29176] ppid=29147 vsize=930480 CPUtime=1.18 cores=4-31
/proc/29152/task/29176/stat : 29176 (plingeling) R 29147 29147 29121 0 -1 4202560 1468 1248 0 0 112 6 0 0 21 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4231065 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=29152/tid=29177] ppid=29147 vsize=930480 CPUtime=1.24 cores=4-31
/proc/29152/task/29177/stat : 29177 (plingeling) R 29147 29147 29121 0 -1 4202560 1447 1248 0 0 120 4 0 0 19 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4301491 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=29152/tid=29178] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29178/stat : 29178 (plingeling) R 29147 29147 29121 0 -1 4202560 1524 1248 0 0 113 6 0 0 21 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332320 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=29152/tid=29179] ppid=29147 vsize=930480 CPUtime=1.22 cores=4-31
/proc/29152/task/29179/stat : 29179 (plingeling) R 29147 29147 29121 0 -1 4202560 1448 1248 0 0 118 4 0 0 19 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4217704 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=29152/tid=29180] ppid=29147 vsize=930480 CPUtime=1.19 cores=4-31
/proc/29152/task/29180/stat : 29180 (plingeling) R 29147 29147 29121 0 -1 4202560 1476 1248 0 0 114 5 0 0 21 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4231564 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=29152/tid=29181] ppid=29147 vsize=930480 CPUtime=1.22 cores=4-31
/proc/29152/task/29181/stat : 29181 (plingeling) R 29147 29147 29121 0 -1 4202560 1472 1248 0 0 119 3 0 0 22 0 29 0 647200210 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4233438 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=29152/tid=29182] ppid=29147 vsize=930480 CPUtime=1.2 cores=4-31
/proc/29152/task/29182/stat : 29182 (plingeling) R 29147 29147 29121 0 -1 4202560 1367 1248 0 0 116 4 0 0 22 0 29 0 647200211 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4233312 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=29152/tid=29183] ppid=29147 vsize=930480 CPUtime=1.22 cores=4-31
/proc/29152/task/29183/stat : 29183 (plingeling) R 29147 29147 29121 0 -1 4202560 1446 1248 0 0 118 4 0 0 22 0 29 0 647200211 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4302050 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=29152/tid=29184] ppid=29147 vsize=930480 CPUtime=1.2 cores=4-31
/proc/29152/task/29184/stat : 29184 (plingeling) R 29147 29147 29121 0 -1 4202560 1303 1248 0 0 116 4 0 0 22 0 29 0 647200211 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4230201 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=29152/tid=29185] ppid=29147 vsize=930480 CPUtime=1.2 cores=4-31
/proc/29152/task/29185/stat : 29185 (plingeling) R 29147 29147 29121 0 -1 4202560 1389 1248 0 0 117 3 0 0 19 0 29 0 647200211 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4218227 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=29152/tid=29186] ppid=29147 vsize=930480 CPUtime=1.21 cores=4-31
/proc/29152/task/29186/stat : 29186 (plingeling) R 29147 29147 29121 0 -1 4202560 1448 1248 0 0 117 4 0 0 22 0 29 0 647200211 952811520 78353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4210765 0 0 4096 17442 0 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 48.1
Current children cumulated vsize (KiB) 2977356

[startup+6.30224 s]
/proc/loadavg: 27.66 31.09 31.74 34/447 29186
/proc/meminfo: memFree=255123616/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
[pid=29148] ppid=29147 vsize=1462304 CPUtime=6.29 cores=0
/proc/29148/stat : 29148 (cryptominisat) R 29147 29147 29121 0 -1 4202496 14485 0 0 0 625 4 0 0 25 0 1 0 647200026 1497399296 9449 268435456000 4194304 4587354 140733212970528 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29148/statm: 365576 9449 336 96 0 362138 0
[pid=29149] ppid=29147 vsize=33480 CPUtime=6.28 cores=1
/proc/29149/stat : 29149 (clasp) R 29147 29147 29121 0 -1 4202496 6533 0 0 0 626 2 0 0 25 0 1 0 647200026 34283520 5683 268435456000 4194304 4785740 140737138674304 18446744073709551615 4312643 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29149/statm: 8370 5683 339 145 0 5415 0
[pid=29150] ppid=29147 vsize=466392 CPUtime=6.28 cores=2
/proc/29150/stat : 29150 (TNM) R 29147 29147 29121 0 -1 4202496 4003 0 0 0 627 1 0 0 25 0 1 0 647200026 477585408 3900 268435456000 134512640 134534564 4289915312 18446744073709551615 134528661 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29150/statm: 116598 3900 95 6 0 115263 0
[pid=29151] ppid=29147 vsize=87932 CPUtime=6.28 cores=3
/proc/29151/stat : 29151 (march_hi) R 29147 29147 29121 0 -1 4202496 49174 0 0 0 617 11 0 0 25 0 1 0 647200026 90042368 15232 268435456000 4194304 4331988 140734958892672 18446744073709551615 258650350841 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29151/statm: 21983 15232 176 34 0 20425 0
[pid=29152] ppid=29147 vsize=968844 CPUtime=125.26 cores=4-31
/proc/29152/stat : 29152 (plingeling) S 29147 29147 29121 0 -1 4202496 115287 1248 0 0 12378 148 0 0 18 0 29 0 647200026 992096256 90047 268435456000 4194304 4356188 140736043052320 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 22 0 0 0
/proc/29152/statm: 242211 90047 158 40 0 240755 0
[pid=29152/tid=29159] ppid=29147 vsize=968844 CPUtime=4.41 cores=4-31
/proc/29152/task/29159/stat : 29159 (plingeling) R 29147 29147 29121 0 -1 4202560 1684 1248 0 0 437 4 0 0 25 0 29 0 647200209 992096256 90047 268435456000 4194304 4356188 140736043052320 18446744073709551615 4280568 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=29152/tid=29160] ppid=29147 vsize=968844 CPUtime=4.4 cores=4-31
/proc/29152/task/29160/stat : 29160 (plingeling) R 29147 29147 29121 0 -1 4202560 2104 1248 0 0 436 4 0 0 25 0 29 0 647200209 992096256 90047 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331407 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29152/tid=29161] ppid=29147 vsize=968844 CPUtime=4.42 cores=4-31
/proc/29152/task/29161/stat : 29161 (plingeling) R 29147 29147 29121 0 -1 4202560 1953 1248 0 0 438 4 0 0 25 0 29 0 647200209 992096256 90047 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=29152/tid=29162] ppid=29147 vsize=968844 CPUtime=4.38 cores=4-31
/proc/29152/task/29162/stat : 29162 (plingeling) R 29147 29147 29121 0 -1 4202560 2148 1248 0 0 433 5 0 0 25 0 29 0 647200209 992096256 90047 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=29152/tid=29163] ppid=29147 vsize=968844 CPUtime=4.39 cores=4-31
/proc/29152/task/29163/stat : 29163 (plingeling) R 29147 29147 29121 0 -1 4202560 1977 1248 0 0 435 4 0 0 25 0 29 0 647200209 992096256 90047 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29152/tid=29164] ppid=29147 vsize=968844 CPUtime=4.39 cores=4-31

################
# More data... #
################

[pid=29150] ppid=29147 vsize=466396 CPUtime=4961.34 cores=2
/proc/29150/stat : 29150 (TNM) R 29147 29147 29121 0 -1 4202496 4012 0 0 0 496133 1 0 0 25 0 1 0 647200026 477589504 3909 268435456000 134512640 134534564 4289915312 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29150/statm: 116599 3909 103 6 0 115264 0
[pid=29151] ppid=29147 vsize=223156 CPUtime=4961.15 cores=3
/proc/29151/stat : 29151 (march_hi) R 29147 29147 29121 0 -1 4202496 76226 0 0 0 496070 45 0 0 25 0 1 0 647200026 228511744 42284 268435456000 4194304 4331988 140734958892672 18446744073709551615 4308792 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29151/statm: 55789 42284 176 34 0 54231 0
[pid=29152] ppid=29147 vsize=2342008 CPUtime=138718 cores=4-31
/proc/29152/stat : 29152 (plingeling) S 29147 29147 29121 0 -1 4202496 6584554 1248 0 0 13867819 3939 0 0 18 0 29 0 647200026 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 22 0 0 0
/proc/29152/statm: 585502 436001 158 40 0 584046 0
[pid=29152/tid=29159] ppid=29147 vsize=2342008 CPUtime=4959.42 cores=4-31
/proc/29152/task/29159/stat : 29159 (plingeling) R 29147 29147 29121 0 -1 4202560 229939 1248 0 0 495794 148 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=29152/tid=29160] ppid=29147 vsize=2342008 CPUtime=4959.43 cores=4-31
/proc/29152/task/29160/stat : 29160 (plingeling) R 29147 29147 29121 0 -1 4202560 243700 1248 0 0 495857 86 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331119 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29152/tid=29161] ppid=29147 vsize=2342008 CPUtime=4958.41 cores=4-31
/proc/29152/task/29161/stat : 29161 (plingeling) R 29147 29147 29121 0 -1 4202560 222282 1248 0 0 495713 128 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331193 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=29152/tid=29162] ppid=29147 vsize=2342008 CPUtime=4959.36 cores=4-31
/proc/29152/task/29162/stat : 29162 (plingeling) R 29147 29147 29121 0 -1 4202560 223716 1248 0 0 495809 127 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=29152/tid=29163] ppid=29147 vsize=2342008 CPUtime=4956.3 cores=4-31
/proc/29152/task/29163/stat : 29163 (plingeling) R 29147 29147 29121 0 -1 4202560 236387 1248 0 0 495519 111 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=29152/tid=29164] ppid=29147 vsize=2342008 CPUtime=4959.32 cores=4-31
/proc/29152/task/29164/stat : 29164 (plingeling) R 29147 29147 29121 0 -1 4202560 236460 1248 0 0 495808 124 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=29152/tid=29165] ppid=29147 vsize=2342008 CPUtime=4959.42 cores=4-31
/proc/29152/task/29165/stat : 29165 (plingeling) R 29147 29147 29121 0 -1 4202560 231762 1248 0 0 495829 113 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29152/tid=29166] ppid=29147 vsize=2342008 CPUtime=4959.41 cores=4-31
/proc/29152/task/29166/stat : 29166 (plingeling) R 29147 29147 29121 0 -1 4202560 200552 1248 0 0 495722 219 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=29152/tid=29167] ppid=29147 vsize=2342008 CPUtime=4959.28 cores=4-31
/proc/29152/task/29167/stat : 29167 (plingeling) R 29147 29147 29121 0 -1 4202560 244569 1248 0 0 495793 135 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=29152/tid=29168] ppid=29147 vsize=2342008 CPUtime=4959.19 cores=4-31
/proc/29152/task/29168/stat : 29168 (plingeling) R 29147 29147 29121 0 -1 4202560 300503 1248 0 0 495752 167 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=29152/tid=29169] ppid=29147 vsize=2342008 CPUtime=4958.2 cores=4-31
/proc/29152/task/29169/stat : 29169 (plingeling) R 29147 29147 29121 0 -1 4202560 228988 1248 0 0 495687 133 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=29152/tid=29170] ppid=29147 vsize=2342008 CPUtime=4958.56 cores=4-31
/proc/29152/task/29170/stat : 29170 (plingeling) R 29147 29147 29121 0 -1 4202560 248496 1248 0 0 495728 128 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=29152/tid=29171] ppid=29147 vsize=2342008 CPUtime=4959.21 cores=4-31
/proc/29152/task/29171/stat : 29171 (plingeling) R 29147 29147 29121 0 -1 4202560 224055 1248 0 0 495797 124 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29152/tid=29172] ppid=29147 vsize=2342008 CPUtime=4959.12 cores=4-31
/proc/29152/task/29172/stat : 29172 (plingeling) R 29147 29147 29121 0 -1 4202560 251735 1248 0 0 495784 128 0 0 25 0 29 0 647200209 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=29152/tid=29173] ppid=29147 vsize=2342008 CPUtime=4934.64 cores=4-31
/proc/29152/task/29173/stat : 29173 (plingeling) R 29147 29147 29121 0 -1 4202560 228533 1248 0 0 493339 125 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=29152/tid=29174] ppid=29147 vsize=2342008 CPUtime=4959.35 cores=4-31
/proc/29152/task/29174/stat : 29174 (plingeling) R 29147 29147 29121 0 -1 4202560 159048 1248 0 0 495774 161 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332272 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=29152/tid=29175] ppid=29147 vsize=2342008 CPUtime=4876.79 cores=4-31
/proc/29152/task/29175/stat : 29175 (plingeling) R 29147 29147 29121 0 -1 4202560 219034 1248 0 0 487575 104 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29152/tid=29176] ppid=29147 vsize=2342008 CPUtime=4940.41 cores=4-31
/proc/29152/task/29176/stat : 29176 (plingeling) R 29147 29147 29121 0 -1 4202560 236169 1248 0 0 493911 130 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=29152/tid=29177] ppid=29147 vsize=2342008 CPUtime=4949.26 cores=4-31
/proc/29152/task/29177/stat : 29177 (plingeling) R 29147 29147 29121 0 -1 4202560 218219 1248 0 0 494804 122 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=29152/tid=29178] ppid=29147 vsize=2342008 CPUtime=4956.14 cores=4-31
/proc/29152/task/29178/stat : 29178 (plingeling) R 29147 29147 29121 0 -1 4202560 237980 1248 0 0 495464 150 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=29152/tid=29179] ppid=29147 vsize=2342008 CPUtime=4959.25 cores=4-31
/proc/29152/task/29179/stat : 29179 (plingeling) R 29147 29147 29121 0 -1 4202560 235686 1248 0 0 495799 126 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=29152/tid=29180] ppid=29147 vsize=2342008 CPUtime=4959.44 cores=4-31
/proc/29152/task/29180/stat : 29180 (plingeling) R 29147 29147 29121 0 -1 4202560 248018 1248 0 0 495784 160 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331193 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=29152/tid=29181] ppid=29147 vsize=2342008 CPUtime=4959.39 cores=4-31
/proc/29152/task/29181/stat : 29181 (plingeling) R 29147 29147 29121 0 -1 4202560 242181 1248 0 0 495782 157 0 0 25 0 29 0 647200210 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=29152/tid=29182] ppid=29147 vsize=2342008 CPUtime=4959.22 cores=4-31
/proc/29152/task/29182/stat : 29182 (plingeling) R 29147 29147 29121 0 -1 4202560 193394 1248 0 0 495707 215 0 0 25 0 29 0 647200211 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=29152/tid=29183] ppid=29147 vsize=2342008 CPUtime=4959.43 cores=4-31
/proc/29152/task/29183/stat : 29183 (plingeling) R 29147 29147 29121 0 -1 4202560 241643 1248 0 0 495811 132 0 0 25 0 29 0 647200211 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=29152/tid=29184] ppid=29147 vsize=2342008 CPUtime=4959.29 cores=4-31
/proc/29152/task/29184/stat : 29184 (plingeling) R 29147 29147 29121 0 -1 4202560 212512 1248 0 0 495789 140 0 0 25 0 29 0 647200211 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=29152/tid=29185] ppid=29147 vsize=2342008 CPUtime=4958.87 cores=4-31
/proc/29152/task/29185/stat : 29185 (plingeling) R 29147 29147 29121 0 -1 4202560 264745 1248 0 0 495727 160 0 0 25 0 29 0 647200211 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=29152/tid=29186] ppid=29147 vsize=2342008 CPUtime=4959.37 cores=4-31
/proc/29152/task/29186/stat : 29186 (plingeling) R 29147 29147 29121 0 -1 4202560 262974 1248 0 0 495780 157 0 0 25 0 29 0 647200211 2398216192 436001 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332225 0 0 4096 17442 0 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 158563
Current children cumulated vsize (KiB) 3652864



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.32 s]
/proc/loadavg: 32.00 32.03 32.01 33/447 29335
/proc/meminfo: memFree=253103380/264257624 swapFree=67111528/67111528
[pid=29147] ppid=29145 vsize=18324 CPUtime=0 cores=0-31
/proc/29147/stat : 29147 (ppfolio) S 29145 29147 29121 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 647200025 18763776 386 268435456000 4194304 4287625 140737023862032 18446744073709551615 258650975999 0 0 4096 65536 13273506547507396607 0 0 17 20 0 0 0
/proc/29147/statm: 4581 386 337 23 0 91 0
[pid=29148] ppid=29147 vsize=434008 CPUtime=5006.39 cores=0
/proc/29148/stat : 29148 (cryptominisat) R 29147 29147 29121 0 -1 4202496 601481 0 0 0 500370 269 0 0 25 0 1 0 647200026 444424192 58828 268435456000 4194304 4587354 140733212970528 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29148/statm: 108502 58828 336 96 0 105064 0
[pid=29149] ppid=29147 vsize=349500 CPUtime=5006.38 cores=1
/proc/29149/stat : 29149 (clasp) R 29147 29147 29121 0 -1 4202496 82331 0 0 0 500587 51 0 0 25 0 1 0 647200026 357888000 81481 268435456000 4194304 4785740 140737138674304 18446744073709551615 4553440 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29149/statm: 87375 81481 339 145 0 84420 0
[pid=29150] ppid=29147 vsize=466396 CPUtime=5006.37 cores=2
/proc/29150/stat : 29150 (TNM) R 29147 29147 29121 0 -1 4202496 4012 0 0 0 500636 1 0 0 25 0 1 0 647200026 477589504 3909 268435456000 134512640 134534564 4289915312 18446744073709551615 134528661 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29150/statm: 116599 3909 103 6 0 115264 0
[pid=29151] ppid=29147 vsize=223156 CPUtime=5006.19 cores=3
/proc/29151/stat : 29151 (march_hi) R 29147 29147 29121 0 -1 4202496 76448 0 0 0 500573 46 0 0 25 0 1 0 647200026 228511744 42506 268435456000 4194304 4331988 140734958892672 18446744073709551615 4299865 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29151/statm: 55789 42506 176 34 0 54231 0
[pid=29152] ppid=29147 vsize=2359416 CPUtime=139977 cores=4-31
/proc/29152/stat : 29152 (plingeling) S 29147 29147 29121 0 -1 4202496 6618314 1248 0 0 13993764 3951 0 0 18 0 29 0 647200026 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 22 0 0 0
/proc/29152/statm: 589854 440353 158 40 0 588398 0
[pid=29152/tid=29159] ppid=29147 vsize=2359416 CPUtime=5004.44 cores=4-31
/proc/29152/task/29159/stat : 29159 (plingeling) R 29147 29147 29121 0 -1 4202560 234547 1248 0 0 500296 148 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=29152/tid=29160] ppid=29147 vsize=2359416 CPUtime=5004.46 cores=4-31
/proc/29152/task/29160/stat : 29160 (plingeling) R 29147 29147 29121 0 -1 4202560 245524 1248 0 0 500360 86 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29152/tid=29161] ppid=29147 vsize=2359416 CPUtime=5003.45 cores=4-31
/proc/29152/task/29161/stat : 29161 (plingeling) R 29147 29147 29121 0 -1 4202560 224010 1248 0 0 500216 129 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=29152/tid=29162] ppid=29147 vsize=2359416 CPUtime=5004.38 cores=4-31
/proc/29152/task/29162/stat : 29162 (plingeling) R 29147 29147 29121 0 -1 4202560 227812 1248 0 0 500311 127 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=29152/tid=29163] ppid=29147 vsize=2359416 CPUtime=5001.33 cores=4-31
/proc/29152/task/29163/stat : 29163 (plingeling) R 29147 29147 29121 0 -1 4202560 236643 1248 0 0 500022 111 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=29152/tid=29164] ppid=29147 vsize=2359416 CPUtime=5004.35 cores=4-31
/proc/29152/task/29164/stat : 29164 (plingeling) R 29147 29147 29121 0 -1 4202560 236460 1248 0 0 500311 124 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=29152/tid=29165] ppid=29147 vsize=2359416 CPUtime=5004.45 cores=4-31
/proc/29152/task/29165/stat : 29165 (plingeling) R 29147 29147 29121 0 -1 4202560 233042 1248 0 0 500332 113 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29152/tid=29166] ppid=29147 vsize=2359416 CPUtime=5004.45 cores=4-31
/proc/29152/task/29166/stat : 29166 (plingeling) R 29147 29147 29121 0 -1 4202560 200552 1248 0 0 500225 220 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331126 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=29152/tid=29167] ppid=29147 vsize=2359416 CPUtime=5004.32 cores=4-31
/proc/29152/task/29167/stat : 29167 (plingeling) R 29147 29147 29121 0 -1 4202560 244825 1248 0 0 500297 135 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=29152/tid=29168] ppid=29147 vsize=2359416 CPUtime=5004.23 cores=4-31
/proc/29152/task/29168/stat : 29168 (plingeling) R 29147 29147 29121 0 -1 4202560 300503 1248 0 0 500256 167 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=29152/tid=29169] ppid=29147 vsize=2359416 CPUtime=5003.24 cores=4-31
/proc/29152/task/29169/stat : 29169 (plingeling) R 29147 29147 29121 0 -1 4202560 229820 1248 0 0 500191 133 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=29152/tid=29170] ppid=29147 vsize=2359416 CPUtime=5003.59 cores=4-31
/proc/29152/task/29170/stat : 29170 (plingeling) R 29147 29147 29121 0 -1 4202560 249392 1248 0 0 500230 129 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=29152/tid=29171] ppid=29147 vsize=2359416 CPUtime=5004.25 cores=4-31
/proc/29152/task/29171/stat : 29171 (plingeling) R 29147 29147 29121 0 -1 4202560 225335 1248 0 0 500300 125 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29152/tid=29172] ppid=29147 vsize=2359416 CPUtime=5004.15 cores=4-31
/proc/29152/task/29172/stat : 29172 (plingeling) R 29147 29147 29121 0 -1 4202560 252759 1248 0 0 500287 128 0 0 25 0 29 0 647200209 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=29152/tid=29173] ppid=29147 vsize=2359416 CPUtime=4979.67 cores=4-31
/proc/29152/task/29173/stat : 29173 (plingeling) R 29147 29147 29121 0 -1 4202560 228533 1248 0 0 497842 125 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=29152/tid=29174] ppid=29147 vsize=2359416 CPUtime=5004.38 cores=4-31
/proc/29152/task/29174/stat : 29174 (plingeling) R 29147 29147 29121 0 -1 4202560 159048 1248 0 0 500277 161 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331345 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=29152/tid=29175] ppid=29147 vsize=2359416 CPUtime=4921.82 cores=4-31
/proc/29152/task/29175/stat : 29175 (plingeling) R 29147 29147 29121 0 -1 4202560 219034 1248 0 0 492078 104 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4281672 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29152/tid=29176] ppid=29147 vsize=2359416 CPUtime=4985.46 cores=4-31
/proc/29152/task/29176/stat : 29176 (plingeling) R 29147 29147 29121 0 -1 4202560 236681 1248 0 0 498415 131 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=29152/tid=29177] ppid=29147 vsize=2359416 CPUtime=4993.01 cores=4-31
/proc/29152/task/29177/stat : 29177 (plingeling) R 29147 29147 29121 0 -1 4202560 219307 1248 0 0 499178 123 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=29152/tid=29178] ppid=29147 vsize=2359416 CPUtime=5001.14 cores=4-31
/proc/29152/task/29178/stat : 29178 (plingeling) R 29147 29147 29121 0 -1 4202560 238748 1248 0 0 499963 151 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=29152/tid=29179] ppid=29147 vsize=2359416 CPUtime=5004.28 cores=4-31
/proc/29152/task/29179/stat : 29179 (plingeling) R 29147 29147 29121 0 -1 4202560 236486 1248 0 0 500302 126 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331170 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=29152/tid=29180] ppid=29147 vsize=2359416 CPUtime=5004.47 cores=4-31
/proc/29152/task/29180/stat : 29180 (plingeling) R 29147 29147 29121 0 -1 4202560 248018 1248 0 0 500287 160 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=29152/tid=29181] ppid=29147 vsize=2359416 CPUtime=5004.43 cores=4-31
/proc/29152/task/29181/stat : 29181 (plingeling) R 29147 29147 29121 0 -1 4202560 242437 1248 0 0 500285 158 0 0 25 0 29 0 647200210 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=29152/tid=29182] ppid=29147 vsize=2359416 CPUtime=5004.25 cores=4-31
/proc/29152/task/29182/stat : 29182 (plingeling) R 29147 29147 29121 0 -1 4202560 201586 1248 0 0 500209 216 0 0 25 0 29 0 647200211 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=29152/tid=29183] ppid=29147 vsize=2359416 CPUtime=5004.46 cores=4-31
/proc/29152/task/29183/stat : 29183 (plingeling) R 29147 29147 29121 0 -1 4202560 241739 1248 0 0 500314 132 0 0 25 0 29 0 647200211 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=29152/tid=29184] ppid=29147 vsize=2359416 CPUtime=5004.32 cores=4-31
/proc/29152/task/29184/stat : 29184 (plingeling) R 29147 29147 29121 0 -1 4202560 214432 1248 0 0 500291 141 0 0 25 0 29 0 647200211 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=29152/tid=29185] ppid=29147 vsize=2359416 CPUtime=5003.91 cores=4-31
/proc/29152/task/29185/stat : 29185 (plingeling) R 29147 29147 29121 0 -1 4202560 265001 1248 0 0 500231 160 0 0 25 0 29 0 647200211 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=29152/tid=29186] ppid=29147 vsize=2359416 CPUtime=5004.4 cores=4-31
/proc/29152/task/29186/stat : 29186 (plingeling) R 29147 29147 29121 0 -1 4202560 264766 1248 0 0 500281 159 0 0 25 0 29 0 647200211 2416041984 440353 268435456000 4194304 4356188 140736043052320 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 3850800

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29147 and gives
#  childrusage.ru_utime.tv_sec=5006
#  childrusage.ru_utime.tv_usec=375915
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=29995
# CPU time returned by wait4() is 5006.41
# while last known CPU time is 160002
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5007.35
CPU time (s): 160002
CPU user time (s): 159959
CPU system time (s): 43.18
CPU usage (%): 3195.35
Max. virtual memory (cumulated for all children) (KiB): 6290412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5006.38
system time used= 0.029995
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4528
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 464
involuntary context switches= 13212

runsolver used 29.5485 second user time and 111.701 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-14 09:12:16
IDJOB=3347573
IDBENCH=83413
IDSOLVER=1808
FILE ID=node200/3347573-1305357136
RUNJOBID= node200-1305357136-29135
PBS_JOBID= 13282114
Free space on /tmp= 62048 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_10_keyfind_1.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347573-1305357136/watcher-3347573-1305357136 -o /tmp/evaluation-result-3347573-1305357136/solver-3347573-1305357136 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347573-1305357136.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= f35f3aee72237c1bf3d673feb4883756
RANDOM SEED=780584006

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255607840 kB
Buffers:        435388 kB
Cached:        7811220 kB
SwapCached:          0 kB
Active:        7382836 kB
Inactive:       932892 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255607840 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           13160 kB
Writeback:           0 kB
AnonPages:       70016 kB
Mapped:          13484 kB
Slab:           249764 kB
PageTables:       3996 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   185992 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62036 MiB
End job on node200 at 2011-05-14 10:35:46