Trace number 3347533

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 156095 5100.04

General information on the benchmark

Nameapplication/fuhs/slp-synthesis-AES/
slp-synthesis-aes-top23.cnf
MD5SUM6ee28191d649bfeef8c7d8c569a3cd59
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark38269.3
Satisfiable
(Un)Satisfiability was proved
Number of variables60823
Number of clauses193450
Sum of the clauses size472824
Maximum clause length2414
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 2117689
Number of clauses of size 374473
Number of clauses of size 4275
Number of clauses of size 521
Number of clauses of size over 5991

Solver Data

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347533-1305288536/watcher-3347533-1305288536 -o /tmp/evaluation-result-3347533-1305288536/solver-3347533-1305288536 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347533-1305288536.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.21 31.02 31.70 1/413 25158
/proc/meminfo: memFree=255628748/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 510 0 0 0 0 0 0 0 21 0 1 0 640340056 18763776 385 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 8847910673057841151 0 0 17 8 0 0 0
/proc/25158/statm: 4581 385 337 23 0 91 0

[startup+0.110356 s]
/proc/loadavg: 27.21 31.02 31.70 1/413 25158
/proc/meminfo: memFree=255628748/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 21 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197619763380224 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18324

[startup+0.200354 s]
/proc/loadavg: 27.21 31.02 31.70 1/413 25158
/proc/meminfo: memFree=255628748/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 21 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197619763380224 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18324

[startup+0.300375 s]
/proc/loadavg: 27.21 31.02 31.70 1/413 25158
/proc/meminfo: memFree=255628748/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 20 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197971950698496 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18324

[startup+0.700378 s]
/proc/loadavg: 27.21 31.02 31.70 1/413 25158
/proc/meminfo: memFree=255628748/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 20 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197971950698496 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18324

[startup+1.5014 s]
/proc/loadavg: 27.21 31.02 31.70 5/418 25169
/proc/meminfo: memFree=255447956/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 20 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197971950698496 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
[pid=25159] ppid=25158 vsize=43884 CPUtime=1.48 cores=0
/proc/25159/stat : 25159 (cryptominisat) R 25158 25158 25132 0 -1 4202496 5916 0 0 0 147 1 0 0 25 0 1 0 640340057 44937216 5720 268435456000 4194304 4587354 140735110702352 18446744073709551615 4237677 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/25159/statm: 10971 5720 303 96 0 7533 0
[pid=25160] ppid=25158 vsize=44744 CPUtime=1.49 cores=1
/proc/25160/stat : 25160 (clasp) R 25158 25158 25132 0 -1 4202496 10893 0 0 0 145 4 0 0 20 0 1 0 640340057 45817856 8343 268435456000 4194304 4785740 140736868060352 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/25160/statm: 11186 8343 338 145 0 8231 0
[pid=25162] ppid=25158 vsize=43064 CPUtime=1.48 cores=3
/proc/25162/stat : 25162 (march_hi) R 25158 25158 25132 0 -1 4202496 54488 0 0 0 136 12 0 0 20 0 1 0 640340057 44097536 7308 268435456000 4194304 4331988 140734522148352 18446744073709551615 4256852 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/25162/statm: 10766 7308 149 34 0 9208 0
[pid=25163] ppid=25158 vsize=176496 CPUtime=1.49 cores=4-31
/proc/25163/stat : 25163 (plingeling) R 25158 25158 25132 0 -1 4202496 42898 1248 0 0 139 9 0 1 21 0 1 0 640340057 180731904 42783 268435456000 4194304 4356188 140735875922992 18446744073709551615 4277469 0 0 4096 17442 0 0 0 17 8 0 0 0
/proc/25163/statm: 44124 42783 129 40 0 42668 0
Current children cumulated CPU time (s) 5.98
Current children cumulated vsize (KiB) 326512

[startup+3.1014 s]
/proc/loadavg: 27.21 31.02 31.70 5/418 25169
/proc/meminfo: memFree=255264068/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 15 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197976245633294 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
[pid=25159] ppid=25158 vsize=67424 CPUtime=3.08 cores=0
/proc/25159/stat : 25159 (cryptominisat) R 25158 25158 25132 0 -1 4202496 11884 0 0 0 306 2 0 0 25 0 1 0 640340057 69042176 11204 268435456000 4194304 4587354 140735110702352 18446744073709551615 4372590 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/25159/statm: 16856 11205 333 96 0 13418 0
[pid=25160] ppid=25158 vsize=44744 CPUtime=3.09 cores=1
/proc/25160/stat : 25160 (clasp) R 25158 25158 25132 0 -1 4202496 10921 0 0 0 305 4 0 0 25 0 1 0 640340057 45817856 8371 268435456000 4194304 4785740 140736868060352 18446744073709551615 4287579 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/25160/statm: 11186 8371 338 145 0 8231 0
[pid=25162] ppid=25158 vsize=137252 CPUtime=3.08 cores=3
/proc/25162/stat : 25162 (march_hi) R 25158 25158 25132 0 -1 4202496 77816 0 0 0 291 17 0 0 25 0 1 0 640340057 140546048 17431 268435456000 4194304 4331988 140734522148352 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/25162/statm: 34313 17431 177 34 0 32755 0
[pid=25163] ppid=25158 vsize=716436 CPUtime=9.13 cores=4-31
/proc/25163/stat : 25163 (plingeling) S 25158 25158 25132 0 -1 4202496 59203 1248 0 0 898 14 0 1 19 0 29 0 640340057 733630464 57156 268435456000 4194304 4356188 140735875922992 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/25163/statm: 179109 57156 137 40 0 177653 0
Current children cumulated CPU time (s) 18.42
Current children cumulated vsize (KiB) 984180

[startup+6.30151 s]
/proc/loadavg: 25.43 30.58 31.56 32/446 25197
/proc/meminfo: memFree=255030308/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 15 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197976245633294 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
[pid=25159] ppid=25158 vsize=77148 CPUtime=6.28 cores=0
/proc/25159/stat : 25159 (cryptominisat) R 25158 25158 25132 0 -1 4202496 18427 0 0 0 624 4 0 0 25 0 1 0 640340057 78999552 14932 268435456000 4194304 4587354 140735110702352 18446744073709551615 4238051 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/25159/statm: 19287 14932 328 96 0 15849 0
[pid=25160] ppid=25158 vsize=46828 CPUtime=6.29 cores=1
/proc/25160/stat : 25160 (clasp) R 25158 25158 25132 0 -1 4202496 11398 0 0 0 625 4 0 0 25 0 1 0 640340057 47951872 8848 268435456000 4194304 4785740 140736868060352 18446744073709551615 4287178 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/25160/statm: 11707 8848 338 145 0 8752 0
[pid=25162] ppid=25158 vsize=144948 CPUtime=6.28 cores=3
/proc/25162/stat : 25162 (march_hi) R 25158 25158 25132 0 -1 4202496 79545 0 0 0 610 18 0 0 25 0 1 0 640340057 148426752 19157 268435456000 4194304 4331988 140734522148352 18446744073709551615 4263314 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/25162/statm: 36237 19157 174 34 0 34679 0
[pid=25163] ppid=25158 vsize=914324 CPUtime=96.51 cores=4-31
/proc/25163/stat : 25163 (plingeling) S 25158 25158 25132 0 -1 4202496 182160 1248 0 0 9557 93 0 1 19 0 29 0 640340057 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/25163/statm: 228581 118491 159 40 0 227125 0
[pid=25163/tid=25170] ppid=25158 vsize=914324 CPUtime=3.44 cores=4-31
/proc/25163/task/25170/stat : 25170 (plingeling) R 25158 25158 25132 0 -1 4202560 5632 1248 0 0 342 1 0 1 21 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4214464 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=25163/tid=25171] ppid=25158 vsize=914324 CPUtime=3.32 cores=4-31
/proc/25163/task/25171/stat : 25171 (plingeling) R 25158 25158 25132 0 -1 4202560 3408 1248 0 0 328 3 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4232872 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25163/tid=25172] ppid=25158 vsize=914324 CPUtime=3.43 cores=4-31
/proc/25163/task/25172/stat : 25172 (plingeling) R 25158 25158 25132 0 -1 4202560 3508 1248 0 0 342 0 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4301779 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=25163/tid=25173] ppid=25158 vsize=914324 CPUtime=3.42 cores=4-31
/proc/25163/task/25173/stat : 25173 (plingeling) R 25158 25158 25132 0 -1 4202560 4118 1248 0 0 338 3 0 1 18 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331407 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=25163/tid=25174] ppid=25158 vsize=914324 CPUtime=3.4 cores=4-31
/proc/25163/task/25174/stat : 25174 (plingeling) R 25158 25158 25132 0 -1 4202560 6249 1248 0 0 337 2 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4216380 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=25163/tid=25175] ppid=25158 vsize=914324 CPUtime=3.28 cores=4-31
/proc/25163/task/25175/stat : 25175 (plingeling) R 25158 25158 25132 0 -1 4202560 5302 1248 0 0 323 4 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4216058 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=25163/tid=25176] ppid=25158 vsize=914324 CPUtime=3.24 cores=4-31
/proc/25163/task/25176/stat : 25176 (plingeling) R 25158 25158 25132 0 -1 4202560 3855 1248 0 0 320 3 0 1 22 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4214624 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=25163/tid=25177] ppid=25158 vsize=914324 CPUtime=3.43 cores=4-31
/proc/25163/task/25177/stat : 25177 (plingeling) R 25158 25158 25132 0 -1 4202560 4641 1248 0 0 340 2 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4232885 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=25163/tid=25178] ppid=25158 vsize=914324 CPUtime=3.41 cores=4-31
/proc/25163/task/25178/stat : 25178 (plingeling) R 25158 25158 25132 0 -1 4202560 3584 1248 0 0 337 3 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4232869 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25163/tid=25179] ppid=25158 vsize=914324 CPUtime=3.32 cores=4-31
/proc/25163/task/25179/stat : 25179 (plingeling) R 25158 25158 25132 0 -1 4202560 6619 1248 0 0 328 3 0 1 22 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4231350 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25163/tid=25180] ppid=25158 vsize=914324 CPUtime=3.23 cores=4-31
/proc/25163/task/25180/stat : 25180 (plingeling) R 25158 25158 25132 0 -1 4202560 5299 1248 0 0 322 0 0 1 25 0 29 0 640340341 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4215750 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=25163/tid=25181] ppid=25158 vsize=914324 CPUtime=3.31 cores=4-31
/proc/25163/task/25181/stat : 25181 (plingeling) R 25158 25158 25132 0 -1 4202560 3579 1248 0 0 327 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331939 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=25163/tid=25182] ppid=25158 vsize=914324 CPUtime=3.36 cores=4-31
/proc/25163/task/25182/stat : 25182 (plingeling) R 25158 25158 25132 0 -1 4202560 3571 1248 0 0 332 3 0 1 21 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4217751 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=25163/tid=25183] ppid=25158 vsize=914324 CPUtime=3.41 cores=4-31
/proc/25163/task/25183/stat : 25183 (plingeling) R 25158 25158 25132 0 -1 4202560 3543 1248 0 0 337 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4217797 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=25163/tid=25184] ppid=25158 vsize=914324 CPUtime=3.35 cores=4-31
/proc/25163/task/25184/stat : 25184 (plingeling) R 25158 25158 25132 0 -1 4202560 3101 1248 0 0 331 3 0 1 23 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4302384 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=25163/tid=25185] ppid=25158 vsize=914324 CPUtime=3.41 cores=4-31
/proc/25163/task/25185/stat : 25185 (plingeling) R 25158 25158 25132 0 -1 4202560 4209 1248 0 0 337 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4248450 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=25163/tid=25186] ppid=25158 vsize=914324 CPUtime=3.41 cores=4-31
/proc/25163/task/25186/stat : 25186 (plingeling) R 25158 25158 25132 0 -1 4202560 3802 1248 0 0 339 1 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4218169 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=25163/tid=25187] ppid=25158 vsize=914324 CPUtime=3.3 cores=4-31
/proc/25163/task/25187/stat : 25187 (plingeling) R 25158 25158 25132 0 -1 4202560 3895 1248 0 0 326 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4210579 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=25163/tid=25188] ppid=25158 vsize=914324 CPUtime=3.41 cores=4-31
/proc/25163/task/25188/stat : 25188 (plingeling) R 25158 25158 25132 0 -1 4202560 3777 1248 0 0 337 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4216213 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=25163/tid=25189] ppid=25158 vsize=914324 CPUtime=3.36 cores=4-31
/proc/25163/task/25189/stat : 25189 (plingeling) R 25158 25158 25132 0 -1 4202560 3242 1248 0 0 334 1 0 1 22 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=25163/tid=25190] ppid=25158 vsize=914324 CPUtime=3.29 cores=4-31
/proc/25163/task/25190/stat : 25190 (plingeling) R 25158 25158 25132 0 -1 4202560 3922 1248 0 0 325 3 0 1 23 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4232633 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=25163/tid=25191] ppid=25158 vsize=914324 CPUtime=3.36 cores=4-31
/proc/25163/task/25191/stat : 25191 (plingeling) R 25158 25158 25132 0 -1 4202560 6607 1248 0 0 330 5 0 1 23 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4233360 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=25163/tid=25192] ppid=25158 vsize=914324 CPUtime=3.42 cores=4-31
/proc/25163/task/25192/stat : 25192 (plingeling) R 25158 25158 25132 0 -1 4202560 5939 1248 0 0 340 1 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4303193 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=25163/tid=25193] ppid=25158 vsize=914324 CPUtime=3.24 cores=4-31
/proc/25163/task/25193/stat : 25193 (plingeling) R 25158 25158 25132 0 -1 4202560 4671 1248 0 0 320 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4232872 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=25163/tid=25194] ppid=25158 vsize=914324 CPUtime=3.22 cores=4-31
/proc/25163/task/25194/stat : 25194 (plingeling) R 25158 25158 25132 0 -1 4202560 4288 1248 0 0 321 0 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4217720 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=25163/tid=25195] ppid=25158 vsize=914324 CPUtime=3.42 cores=4-31
/proc/25163/task/25195/stat : 25195 (plingeling) R 25158 25158 25132 0 -1 4202560 4024 1248 0 0 338 3 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4232296 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=25163/tid=25196] ppid=25158 vsize=914324 CPUtime=3.28 cores=4-31
/proc/25163/task/25196/stat : 25196 (plingeling) R 25158 25158 25132 0 -1 4202560 5434 1248 0 0 325 2 0 1 25 0 29 0 640340342 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4301757 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25163/tid=25197] ppid=25158 vsize=914324 CPUtime=3.22 cores=4-31
/proc/25163/task/25197/stat : 25197 (plingeling) R 25158 25158 25132 0 -1 4202560 4814 1248 0 0 318 3 0 1 25 0 29 0 640340343 936267776 118491 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 16 0 0 0
Current children cumulated CPU time (s) 115.4
Current children cumulated vsize (KiB) 1201572

[startup+12.7017 s]
/proc/loadavg: 25.95 30.61 31.56 32/446 25197
/proc/meminfo: memFree=254881048/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.04 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 0 1 3 15 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197976245633294 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
[pid=25159] ppid=25158 vsize=114652 CPUtime=12.68 cores=0
/proc/25159/stat : 25159 (cryptominisat) R 25158 25158 25132 0 -1 4202496 19364 0 0 0 1263 5 0 0 25 0 1 0 640340057 117403648 15854 268435456000 4194304 4587354 140735110702352 18446744073709551615 4449074 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/25159/statm: 28663 15854 334 96 0 25225 0
[pid=25160] ppid=25158 vsize=53360 CPUtime=12.69 cores=1
/proc/25160/stat : 25160 (clasp) R 25158 25158 25132 0 -1 4202496 13020 0 0 0 1264 5 0 0 25 0 1 0 640340057 54640640 10470 268435456000 4194304 4785740 140736868060352 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/25160/statm: 13340 10470 338 145 0 10385 0
[pid=25162] ppid=25158 vsize=145648 CPUtime=12.68 cores=3
/proc/25162/stat : 25162 (march_hi) R 25158 25158 25132 0 -1 4202496 79700 0 0 0 1250 18 0 0 25 0 1 0 640340057 149143552 19312 268435456000 4194304 4331988 140734522148352 18446744073709551615 4299264 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/25162/statm: 36412 19312 174 34 0 34854 0
[pid=25163] ppid=25158 vsize=995200 CPUtime=275.49 cores=4-31
/proc/25163/stat : 25163 (plingeling) S 25158 25158 25132 0 -1 4202496 202806 1248 0 0 27451 97 0 1 19 0 29 0 640340057 1019084800 137646 268435456000 4194304 4356188 140735875922992 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=252963632/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.05 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 1 1 3 15 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197976245633294 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
[pid=25159] ppid=25158 vsize=1930436 CPUtime=5098.99 cores=0
/proc/25159/stat : 25159 (cryptominisat) R 25158 25158 25132 0 -1 4202496 556803 0 0 0 509549 350 0 0 25 0 1 0 640340057 1976766464 27695 268435456000 4194304 4587354 140735110702352 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/25159/statm: 482609 27695 334 96 0 479171 0
[pid=25160] ppid=25158 vsize=314068 CPUtime=5099.05 cores=1
/proc/25160/stat : 25160 (clasp) R 25158 25158 25132 0 -1 4202496 76343 0 0 0 509854 51 0 0 25 0 1 0 640340057 321605632 73793 268435456000 4194304 4785740 140736868060352 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/25160/statm: 78517 73793 338 145 0 75562 0
[pid=25162] ppid=25158 vsize=163480 CPUtime=5099.05 cores=3
/proc/25162/stat : 25162 (march_hi) R 25158 25158 25132 0 -1 4202496 83690 0 0 0 509883 22 0 0 25 0 1 0 640340057 167403520 23302 268435456000 4194304 4331988 140734522148352 18446744073709551615 4265456 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/25162/statm: 40870 23302 174 34 0 39312 0
[pid=25163] ppid=25158 vsize=2610184 CPUtime=140798 cores=4-31
/proc/25163/stat : 25163 (plingeling) S 25158 25158 25132 0 -1 4202496 4689104 1248 0 0 14076661 3154 0 1 19 0 29 0 640340057 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/25163/statm: 652546 539529 159 40 0 651090 0
[pid=25163/tid=25170] ppid=25158 vsize=2610184 CPUtime=5029.45 cores=4-31
/proc/25163/task/25170/stat : 25170 (plingeling) R 25158 25158 25132 0 -1 4202560 195103 1248 0 0 502827 117 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4298488 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=25163/tid=25171] ppid=25158 vsize=2610184 CPUtime=5078.16 cores=4-31
/proc/25163/task/25171/stat : 25171 (plingeling) R 25158 25158 25132 0 -1 4202560 170053 1248 0 0 507699 116 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4279742 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=25163/tid=25172] ppid=25158 vsize=2610184 CPUtime=4859.19 cores=4-31
/proc/25163/task/25172/stat : 25172 (plingeling) R 25158 25158 25132 0 -1 4202560 212673 1248 0 0 485806 112 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=25163/tid=25173] ppid=25158 vsize=2610184 CPUtime=5047.99 cores=4-31
/proc/25163/task/25173/stat : 25173 (plingeling) R 25158 25158 25132 0 -1 4202560 40220 1248 0 0 504758 40 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=25163/tid=25174] ppid=25158 vsize=2610184 CPUtime=4938.5 cores=4-31
/proc/25163/task/25174/stat : 25174 (plingeling) R 25158 25158 25132 0 -1 4202560 150007 1248 0 0 493739 110 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=25163/tid=25175] ppid=25158 vsize=2610184 CPUtime=5049.75 cores=4-31
/proc/25163/task/25175/stat : 25175 (plingeling) R 25158 25158 25132 0 -1 4202560 178499 1248 0 0 504833 141 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25163/tid=25176] ppid=25158 vsize=2610184 CPUtime=5070.68 cores=4-31
/proc/25163/task/25176/stat : 25176 (plingeling) R 25158 25158 25132 0 -1 4202560 157498 1248 0 0 506960 107 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=25163/tid=25177] ppid=25158 vsize=2610184 CPUtime=5053.31 cores=4-31
/proc/25163/task/25177/stat : 25177 (plingeling) R 25158 25158 25132 0 -1 4202560 225398 1248 0 0 505114 216 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=25163/tid=25178] ppid=25158 vsize=2610184 CPUtime=5044.96 cores=4-31
/proc/25163/task/25178/stat : 25178 (plingeling) R 25158 25158 25132 0 -1 4202560 205628 1248 0 0 504368 127 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=25163/tid=25179] ppid=25158 vsize=2610184 CPUtime=5023.64 cores=4-31
/proc/25163/task/25179/stat : 25179 (plingeling) R 25158 25158 25132 0 -1 4202560 178950 1248 0 0 502224 139 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=25163/tid=25180] ppid=25158 vsize=2610184 CPUtime=5057.85 cores=4-31
/proc/25163/task/25180/stat : 25180 (plingeling) R 25158 25158 25132 0 -1 4202560 177756 1248 0 0 505696 88 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=25163/tid=25181] ppid=25158 vsize=2610184 CPUtime=5070.5 cores=4-31
/proc/25163/task/25181/stat : 25181 (plingeling) R 25158 25158 25132 0 -1 4202560 23935 1248 0 0 507032 17 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=25163/tid=25182] ppid=25158 vsize=2610184 CPUtime=5096.11 cores=4-31
/proc/25163/task/25182/stat : 25182 (plingeling) R 25158 25158 25132 0 -1 4202560 144945 1248 0 0 509524 86 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=25163/tid=25183] ppid=25158 vsize=2610184 CPUtime=5096.04 cores=4-31
/proc/25163/task/25183/stat : 25183 (plingeling) R 25158 25158 25132 0 -1 4202560 243989 1248 0 0 509410 193 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=25163/tid=25184] ppid=25158 vsize=2610184 CPUtime=5053.03 cores=4-31
/proc/25163/task/25184/stat : 25184 (plingeling) R 25158 25158 25132 0 -1 4202560 195700 1248 0 0 505168 134 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=25163/tid=25185] ppid=25158 vsize=2610184 CPUtime=5096.09 cores=4-31
/proc/25163/task/25185/stat : 25185 (plingeling) R 25158 25158 25132 0 -1 4202560 239278 1248 0 0 509392 216 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=25163/tid=25186] ppid=25158 vsize=2610184 CPUtime=5096 cores=4-31
/proc/25163/task/25186/stat : 25186 (plingeling) R 25158 25158 25132 0 -1 4202560 195323 1248 0 0 509461 138 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=25163/tid=25187] ppid=25158 vsize=2610184 CPUtime=4873.35 cores=4-31
/proc/25163/task/25187/stat : 25187 (plingeling) R 25158 25158 25132 0 -1 4202560 139843 1248 0 0 487227 107 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=25163/tid=25188] ppid=25158 vsize=2610184 CPUtime=5096.09 cores=4-31
/proc/25163/task/25188/stat : 25188 (plingeling) R 25158 25158 25132 0 -1 4202560 172671 1248 0 0 509500 108 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25163/tid=25189] ppid=25158 vsize=2610184 CPUtime=4897.76 cores=4-31
/proc/25163/task/25189/stat : 25189 (plingeling) R 25158 25158 25132 0 -1 4202560 33354 1248 0 0 489754 21 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4234276 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25163/tid=25190] ppid=25158 vsize=2610184 CPUtime=4922.54 cores=4-31
/proc/25163/task/25190/stat : 25190 (plingeling) R 25158 25158 25132 0 -1 4202560 182595 1248 0 0 492137 116 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331345 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=25163/tid=25191] ppid=25158 vsize=2610184 CPUtime=5079.58 cores=4-31
/proc/25163/task/25191/stat : 25191 (plingeling) R 25158 25158 25132 0 -1 4202560 206866 1248 0 0 507852 105 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331309 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=25163/tid=25192] ppid=25158 vsize=2610184 CPUtime=5094.4 cores=4-31
/proc/25163/task/25192/stat : 25192 (plingeling) R 25158 25158 25132 0 -1 4202560 165314 1248 0 0 509350 89 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=25163/tid=25193] ppid=25158 vsize=2610184 CPUtime=5093.63 cores=4-31
/proc/25163/task/25193/stat : 25193 (plingeling) R 25158 25158 25132 0 -1 4202560 278092 1248 0 0 509110 252 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=25163/tid=25194] ppid=25158 vsize=2610184 CPUtime=4986.86 cores=4-31
/proc/25163/task/25194/stat : 25194 (plingeling) R 25158 25158 25132 0 -1 4202560 167117 1248 0 0 498635 50 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25163/tid=25195] ppid=25158 vsize=2610184 CPUtime=5057.95 cores=4-31
/proc/25163/task/25195/stat : 25195 (plingeling) R 25158 25158 25132 0 -1 4202560 185030 1248 0 0 505677 117 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=25163/tid=25196] ppid=25158 vsize=2610184 CPUtime=5062.05 cores=4-31
/proc/25163/task/25196/stat : 25196 (plingeling) R 25158 25158 25132 0 -1 4202560 128582 1248 0 0 506165 39 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=25163/tid=25197] ppid=25158 vsize=2610184 CPUtime=4869.91 cores=4-31
/proc/25163/task/25197/stat : 25197 (plingeling) R 25158 25158 25132 0 -1 4202560 37158 1248 0 0 486965 25 0 1 25 0 29 0 640340343 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 20 0 0 0
Current children cumulated CPU time (s) 156095
Current children cumulated vsize (KiB) 5036492

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100 s]
/proc/loadavg: 31.01 31.02 31.00 32/446 25348
/proc/meminfo: memFree=252963632/264257624 swapFree=67111528/67111528
[pid=25158] ppid=25156 vsize=18324 CPUtime=0.05 cores=0-31
/proc/25158/stat : 25158 (ppfolio) S 25156 25158 25132 0 -1 4202496 511 1194 0 0 0 1 1 3 15 0 1 0 640340056 18763776 386 268435456000 4194304 4287625 140734669905152 18446744073709551615 258650975999 0 0 4096 65536 9224197976245633294 0 0 17 2 0 0 0
/proc/25158/statm: 4581 386 338 23 0 91 0
[pid=25159] ppid=25158 vsize=1930436 CPUtime=5098.99 cores=0
/proc/25159/stat : 25159 (cryptominisat) R 25158 25158 25132 0 -1 4202496 556803 0 0 0 509549 350 0 0 25 0 1 0 640340057 1976766464 27695 268435456000 4194304 4587354 140735110702352 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/25159/statm: 482609 27695 334 96 0 479171 0
[pid=25160] ppid=25158 vsize=314068 CPUtime=5099.05 cores=1
/proc/25160/stat : 25160 (clasp) R 25158 25158 25132 0 -1 4202496 76343 0 0 0 509854 51 0 0 25 0 1 0 640340057 321605632 73793 268435456000 4194304 4785740 140736868060352 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/25160/statm: 78517 73793 338 145 0 75562 0
[pid=25162] ppid=25158 vsize=163480 CPUtime=5099.05 cores=3
/proc/25162/stat : 25162 (march_hi) R 25158 25158 25132 0 -1 4202496 83690 0 0 0 509883 22 0 0 25 0 1 0 640340057 167403520 23302 268435456000 4194304 4331988 140734522148352 18446744073709551615 4265456 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/25162/statm: 40870 23302 174 34 0 39312 0
[pid=25163] ppid=25158 vsize=2610184 CPUtime=140798 cores=4-31
/proc/25163/stat : 25163 (plingeling) S 25158 25158 25132 0 -1 4202496 4689104 1248 0 0 14076661 3154 0 1 19 0 29 0 640340057 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/25163/statm: 652546 539529 159 40 0 651090 0
[pid=25163/tid=25170] ppid=25158 vsize=2610184 CPUtime=5029.45 cores=4-31
/proc/25163/task/25170/stat : 25170 (plingeling) R 25158 25158 25132 0 -1 4202560 195103 1248 0 0 502827 117 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4298488 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=25163/tid=25171] ppid=25158 vsize=2610184 CPUtime=5078.16 cores=4-31
/proc/25163/task/25171/stat : 25171 (plingeling) R 25158 25158 25132 0 -1 4202560 170053 1248 0 0 507699 116 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4279742 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=25163/tid=25172] ppid=25158 vsize=2610184 CPUtime=4859.19 cores=4-31
/proc/25163/task/25172/stat : 25172 (plingeling) R 25158 25158 25132 0 -1 4202560 212673 1248 0 0 485806 112 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=25163/tid=25173] ppid=25158 vsize=2610184 CPUtime=5047.99 cores=4-31
/proc/25163/task/25173/stat : 25173 (plingeling) R 25158 25158 25132 0 -1 4202560 40220 1248 0 0 504758 40 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=25163/tid=25174] ppid=25158 vsize=2610184 CPUtime=4938.5 cores=4-31
/proc/25163/task/25174/stat : 25174 (plingeling) R 25158 25158 25132 0 -1 4202560 150007 1248 0 0 493739 110 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=25163/tid=25175] ppid=25158 vsize=2610184 CPUtime=5049.75 cores=4-31
/proc/25163/task/25175/stat : 25175 (plingeling) R 25158 25158 25132 0 -1 4202560 178499 1248 0 0 504833 141 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=25163/tid=25176] ppid=25158 vsize=2610184 CPUtime=5070.68 cores=4-31
/proc/25163/task/25176/stat : 25176 (plingeling) R 25158 25158 25132 0 -1 4202560 157498 1248 0 0 506960 107 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=25163/tid=25177] ppid=25158 vsize=2610184 CPUtime=5053.31 cores=4-31
/proc/25163/task/25177/stat : 25177 (plingeling) R 25158 25158 25132 0 -1 4202560 225398 1248 0 0 505114 216 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=25163/tid=25178] ppid=25158 vsize=2610184 CPUtime=5044.96 cores=4-31
/proc/25163/task/25178/stat : 25178 (plingeling) R 25158 25158 25132 0 -1 4202560 205628 1248 0 0 504368 127 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=25163/tid=25179] ppid=25158 vsize=2610184 CPUtime=5023.64 cores=4-31
/proc/25163/task/25179/stat : 25179 (plingeling) R 25158 25158 25132 0 -1 4202560 178950 1248 0 0 502224 139 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=25163/tid=25180] ppid=25158 vsize=2610184 CPUtime=5057.85 cores=4-31
/proc/25163/task/25180/stat : 25180 (plingeling) R 25158 25158 25132 0 -1 4202560 177756 1248 0 0 505696 88 0 1 25 0 29 0 640340341 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=25163/tid=25181] ppid=25158 vsize=2610184 CPUtime=5070.5 cores=4-31
/proc/25163/task/25181/stat : 25181 (plingeling) R 25158 25158 25132 0 -1 4202560 23935 1248 0 0 507032 17 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=25163/tid=25182] ppid=25158 vsize=2610184 CPUtime=5096.11 cores=4-31
/proc/25163/task/25182/stat : 25182 (plingeling) R 25158 25158 25132 0 -1 4202560 144945 1248 0 0 509524 86 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=25163/tid=25183] ppid=25158 vsize=2610184 CPUtime=5096.04 cores=4-31
/proc/25163/task/25183/stat : 25183 (plingeling) R 25158 25158 25132 0 -1 4202560 243989 1248 0 0 509410 193 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=25163/tid=25184] ppid=25158 vsize=2610184 CPUtime=5053.03 cores=4-31
/proc/25163/task/25184/stat : 25184 (plingeling) R 25158 25158 25132 0 -1 4202560 195700 1248 0 0 505168 134 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=25163/tid=25185] ppid=25158 vsize=2610184 CPUtime=5096.09 cores=4-31
/proc/25163/task/25185/stat : 25185 (plingeling) R 25158 25158 25132 0 -1 4202560 239278 1248 0 0 509392 216 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=25163/tid=25186] ppid=25158 vsize=2610184 CPUtime=5096 cores=4-31
/proc/25163/task/25186/stat : 25186 (plingeling) R 25158 25158 25132 0 -1 4202560 195323 1248 0 0 509461 138 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=25163/tid=25187] ppid=25158 vsize=2610184 CPUtime=4873.35 cores=4-31
/proc/25163/task/25187/stat : 25187 (plingeling) R 25158 25158 25132 0 -1 4202560 139843 1248 0 0 487227 107 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=25163/tid=25188] ppid=25158 vsize=2610184 CPUtime=5096.09 cores=4-31
/proc/25163/task/25188/stat : 25188 (plingeling) R 25158 25158 25132 0 -1 4202560 172671 1248 0 0 509500 108 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25163/tid=25189] ppid=25158 vsize=2610184 CPUtime=4897.76 cores=4-31
/proc/25163/task/25189/stat : 25189 (plingeling) R 25158 25158 25132 0 -1 4202560 33354 1248 0 0 489754 21 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4234276 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25163/tid=25190] ppid=25158 vsize=2610184 CPUtime=4922.54 cores=4-31
/proc/25163/task/25190/stat : 25190 (plingeling) R 25158 25158 25132 0 -1 4202560 182595 1248 0 0 492137 116 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331345 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=25163/tid=25191] ppid=25158 vsize=2610184 CPUtime=5079.58 cores=4-31
/proc/25163/task/25191/stat : 25191 (plingeling) R 25158 25158 25132 0 -1 4202560 206866 1248 0 0 507852 105 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331309 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=25163/tid=25192] ppid=25158 vsize=2610184 CPUtime=5094.4 cores=4-31
/proc/25163/task/25192/stat : 25192 (plingeling) R 25158 25158 25132 0 -1 4202560 165314 1248 0 0 509350 89 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=25163/tid=25193] ppid=25158 vsize=2610184 CPUtime=5093.63 cores=4-31
/proc/25163/task/25193/stat : 25193 (plingeling) R 25158 25158 25132 0 -1 4202560 278092 1248 0 0 509110 252 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=25163/tid=25194] ppid=25158 vsize=2610184 CPUtime=4986.86 cores=4-31
/proc/25163/task/25194/stat : 25194 (plingeling) R 25158 25158 25132 0 -1 4202560 167117 1248 0 0 498635 50 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25163/tid=25195] ppid=25158 vsize=2610184 CPUtime=5057.95 cores=4-31
/proc/25163/task/25195/stat : 25195 (plingeling) R 25158 25158 25132 0 -1 4202560 185030 1248 0 0 505677 117 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=25163/tid=25196] ppid=25158 vsize=2610184 CPUtime=5062.05 cores=4-31
/proc/25163/task/25196/stat : 25196 (plingeling) R 25158 25158 25132 0 -1 4202560 128582 1248 0 0 506165 39 0 1 25 0 29 0 640340342 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=25163/tid=25197] ppid=25158 vsize=2610184 CPUtime=4869.91 cores=4-31
/proc/25163/task/25197/stat : 25197 (plingeling) R 25158 25158 25132 0 -1 4202560 37158 1248 0 0 486965 25 0 1 25 0 29 0 640340343 2672828416 539529 268435456000 4194304 4356188 140735875922992 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 20 0 0 0
Current children cumulated CPU time (s) 156095
Current children cumulated vsize (KiB) 5036492

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 25158 and gives
#  childrusage.ru_utime.tv_sec=10194
#  childrusage.ru_utime.tv_usec=352222
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=803421
# CPU time returned by wait4() is 10198.2
# while last known CPU time is 156095
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5100.04
CPU time (s): 156095
CPU user time (s): 156059
CPU system time (s): 35.82
CPU usage (%): 3060.67
Max. virtual memory (cumulated for all children) (KiB): 6885684

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 10194.4
system time used= 3.80342
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 642198
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 483
involuntary context switches= 10687

runsolver used 27.7988 second user time and 116.741 second system time

The end

Launcher Data

Begin job on node200 at 2011-05-13 14:08:56
IDJOB=3347533
IDBENCH=83114
IDSOLVER=1808
FILE ID=node200/3347533-1305288536
RUNJOBID= node200-1305288536-25146
PBS_JOBID= 13282107
Free space on /tmp= 62048 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/application/fuhs/slp-synthesis-AES/slp-synthesis-aes-top23.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347533-1305288536/watcher-3347533-1305288536 -o /tmp/evaluation-result-3347533-1305288536/solver-3347533-1305288536 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347533-1305288536.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 6ee28191d649bfeef8c7d8c569a3cd59
RANDOM SEED=32907298

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255629028 kB
Buffers:        432696 kB
Cached:        7797880 kB
SwapCached:          0 kB
Active:        7377416 kB
Inactive:       922056 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255629028 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           13096 kB
Writeback:           0 kB
AnonPages:       70152 kB
Mapped:          13428 kB
Slab:           249500 kB
PageTables:       4852 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   187948 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62032 MiB
End job on node200 at 2011-05-13 15:33:59