Trace number 3347239

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160004 5009.46

General information on the benchmark

NameAPPLICATIONS/bitverif/countbitswegner/
countbitswegner128.cnf
MD5SUM4b75ef7fb46928f50793af1d74c176e2
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark129083
Satisfiable
(Un)Satisfiability was proved
Number of variables98048
Number of clauses293759
Sum of the clauses size685435
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2195838
Number of clauses of size 397919
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347239-1304917794/watcher-3347239-1304917794 -o /tmp/evaluation-result-3347239-1304917794/solver-3347239-1304917794 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347239-1304917794.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.28 31.09 31.71 1/413 24059
/proc/meminfo: memFree=256745424/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 20 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0

[startup+0.0394771 s]
/proc/loadavg: 27.28 31.09 31.71 1/413 24059
/proc/meminfo: memFree=256745424/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 20 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100561 s]
/proc/loadavg: 27.28 31.09 31.71 1/413 24059
/proc/meminfo: memFree=256745424/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 20 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.30056 s]
/proc/loadavg: 27.28 31.09 31.71 1/413 24059
/proc/meminfo: memFree=256745424/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 18 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700586 s]
/proc/loadavg: 27.28 31.09 31.71 1/413 24059
/proc/meminfo: memFree=256745424/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 18 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.5006 s]
/proc/loadavg: 27.28 31.09 31.71 6/419 24070
/proc/meminfo: memFree=256491028/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 18 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
[pid=24060] ppid=24059 vsize=58104 CPUtime=1.48 cores=0
/proc/24060/stat : 24060 (cryptominisat) R 24059 24059 24033 0 -1 4202496 9140 0 0 0 145 3 0 0 20 0 1 0 603265850 59498496 8833 268435456000 4194304 4587354 140735678391152 18446744073709551615 4237686 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24060/statm: 14526 8833 304 96 0 11088 0
[pid=24061] ppid=24059 vsize=66544 CPUtime=1.48 cores=1
/proc/24061/stat : 24061 (clasp) R 24059 24059 24033 0 -1 4202496 17402 0 0 0 143 5 0 0 20 0 1 0 603265850 68141056 13680 268435456000 4194304 4785740 140736765949728 18446744073709551615 4546313 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/24061/statm: 16636 13680 330 145 0 13681 0
[pid=24062] ppid=24059 vsize=472328 CPUtime=1.48 cores=2
/proc/24062/stat : 24062 (TNM) R 24059 24059 24033 0 -1 4202496 10296 0 0 0 144 4 0 0 20 0 1 0 603265850 483663872 10194 268435456000 134512640 134534564 4291322608 18446744073709551615 134529352 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/24062/statm: 118082 10194 95 6 0 116747 0
[pid=24063] ppid=24059 vsize=67016 CPUtime=1.48 cores=3
/proc/24063/stat : 24063 (march_hi) R 24059 24059 24033 0 -1 4202496 52797 0 0 0 134 14 0 0 20 0 1 0 603265850 68624384 12451 268435456000 4194304 4331988 140735843830752 18446744073709551615 4252197 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24063/statm: 16754 12451 140 34 0 15196 0
[pid=24064] ppid=24059 vsize=120104 CPUtime=1.49 cores=4-31
/proc/24064/stat : 24064 (plingeling) R 24059 24059 24033 0 -1 4202496 28800 1240 0 0 139 9 0 1 21 0 1 0 603265850 122986496 28685 268435456000 4194304 4356188 140735602176112 18446744073709551615 4277958 0 0 4096 17442 0 0 0 17 9 0 0 0
/proc/24064/statm: 30026 28685 129 40 0 28570 0
Current children cumulated CPU time (s) 7.41
Current children cumulated vsize (KiB) 802420

[startup+3.10061 s]
/proc/loadavg: 25.50 30.66 31.57 6/419 24070
/proc/meminfo: memFree=256290148/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
[pid=24060] ppid=24059 vsize=76892 CPUtime=3.08 cores=0
/proc/24060/stat : 24060 (cryptominisat) R 24059 24059 24033 0 -1 4202496 13807 0 0 0 304 4 0 0 25 0 1 0 603265850 78737408 13500 268435456000 4194304 4587354 140735678391152 18446744073709551615 4367567 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24060/statm: 19223 13500 310 96 0 15785 0
[pid=24061] ppid=24059 vsize=66544 CPUtime=3.08 cores=1
/proc/24061/stat : 24061 (clasp) R 24059 24059 24033 0 -1 4202496 17410 0 0 0 303 5 0 0 25 0 1 0 603265850 68141056 13688 268435456000 4194304 4785740 140736765949728 18446744073709551615 4553402 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/24061/statm: 16636 13688 338 145 0 13681 0
[pid=24062] ppid=24059 vsize=476156 CPUtime=3.08 cores=2
/proc/24062/stat : 24062 (TNM) R 24059 24059 24033 0 -1 4202496 10297 0 0 0 304 4 0 0 25 0 1 0 603265850 487583744 10195 268435456000 134512640 134534564 4291322608 18446744073709551615 134528330 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/24062/statm: 119039 10195 95 6 0 117704 0
[pid=24063] ppid=24059 vsize=190752 CPUtime=3.08 cores=3
/proc/24063/stat : 24063 (march_hi) R 24059 24059 24033 0 -1 4202496 83347 0 0 0 289 19 0 0 25 0 1 0 603265850 195330048 22093 268435456000 4194304 4331988 140735843830752 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24063/statm: 47688 22093 168 34 0 46130 0
[pid=24064] ppid=24059 vsize=342880 CPUtime=3.08 cores=4-31
/proc/24064/stat : 24064 (plingeling) R 24059 24059 24033 0 -1 4202496 84494 1240 0 0 288 19 0 1 25 0 1 0 603265850 351109120 84379 268435456000 4194304 4356188 140735602176112 18446744073709551615 4278302 0 0 4096 17442 0 0 0 17 9 0 0 0
/proc/24064/statm: 85720 84379 129 40 0 84264 0
Current children cumulated CPU time (s) 15.4
Current children cumulated vsize (KiB) 1171548

[startup+6.30169 s]
/proc/loadavg: 25.50 30.66 31.57 34/447 24098
/proc/meminfo: memFree=255846948/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
[pid=24060] ppid=24059 vsize=76892 CPUtime=6.28 cores=0
/proc/24060/stat : 24060 (cryptominisat) R 24059 24059 24033 0 -1 4202496 13807 0 0 0 624 4 0 0 25 0 1 0 603265850 78737408 13500 268435456000 4194304 4587354 140735678391152 18446744073709551615 4421086 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24060/statm: 19223 13500 310 96 0 15785 0
[pid=24061] ppid=24059 vsize=66544 CPUtime=6.28 cores=1
/proc/24061/stat : 24061 (clasp) R 24059 24059 24033 0 -1 4202496 17410 0 0 0 623 5 0 0 25 0 1 0 603265850 68141056 13688 268435456000 4194304 4785740 140736765949728 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/24061/statm: 16636 13688 338 145 0 13681 0
[pid=24062] ppid=24059 vsize=476156 CPUtime=6.28 cores=2
/proc/24062/stat : 24062 (TNM) R 24059 24059 24033 0 -1 4202496 10297 0 0 0 624 4 0 0 25 0 1 0 603265850 487583744 10195 268435456000 134512640 134534564 4291322608 18446744073709551615 134528661 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/24062/statm: 119039 10195 95 6 0 117704 0
[pid=24063] ppid=24059 vsize=202728 CPUtime=6.29 cores=3
/proc/24063/stat : 24063 (march_hi) R 24059 24059 24033 0 -1 4202496 90026 0 0 0 608 21 0 0 25 0 1 0 603265850 207593472 28772 268435456000 4194304 4331988 140735843830752 18446744073709551615 4229294 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24063/statm: 50682 28772 172 34 0 49124 0
[pid=24064] ppid=24059 vsize=1180048 CPUtime=56.58 cores=4-31
/proc/24064/stat : 24064 (plingeling) S 24059 24059 24033 0 -1 4202496 273482 1240 0 0 5404 253 0 1 20 0 29 0 603265850 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 13 0 0 0
/proc/24064/statm: 295012 170754 147 40 0 293556 0
[pid=24064/tid=24071] ppid=24059 vsize=1180048 CPUtime=1.98 cores=4-31
/proc/24064/task/24071/stat : 24071 (plingeling) R 24059 24059 24033 0 -1 4202560 7496 1240 0 0 189 8 0 1 20 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244324 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24064/tid=24072] ppid=24059 vsize=1180048 CPUtime=1.87 cores=4-31
/proc/24064/task/24072/stat : 24072 (plingeling) R 24059 24059 24033 0 -1 4202560 5739 1240 0 0 180 6 0 1 19 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4259314 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24064/tid=24073] ppid=24059 vsize=1180048 CPUtime=1.81 cores=4-31
/proc/24064/task/24073/stat : 24073 (plingeling) R 24059 24059 24033 0 -1 4202560 8371 1240 0 0 169 11 0 1 18 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=24064/tid=24074] ppid=24059 vsize=1180048 CPUtime=1.82 cores=4-31
/proc/24064/task/24074/stat : 24074 (plingeling) R 24059 24059 24033 0 -1 4202560 6431 1240 0 0 170 11 0 1 21 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=24064/tid=24075] ppid=24059 vsize=1180048 CPUtime=1.89 cores=4-31
/proc/24064/task/24075/stat : 24075 (plingeling) R 24059 24059 24033 0 -1 4202560 6544 1240 0 0 180 8 0 1 22 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=24064/tid=24076] ppid=24059 vsize=1180048 CPUtime=1.83 cores=4-31
/proc/24064/task/24076/stat : 24076 (plingeling) R 24059 24059 24033 0 -1 4202560 5510 1240 0 0 174 8 0 1 19 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4260351 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=24064/tid=24077] ppid=24059 vsize=1180048 CPUtime=1.81 cores=4-31
/proc/24064/task/24077/stat : 24077 (plingeling) R 24059 24059 24033 0 -1 4202560 6303 1240 0 0 170 10 0 1 18 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=24064/tid=24078] ppid=24059 vsize=1180048 CPUtime=1.91 cores=4-31
/proc/24064/task/24078/stat : 24078 (plingeling) R 24059 24059 24033 0 -1 4202560 6165 1240 0 0 183 7 0 1 18 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4259314 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=24064/tid=24079] ppid=24059 vsize=1180048 CPUtime=1.88 cores=4-31
/proc/24064/task/24079/stat : 24079 (plingeling) R 24059 24059 24033 0 -1 4202560 5933 1240 0 0 179 8 0 1 19 0 29 0 603266277 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4261279 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=24064/tid=24080] ppid=24059 vsize=1180048 CPUtime=1.91 cores=4-31
/proc/24064/task/24080/stat : 24080 (plingeling) R 24059 24059 24033 0 -1 4202560 6249 1240 0 0 184 6 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=24064/tid=24081] ppid=24059 vsize=1180048 CPUtime=1.9 cores=4-31
/proc/24064/task/24081/stat : 24081 (plingeling) R 24059 24059 24033 0 -1 4202560 4119 1240 0 0 181 8 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4260341 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=24064/tid=24082] ppid=24059 vsize=1180048 CPUtime=1.82 cores=4-31
/proc/24064/task/24082/stat : 24082 (plingeling) R 24059 24059 24033 0 -1 4202560 4311 1240 0 0 169 12 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=24064/tid=24083] ppid=24059 vsize=1180048 CPUtime=1.81 cores=4-31
/proc/24064/task/24083/stat : 24083 (plingeling) R 24059 24059 24033 0 -1 4202560 6160 1240 0 0 171 9 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=24064/tid=24084] ppid=24059 vsize=1180048 CPUtime=1.9 cores=4-31
/proc/24064/task/24084/stat : 24084 (plingeling) R 24059 24059 24033 0 -1 4202560 4236 1240 0 0 183 6 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4267684 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=24064/tid=24085] ppid=24059 vsize=1180048 CPUtime=1.85 cores=4-31
/proc/24064/task/24085/stat : 24085 (plingeling) R 24059 24059 24033 0 -1 4202560 6896 1240 0 0 176 8 0 1 25 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4260324 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=24064/tid=24086] ppid=24059 vsize=1180048 CPUtime=1.86 cores=4-31
/proc/24064/task/24086/stat : 24086 (plingeling) R 24059 24059 24033 0 -1 4202560 4571 1240 0 0 177 8 0 1 18 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4260245 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=24064/tid=24087] ppid=24059 vsize=1180048 CPUtime=1.93 cores=4-31
/proc/24064/task/24087/stat : 24087 (plingeling) R 24059 24059 24033 0 -1 4202560 6269 1240 0 0 190 2 0 1 25 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=24064/tid=24088] ppid=24059 vsize=1180048 CPUtime=1.84 cores=4-31
/proc/24064/task/24088/stat : 24088 (plingeling) R 24059 24059 24033 0 -1 4202560 4396 1240 0 0 175 8 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4213005 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=24064/tid=24089] ppid=24059 vsize=1180048 CPUtime=1.88 cores=4-31
/proc/24064/task/24089/stat : 24089 (plingeling) R 24059 24059 24033 0 -1 4202560 7068 1240 0 0 180 7 0 1 18 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4267675 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24064/tid=24090] ppid=24059 vsize=1180048 CPUtime=1.85 cores=4-31
/proc/24064/task/24090/stat : 24090 (plingeling) R 24059 24059 24033 0 -1 4202560 4501 1240 0 0 183 1 0 1 18 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4261920 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=24064/tid=24091] ppid=24059 vsize=1180048 CPUtime=1.89 cores=4-31
/proc/24064/task/24091/stat : 24091 (plingeling) R 24059 24059 24033 0 -1 4202560 4971 1240 0 0 181 7 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244324 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=24064/tid=24092] ppid=24059 vsize=1180048 CPUtime=1.81 cores=4-31
/proc/24064/task/24092/stat : 24092 (plingeling) R 24059 24059 24033 0 -1 4202560 6778 1240 0 0 172 8 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4245442 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=24064/tid=24093] ppid=24059 vsize=1180048 CPUtime=1.84 cores=4-31
/proc/24064/task/24093/stat : 24093 (plingeling) R 24059 24059 24033 0 -1 4202560 7753 1240 0 0 173 10 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24064/tid=24094] ppid=24059 vsize=1180048 CPUtime=1.88 cores=4-31
/proc/24064/task/24094/stat : 24094 (plingeling) R 24059 24059 24033 0 -1 4202560 5251 1240 0 0 180 7 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4261931 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=24064/tid=24095] ppid=24059 vsize=1180048 CPUtime=1.93 cores=4-31
/proc/24064/task/24095/stat : 24095 (plingeling) R 24059 24059 24033 0 -1 4202560 5165 1240 0 0 185 7 0 1 20 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244402 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=24064/tid=24096] ppid=24059 vsize=1180048 CPUtime=1.81 cores=4-31
/proc/24064/task/24096/stat : 24096 (plingeling) R 24059 24059 24033 0 -1 4202560 3512 1240 0 0 175 5 0 1 19 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4264383 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=24064/tid=24097] ppid=24059 vsize=1180048 CPUtime=1.97 cores=4-31
/proc/24064/task/24097/stat : 24097 (plingeling) R 24059 24059 24033 0 -1 4202560 4552 1240 0 0 188 8 0 1 25 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4260341 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=24064/tid=24098] ppid=24059 vsize=1180048 CPUtime=1.86 cores=4-31
/proc/24064/task/24098/stat : 24098 (plingeling) R 24059 24059 24033 0 -1 4202560 5029 1240 0 0 177 8 0 1 25 0 29 0 603266278 1208369152 170754 268435456000 4194304 4356188 140735602176112 18446744073709551615 4261931 0 0 4096 17442 0 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 81.71
Current children cumulated vsize (KiB) 2020692

[startup+12.7019 s]
/proc/loadavg: 26.50 30.70 31.57 33/447 24098
/proc/meminfo: memFree=255558092/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 0 0 0 15 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
[pid=24060] ppid=24059 vsize=158036 CPUtime=12.68 cores=0
/proc/24060/stat : 24060 (cryptominisat) R 24059 24059 24033 0 -1 4202496 32171 0 0 0 1259 9 0 0 25 0 1 0 603265850 161828864 25101 268435456000 4194304 4587354 140735678391152 18446744073709551615 4238268 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=24062] ppid=24059 vsize=476160 CPUtime=4961.35 cores=2
/proc/24062/stat : 24062 (TNM) R 24059 24059 24033 0 -1 4202496 10307 0 0 0 496131 4 0 0 25 0 1 0 603265850 487587840 10205 268435456000 134512640 134534564 4291322608 18446744073709551615 134529031 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/24062/statm: 119040 10205 103 6 0 117705 0
[pid=24063] ppid=24059 vsize=1182480 CPUtime=4961.35 cores=3
/proc/24063/stat : 24063 (march_hi) R 24059 24059 24033 0 -1 4202496 265979 0 0 0 496017 118 0 0 25 0 1 0 603265850 1210859520 204440 268435456000 4194304 4331988 140735843830752 18446744073709551615 4228968 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24063/statm: 295620 204440 246 34 0 294062 0
[pid=24064] ppid=24059 vsize=4052176 CPUtime=138651 cores=4-31
/proc/24064/stat : 24064 (plingeling) S 24059 24059 24033 0 -1 4202496 3717368 1240 0 0 13862966 2165 0 1 20 0 29 0 603265850 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 13 0 0 0
/proc/24064/statm: 1013044 884533 159 40 0 1011588 0
[pid=24064/tid=24071] ppid=24059 vsize=4052176 CPUtime=4957.05 cores=4-31
/proc/24064/task/24071/stat : 24071 (plingeling) R 24059 24059 24033 0 -1 4202560 181225 1240 0 0 495604 100 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24064/tid=24072] ppid=24059 vsize=4052176 CPUtime=4956.94 cores=4-31
/proc/24064/task/24072/stat : 24072 (plingeling) R 24059 24059 24033 0 -1 4202560 44497 1240 0 0 495678 15 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24064/tid=24073] ppid=24059 vsize=4052176 CPUtime=4956.65 cores=4-31
/proc/24064/task/24073/stat : 24073 (plingeling) R 24059 24059 24033 0 -1 4202560 259217 1240 0 0 495523 141 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=24064/tid=24074] ppid=24059 vsize=4052176 CPUtime=4956.57 cores=4-31
/proc/24064/task/24074/stat : 24074 (plingeling) R 24059 24059 24033 0 -1 4202560 105117 1240 0 0 495548 108 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=24064/tid=24075] ppid=24059 vsize=4052176 CPUtime=4956.91 cores=4-31
/proc/24064/task/24075/stat : 24075 (plingeling) R 24059 24059 24033 0 -1 4202560 44085 1240 0 0 495656 34 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=24064/tid=24076] ppid=24059 vsize=4052176 CPUtime=4953.06 cores=4-31
/proc/24064/task/24076/stat : 24076 (plingeling) R 24059 24059 24033 0 -1 4202560 217717 1240 0 0 495121 184 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4279738 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=24064/tid=24077] ppid=24059 vsize=4052176 CPUtime=4956.77 cores=4-31
/proc/24064/task/24077/stat : 24077 (plingeling) R 24059 24059 24033 0 -1 4202560 233681 1240 0 0 495529 147 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=24064/tid=24078] ppid=24059 vsize=4052176 CPUtime=4956.98 cores=4-31
/proc/24064/task/24078/stat : 24078 (plingeling) R 24059 24059 24033 0 -1 4202560 159492 1240 0 0 495572 125 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4284014 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=24064/tid=24079] ppid=24059 vsize=4052176 CPUtime=4938.25 cores=4-31
/proc/24064/task/24079/stat : 24079 (plingeling) R 24059 24059 24033 0 -1 4202560 99677 1240 0 0 493760 64 0 1 25 0 29 0 603266277 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244409 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=24064/tid=24080] ppid=24059 vsize=4052176 CPUtime=4956.89 cores=4-31
/proc/24064/task/24080/stat : 24080 (plingeling) R 24059 24059 24033 0 -1 4202560 156189 1240 0 0 495597 91 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4280034 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=24064/tid=24081] ppid=24059 vsize=4052176 CPUtime=4956.96 cores=4-31
/proc/24064/task/24081/stat : 24081 (plingeling) R 24059 24059 24033 0 -1 4202560 52297 1240 0 0 495677 18 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=24064/tid=24082] ppid=24059 vsize=4052176 CPUtime=4956.9 cores=4-31
/proc/24064/task/24082/stat : 24082 (plingeling) R 24059 24059 24033 0 -1 4202560 50545 1240 0 0 495659 30 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=24064/tid=24083] ppid=24059 vsize=4052176 CPUtime=4910.89 cores=4-31
/proc/24064/task/24083/stat : 24083 (plingeling) R 24059 24059 24033 0 -1 4202560 116232 1240 0 0 491005 83 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4279478 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=24064/tid=24084] ppid=24059 vsize=4052176 CPUtime=4893.96 cores=4-31
/proc/24064/task/24084/stat : 24084 (plingeling) R 24059 24059 24033 0 -1 4202560 62166 1240 0 0 489367 28 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=24064/tid=24085] ppid=24059 vsize=4052176 CPUtime=4956.86 cores=4-31
/proc/24064/task/24085/stat : 24085 (plingeling) R 24059 24059 24033 0 -1 4202560 133724 1240 0 0 495633 52 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283912 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=24064/tid=24086] ppid=24059 vsize=4052176 CPUtime=4956.92 cores=4-31
/proc/24064/task/24086/stat : 24086 (plingeling) R 24059 24059 24033 0 -1 4202560 89557 1240 0 0 495633 58 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=24064/tid=24087] ppid=24059 vsize=4052176 CPUtime=4956.79 cores=4-31
/proc/24064/task/24087/stat : 24087 (plingeling) R 24059 24059 24033 0 -1 4202560 68771 1240 0 0 495654 24 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4330805 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=24064/tid=24088] ppid=24059 vsize=4052176 CPUtime=4956.84 cores=4-31
/proc/24064/task/24088/stat : 24088 (plingeling) R 24059 24059 24033 0 -1 4202560 185788 1240 0 0 495521 162 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=24064/tid=24089] ppid=24059 vsize=4052176 CPUtime=4955.05 cores=4-31
/proc/24064/task/24089/stat : 24089 (plingeling) R 24059 24059 24033 0 -1 4202560 102735 1240 0 0 495460 44 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4210992 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24064/tid=24090] ppid=24059 vsize=4052176 CPUtime=4956.91 cores=4-31
/proc/24064/task/24090/stat : 24090 (plingeling) R 24059 24059 24033 0 -1 4202560 168177 1240 0 0 495649 41 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4324019 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=24064/tid=24091] ppid=24059 vsize=4052176 CPUtime=4956.89 cores=4-31
/proc/24064/task/24091/stat : 24091 (plingeling) R 24059 24059 24033 0 -1 4202560 203384 1240 0 0 495613 75 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=24064/tid=24092] ppid=24059 vsize=4052176 CPUtime=4956.7 cores=4-31
/proc/24064/task/24092/stat : 24092 (plingeling) R 24059 24059 24033 0 -1 4202560 298401 1240 0 0 495534 135 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4332165 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=24064/tid=24093] ppid=24059 vsize=4052176 CPUtime=4956.9 cores=4-31
/proc/24064/task/24093/stat : 24093 (plingeling) R 24059 24059 24033 0 -1 4202560 174459 1240 0 0 495603 86 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24064/tid=24094] ppid=24059 vsize=4052176 CPUtime=4956.95 cores=4-31
/proc/24064/task/24094/stat : 24094 (plingeling) R 24059 24059 24033 0 -1 4202560 136575 1240 0 0 495569 125 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=24064/tid=24095] ppid=24059 vsize=4052176 CPUtime=4957 cores=4-31
/proc/24064/task/24095/stat : 24095 (plingeling) R 24059 24059 24033 0 -1 4202560 114104 1240 0 0 495622 77 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4238570 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=24064/tid=24096] ppid=24059 vsize=4052176 CPUtime=4956.88 cores=4-31
/proc/24064/task/24096/stat : 24096 (plingeling) R 24059 24059 24033 0 -1 4202560 30138 1240 0 0 495666 21 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4242651 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=24064/tid=24097] ppid=24059 vsize=4052176 CPUtime=4944.65 cores=4-31
/proc/24064/task/24097/stat : 24097 (plingeling) R 24059 24059 24033 0 -1 4202560 45910 1240 0 0 494439 25 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4211032 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=24064/tid=24098] ppid=24059 vsize=4052176 CPUtime=4956.92 cores=4-31
/proc/24064/task/24098/stat : 24098 (plingeling) R 24059 24059 24033 0 -1 4202560 70305 1240 0 0 495661 30 0 1 25 0 29 0 603266278 4149428224 884533 268435456000 4194304 4356188 140735602176112 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 158497
Current children cumulated vsize (KiB) 8254420



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.42 s]
/proc/loadavg: 32.11 32.15 32.10 33/447 24245
/proc/meminfo: memFree=250845836/264257624 swapFree=67111528/67111528
[pid=24059] ppid=24057 vsize=18324 CPUtime=0.01 cores=0-31
/proc/24059/stat : 24059 (ppfolio) S 24057 24059 24033 0 -1 4202496 515 0 0 0 0 1 0 0 15 0 1 0 603265849 18763776 386 268435456000 4194304 4287625 140736174856320 18446744073709551615 258650975999 0 0 4096 65536 8875434335319097343 0 0 17 4 0 0 0
/proc/24059/statm: 4581 386 337 23 0 91 0
[pid=24060] ppid=24059 vsize=1400064 CPUtime=5008.45 cores=0
/proc/24060/stat : 24060 (cryptominisat) R 24059 24059 24033 0 -1 4202496 825004 0 0 0 500455 390 0 0 25 0 1 0 603265850 1433665536 88221 268435456000 4194304 4587354 140735678391152 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24060/statm: 350016 88221 336 96 0 346578 0
[pid=24061] ppid=24059 vsize=1125216 CPUtime=5008.45 cores=1
/proc/24061/stat : 24061 (clasp) R 24059 24059 24033 0 -1 4202496 280348 0 0 0 500750 95 0 0 25 0 1 0 603265850 1152221184 276626 268435456000 4194304 4785740 140736765949728 18446744073709551615 4286964 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/24061/statm: 281304 276626 338 145 0 278349 0
[pid=24062] ppid=24059 vsize=476160 CPUtime=5008.48 cores=2
/proc/24062/stat : 24062 (TNM) R 24059 24059 24033 0 -1 4202496 10307 0 0 0 500844 4 0 0 25 0 1 0 603265850 487587840 10205 268435456000 134512640 134534564 4291322608 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/24062/statm: 119040 10205 103 6 0 117705 0
[pid=24063] ppid=24059 vsize=1182480 CPUtime=5008.49 cores=3
/proc/24063/stat : 24063 (march_hi) R 24059 24059 24033 0 -1 4202496 266281 0 0 0 500730 119 0 0 25 0 1 0 603265850 1210859520 204742 268435456000 4194304 4331988 140735843830752 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24063/statm: 295620 204742 246 34 0 294062 0
[pid=24064] ppid=24059 vsize=4138140 CPUtime=139970 cores=4-31
/proc/24064/stat : 24064 (plingeling) S 24059 24059 24033 0 -1 4202496 3781510 1240 0 0 13994778 2183 0 1 20 0 29 0 603265850 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 13 0 0 0
/proc/24064/statm: 1034535 894247 159 40 0 1033079 0
[pid=24064/tid=24071] ppid=24059 vsize=4138140 CPUtime=5004.18 cores=4-31
/proc/24064/task/24071/stat : 24071 (plingeling) R 24059 24059 24033 0 -1 4202560 187394 1240 0 0 500316 101 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=24064/tid=24072] ppid=24059 vsize=4138140 CPUtime=5004.07 cores=4-31
/proc/24064/task/24072/stat : 24072 (plingeling) R 24059 24059 24033 0 -1 4202560 44497 1240 0 0 500391 15 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=24064/tid=24073] ppid=24059 vsize=4138140 CPUtime=5003.78 cores=4-31
/proc/24064/task/24073/stat : 24073 (plingeling) R 24059 24059 24033 0 -1 4202560 262296 1240 0 0 500236 141 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=24064/tid=24074] ppid=24059 vsize=4138140 CPUtime=5003.71 cores=4-31
/proc/24064/task/24074/stat : 24074 (plingeling) R 24059 24059 24033 0 -1 4202560 105117 1240 0 0 500261 109 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=24064/tid=24075] ppid=24059 vsize=4138140 CPUtime=5004.05 cores=4-31
/proc/24064/task/24075/stat : 24075 (plingeling) R 24059 24059 24033 0 -1 4202560 44085 1240 0 0 500370 34 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4236939 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=24064/tid=24076] ppid=24059 vsize=4138140 CPUtime=5000.15 cores=4-31
/proc/24064/task/24076/stat : 24076 (plingeling) R 24059 24059 24033 0 -1 4202560 223861 1240 0 0 499828 186 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4284016 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=24064/tid=24077] ppid=24059 vsize=4138140 CPUtime=5003.9 cores=4-31
/proc/24064/task/24077/stat : 24077 (plingeling) R 24059 24059 24033 0 -1 4202560 239441 1240 0 0 500240 149 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=24064/tid=24078] ppid=24059 vsize=4138140 CPUtime=5004.1 cores=4-31
/proc/24064/task/24078/stat : 24078 (plingeling) R 24059 24059 24033 0 -1 4202560 163588 1240 0 0 500284 125 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4279427 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=24064/tid=24079] ppid=24059 vsize=4138140 CPUtime=4985.38 cores=4-31
/proc/24064/task/24079/stat : 24079 (plingeling) R 24059 24059 24033 0 -1 4202560 101213 1240 0 0 498473 64 0 1 25 0 29 0 603266277 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=24064/tid=24080] ppid=24059 vsize=4138140 CPUtime=5004.02 cores=4-31
/proc/24064/task/24080/stat : 24080 (plingeling) R 24059 24059 24033 0 -1 4202560 160029 1240 0 0 500310 91 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283885 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=24064/tid=24081] ppid=24059 vsize=4138140 CPUtime=5004.09 cores=4-31
/proc/24064/task/24081/stat : 24081 (plingeling) R 24059 24059 24033 0 -1 4202560 52297 1240 0 0 500390 18 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=24064/tid=24082] ppid=24059 vsize=4138140 CPUtime=5004.03 cores=4-31
/proc/24064/task/24082/stat : 24082 (plingeling) R 24059 24059 24033 0 -1 4202560 50545 1240 0 0 500372 30 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=24064/tid=24083] ppid=24059 vsize=4138140 CPUtime=4958.02 cores=4-31
/proc/24064/task/24083/stat : 24083 (plingeling) R 24059 24059 24033 0 -1 4202560 120840 1240 0 0 495717 84 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=24064/tid=24084] ppid=24059 vsize=4138140 CPUtime=4941.1 cores=4-31
/proc/24064/task/24084/stat : 24084 (plingeling) R 24059 24059 24033 0 -1 4202560 67286 1240 0 0 494079 30 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4332322 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=24064/tid=24085] ppid=24059 vsize=4138140 CPUtime=5004 cores=4-31
/proc/24064/task/24085/stat : 24085 (plingeling) R 24059 24059 24033 0 -1 4202560 135260 1240 0 0 500346 53 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283885 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=24064/tid=24086] ppid=24059 vsize=4138140 CPUtime=5004.05 cores=4-31
/proc/24064/task/24086/stat : 24086 (plingeling) R 24059 24059 24033 0 -1 4202560 93653 1240 0 0 500345 59 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=24064/tid=24087] ppid=24059 vsize=4138140 CPUtime=5003.92 cores=4-31
/proc/24064/task/24087/stat : 24087 (plingeling) R 24059 24059 24033 0 -1 4202560 70819 1240 0 0 500367 24 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=24064/tid=24088] ppid=24059 vsize=4138140 CPUtime=5003.97 cores=4-31
/proc/24064/task/24088/stat : 24088 (plingeling) R 24059 24059 24033 0 -1 4202560 185788 1240 0 0 500234 162 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=24064/tid=24089] ppid=24059 vsize=4138140 CPUtime=5002.19 cores=4-31
/proc/24064/task/24089/stat : 24089 (plingeling) R 24059 24059 24033 0 -1 4202560 105807 1240 0 0 500173 45 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331203 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=24064/tid=24090] ppid=24059 vsize=4138140 CPUtime=5004.05 cores=4-31
/proc/24064/task/24090/stat : 24090 (plingeling) R 24059 24059 24033 0 -1 4202560 171249 1240 0 0 500362 42 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4244636 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=24064/tid=24091] ppid=24059 vsize=4138140 CPUtime=5004.02 cores=4-31
/proc/24064/task/24091/stat : 24091 (plingeling) R 24059 24059 24033 0 -1 4202560 204152 1240 0 0 500326 75 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=24064/tid=24092] ppid=24059 vsize=4138140 CPUtime=5003.83 cores=4-31
/proc/24064/task/24092/stat : 24092 (plingeling) R 24059 24059 24033 0 -1 4202560 298401 1240 0 0 500247 135 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331069 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=24064/tid=24093] ppid=24059 vsize=4138140 CPUtime=5004.04 cores=4-31
/proc/24064/task/24093/stat : 24093 (plingeling) R 24059 24059 24033 0 -1 4202560 183645 1240 0 0 500312 91 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=24064/tid=24094] ppid=24059 vsize=4138140 CPUtime=5004.08 cores=4-31
/proc/24064/task/24094/stat : 24094 (plingeling) R 24059 24059 24033 0 -1 4202560 136575 1240 0 0 500282 125 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=24064/tid=24095] ppid=24059 vsize=4138140 CPUtime=5004.13 cores=4-31
/proc/24064/task/24095/stat : 24095 (plingeling) R 24059 24059 24033 0 -1 4202560 114116 1240 0 0 500335 77 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=24064/tid=24096] ppid=24059 vsize=4138140 CPUtime=5004.01 cores=4-31
/proc/24064/task/24096/stat : 24096 (plingeling) R 24059 24059 24033 0 -1 4202560 30138 1240 0 0 500379 21 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=24064/tid=24097] ppid=24059 vsize=4138140 CPUtime=4990.43 cores=4-31
/proc/24064/task/24097/stat : 24097 (plingeling) R 24059 24059 24033 0 -1 4202560 45910 1240 0 0 499017 25 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4284190 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=24064/tid=24098] ppid=24059 vsize=4138140 CPUtime=5004.05 cores=4-31
/proc/24064/task/24098/stat : 24098 (plingeling) R 24059 24059 24033 0 -1 4202560 70305 1240 0 0 500374 30 0 1 25 0 29 0 603266278 4237455360 894247 268435456000 4194304 4356188 140735602176112 18446744073709551615 4284008 0 0 4096 17442 0 0 0 -1 30 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 8340384

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 24059 and gives
#  childrusage.ru_utime.tv_sec=5008
#  childrusage.ru_utime.tv_usec=456598
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=65989
# CPU time returned by wait4() is 5008.52
# while last known CPU time is 160004
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5009.46
CPU time (s): 160004
CPU user time (s): 159976
CPU system time (s): 27.93
CPU usage (%): 3194.03
Max. virtual memory (cumulated for all children) (KiB): 9457460

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5008.46
system time used= 0.065989
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 10824
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 533
involuntary context switches= 12509

runsolver used 30.0684 second user time and 111.477 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-09 07:09:54
IDJOB=3347239
IDBENCH=70799
IDSOLVER=1808
FILE ID=node200/3347239-1304917794
RUNJOBID= node200-1304917794-24047
PBS_JOBID= 13281998
Free space on /tmp= 62048 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT09/APPLICATIONS/bitverif/countbitswegner/countbitswegner128.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347239-1304917794/watcher-3347239-1304917794 -o /tmp/evaluation-result-3347239-1304917794/solver-3347239-1304917794 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347239-1304917794.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 4b75ef7fb46928f50793af1d74c176e2
RANDOM SEED=227910541

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      256745704 kB
Buffers:        363380 kB
Cached:        6800880 kB
SwapCached:          0 kB
Active:        6341592 kB
Inactive:       891708 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      256745704 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           14704 kB
Writeback:           0 kB
AnonPages:       68940 kB
Mapped:          14400 kB
Slab:           191964 kB
PageTables:       4412 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   184988 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62036 MiB
End job on node200 at 2011-05-09 08:33:26