Trace number 3347357

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160003 5013.94

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
openstacks-sequencedstrips-nonadl-nonnegated-os-sequencedstrips-p30_1.045-NOTKNOWN.cnf
MD5SUM0385ba51f44fb47d0b0726e26312816f
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables171676
Number of clauses858766
Sum of the clauses size2046721
Maximum clause length63
Minimum clause length1
Number of clauses of size 1211
Number of clauses of size 2847665
Number of clauses of size 32700
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58190

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347357-1305045179/watcher-3347357-1305045179 -o /tmp/evaluation-result-3347357-1305045179/solver-3347357-1305045179 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347357-1305045179.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 23.02 29.92 31.24 1/413 4118
/proc/meminfo: memFree=256305100/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0

[startup+0.075757 s]
/proc/loadavg: 23.02 29.92 31.24 1/413 4118
/proc/meminfo: memFree=256305100/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100761 s]
/proc/loadavg: 23.02 29.92 31.24 1/413 4118
/proc/meminfo: memFree=256305100/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300788 s]
/proc/loadavg: 23.02 29.92 31.24 1/413 4118
/proc/meminfo: memFree=256305100/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700774 s]
/proc/loadavg: 23.02 29.92 31.24 1/413 4118
/proc/meminfo: memFree=256305100/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 16 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50079 s]
/proc/loadavg: 23.02 29.92 31.24 6/419 4129
/proc/meminfo: memFree=255980144/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
[pid=4119] ppid=4118 vsize=120356 CPUtime=1.49 cores=0
/proc/4119/stat : 4119 (cryptominisat) R 4118 4118 4092 0 -1 4202496 23407 0 0 0 143 6 0 0 20 0 1 0 616004433 123244544 23058 268435456000 4194304 4587354 140733938499328 18446744073709551615 4364488 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/4119/statm: 30089 23058 309 96 0 26651 0
[pid=4120] ppid=4118 vsize=93928 CPUtime=1.49 cores=1
/proc/4120/stat : 4120 (clasp) R 4118 4118 4092 0 -1 4202496 24656 0 0 0 142 7 0 0 20 0 1 0 616004433 96182272 20051 268435456000 4194304 4785740 140733692325440 18446744073709551615 4495623 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/4120/statm: 23482 20051 317 145 0 20527 0
[pid=4121] ppid=4118 vsize=500048 CPUtime=1.48 cores=2
/proc/4121/stat : 4121 (TNM) R 4118 4118 4092 0 -1 4202496 16489 0 0 0 143 5 0 0 20 0 1 0 616004433 512049152 16386 268435456000 134512640 134534564 4289880192 18446744073709551615 134524580 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4121/statm: 125012 16386 91 6 0 123677 0
[pid=4122] ppid=4118 vsize=138056 CPUtime=1.48 cores=3
/proc/4122/stat : 4122 (march_hi) R 4118 4118 4092 0 -1 4194304 36750 0 0 0 137 11 0 0 20 0 1 0 616004433 141369344 24669 268435456000 4194304 4331988 140736717502608 18446744073709551615 4252200 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/4122/statm: 34514 24669 126 34 0 32956 0
[pid=4123] ppid=4118 vsize=150876 CPUtime=1.49 cores=4-31
/proc/4123/stat : 4123 (plingeling) R 4118 4118 4092 0 -1 4202496 36478 1238 0 0 139 9 0 1 21 0 1 0 616004433 154497024 36362 268435456000 4194304 4356188 140736724986944 18446744073709551615 4277939 0 0 4096 17442 0 0 0 17 12 0 0 0
/proc/4123/statm: 37719 36362 129 40 0 36263 0
Current children cumulated CPU time (s) 7.43
Current children cumulated vsize (KiB) 1021588

[startup+3.10084 s]
/proc/loadavg: 23.02 29.92 31.24 6/419 4129
/proc/meminfo: memFree=255802592/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
[pid=4119] ppid=4118 vsize=120740 CPUtime=3.09 cores=0
/proc/4119/stat : 4119 (cryptominisat) R 4118 4118 4092 0 -1 4202496 23503 0 0 0 303 6 0 0 25 0 1 0 616004433 123637760 23154 268435456000 4194304 4587354 140733938499328 18446744073709551615 4248827 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/4119/statm: 30185 23154 310 96 0 26747 0
[pid=4120] ppid=4118 vsize=81748 CPUtime=3.08 cores=1
/proc/4120/stat : 4120 (clasp) R 4118 4118 4092 0 -1 4202496 24780 0 0 0 301 7 0 0 25 0 1 0 616004433 83709952 17234 268435456000 4194304 4785740 140733692325440 18446744073709551615 4552626 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/4120/statm: 20437 17234 338 145 0 17482 0
[pid=4121] ppid=4118 vsize=530408 CPUtime=3.08 cores=2
/proc/4121/stat : 4121 (TNM) R 4118 4118 4092 0 -1 4202496 30802 0 0 0 300 8 0 0 25 0 1 0 616004433 543137792 30699 268435456000 134512640 134534564 4289880192 18446744073709551615 134529953 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4121/statm: 132602 30699 95 6 0 131267 0
[pid=4122] ppid=4118 vsize=137204 CPUtime=3.08 cores=3
/proc/4122/stat : 4122 (march_hi) R 4118 4118 4092 0 -1 4194304 68134 0 0 0 291 17 0 0 25 0 1 0 616004433 140496896 24464 268435456000 4194304 4331988 140736717502608 18446744073709551615 4252174 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/4122/statm: 34301 24464 127 34 0 32743 0
[pid=4123] ppid=4118 vsize=297452 CPUtime=3.09 cores=4-31
/proc/4123/stat : 4123 (plingeling) R 4118 4118 4092 0 -1 4202496 73186 1238 0 0 292 16 0 1 25 0 1 0 616004433 304590848 73022 268435456000 4194304 4356188 140736724986944 18446744073709551615 4277785 0 0 4096 17442 0 0 0 17 12 0 0 0
/proc/4123/statm: 74363 73022 129 40 0 72907 0
Current children cumulated CPU time (s) 15.42
Current children cumulated vsize (KiB) 1185876

[startup+6.30099 s]
/proc/loadavg: 21.57 29.50 31.10 6/419 4129
/proc/meminfo: memFree=255501976/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
[pid=4119] ppid=4118 vsize=163016 CPUtime=6.28 cores=0
/proc/4119/stat : 4119 (cryptominisat) R 4118 4118 4092 0 -1 4202496 46532 0 0 0 617 11 0 0 25 0 1 0 616004433 166928384 33369 268435456000 4194304 4587354 140733938499328 18446744073709551615 4236583 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/4119/statm: 40754 33369 328 96 0 37316 0
[pid=4120] ppid=4118 vsize=119744 CPUtime=6.29 cores=1
/proc/4120/stat : 4120 (clasp) R 4118 4118 4092 0 -1 4202496 34379 0 0 0 619 10 0 0 25 0 1 0 616004433 122617856 26833 268435456000 4194304 4785740 140733692325440 18446744073709551615 258650616632 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/4120/statm: 29936 26833 338 145 0 26981 0
[pid=4121] ppid=4118 vsize=530408 CPUtime=6.28 cores=2
/proc/4121/stat : 4121 (TNM) R 4118 4118 4092 0 -1 4202496 30802 0 0 0 620 8 0 0 25 0 1 0 616004433 543137792 30699 268435456000 134512640 134534564 4289880192 18446744073709551615 134529640 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4121/statm: 132602 30699 95 6 0 131267 0
[pid=4122] ppid=4118 vsize=118328 CPUtime=6.28 cores=3
/proc/4122/stat : 4122 (march_hi) R 4118 4118 4092 0 -1 4194304 132548 0 0 0 597 31 0 0 25 0 1 0 616004433 121167872 19762 268435456000 4194304 4331988 140736717502608 18446744073709551615 258650614935 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/4122/statm: 29582 19762 127 34 0 28024 0
[pid=4123] ppid=4118 vsize=491256 CPUtime=6.29 cores=4-31
/proc/4123/stat : 4123 (plingeling) R 4118 4118 4092 0 -1 4202496 121765 1238 0 0 602 26 0 1 25 0 1 0 616004433 503046144 121473 268435456000 4194304 4356188 140736724986944 18446744073709551615 4278302 0 0 4096 17442 0 0 0 17 12 0 0 0
/proc/4123/statm: 122814 121473 129 40 0 121358 0
Current children cumulated CPU time (s) 31.42
Current children cumulated vsize (KiB) 1441076

[startup+12.7025 s]
/proc/loadavg: 22.49 29.56 31.11 33/447 4157
/proc/meminfo: memFree=255068580/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 8 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
[pid=4119] ppid=4118 vsize=163016 CPUtime=12.68 cores=0
/proc/4119/stat : 4119 (cryptominisat) R 4118 4118 4092 0 -1 4202496 47443 0 0 0 1257 11 0 0 25 0 1 0 616004433 166928384 34280 268435456000 4194304 4587354 140733938499328 18446744073709551615 4236595 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/4119/statm: 40754 34280 328 96 0 37316 0
[pid=4120] ppid=4118 vsize=189256 CPUtime=12.69 cores=1
/proc/4120/stat : 4120 (clasp) R 4118 4118 4092 0 -1 4202496 51714 0 0 0 1254 15 0 0 25 0 1 0 616004433 193798144 44168 268435456000 4194304 4785740 140733692325440 18446744073709551615 4287030 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/4120/statm: 47314 44168 338 145 0 44359 0
[pid=4121] ppid=4118 vsize=530408 CPUtime=12.68 cores=2
/proc/4121/stat : 4121 (TNM) R 4118 4118 4092 0 -1 4202496 30802 0 0 0 1260 8 0 0 25 0 1 0 616004433 543137792 30699 268435456000 134512640 134534564 4289880192 18446744073709551615 134530669 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4121/statm: 132602 30699 95 6 0 131267 0
[pid=4122] ppid=4118 vsize=120844 CPUtime=12.69 cores=3
/proc/4122/stat : 4122 (march_hi) R 4118 4118 4092 0 -1 4194304 238911 0 0 0 1211 58 0 0 25 0 1 0 616004433 123744256 20391 268435456000 4194304 4331988 140736717502608 18446744073709551615 4262783 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/4122/statm: 30211 20391 127 34 0 28653 0
[pid=4123] ppid=4118 vsize=1307340 CPUtime=107.06 cores=4-31
/proc/4123/stat : 4123 (plingeling) S 4118 4118 4092 0 -1 4202496 335377 1238 0 0 10578 127 0 1 22 0 29 0 616004433 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/4123/statm: 326835 180169 159 40 0 325379 0
[pid=4123/tid=4130] ppid=4118 vsize=1307340 CPUtime=3.48 cores=4-31
/proc/4123/task/4130/stat : 4130 (plingeling) R 4118 4118 4092 0 -1 4202560 7909 1238 0 0 344 3 0 1 20 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210562 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=4123/tid=4131] ppid=4118 vsize=1307340 CPUtime=3.54 cores=4-31
/proc/4123/task/4131/stat : 4131 (plingeling) R 4118 4118 4092 0 -1 4202560 6910 1238 0 0 352 1 0 1 25 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4288427 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=4123/tid=4132] ppid=4118 vsize=1307340 CPUtime=3.58 cores=4-31
/proc/4123/task/4132/stat : 4132 (plingeling) R 4118 4118 4092 0 -1 4202560 12984 1238 0 0 349 8 0 1 25 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4217704 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=4123/tid=4133] ppid=4118 vsize=1307340 CPUtime=3.54 cores=4-31
/proc/4123/task/4133/stat : 4133 (plingeling) R 4118 4118 4092 0 -1 4202560 6782 1238 0 0 351 2 0 1 25 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=4123/tid=4134] ppid=4118 vsize=1307340 CPUtime=3.54 cores=4-31
/proc/4123/task/4134/stat : 4134 (plingeling) R 4118 4118 4092 0 -1 4202560 4093 1238 0 0 352 1 0 1 25 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4286843 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=4123/tid=4135] ppid=4118 vsize=1307340 CPUtime=3.47 cores=4-31
/proc/4123/task/4135/stat : 4135 (plingeling) R 4118 4118 4092 0 -1 4202560 449 1238 0 0 346 0 0 1 25 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4290695 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=4123/tid=4136] ppid=4118 vsize=1307340 CPUtime=3.55 cores=4-31
/proc/4123/task/4136/stat : 4136 (plingeling) R 4118 4118 4092 0 -1 4202560 3218 1238 0 0 353 1 0 1 25 0 29 0 616005344 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4322024 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=4123/tid=4137] ppid=4118 vsize=1307340 CPUtime=3.52 cores=4-31
/proc/4123/task/4137/stat : 4137 (plingeling) R 4118 4118 4092 0 -1 4202560 6282 1238 0 0 349 2 0 1 20 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4290607 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=4123/tid=4138] ppid=4118 vsize=1307340 CPUtime=3.39 cores=4-31
/proc/4123/task/4138/stat : 4138 (plingeling) D 4118 4118 4092 0 -1 4202560 8096 1238 0 0 331 7 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=4123/tid=4139] ppid=4118 vsize=1307340 CPUtime=3.39 cores=4-31
/proc/4123/task/4139/stat : 4139 (plingeling) D 4118 4118 4092 0 -1 4202560 5177 1238 0 0 331 7 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=4123/tid=4140] ppid=4118 vsize=1307340 CPUtime=3.54 cores=4-31
/proc/4123/task/4140/stat : 4140 (plingeling) R 4118 4118 4092 0 -1 4202560 1474 1238 0 0 352 1 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4289578 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=4123/tid=4141] ppid=4118 vsize=1307340 CPUtime=3.53 cores=4-31
/proc/4123/task/4141/stat : 4141 (plingeling) R 4118 4118 4092 0 -1 4202560 2881 1238 0 0 351 1 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281672 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=4123/tid=4142] ppid=4118 vsize=1307340 CPUtime=3.52 cores=4-31
/proc/4123/task/4142/stat : 4142 (plingeling) R 4118 4118 4092 0 -1 4202560 3842 1238 0 0 350 1 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4290639 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=4123/tid=4143] ppid=4118 vsize=1307340 CPUtime=3.38 cores=4-31
/proc/4123/task/4143/stat : 4143 (plingeling) D 4118 4118 4092 0 -1 4202560 9484 1238 0 0 330 7 0 1 20 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=4123/tid=4144] ppid=4118 vsize=1307340 CPUtime=3.53 cores=4-31
/proc/4123/task/4144/stat : 4144 (plingeling) R 4118 4118 4092 0 -1 4202560 3069 1238 0 0 351 1 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4287203 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=4123/tid=4145] ppid=4118 vsize=1307340 CPUtime=3.54 cores=4-31
/proc/4123/task/4145/stat : 4145 (plingeling) R 4118 4118 4092 0 -1 4202560 9349 1238 0 0 349 4 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=4123/tid=4146] ppid=4118 vsize=1307340 CPUtime=3.53 cores=4-31
/proc/4123/task/4146/stat : 4146 (plingeling) R 4118 4118 4092 0 -1 4202560 9075 1238 0 0 349 3 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4290660 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=4123/tid=4147] ppid=4118 vsize=1307340 CPUtime=3.52 cores=4-31
/proc/4123/task/4147/stat : 4147 (plingeling) R 4118 4118 4092 0 -1 4202560 2538 1238 0 0 350 1 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4286699 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=4123/tid=4148] ppid=4118 vsize=1307340 CPUtime=3.51 cores=4-31
/proc/4123/task/4148/stat : 4148 (plingeling) R 4118 4118 4092 0 -1 4202560 7899 1238 0 0 347 3 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210610 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=4123/tid=4149] ppid=4118 vsize=1307340 CPUtime=3.52 cores=4-31
/proc/4123/task/4149/stat : 4149 (plingeling) R 4118 4118 4092 0 -1 4202560 3587 1238 0 0 350 1 0 1 20 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4334609 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=4123/tid=4150] ppid=4118 vsize=1307340 CPUtime=3.5 cores=4-31
/proc/4123/task/4150/stat : 4150 (plingeling) R 4118 4118 4092 0 -1 4202560 4020 1238 0 0 347 2 0 1 25 0 29 0 616005345 1338716160 180169 268435456000 4194304 4356188 140736724986944 18446744073709551615 4289310 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=4123/tid=4151] ppid=4118 vsize=1307340 CPUtime=3.55 cores=4-31

################
# More data... #
################

/proc/4119/statm: 161088 93931 338 96 0 157650 0
[pid=4120] ppid=4118 vsize=3486320 CPUtime=5012.96 cores=1
/proc/4120/stat : 4120 (clasp) R 4118 4118 4092 0 -1 4202496 874327 0 0 0 501002 294 0 0 25 0 1 0 616004433 3569991680 866781 268435456000 4194304 4785740 140733692325440 18446744073709551615 4303607 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/4120/statm: 871580 866781 338 145 0 868625 0
[pid=4121] ppid=4118 vsize=530408 CPUtime=5012.96 cores=2
/proc/4121/stat : 4121 (TNM) R 4118 4118 4092 0 -1 4202496 30802 0 0 0 501288 8 0 0 25 0 1 0 616004433 543137792 30699 268435456000 134512640 134534564 4289880192 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4121/statm: 132602 30699 95 6 0 131267 0
[pid=4122] ppid=4118 vsize=412840 CPUtime=5012.93 cores=3
/proc/4122/stat : 4122 (march_hi) R 4118 4118 4092 0 -1 4202496 441568 0 0 0 501183 110 0 0 25 0 1 0 616004433 422748160 52316 268435456000 4194304 4331988 140736717502608 18446744073709551615 4228270 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/4122/statm: 103210 52316 228 34 0 101652 0
[pid=4123] ppid=4118 vsize=9257416 CPUtime=139951 cores=4-31
/proc/4123/stat : 4123 (plingeling) S 4118 4118 4092 0 -1 4202496 20916676 1238 0 0 13981259 13823 0 1 22 0 29 0 616004433 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/4123/statm: 2314354 2168250 159 40 0 2312898 0
[pid=4123/tid=4130] ppid=4118 vsize=9257416 CPUtime=5001.38 cores=4-31
/proc/4123/task/4130/stat : 4130 (plingeling) R 4118 4118 4092 0 -1 4202560 869171 1238 0 0 499615 522 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4236874 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=4123/tid=4131] ppid=4118 vsize=9257416 CPUtime=4998.06 cores=4-31
/proc/4123/task/4131/stat : 4131 (plingeling) R 4118 4118 4092 0 -1 4202560 665718 1238 0 0 499466 339 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=4123/tid=4132] ppid=4118 vsize=9257416 CPUtime=5003.38 cores=4-31
/proc/4123/task/4132/stat : 4132 (plingeling) R 4118 4118 4092 0 -1 4202560 906112 1238 0 0 499811 526 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=4123/tid=4133] ppid=4118 vsize=9257416 CPUtime=4988.05 cores=4-31
/proc/4123/task/4133/stat : 4133 (plingeling) R 4118 4118 4092 0 -1 4202560 662490 1238 0 0 498395 409 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281672 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=4123/tid=4134] ppid=4118 vsize=9257416 CPUtime=5003.05 cores=4-31
/proc/4123/task/4134/stat : 4134 (plingeling) R 4118 4118 4092 0 -1 4202560 653461 1238 0 0 499943 361 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4283998 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=4123/tid=4135] ppid=4118 vsize=9257416 CPUtime=5003 cores=4-31
/proc/4123/task/4135/stat : 4135 (plingeling) R 4118 4118 4092 0 -1 4202560 686473 1238 0 0 499926 373 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210587 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=4123/tid=4136] ppid=4118 vsize=9257416 CPUtime=5002.52 cores=4-31
/proc/4123/task/4136/stat : 4136 (plingeling) R 4118 4118 4092 0 -1 4202560 860789 1238 0 0 499685 566 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281627 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=4123/tid=4137] ppid=4118 vsize=9257416 CPUtime=5003.2 cores=4-31
/proc/4123/task/4137/stat : 4137 (plingeling) R 4118 4118 4092 0 -1 4202560 790743 1238 0 0 499466 853 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=4123/tid=4138] ppid=4118 vsize=9257416 CPUtime=5003.06 cores=4-31
/proc/4123/task/4138/stat : 4138 (plingeling) R 4118 4118 4092 0 -1 4202560 887420 1238 0 0 499889 416 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281706 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=4123/tid=4139] ppid=4118 vsize=9257416 CPUtime=5001.19 cores=4-31
/proc/4123/task/4139/stat : 4139 (plingeling) R 4118 4118 4092 0 -1 4202560 707268 1238 0 0 499664 454 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4283905 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=4123/tid=4140] ppid=4118 vsize=9257416 CPUtime=5002.82 cores=4-31
/proc/4123/task/4140/stat : 4140 (plingeling) R 4118 4118 4092 0 -1 4202560 722591 1238 0 0 499747 534 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=4123/tid=4141] ppid=4118 vsize=9257416 CPUtime=5003.34 cores=4-31
/proc/4123/task/4141/stat : 4141 (plingeling) R 4118 4118 4092 0 -1 4202560 381087 1238 0 0 500146 187 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281639 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=4123/tid=4142] ppid=4118 vsize=9257416 CPUtime=4985.98 cores=4-31
/proc/4123/task/4142/stat : 4142 (plingeling) R 4118 4118 4092 0 -1 4202560 561227 1238 0 0 498306 291 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281681 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=4123/tid=4143] ppid=4118 vsize=9257416 CPUtime=5003.21 cores=4-31
/proc/4123/task/4143/stat : 4143 (plingeling) R 4118 4118 4092 0 -1 4202560 907731 1238 0 0 499734 586 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4236874 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=4123/tid=4144] ppid=4118 vsize=9257416 CPUtime=4992.83 cores=4-31
/proc/4123/task/4144/stat : 4144 (plingeling) R 4118 4118 4092 0 -1 4202560 726126 1238 0 0 498891 391 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281793 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=4123/tid=4145] ppid=4118 vsize=9257416 CPUtime=5002.52 cores=4-31
/proc/4123/task/4145/stat : 4145 (plingeling) R 4118 4118 4092 0 -1 4202560 776015 1238 0 0 499400 851 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=4123/tid=4146] ppid=4118 vsize=9257416 CPUtime=5003.31 cores=4-31
/proc/4123/task/4146/stat : 4146 (plingeling) R 4118 4118 4092 0 -1 4202560 731320 1238 0 0 499864 466 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4284205 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=4123/tid=4147] ppid=4118 vsize=9257416 CPUtime=5003.2 cores=4-31
/proc/4123/task/4147/stat : 4147 (plingeling) R 4118 4118 4092 0 -1 4202560 632116 1238 0 0 499970 349 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210575 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=4123/tid=4148] ppid=4118 vsize=9257416 CPUtime=5003.04 cores=4-31
/proc/4123/task/4148/stat : 4148 (plingeling) R 4118 4118 4092 0 -1 4202560 857855 1238 0 0 499686 617 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4237401 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=4123/tid=4149] ppid=4118 vsize=9257416 CPUtime=5002.36 cores=4-31
/proc/4123/task/4149/stat : 4149 (plingeling) R 4118 4118 4092 0 -1 4202560 685060 1238 0 0 499734 501 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4273757 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=4123/tid=4150] ppid=4118 vsize=9257416 CPUtime=4981.72 cores=4-31
/proc/4123/task/4150/stat : 4150 (plingeling) R 4118 4118 4092 0 -1 4202560 896832 1238 0 0 497605 566 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4211010 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=4123/tid=4151] ppid=4118 vsize=9257416 CPUtime=5003.18 cores=4-31
/proc/4123/task/4151/stat : 4151 (plingeling) R 4118 4118 4092 0 -1 4202560 754381 1238 0 0 499780 537 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210596 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=4123/tid=4152] ppid=4118 vsize=9257416 CPUtime=4986.54 cores=4-31
/proc/4123/task/4152/stat : 4152 (plingeling) R 4118 4118 4092 0 -1 4202560 638481 1238 0 0 498307 346 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281692 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=4123/tid=4153] ppid=4118 vsize=9257416 CPUtime=5000.92 cores=4-31
/proc/4123/task/4153/stat : 4153 (plingeling) R 4118 4118 4092 0 -1 4202560 754366 1238 0 0 499337 754 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331328 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=4123/tid=4154] ppid=4118 vsize=9257416 CPUtime=4980.96 cores=4-31
/proc/4123/task/4154/stat : 4154 (plingeling) R 4118 4118 4092 0 -1 4202560 758686 1238 0 0 497473 622 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281498 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=4123/tid=4155] ppid=4118 vsize=9257416 CPUtime=4973.61 cores=4-31
/proc/4123/task/4155/stat : 4155 (plingeling) R 4118 4118 4092 0 -1 4202560 756848 1238 0 0 496950 410 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=4123/tid=4156] ppid=4118 vsize=9257416 CPUtime=5002.08 cores=4-31
/proc/4123/task/4156/stat : 4156 (plingeling) R 4118 4118 4092 0 -1 4202560 839184 1238 0 0 499735 472 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281593 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=4123/tid=4157] ppid=4118 vsize=9257416 CPUtime=5003.42 cores=4-31
/proc/4123/task/4157/stat : 4157 (plingeling) R 4118 4118 4092 0 -1 4202560 670725 1238 0 0 499867 474 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 24 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 14349660

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5013.9 s]
/proc/loadavg: 32.07 32.07 32.07 33/447 4303
/proc/meminfo: memFree=243436560/264257624 swapFree=67111528/67111528
[pid=4118] ppid=4116 vsize=18324 CPUtime=0.05 cores=0-31
/proc/4118/stat : 4118 (ppfolio) S 4116 4118 4092 0 -1 4202496 514 0 0 0 1 4 0 0 15 0 1 0 616004433 18763776 386 268435456000 4194304 4287625 140736731690032 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 0 0 0 0
/proc/4118/statm: 4581 386 337 23 0 91 0
[pid=4119] ppid=4118 vsize=644352 CPUtime=5012.91 cores=0
/proc/4119/stat : 4119 (cryptominisat) R 4118 4118 4092 0 -1 4202496 6823137 0 0 0 499440 1851 0 0 25 0 1 0 616004433 659816448 93931 268435456000 4194304 4587354 140733938499328 18446744073709551615 4241114 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/4119/statm: 161088 93931 338 96 0 157650 0
[pid=4120] ppid=4118 vsize=3486320 CPUtime=5012.96 cores=1
/proc/4120/stat : 4120 (clasp) R 4118 4118 4092 0 -1 4202496 874327 0 0 0 501002 294 0 0 25 0 1 0 616004433 3569991680 866781 268435456000 4194304 4785740 140733692325440 18446744073709551615 4303607 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/4120/statm: 871580 866781 338 145 0 868625 0
[pid=4121] ppid=4118 vsize=530408 CPUtime=5012.96 cores=2
/proc/4121/stat : 4121 (TNM) R 4118 4118 4092 0 -1 4202496 30802 0 0 0 501288 8 0 0 25 0 1 0 616004433 543137792 30699 268435456000 134512640 134534564 4289880192 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4121/statm: 132602 30699 95 6 0 131267 0
[pid=4122] ppid=4118 vsize=412840 CPUtime=5012.93 cores=3
/proc/4122/stat : 4122 (march_hi) R 4118 4118 4092 0 -1 4202496 441568 0 0 0 501183 110 0 0 25 0 1 0 616004433 422748160 52316 268435456000 4194304 4331988 140736717502608 18446744073709551615 4228270 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/4122/statm: 103210 52316 228 34 0 101652 0
[pid=4123] ppid=4118 vsize=9257416 CPUtime=139951 cores=4-31
/proc/4123/stat : 4123 (plingeling) S 4118 4118 4092 0 -1 4202496 20916676 1238 0 0 13981259 13823 0 1 22 0 29 0 616004433 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 26 0 0 0
/proc/4123/statm: 2314354 2168250 159 40 0 2312898 0
[pid=4123/tid=4130] ppid=4118 vsize=9257416 CPUtime=5001.38 cores=4-31
/proc/4123/task/4130/stat : 4130 (plingeling) R 4118 4118 4092 0 -1 4202560 869171 1238 0 0 499615 522 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4236874 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=4123/tid=4131] ppid=4118 vsize=9257416 CPUtime=4998.06 cores=4-31
/proc/4123/task/4131/stat : 4131 (plingeling) R 4118 4118 4092 0 -1 4202560 665718 1238 0 0 499466 339 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=4123/tid=4132] ppid=4118 vsize=9257416 CPUtime=5003.38 cores=4-31
/proc/4123/task/4132/stat : 4132 (plingeling) R 4118 4118 4092 0 -1 4202560 906112 1238 0 0 499811 526 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=4123/tid=4133] ppid=4118 vsize=9257416 CPUtime=4988.05 cores=4-31
/proc/4123/task/4133/stat : 4133 (plingeling) R 4118 4118 4092 0 -1 4202560 662490 1238 0 0 498395 409 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281672 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=4123/tid=4134] ppid=4118 vsize=9257416 CPUtime=5003.05 cores=4-31
/proc/4123/task/4134/stat : 4134 (plingeling) R 4118 4118 4092 0 -1 4202560 653461 1238 0 0 499943 361 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4283998 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=4123/tid=4135] ppid=4118 vsize=9257416 CPUtime=5003 cores=4-31
/proc/4123/task/4135/stat : 4135 (plingeling) R 4118 4118 4092 0 -1 4202560 686473 1238 0 0 499926 373 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210587 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=4123/tid=4136] ppid=4118 vsize=9257416 CPUtime=5002.52 cores=4-31
/proc/4123/task/4136/stat : 4136 (plingeling) R 4118 4118 4092 0 -1 4202560 860789 1238 0 0 499685 566 0 1 25 0 29 0 616005344 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281627 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=4123/tid=4137] ppid=4118 vsize=9257416 CPUtime=5003.2 cores=4-31
/proc/4123/task/4137/stat : 4137 (plingeling) R 4118 4118 4092 0 -1 4202560 790743 1238 0 0 499466 853 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=4123/tid=4138] ppid=4118 vsize=9257416 CPUtime=5003.06 cores=4-31
/proc/4123/task/4138/stat : 4138 (plingeling) R 4118 4118 4092 0 -1 4202560 887420 1238 0 0 499889 416 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281706 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=4123/tid=4139] ppid=4118 vsize=9257416 CPUtime=5001.19 cores=4-31
/proc/4123/task/4139/stat : 4139 (plingeling) R 4118 4118 4092 0 -1 4202560 707268 1238 0 0 499664 454 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4283905 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=4123/tid=4140] ppid=4118 vsize=9257416 CPUtime=5002.82 cores=4-31
/proc/4123/task/4140/stat : 4140 (plingeling) R 4118 4118 4092 0 -1 4202560 722591 1238 0 0 499747 534 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=4123/tid=4141] ppid=4118 vsize=9257416 CPUtime=5003.34 cores=4-31
/proc/4123/task/4141/stat : 4141 (plingeling) R 4118 4118 4092 0 -1 4202560 381087 1238 0 0 500146 187 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281639 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=4123/tid=4142] ppid=4118 vsize=9257416 CPUtime=4985.98 cores=4-31
/proc/4123/task/4142/stat : 4142 (plingeling) R 4118 4118 4092 0 -1 4202560 561227 1238 0 0 498306 291 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281681 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=4123/tid=4143] ppid=4118 vsize=9257416 CPUtime=5003.21 cores=4-31
/proc/4123/task/4143/stat : 4143 (plingeling) R 4118 4118 4092 0 -1 4202560 907731 1238 0 0 499734 586 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4236874 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=4123/tid=4144] ppid=4118 vsize=9257416 CPUtime=4992.83 cores=4-31
/proc/4123/task/4144/stat : 4144 (plingeling) R 4118 4118 4092 0 -1 4202560 726126 1238 0 0 498891 391 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281793 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=4123/tid=4145] ppid=4118 vsize=9257416 CPUtime=5002.52 cores=4-31
/proc/4123/task/4145/stat : 4145 (plingeling) R 4118 4118 4092 0 -1 4202560 776015 1238 0 0 499400 851 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=4123/tid=4146] ppid=4118 vsize=9257416 CPUtime=5003.31 cores=4-31
/proc/4123/task/4146/stat : 4146 (plingeling) R 4118 4118 4092 0 -1 4202560 731320 1238 0 0 499864 466 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4284205 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=4123/tid=4147] ppid=4118 vsize=9257416 CPUtime=5003.2 cores=4-31
/proc/4123/task/4147/stat : 4147 (plingeling) R 4118 4118 4092 0 -1 4202560 632116 1238 0 0 499970 349 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210575 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=4123/tid=4148] ppid=4118 vsize=9257416 CPUtime=5003.04 cores=4-31
/proc/4123/task/4148/stat : 4148 (plingeling) R 4118 4118 4092 0 -1 4202560 857855 1238 0 0 499686 617 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4237401 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=4123/tid=4149] ppid=4118 vsize=9257416 CPUtime=5002.36 cores=4-31
/proc/4123/task/4149/stat : 4149 (plingeling) R 4118 4118 4092 0 -1 4202560 685060 1238 0 0 499734 501 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4273757 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=4123/tid=4150] ppid=4118 vsize=9257416 CPUtime=4981.72 cores=4-31
/proc/4123/task/4150/stat : 4150 (plingeling) R 4118 4118 4092 0 -1 4202560 896832 1238 0 0 497605 566 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4211010 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=4123/tid=4151] ppid=4118 vsize=9257416 CPUtime=5003.18 cores=4-31
/proc/4123/task/4151/stat : 4151 (plingeling) R 4118 4118 4092 0 -1 4202560 754381 1238 0 0 499780 537 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4210596 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=4123/tid=4152] ppid=4118 vsize=9257416 CPUtime=4986.54 cores=4-31
/proc/4123/task/4152/stat : 4152 (plingeling) R 4118 4118 4092 0 -1 4202560 638481 1238 0 0 498307 346 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281692 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=4123/tid=4153] ppid=4118 vsize=9257416 CPUtime=5000.92 cores=4-31
/proc/4123/task/4153/stat : 4153 (plingeling) R 4118 4118 4092 0 -1 4202560 754366 1238 0 0 499337 754 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331328 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=4123/tid=4154] ppid=4118 vsize=9257416 CPUtime=4980.96 cores=4-31
/proc/4123/task/4154/stat : 4154 (plingeling) R 4118 4118 4092 0 -1 4202560 758686 1238 0 0 497473 622 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281498 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=4123/tid=4155] ppid=4118 vsize=9257416 CPUtime=4973.61 cores=4-31
/proc/4123/task/4155/stat : 4155 (plingeling) R 4118 4118 4092 0 -1 4202560 756848 1238 0 0 496950 410 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=4123/tid=4156] ppid=4118 vsize=9257416 CPUtime=5002.08 cores=4-31
/proc/4123/task/4156/stat : 4156 (plingeling) R 4118 4118 4092 0 -1 4202560 839184 1238 0 0 499735 472 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4281593 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=4123/tid=4157] ppid=4118 vsize=9257416 CPUtime=5003.42 cores=4-31
/proc/4123/task/4157/stat : 4157 (plingeling) R 4118 4118 4092 0 -1 4202560 670725 1238 0 0 499867 474 0 1 25 0 29 0 616005345 9479593984 2168250 268435456000 4194304 4356188 140736724986944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 24 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 14349660

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 4118 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=15997
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=48992
# CPU time returned by wait4() is 0.064989
# while last known CPU time is 160003
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5013.94
CPU time (s): 160003
CPU user time (s): 159842
CPU system time (s): 160.91
CPU usage (%): 3191.16
Max. virtual memory (cumulated for all children) (KiB): 14828016

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.015997
system time used= 0.048992
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 514
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 11386
involuntary context switches= 2

runsolver used 26.594 second user time and 114.718 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-10 18:33:00
IDJOB=3347357
IDBENCH=82369
IDSOLVER=1808
FILE ID=node200/3347357-1305045179
RUNJOBID= node200-1305045179-4106
PBS_JOBID= 13282044
Free space on /tmp= 62036 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/application/rintanen/SATPlanning/openstacks-sequencedstrips-nonadl-nonnegated-os-sequencedstrips-p30_1.045-NOTKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347357-1305045179/watcher-3347357-1305045179 -o /tmp/evaluation-result-3347357-1305045179/solver-3347357-1305045179 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347357-1305045179.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 0385ba51f44fb47d0b0726e26312816f
RANDOM SEED=650135190

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      256305388 kB
Buffers:        379524 kB
Cached:        7227096 kB
SwapCached:          0 kB
Active:        6741532 kB
Inactive:       933864 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      256305388 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           25512 kB
Writeback:           0 kB
AnonPages:       69940 kB
Mapped:          13484 kB
Slab:           194508 kB
PageTables:       4720 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   189828 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62024 MiB
End job on node200 at 2011-05-10 19:56:36