Trace number 3472359

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 157155 5103.38

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.070-NOTKNOWN.cnf
MD5SUMf52ec4dacdcf65c3aedb61fb46ade01b
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark24553.6
Satisfiable
(Un)Satisfiability was proved
Number of variables153284
Number of clauses2473656
Sum of the clauses size5647736
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22387280
Number of clauses of size 36720
Number of clauses of size 431080
Number of clauses of size 513440
Number of clauses of size over 534440

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...511423/instance-3472359-1307511423.cnf
3.09/3.18	c Solving...
157151.47/5100.03	
157151.47/5100.03	*** INTERRUPTED! ***
157151.47/5100.03	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472359-1307511423/watcher-3472359-1307511423 -o /tmp/evaluation-result-3472359-1307511423/solver-3472359-1307511423 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472359-1307511423.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 17.34 26.44 28.76 1/413 21068
/proc/meminfo: memFree=262959688/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=40344 CPUtime=0.02 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 4807 0 0 0 2 0 0 0 21 0 1 0 16529390 41312256 4736 270582939648 4194304 4964800 140735826949248 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 10086 4744 373 189 0 4960 0

[startup+0.0660501 s]
/proc/loadavg: 17.34 26.44 28.76 1/413 21068
/proc/meminfo: memFree=262959688/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=51132 CPUtime=0.06 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 9148 0 0 0 5 1 0 0 21 0 1 0 16529390 52359168 7880 270582939648 4194304 4964800 140735826949248 18446744073709551615 258650955760 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 12783 7880 386 189 0 7657 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 51132

[startup+0.101047 s]
/proc/loadavg: 17.34 26.44 28.76 1/413 21068
/proc/meminfo: memFree=262959688/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=51536 CPUtime=0.09 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 9250 0 0 0 8 1 0 0 21 0 1 0 16529390 52772864 7982 270582939648 4194304 4964800 140735826949248 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 12884 7984 386 189 0 7758 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 51536

[startup+0.301051 s]
/proc/loadavg: 17.34 26.44 28.76 1/413 21068
/proc/meminfo: memFree=262959688/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=75224 CPUtime=0.29 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 17577 0 0 0 26 3 0 0 22 0 1 0 16529390 77029376 13639 270582939648 4194304 4964800 140735826949248 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 18806 13640 386 189 0 13680 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 75224

[startup+0.701075 s]
/proc/loadavg: 17.34 26.44 28.76 1/413 21068
/proc/meminfo: memFree=262959688/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=128732 CPUtime=0.69 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 36656 0 0 0 61 8 0 0 23 0 1 0 16529390 131821568 26223 270582939648 4194304 4964800 140735826949248 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 32183 26226 386 189 0 27057 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 128732

[startup+1.5011 s]
/proc/loadavg: 17.34 26.44 28.76 2/414 21069
/proc/meminfo: memFree=262818180/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=191124 CPUtime=1.49 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 54066 0 0 0 137 12 0 0 25 0 1 0 16529390 195710976 42099 270582939648 4194304 4964800 140735826949248 18446744073709551615 4602553 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 47781 42099 397 189 0 42655 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 191124

[startup+3.10115 s]
/proc/loadavg: 16.03 26.02 28.61 2/414 21069
/proc/meminfo: memFree=262791748/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=191124 CPUtime=3.09 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 54094 0 0 0 297 12 0 0 25 0 1 0 16529390 195710976 42127 270582939648 4194304 4964800 140735826949248 18446744073709551615 258650615289 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21068/statm: 47781 42127 403 189 0 42655 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 191124

[startup+6.30222 s]
/proc/loadavg: 16.03 26.02 28.61 5/445 21100
/proc/meminfo: memFree=262215492/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=2132672 CPUtime=28.36 cores=0-31
/proc/21068/stat : 21068 (claspmt) S 21066 21068 20866 0 -1 4202496 239458 0 0 0 1438 1398 0 0 18 0 32 0 16529390 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 16 0 0 0
/proc/21068/statm: 533168 185306 440 189 0 527517 0
[pid=21068/tid=21070] ppid=21066 vsize=2132672 CPUtime=0.58 cores=0-31
/proc/21068/task/21070/stat : 21070 (claspmt) S 21066 21068 20866 0 -1 4202560 5130 0 0 0 8 50 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498707733984 0 0 -1 1 0 0 0
[pid=21068/tid=21071] ppid=21066 vsize=2132672 CPUtime=0.66 cores=0-31
/proc/21068/task/21071/stat : 21071 (claspmt) S 21066 21068 20866 0 -1 4202560 6908 0 0 0 19 47 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604505758277088 0 0 -1 6 0 0 0
[pid=21068/tid=21072] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21072/stat : 21072 (claspmt) S 21066 21068 20866 0 -1 4202560 6818 0 0 0 18 51 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498531786208 0 0 -1 12 0 0 0
[pid=21068/tid=21073] ppid=21066 vsize=2132672 CPUtime=0.5 cores=0-31
/proc/21068/task/21073/stat : 21073 (claspmt) S 21066 21068 20866 0 -1 4202560 5411 0 0 0 8 42 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475777504 0 0 -1 11 0 0 0
[pid=21068/tid=21074] ppid=21066 vsize=2132672 CPUtime=2.07 cores=0-31
/proc/21068/task/21074/stat : 21074 (claspmt) R 21066 21068 20866 0 -1 4202560 6553 0 0 0 158 49 0 0 19 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 4651391 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=21068/tid=21075] ppid=21066 vsize=2132672 CPUtime=0.59 cores=0-31
/proc/21068/task/21075/stat : 21075 (claspmt) S 21066 21068 20866 0 -1 4202560 6490 0 0 0 12 47 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475761120 0 0 -1 0 0 0 0
[pid=21068/tid=21076] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21076/stat : 21076 (claspmt) S 21066 21068 20866 0 -1 4202560 7443 0 0 0 19 50 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475769312 0 0 -1 18 0 0 0
[pid=21068/tid=21077] ppid=21066 vsize=2132672 CPUtime=0.49 cores=0-31
/proc/21068/task/21077/stat : 21077 (claspmt) S 21066 21068 20866 0 -1 4202560 5156 0 0 0 7 42 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604500654161376 0 0 -1 19 0 0 0
[pid=21068/tid=21078] ppid=21066 vsize=2132672 CPUtime=0.54 cores=0-31
/proc/21068/task/21078/stat : 21078 (claspmt) S 21066 21068 20866 0 -1 4202560 5934 0 0 0 9 45 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498707742176 0 0 -1 6 0 0 0
[pid=21068/tid=21079] ppid=21066 vsize=2132672 CPUtime=0.62 cores=0-31
/proc/21068/task/21079/stat : 21079 (claspmt) S 21066 21068 20866 0 -1 4202560 7173 0 0 0 13 49 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604499855105504 0 0 -1 20 0 0 0
[pid=21068/tid=21080] ppid=21066 vsize=2132672 CPUtime=0.63 cores=0-31
/proc/21068/task/21080/stat : 21080 (claspmt) S 21066 21068 20866 0 -1 4202560 6804 0 0 0 20 43 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604504589622752 0 0 -1 21 0 0 0
[pid=21068/tid=21081] ppid=21066 vsize=2132672 CPUtime=0.42 cores=0-31
/proc/21068/task/21081/stat : 21081 (claspmt) S 21066 21068 20866 0 -1 4202560 5084 0 0 0 4 38 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502438788576 0 0 -1 24 0 0 0
[pid=21068/tid=21082] ppid=21066 vsize=2132672 CPUtime=2.4 cores=0-31
/proc/21068/task/21082/stat : 21082 (claspmt) R 21066 21068 20866 0 -1 4202560 5602 0 0 0 195 45 0 0 19 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21068/tid=21083] ppid=21066 vsize=2132672 CPUtime=0.71 cores=0-31
/proc/21068/task/21083/stat : 21083 (claspmt) S 21066 21068 20866 0 -1 4202560 7002 0 0 0 19 52 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498531810784 0 0 -1 26 0 0 0
[pid=21068/tid=21084] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21084/stat : 21084 (claspmt) S 21066 21068 20866 0 -1 4202560 7034 0 0 0 22 47 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502288997856 0 0 -1 14 0 0 0
[pid=21068/tid=21085] ppid=21066 vsize=2132672 CPUtime=0.48 cores=0-31
/proc/21068/task/21085/stat : 21085 (claspmt) S 21066 21068 20866 0 -1 4202560 4788 0 0 0 8 40 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502289006048 0 0 -1 22 0 0 0
[pid=21068/tid=21086] ppid=21066 vsize=2132672 CPUtime=0.51 cores=0-31
/proc/21068/task/21086/stat : 21086 (claspmt) S 21066 21068 20866 0 -1 4202560 5575 0 0 0 5 46 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604500654153184 0 0 -1 16 0 0 0
[pid=21068/tid=21087] ppid=21066 vsize=2132672 CPUtime=0.12 cores=0-31
/proc/21068/task/21087/stat : 21087 (claspmt) S 21066 21068 20866 0 -1 4202560 1715 0 0 0 12 0 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604572833545696 0 0 -1 16 0 0 0
[pid=21068/tid=21088] ppid=21066 vsize=2132672 CPUtime=0.58 cores=0-31
/proc/21068/task/21088/stat : 21088 (claspmt) S 21066 21068 20866 0 -1 4202560 6138 0 0 0 12 46 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502438804960 0 0 -1 4 0 0 0
[pid=21068/tid=21089] ppid=21066 vsize=2132672 CPUtime=0.5 cores=0-31
/proc/21068/task/21089/stat : 21089 (claspmt) S 21066 21068 20866 0 -1 4202560 5071 0 0 0 7 43 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475752928 0 0 -1 27 0 0 0
[pid=21068/tid=21090] ppid=21066 vsize=2132672 CPUtime=2.89 cores=0-31
/proc/21068/task/21090/stat : 21090 (claspmt) R 21066 21068 20866 0 -1 4202560 3751 0 0 0 260 29 0 0 21 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 4402731 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=21068/tid=21091] ppid=21066 vsize=2132672 CPUtime=0.65 cores=0-31
/proc/21068/task/21091/stat : 21091 (claspmt) S 21066 21068 20866 0 -1 4202560 6944 0 0 0 20 45 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498767592928 0 0 -1 7 0 0 0
[pid=21068/tid=21092] ppid=21066 vsize=2132672 CPUtime=0.67 cores=0-31
/proc/21068/task/21092/stat : 21092 (claspmt) S 21066 21068 20866 0 -1 4202560 5933 0 0 0 22 45 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498767584736 0 0 -1 13 0 0 0
[pid=21068/tid=21093] ppid=21066 vsize=2132672 CPUtime=0.51 cores=0-31
/proc/21068/task/21093/stat : 21093 (claspmt) S 21066 21068 20866 0 -1 4202560 5603 0 0 0 7 44 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502438780384 0 0 -1 23 0 0 0
[pid=21068/tid=21094] ppid=21066 vsize=2132672 CPUtime=0.48 cores=0-31
/proc/21068/task/21094/stat : 21094 (claspmt) S 21066 21068 20866 0 -1 4202560 4978 0 0 0 7 41 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498531794400 0 0 -1 15 0 0 0
[pid=21068/tid=21095] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21095/stat : 21095 (claspmt) S 21066 21068 20866 0 -1 4202560 6661 0 0 0 22 47 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502289014240 0 0 -1 23 0 0 0
[pid=21068/tid=21096] ppid=21066 vsize=2132672 CPUtime=0.68 cores=0-31
/proc/21068/task/21096/stat : 21096 (claspmt) S 21066 21068 20866 0 -1 4202560 6905 0 0 0 18 50 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604499855097312 0 0 -1 7 0 0 0
[pid=21068/tid=21097] ppid=21066 vsize=2132672 CPUtime=0.49 cores=0-31
/proc/21068/task/21097/stat : 21097 (claspmt) S 21066 21068 20866 0 -1 4202560 5254 0 0 0 5 44 0 0 18 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604503085579744 0 0 -1 21 0 0 0
[pid=21068/tid=21098] ppid=21066 vsize=2132672 CPUtime=2.02 cores=0-31
/proc/21068/task/21098/stat : 21098 (claspmt) R 21066 21068 20866 0 -1 4202560 7461 0 0 0 152 50 0 0 19 0 32 0 16529708 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=21068/tid=21099] ppid=21066 vsize=2132672 CPUtime=0.67 cores=0-31
/proc/21068/task/21099/stat : 21099 (claspmt) S 21066 21068 20866 0 -1 4202560 7023 0 0 0 20 47 0 0 18 0 32 0 16529709 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604505758301664 0 0 -1 29 0 0 0
[pid=21068/tid=21100] ppid=21066 vsize=2132672 CPUtime=0.65 cores=0-31
/proc/21068/task/21100/stat : 21100 (claspmt) S 21066 21068 20866 0 -1 4202560 6919 0 0 0 13 52 0 0 18 0 32 0 16529709 2183856128 185306 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604504589614560 0 0 -1 12 0 0 0
Current children cumulated CPU time (s) 28.36
Current children cumulated vsize (KiB) 2132672

[startup+12.7014 s]
/proc/loadavg: 14.18 25.29 28.35 5/445 21100
/proc/meminfo: memFree=262212652/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=2132672 CPUtime=53.96 cores=0-31
/proc/21068/stat : 21068 (claspmt) S 21066 21068 20866 0 -1 4202496 240040 0 0 0 3996 1400 0 0 18 0 32 0 16529390 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 16 0 0 0
/proc/21068/statm: 533168 185888 440 189 0 527517 0
[pid=21068/tid=21070] ppid=21066 vsize=2132672 CPUtime=0.58 cores=0-31
/proc/21068/task/21070/stat : 21070 (claspmt) S 21066 21068 20866 0 -1 4202560 5130 0 0 0 8 50 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498707733984 0 0 -1 1 0 0 0
[pid=21068/tid=21071] ppid=21066 vsize=2132672 CPUtime=0.66 cores=0-31
/proc/21068/task/21071/stat : 21071 (claspmt) S 21066 21068 20866 0 -1 4202560 6908 0 0 0 19 47 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604505758277088 0 0 -1 6 0 0 0
[pid=21068/tid=21072] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21072/stat : 21072 (claspmt) S 21066 21068 20866 0 -1 4202560 6818 0 0 0 18 51 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498531786208 0 0 -1 12 0 0 0
[pid=21068/tid=21073] ppid=21066 vsize=2132672 CPUtime=0.5 cores=0-31
/proc/21068/task/21073/stat : 21073 (claspmt) S 21066 21068 20866 0 -1 4202560 5411 0 0 0 8 42 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475777504 0 0 -1 11 0 0 0
[pid=21068/tid=21074] ppid=21066 vsize=2132672 CPUtime=8.47 cores=0-31
/proc/21068/task/21074/stat : 21074 (claspmt) R 21066 21068 20866 0 -1 4202560 6746 0 0 0 798 49 0 0 25 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=21068/tid=21075] ppid=21066 vsize=2132672 CPUtime=0.59 cores=0-31
/proc/21068/task/21075/stat : 21075 (claspmt) S 21066 21068 20866 0 -1 4202560 6490 0 0 0 12 47 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475761120 0 0 -1 0 0 0 0
[pid=21068/tid=21076] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21076/stat : 21076 (claspmt) S 21066 21068 20866 0 -1 4202560 7443 0 0 0 19 50 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475769312 0 0 -1 18 0 0 0
[pid=21068/tid=21077] ppid=21066 vsize=2132672 CPUtime=0.49 cores=0-31
/proc/21068/task/21077/stat : 21077 (claspmt) S 21066 21068 20866 0 -1 4202560 5156 0 0 0 7 42 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604500654161376 0 0 -1 19 0 0 0
[pid=21068/tid=21078] ppid=21066 vsize=2132672 CPUtime=0.54 cores=0-31
/proc/21068/task/21078/stat : 21078 (claspmt) S 21066 21068 20866 0 -1 4202560 5934 0 0 0 9 45 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498707742176 0 0 -1 6 0 0 0
[pid=21068/tid=21079] ppid=21066 vsize=2132672 CPUtime=0.62 cores=0-31
/proc/21068/task/21079/stat : 21079 (claspmt) S 21066 21068 20866 0 -1 4202560 7173 0 0 0 13 49 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604499855105504 0 0 -1 20 0 0 0
[pid=21068/tid=21080] ppid=21066 vsize=2132672 CPUtime=0.63 cores=0-31
/proc/21068/task/21080/stat : 21080 (claspmt) S 21066 21068 20866 0 -1 4202560 6804 0 0 0 20 43 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604504589622752 0 0 -1 21 0 0 0
[pid=21068/tid=21081] ppid=21066 vsize=2132672 CPUtime=0.42 cores=0-31
/proc/21068/task/21081/stat : 21081 (claspmt) S 21066 21068 20866 0 -1 4202560 5084 0 0 0 4 38 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502438788576 0 0 -1 24 0 0 0
[pid=21068/tid=21082] ppid=21066 vsize=2132672 CPUtime=8.79 cores=0-31
/proc/21068/task/21082/stat : 21082 (claspmt) R 21066 21068 20866 0 -1 4202560 5787 0 0 0 834 45 0 0 25 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 4678045 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21068/tid=21083] ppid=21066 vsize=2132672 CPUtime=0.71 cores=0-31
/proc/21068/task/21083/stat : 21083 (claspmt) S 21066 21068 20866 0 -1 4202560 7002 0 0 0 19 52 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498531810784 0 0 -1 26 0 0 0
[pid=21068/tid=21084] ppid=21066 vsize=2132672 CPUtime=0.69 cores=0-31
/proc/21068/task/21084/stat : 21084 (claspmt) S 21066 21068 20866 0 -1 4202560 7034 0 0 0 22 47 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502288997856 0 0 -1 14 0 0 0
[pid=21068/tid=21085] ppid=21066 vsize=2132672 CPUtime=0.48 cores=0-31
/proc/21068/task/21085/stat : 21085 (claspmt) S 21066 21068 20866 0 -1 4202560 4788 0 0 0 8 40 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502289006048 0 0 -1 22 0 0 0
[pid=21068/tid=21086] ppid=21066 vsize=2132672 CPUtime=0.51 cores=0-31
/proc/21068/task/21086/stat : 21086 (claspmt) S 21066 21068 20866 0 -1 4202560 5575 0 0 0 5 46 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604500654153184 0 0 -1 16 0 0 0
[pid=21068/tid=21087] ppid=21066 vsize=2132672 CPUtime=0.12 cores=0-31
/proc/21068/task/21087/stat : 21087 (claspmt) S 21066 21068 20866 0 -1 4202560 1715 0 0 0 12 0 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604572833545696 0 0 -1 16 0 0 0
[pid=21068/tid=21088] ppid=21066 vsize=2132672 CPUtime=0.58 cores=0-31
/proc/21068/task/21088/stat : 21088 (claspmt) S 21066 21068 20866 0 -1 4202560 6138 0 0 0 12 46 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604502438804960 0 0 -1 4 0 0 0
[pid=21068/tid=21089] ppid=21066 vsize=2132672 CPUtime=0.5 cores=0-31
/proc/21068/task/21089/stat : 21089 (claspmt) S 21066 21068 20866 0 -1 4202560 5071 0 0 0 7 43 0 0 18 0 32 0 16529708 2183856128 185888 270582939648 4194304 4964800 140735826949248 18446744073709551615 258662772457 0 0 4096 18946 18446604498475752928 0 0 -1 27 0 0 0
[pid=21068/tid=21090] ppid=21066 vsize=2132672 CPUtime=9.29 cores=0-31

################
# More data... #
################

/proc/21068/task/21081/stat : 21081 (claspmt) R 21066 21068 20866 0 -1 4202560 222827 0 0 0 486034 512 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=21068/tid=21082] ppid=21066 vsize=29223064 CPUtime=5077.4 cores=0-31
/proc/21068/task/21082/stat : 21082 (claspmt) R 21066 21068 20866 0 -1 4202560 6429 0 0 0 507693 47 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4399339 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21068/tid=21083] ppid=21066 vsize=29223064 CPUtime=4853.41 cores=0-31
/proc/21068/task/21083/stat : 21083 (claspmt) R 21066 21068 20866 0 -1 4202560 177292 0 0 0 484944 397 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=21068/tid=21084] ppid=21066 vsize=29223064 CPUtime=4871.45 cores=0-31
/proc/21068/task/21084/stat : 21084 (claspmt) R 21066 21068 20866 0 -1 4202560 327311 0 0 0 486396 749 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670640 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21068/tid=21085] ppid=21066 vsize=29223064 CPUtime=4871.44 cores=0-31
/proc/21068/task/21085/stat : 21085 (claspmt) R 21066 21068 20866 0 -1 4202560 78010 0 0 0 486871 273 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21068/tid=21086] ppid=21066 vsize=29223064 CPUtime=4872.28 cores=0-31
/proc/21068/task/21086/stat : 21086 (claspmt) R 21066 21068 20866 0 -1 4202560 84589 0 0 0 486940 288 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=21068/tid=21087] ppid=21066 vsize=29223064 CPUtime=4867.39 cores=0-31
/proc/21068/task/21087/stat : 21087 (claspmt) R 21066 21068 20866 0 -1 4202560 721818 0 0 0 485783 956 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=21068/tid=21088] ppid=21066 vsize=29223064 CPUtime=4871.95 cores=0-31
/proc/21068/task/21088/stat : 21088 (claspmt) R 21066 21068 20866 0 -1 4202560 188449 0 0 0 486757 438 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21068/tid=21089] ppid=21066 vsize=29223064 CPUtime=4870.67 cores=0-31
/proc/21068/task/21089/stat : 21089 (claspmt) R 21066 21068 20866 0 -1 4202560 156900 0 0 0 486665 402 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=21068/tid=21090] ppid=21066 vsize=29223064 CPUtime=5077.93 cores=0-31
/proc/21068/task/21090/stat : 21090 (claspmt) R 21066 21068 20866 0 -1 4202560 3866 0 0 0 507756 37 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=21068/tid=21091] ppid=21066 vsize=29223064 CPUtime=4862.35 cores=0-31
/proc/21068/task/21091/stat : 21091 (claspmt) R 21066 21068 20866 0 -1 4202560 204476 0 0 0 485748 487 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21068/tid=21092] ppid=21066 vsize=29223064 CPUtime=4867.11 cores=0-31
/proc/21068/task/21092/stat : 21092 (claspmt) R 21066 21068 20866 0 -1 4202560 361636 0 0 0 485914 797 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=21068/tid=21093] ppid=21066 vsize=29223064 CPUtime=4859.64 cores=0-31
/proc/21068/task/21093/stat : 21093 (claspmt) R 21066 21068 20866 0 -1 4202560 100749 0 0 0 485622 342 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=21068/tid=21094] ppid=21066 vsize=29223064 CPUtime=4858.14 cores=0-31
/proc/21068/task/21094/stat : 21094 (claspmt) R 21066 21068 20866 0 -1 4202560 84978 0 0 0 485529 285 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=21068/tid=21095] ppid=21066 vsize=29223064 CPUtime=4869.03 cores=0-31
/proc/21068/task/21095/stat : 21095 (claspmt) R 21066 21068 20866 0 -1 4202560 583764 0 0 0 486130 773 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4363491 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21068/tid=21096] ppid=21066 vsize=29223064 CPUtime=4871.65 cores=0-31
/proc/21068/task/21096/stat : 21096 (claspmt) R 21066 21068 20866 0 -1 4202560 212906 0 0 0 486691 474 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=21068/tid=21097] ppid=21066 vsize=29223064 CPUtime=4871.9 cores=0-31
/proc/21068/task/21097/stat : 21097 (claspmt) R 21066 21068 20866 0 -1 4202560 173380 0 0 0 486787 403 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=21068/tid=21098] ppid=21066 vsize=29223064 CPUtime=5075.4 cores=0-31
/proc/21068/task/21098/stat : 21098 (claspmt) R 21066 21068 20866 0 -1 4202560 8304 0 0 0 507489 51 0 0 25 0 32 0 16529708 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21068/tid=21099] ppid=21066 vsize=29223064 CPUtime=4871.26 cores=0-31
/proc/21068/task/21099/stat : 21099 (claspmt) R 21066 21068 20866 0 -1 4202560 229131 0 0 0 486605 521 0 0 25 0 32 0 16529709 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=21068/tid=21100] ppid=21066 vsize=29223064 CPUtime=4869.86 cores=0-31
/proc/21068/task/21100/stat : 21100 (claspmt) R 21066 21068 20866 0 -1 4202560 340964 0 0 0 486255 731 0 0 25 0 32 0 16529709 29924417536 7027342 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 28 0 0 0
Current children cumulated CPU time (s) 156585
Current children cumulated vsize (KiB) 29223064



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 32.06 32.06 32.00 33/445 21260
/proc/meminfo: memFree=234797928/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=29223064 CPUtime=157151 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 7091183 0 0 0 15700771 14376 0 0 25 0 32 0 16529390 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669509 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/21068/statm: 7305766 7027349 463 189 0 7300115 0
[pid=21068/tid=21070] ppid=21066 vsize=29223064 CPUtime=4860.4 cores=0-31
/proc/21068/task/21070/stat : 21070 (claspmt) R 21066 21068 20866 0 -1 4202560 94207 0 0 0 485719 321 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4668614 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=21068/tid=21071] ppid=21066 vsize=29223064 CPUtime=4889.33 cores=0-31
/proc/21068/task/21071/stat : 21071 (claspmt) R 21066 21068 20866 0 -1 4202560 873340 0 0 0 487780 1153 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4363459 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21068/tid=21072] ppid=21066 vsize=29223064 CPUtime=4888.64 cores=0-31
/proc/21068/task/21072/stat : 21072 (claspmt) R 21066 21068 20866 0 -1 4202560 170505 0 0 0 488487 377 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=21068/tid=21073] ppid=21066 vsize=29223064 CPUtime=4889.83 cores=0-31
/proc/21068/task/21073/stat : 21073 (claspmt) R 21066 21068 20866 0 -1 4202560 206260 0 0 0 488509 474 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=21068/tid=21074] ppid=21066 vsize=29223064 CPUtime=5093.16 cores=0-31
/proc/21068/task/21074/stat : 21074 (claspmt) R 21066 21068 20866 0 -1 4202560 7411 0 0 0 509266 50 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=21068/tid=21075] ppid=21066 vsize=29223064 CPUtime=4888.47 cores=0-31
/proc/21068/task/21075/stat : 21075 (claspmt) R 21066 21068 20866 0 -1 4202560 193822 0 0 0 488418 429 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=21068/tid=21076] ppid=21066 vsize=29223064 CPUtime=4889.93 cores=0-31
/proc/21068/task/21076/stat : 21076 (claspmt) R 21066 21068 20866 0 -1 4202560 246219 0 0 0 488404 589 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669280 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=21068/tid=21077] ppid=21066 vsize=29223064 CPUtime=4889.99 cores=0-31
/proc/21068/task/21077/stat : 21077 (claspmt) R 21066 21068 20866 0 -1 4202560 96102 0 0 0 488687 312 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=21068/tid=21078] ppid=21066 vsize=29223064 CPUtime=4889.96 cores=0-31
/proc/21068/task/21078/stat : 21078 (claspmt) R 21066 21068 20866 0 -1 4202560 68896 0 0 0 488752 244 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669386 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=21068/tid=21079] ppid=21066 vsize=29223064 CPUtime=4861.62 cores=0-31
/proc/21068/task/21079/stat : 21079 (claspmt) R 21066 21068 20866 0 -1 4202560 553769 0 0 0 485399 763 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=21068/tid=21080] ppid=21066 vsize=29223064 CPUtime=4886.96 cores=0-31
/proc/21068/task/21080/stat : 21080 (claspmt) R 21066 21068 20866 0 -1 4202560 165833 0 0 0 488290 406 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=21068/tid=21081] ppid=21066 vsize=29223064 CPUtime=4883.18 cores=0-31
/proc/21068/task/21081/stat : 21081 (claspmt) R 21066 21068 20866 0 -1 4202560 222827 0 0 0 487806 512 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=21068/tid=21082] ppid=21066 vsize=29223064 CPUtime=5095.12 cores=0-31
/proc/21068/task/21082/stat : 21082 (claspmt) R 21066 21068 20866 0 -1 4202560 6429 0 0 0 509465 47 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669445 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21068/tid=21083] ppid=21066 vsize=29223064 CPUtime=4871.13 cores=0-31
/proc/21068/task/21083/stat : 21083 (claspmt) R 21066 21068 20866 0 -1 4202560 177292 0 0 0 486716 397 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669512 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=21068/tid=21084] ppid=21066 vsize=29223064 CPUtime=4889.17 cores=0-31
/proc/21068/task/21084/stat : 21084 (claspmt) R 21066 21068 20866 0 -1 4202560 327311 0 0 0 488168 749 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21068/tid=21085] ppid=21066 vsize=29223064 CPUtime=4889.16 cores=0-31
/proc/21068/task/21085/stat : 21085 (claspmt) R 21066 21068 20866 0 -1 4202560 78010 0 0 0 488643 273 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21068/tid=21086] ppid=21066 vsize=29223064 CPUtime=4890 cores=0-31
/proc/21068/task/21086/stat : 21086 (claspmt) R 21066 21068 20866 0 -1 4202560 84590 0 0 0 488712 288 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=21068/tid=21087] ppid=21066 vsize=29223064 CPUtime=4885.11 cores=0-31
/proc/21068/task/21087/stat : 21087 (claspmt) R 21066 21068 20866 0 -1 4202560 721824 0 0 0 487555 956 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=21068/tid=21088] ppid=21066 vsize=29223064 CPUtime=4889.66 cores=0-31
/proc/21068/task/21088/stat : 21088 (claspmt) R 21066 21068 20866 0 -1 4202560 188449 0 0 0 488528 438 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669476 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21068/tid=21089] ppid=21066 vsize=29223064 CPUtime=4888.39 cores=0-31
/proc/21068/task/21089/stat : 21089 (claspmt) R 21066 21068 20866 0 -1 4202560 156900 0 0 0 488437 402 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=21068/tid=21090] ppid=21066 vsize=29223064 CPUtime=5095.65 cores=0-31
/proc/21068/task/21090/stat : 21090 (claspmt) R 21066 21068 20866 0 -1 4202560 3866 0 0 0 509528 37 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=21068/tid=21091] ppid=21066 vsize=29223064 CPUtime=4880.07 cores=0-31
/proc/21068/task/21091/stat : 21091 (claspmt) R 21066 21068 20866 0 -1 4202560 204476 0 0 0 487520 487 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21068/tid=21092] ppid=21066 vsize=29223064 CPUtime=4884.83 cores=0-31
/proc/21068/task/21092/stat : 21092 (claspmt) R 21066 21068 20866 0 -1 4202560 361636 0 0 0 487686 797 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=21068/tid=21093] ppid=21066 vsize=29223064 CPUtime=4877.36 cores=0-31
/proc/21068/task/21093/stat : 21093 (claspmt) R 21066 21068 20866 0 -1 4202560 100749 0 0 0 487394 342 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=21068/tid=21094] ppid=21066 vsize=29223064 CPUtime=4875.86 cores=0-31
/proc/21068/task/21094/stat : 21094 (claspmt) R 21066 21068 20866 0 -1 4202560 84978 0 0 0 487301 285 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=21068/tid=21095] ppid=21066 vsize=29223064 CPUtime=4886.75 cores=0-31
/proc/21068/task/21095/stat : 21095 (claspmt) R 21066 21068 20866 0 -1 4202560 583764 0 0 0 487902 773 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4361293 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21068/tid=21096] ppid=21066 vsize=29223064 CPUtime=4889.36 cores=0-31
/proc/21068/task/21096/stat : 21096 (claspmt) R 21066 21068 20866 0 -1 4202560 212906 0 0 0 488462 474 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=21068/tid=21097] ppid=21066 vsize=29223064 CPUtime=4889.62 cores=0-31
/proc/21068/task/21097/stat : 21097 (claspmt) R 21066 21068 20866 0 -1 4202560 173380 0 0 0 488559 403 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=21068/tid=21098] ppid=21066 vsize=29223064 CPUtime=5093.12 cores=0-31
/proc/21068/task/21098/stat : 21098 (claspmt) R 21066 21068 20866 0 -1 4202560 8304 0 0 0 509261 51 0 0 25 0 32 0 16529708 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21068/tid=21099] ppid=21066 vsize=29223064 CPUtime=4888.98 cores=0-31
/proc/21068/task/21099/stat : 21099 (claspmt) R 21066 21068 20866 0 -1 4202560 229131 0 0 0 488377 521 0 0 25 0 32 0 16529709 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=21068/tid=21100] ppid=21066 vsize=29223064 CPUtime=4887.58 cores=0-31
/proc/21068/task/21100/stat : 21100 (claspmt) R 21066 21068 20866 0 -1 4202560 340964 0 0 0 488027 731 0 0 25 0 32 0 16529709 29924417536 7027349 270582939648 4194304 4964800 140735826949248 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 28 0 0 0
Current children cumulated CPU time (s) 157151
Current children cumulated vsize (KiB) 29223064

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -21068

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 32.06 32.06 32.00 2/415 21261
/proc/meminfo: memFree=234797936/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=29132864 CPUtime=157153 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4202496 7091194 0 0 0 15700875 14377 0 0 25 0 1 0 16529390 29832052736 7027316 270582939648 4194304 4964800 140735826949248 18446744073709551615 4627081 0 0 4096 18946 0 0 0 17 24 0 0 0
/proc/21068/statm: 7283216 7027316 474 189 0 7277565 0
Current children cumulated CPU time (s) 157153
Current children cumulated vsize (KiB) 29132864

[startup+5102.2 s]
/proc/loadavg: 32.06 32.06 32.00 2/415 21261
/proc/meminfo: memFree=234798168/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=0 CPUtime=157154 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4203524 7091194 0 0 0 15701014 14394 0 0 25 0 1 0 16529390 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/21068/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 157154
Current children cumulated vsize (KiB) 0

[startup+5103 s]
/proc/loadavg: 29.57 31.54 31.84 2/414 21261
/proc/meminfo: memFree=249566600/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=0 CPUtime=157155 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4203524 7091194 0 0 0 15701014 14474 0 0 25 0 1 0 16529390 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/21068/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 157155
Current children cumulated vsize (KiB) 0

[startup+5103.2 s]
/proc/loadavg: 29.57 31.54 31.84 2/414 21261
/proc/meminfo: memFree=249566600/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=0 CPUtime=157155 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4203524 7091194 0 0 0 15701014 14494 0 0 25 0 1 0 16529390 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/21068/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 157155
Current children cumulated vsize (KiB) 0

[startup+5103.3 s]
/proc/loadavg: 29.57 31.54 31.84 2/414 21261
/proc/meminfo: memFree=249566600/264257624 swapFree=67111528/67111528
[pid=21068] ppid=21066 vsize=0 CPUtime=157155 cores=0-31
/proc/21068/stat : 21068 (claspmt) R 21066 21068 20866 0 -1 4203524 7091194 0 0 0 15701014 14504 0 0 25 0 1 0 16529390 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/21068/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 157155
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5103.38
CPU time (s): 157155
CPU user time (s): 157010
CPU system time (s): 145.118
CPU usage (%): 3079.44
Max. virtual memory (cumulated for all children) (KiB): 29223064

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 157010
system time used= 145.118
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 7091194
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 438791
involuntary context switches= 375647

runsolver used 29.0246 second user time and 114.386 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-08 07:37:05
IDJOB=3472359
IDBENCH=82363
IDSOLVER=1961
FILE ID=node200/3472359-1307511423
RUNJOBID= node200-1307511318-20880
PBS_JOBID= 13509206
Free space on /tmp= 61700 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.070-NOTKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472359-1307511423/watcher-3472359-1307511423 -o /tmp/evaluation-result-3472359-1307511423/solver-3472359-1307511423 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472359-1307511423.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= f52ec4dacdcf65c3aedb61fb46ade01b
RANDOM SEED=861009266

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      262959960 kB
Buffers:        164796 kB
Cached:         866724 kB
SwapCached:          0 kB
Active:         236972 kB
Inactive:       862588 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      262959960 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           44844 kB
Writeback:           0 kB
AnonPages:       68028 kB
Mapped:          14532 kB
Slab:           121512 kB
PageTables:       4456 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   176400 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61700 MiB
End job on node200 at 2011-06-08 09:02:09