Trace number 3371029

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.51 1202.68

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_60_20_50_3-dir.shuffled-as.sat03-114.cnf
MD5SUM9ffa0325c9e00b41c025f41d20ec5a0c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark4252.27
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses49070
Sum of the clauses size98700
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 249050
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 520

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3371029-1305429968/watcher-3371029-1305429968 -o /tmp/evaluation-result-3371029-1305429968/solver-3371029-1305429968 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3371029-1305429968.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.80 7.70 6.32 2/272 2402
/proc/meminfo: memFree=20971016/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=9316 CPUtime=0 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) R 2400 2402 27712 0 -1 4202496 422 1255 0 0 0 0 0 0 24 0 1 0 654596541 9539584 351 33554432000 4194304 4417452 140734342002256 18446744073709551615 237278879488 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/2402/statm: 2329 351 147 55 0 215 0

[startup+0.0307219 s]
/proc/loadavg: 6.80 7.70 6.32 2/272 2402
/proc/meminfo: memFree=20971016/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=13268 CPUtime=0.02 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) R 2400 2402 27712 0 -1 4202496 1451 1255 0 0 2 0 0 0 24 0 1 0 654596541 13586432 1340 33554432000 4194304 4417452 140734342002256 18446744073709551615 4235981 0 0 4096 17442 0 0 0 17 1 0 0 0
/proc/2402/statm: 3317 1340 148 55 0 1203 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 13268

[startup+0.100712 s]
/proc/loadavg: 6.80 7.70 6.32 2/272 2402
/proc/meminfo: memFree=20971016/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=57896 CPUtime=0.28 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 3519 1255 0 0 28 0 0 0 24 0 5 0 654596541 59285504 2130 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 14474 2130 176 55 0 12360 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 57896

[startup+0.300675 s]
/proc/loadavg: 6.80 7.70 6.32 2/272 2402
/proc/meminfo: memFree=20971016/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=60292 CPUtime=1.08 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 4547 1255 0 0 107 1 0 0 24 0 5 0 654596541 61739008 2859 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 15073 2859 179 55 0 12959 0
Current children cumulated CPU time (s) 1.08
Current children cumulated vsize (KiB) 60292

[startup+0.700603 s]
/proc/loadavg: 6.80 7.70 6.32 2/272 2402
/proc/meminfo: memFree=20971016/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=62748 CPUtime=2.68 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 5514 1255 0 0 267 1 0 0 24 0 5 0 654596541 64253952 3500 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 15687 3500 179 55 0 13573 0
Current children cumulated CPU time (s) 2.68
Current children cumulated vsize (KiB) 62748

[startup+1.50046 s]
/proc/loadavg: 6.80 7.70 6.32 10/281 2416
/proc/meminfo: memFree=20931180/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=70256 CPUtime=5.87 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 8370 1255 0 0 586 1 0 0 24 0 5 0 654596541 71942144 5225 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 17564 5225 179 55 0 15450 0
[pid=2402/tid=2409] ppid=2400 vsize=70256 CPUtime=1.46 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 645 1255 0 0 146 0 0 0 25 0 5 0 654596545 71942144 5225 33554432000 4194304 4417452 140734342002256 18446744073709551615 4337184 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=70256 CPUtime=1.45 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 501 1255 0 0 145 0 0 0 25 0 5 0 654596545 71942144 5225 33554432000 4194304 4417452 140734342002256 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=70256 CPUtime=1.46 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 1220 1255 0 0 146 0 0 0 25 0 5 0 654596545 71942144 5225 33554432000 4194304 4417452 140734342002256 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=70256 CPUtime=1.46 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 4541 1255 0 0 145 1 0 0 25 0 5 0 654596545 71942144 5225 33554432000 4194304 4417452 140734342002256 18446744073709551615 4356867 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5.87
Current children cumulated vsize (KiB) 70256

[startup+3.10118 s]
/proc/loadavg: 6.80 7.70 6.32 10/281 2416
/proc/meminfo: memFree=20922004/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=75232 CPUtime=12.26 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 12874 1255 0 0 1224 2 0 0 24 0 5 0 654596541 77037568 6594 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 18808 6594 179 55 0 16694 0
[pid=2402/tid=2409] ppid=2400 vsize=75232 CPUtime=3.06 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 956 1255 0 0 306 0 0 0 25 0 5 0 654596545 77037568 6594 33554432000 4194304 4417452 140734342002256 18446744073709551615 4236319 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=75232 CPUtime=3.04 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 501 1255 0 0 304 0 0 0 25 0 5 0 654596545 77037568 6594 33554432000 4194304 4417452 140734342002256 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=75232 CPUtime=3.06 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 2084 1255 0 0 306 0 0 0 25 0 5 0 654596545 77037568 6594 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=75232 CPUtime=3.05 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 7870 1255 0 0 304 1 0 0 25 0 5 0 654596545 77037568 6594 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364896 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.26
Current children cumulated vsize (KiB) 75232

[startup+6.30061 s]
/proc/loadavg: 6.90 7.70 6.33 10/281 2416
/proc/meminfo: memFree=20894784/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=86156 CPUtime=25.04 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 20833 1255 0 0 2500 4 0 0 24 0 5 0 654596541 88223744 9325 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 21539 9325 179 55 0 19425 0
[pid=2402/tid=2409] ppid=2400 vsize=86156 CPUtime=6.26 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 1336 1255 0 0 626 0 0 0 25 0 5 0 654596545 88223744 9325 33554432000 4194304 4417452 140734342002256 18446744073709551615 4236722 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=86156 CPUtime=6.21 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 757 1255 0 0 621 0 0 0 25 0 5 0 654596545 88223744 9325 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=86156 CPUtime=6.25 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 3777 1255 0 0 625 0 0 0 25 0 5 0 654596545 88223744 9325 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=86156 CPUtime=6.25 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 13500 1255 0 0 623 2 0 0 25 0 5 0 654596545 88223744 9325 33554432000 4194304 4417452 140734342002256 18446744073709551615 4250623 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.04
Current children cumulated vsize (KiB) 86156

[startup+12.7005 s]
/proc/loadavg: 6.99 7.71 6.33 10/281 2416
/proc/meminfo: memFree=20824216/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=119416 CPUtime=50.58 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 37706 1255 0 0 5052 6 0 0 24 0 5 0 654596541 122281984 17640 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 29854 17640 179 55 0 27740 0
[pid=2402/tid=2409] ppid=2400 vsize=119416 CPUtime=12.66 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 4543 1255 0 0 1266 0 0 0 25 0 5 0 654596545 122281984 17640 33554432000 4194304 4417452 140734342002256 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=119416 CPUtime=12.56 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 982 1255 0 0 1256 0 0 0 25 0 5 0 654596545 122281984 17640 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=119416 CPUtime=12.66 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 6935 1255 0 0 1265 1 0 0 25 0 5 0 654596545 122281984 17640 33554432000 4194304 4417452 140734342002256 18446744073709551615 4352076 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=119416 CPUtime=12.66 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 23783 1255 0 0 1262 4 0 0 25 0 5 0 654596545 122281984 17640 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.58
Current children cumulated vsize (KiB) 119416

[startup+25.5012 s]
/proc/loadavg: 7.21 7.72 6.36 10/281 2417
/proc/meminfo: memFree=20677048/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=205668 CPUtime=101.67 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 77391 1255 0 0 10155 12 0 0 24 0 5 0 654596541 210604032 39204 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 51417 39204 180 55 0 49303 0
[pid=2402/tid=2409] ppid=2400 vsize=205668 CPUtime=25.46 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 12082 1255 0 0 2545 1 0 0 25 0 5 0 654596545 210604032 39204 33554432000 4194304 4417452 140734342002256 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=205668 CPUtime=25.24 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 7053 1255 0 0 2524 0 0 0 25 0 5 0 654596545 210604032 39204 33554432000 4194304 4417452 140734342002256 18446744073709551615 4305920 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=205668 CPUtime=25.45 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 17411 1255 0 0 2543 2 0 0 25 0 5 0 654596545 210604032 39204 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=205668 CPUtime=25.46 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 39382 1255 0 0 2539 7 0 0 25 0 5 0 654596545 210604032 39204 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.67
Current children cumulated vsize (KiB) 205668

[startup+51.1006 s]
/proc/loadavg: 7.63 7.78 6.41 10/281 2417
/proc/meminfo: memFree=20560084/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=244164 CPUtime=203.87 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 150408 1255 0 0 20363 24 0 0 24 0 5 0 654596541 250023936 48828 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 61041 48828 180 55 0 58927 0
[pid=2402/tid=2409] ppid=2400 vsize=244164 CPUtime=51.06 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 35477 1255 0 0 5102 4 0 0 25 0 5 0 654596545 250023936 48828 33554432000 4194304 4417452 140734342002256 18446744073709551615 4321362 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=244164 CPUtime=50.64 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 22211 1255 0 0 5061 3 0 0 25 0 5 0 654596545 250023936 48828 33554432000 4194304 4417452 140734342002256 18446744073709551615 4309503 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=244164 CPUtime=51.06 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 31108 1255 0 0 5101 5 0 0 25 0 5 0 654596545 250023936 48828 33554432000 4194304 4417452 140734342002256 18446744073709551615 4255584 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=244164 CPUtime=51.05 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 60149 1255 0 0 5095 10 0 0 25 0 5 0 654596545 250023936 48828 33554432000 4194304 4417452 140734342002256 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 203.87
Current children cumulated vsize (KiB) 244164

[startup+102.311 s]
/proc/loadavg: 7.84 7.81 6.49 10/281 2419
/proc/meminfo: memFree=20348772/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=370708 CPUtime=408.29 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 269366 1255 0 0 40784 45 0 0 24 0 5 0 654596541 379604992 80464 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 92677 80464 180 55 0 90563 0
[pid=2402/tid=2409] ppid=2400 vsize=370708 CPUtime=102.27 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 64185 1255 0 0 10218 9 0 0 25 0 5 0 654596545 379604992 80464 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=370708 CPUtime=101.42 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 54821 1255 0 0 10134 8 0 0 25 0 5 0 654596545 379604992 80464 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=370708 CPUtime=102.26 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 61385 1255 0 0 10215 11 0 0 25 0 5 0 654596545 379604992 80464 33554432000 4194304 4417452 140734342002256 18446744073709551615 4310188 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=370708 CPUtime=102.27 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 87512 1255 0 0 10212 15 0 0 25 0 5 0 654596545 379604992 80464 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 408.29
Current children cumulated vsize (KiB) 370708

[startup+162.301 s]
/proc/loadavg: 8.08 7.87 6.60 10/281 2420
/proc/meminfo: memFree=20144268/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=445792 CPUtime=647.76 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 390980 1255 0 0 64713 63 0 0 24 0 5 0 654596541 456491008 99235 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 111448 99235 180 55 0 109334 0
[pid=2402/tid=2409] ppid=2400 vsize=445792 CPUtime=162.26 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 96727 1255 0 0 16212 14 0 0 25 0 5 0 654596545 456491008 99235 33554432000 4194304 4417452 140734342002256 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=445792 CPUtime=160.92 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 83351 1255 0 0 16080 12 0 0 25 0 5 0 654596545 456491008 99235 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=445792 CPUtime=162.26 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 92568 1255 0 0 16210 16 0 0 25 0 5 0 654596545 456491008 99235 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=445792 CPUtime=162.26 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 116871 1255 0 0 16207 19 0 0 25 0 5 0 654596545 456491008 99235 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 647.76
Current children cumulated vsize (KiB) 445792


################
# More data... #
################

[pid=2402/tid=2410] ppid=2400 vsize=931028 CPUtime=696.44 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 239181 1255 0 0 69607 37 0 0 25 0 5 0 654596545 953372672 220544 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=931028 CPUtime=702.27 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 324898 1255 0 0 70173 54 0 0 25 0 5 0 654596545 953372672 220544 33554432000 4194304 4417452 140734342002256 18446744073709551615 4306345 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=931028 CPUtime=702.27 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 309216 1255 0 0 70178 49 0 0 25 0 5 0 654596545 953372672 220544 33554432000 4194304 4417452 140734342002256 18446744073709551615 4344138 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2803.3
Current children cumulated vsize (KiB) 931028

[startup+762.31 s]
/proc/loadavg: 8.10 8.07 7.30 10/281 2436
/proc/meminfo: memFree=18718200/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1184484 CPUtime=3042.84 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1391688 1255 0 0 304060 224 0 0 24 0 5 0 654596541 1212911616 283908 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 296121 283908 180 55 0 294007 0
[pid=2402/tid=2409] ppid=2400 vsize=1184484 CPUtime=762.26 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 442237 1255 0 0 76157 69 0 0 25 0 5 0 654596545 1212911616 283908 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1184484 CPUtime=755.96 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 268469 1255 0 0 75553 43 0 0 25 0 5 0 654596545 1212911616 283908 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1184484 CPUtime=762.28 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 353319 1255 0 0 76170 58 0 0 25 0 5 0 654596545 1212911616 283908 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1184484 CPUtime=762.28 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 326200 1255 0 0 76176 52 0 0 25 0 5 0 654596545 1212911616 283908 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3042.84
Current children cumulated vsize (KiB) 1184484

[startup+822.3 s]
/proc/loadavg: 8.07 8.07 7.34 10/281 2438
/proc/meminfo: memFree=18831636/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1022216 CPUtime=3282.3 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1449415 1255 0 0 327994 236 0 0 24 0 5 0 654596541 1046749184 243341 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 255554 243341 180 55 0 253440 0
[pid=2402/tid=2409] ppid=2400 vsize=1022216 CPUtime=822.26 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 450449 1255 0 0 82155 71 0 0 25 0 5 0 654596545 1046749184 243341 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1022216 CPUtime=815.43 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 271872 1255 0 0 81499 44 0 0 25 0 5 0 654596545 1046749184 243341 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1022216 CPUtime=822.28 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 388859 1255 0 0 82163 65 0 0 25 0 5 0 654596545 1046749184 243341 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1022216 CPUtime=822.27 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 336772 1255 0 0 82173 54 0 0 25 0 5 0 654596545 1046749184 243341 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3282.3
Current children cumulated vsize (KiB) 1022216

[startup+882.301 s]
/proc/loadavg: 8.17 8.10 7.40 10/281 2440
/proc/meminfo: memFree=19007332/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=952504 CPUtime=3521.8 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1480750 1255 0 0 351938 242 0 0 24 0 5 0 654596541 975364096 225913 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 238126 225913 180 55 0 236012 0
[pid=2402/tid=2409] ppid=2400 vsize=952504 CPUtime=882.25 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 453861 1255 0 0 88154 71 0 0 25 0 5 0 654596545 975364096 225913 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=952504 CPUtime=874.93 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 288185 1255 0 0 87446 47 0 0 25 0 5 0 654596545 975364096 225913 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=952504 CPUtime=882.28 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 391471 1255 0 0 88162 66 0 0 25 0 5 0 654596545 975364096 225913 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=952504 CPUtime=882.27 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 345770 1255 0 0 88172 55 0 0 25 0 5 0 654596545 975364096 225913 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3521.8
Current children cumulated vsize (KiB) 952504

[startup+942.301 s]
/proc/loadavg: 8.21 8.11 7.44 9/281 2441
/proc/meminfo: memFree=18845244/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1070380 CPUtime=3761.31 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1530699 1255 0 0 375882 249 0 0 24 0 5 0 654596541 1096069120 255382 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 267595 255382 180 55 0 265481 0
[pid=2402/tid=2409] ppid=2400 vsize=1070380 CPUtime=942.25 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 464781 1255 0 0 94152 73 0 0 25 0 5 0 654596545 1096069120 255382 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1070380 CPUtime=934.44 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 296377 1255 0 0 93396 48 0 0 25 0 5 0 654596545 1096069120 255382 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1070380 CPUtime=942.27 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 399476 1255 0 0 94160 67 0 0 25 0 5 0 654596545 1096069120 255382 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1070380 CPUtime=942.27 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 368602 1255 0 0 94169 58 0 0 25 0 5 0 654596545 1096069120 255382 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3761.31
Current children cumulated vsize (KiB) 1070380

[startup+1002.3 s]
/proc/loadavg: 8.14 8.10 7.48 10/281 2443
/proc/meminfo: memFree=18746036/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1179088 CPUtime=4000.81 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1590637 1255 0 0 399823 258 0 0 24 0 5 0 654596541 1207386112 282559 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 294772 282559 180 55 0 292658 0
[pid=2402/tid=2409] ppid=2400 vsize=1179088 CPUtime=1002.25 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 489278 1255 0 0 100148 77 0 0 25 0 5 0 654596545 1207386112 282559 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1179088 CPUtime=993.94 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 303248 1255 0 0 99345 49 0 0 25 0 5 0 654596545 1207386112 282559 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1179088 CPUtime=1002.28 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 411662 1255 0 0 100158 70 0 0 25 0 5 0 654596545 1207386112 282559 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1179088 CPUtime=1002.27 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 384986 1255 0 0 100167 60 0 0 25 0 5 0 654596545 1207386112 282559 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4000.81
Current children cumulated vsize (KiB) 1179088

[startup+1062.3 s]
/proc/loadavg: 8.12 8.10 7.51 10/281 2444
/proc/meminfo: memFree=18574532/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1155136 CPUtime=4240.34 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1601033 1255 0 0 423774 260 0 0 24 0 5 0 654596541 1182859264 276571 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 288784 276571 180 55 0 286670 0
[pid=2402/tid=2409] ppid=2400 vsize=1155136 CPUtime=1062.25 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 489278 1255 0 0 106148 77 0 0 25 0 5 0 654596545 1182859264 276571 33554432000 4194304 4417452 140734342002256 18446744073709551615 4346490 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1155136 CPUtime=1053.46 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 303248 1255 0 0 105297 49 0 0 25 0 5 0 654596545 1182859264 276571 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1155136 CPUtime=1062.28 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 413866 1255 0 0 106158 70 0 0 25 0 5 0 654596545 1182859264 276571 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1155136 CPUtime=1062.28 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 393178 1255 0 0 106166 62 0 0 25 0 5 0 654596545 1182859264 276571 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4240.34
Current children cumulated vsize (KiB) 1155136

[startup+1122.3 s]
/proc/loadavg: 8.04 8.08 7.54 10/281 2446
/proc/meminfo: memFree=18303824/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1195384 CPUtime=4479.84 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1667516 1255 0 0 447715 269 0 0 24 0 5 0 654596541 1224073216 286633 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 298846 286633 180 55 0 296732 0
[pid=2402/tid=2409] ppid=2400 vsize=1195384 CPUtime=1122.25 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 521061 1255 0 0 112143 82 0 0 25 0 5 0 654596545 1224073216 286633 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1195384 CPUtime=1112.96 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 303248 1255 0 0 111247 49 0 0 25 0 5 0 654596545 1224073216 286633 33554432000 4194304 4417452 140734342002256 18446744073709551615 4249583 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1195384 CPUtime=1122.29 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 440374 1255 0 0 112155 74 0 0 25 0 5 0 654596545 1224073216 286633 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1195384 CPUtime=1122.28 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 401370 1255 0 0 112166 62 0 0 25 0 5 0 654596545 1224073216 286633 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4479.84
Current children cumulated vsize (KiB) 1195384

[startup+1182.3 s]
/proc/loadavg: 8.21 8.11 7.58 10/281 2448
/proc/meminfo: memFree=18177204/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1375008 CPUtime=4719.36 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1746214 1255 0 0 471657 279 0 0 24 0 5 0 654596541 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 343752 331539 180 55 0 341638 0
[pid=2402/tid=2409] ppid=2400 vsize=1375008 CPUtime=1182.26 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 521061 1255 0 0 118144 82 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1375008 CPUtime=1172.48 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 338148 1255 0 0 117194 54 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1375008 CPUtime=1182.29 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 467788 1255 0 0 118152 77 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1375008 CPUtime=1182.28 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 417754 1255 0 0 118164 64 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4366400 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4719.36
Current children cumulated vsize (KiB) 1375008



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.6 s]
/proc/loadavg: 8.15 8.10 7.59 10/281 2448
/proc/meminfo: memFree=18144088/32950904 swapFree=67057400/67111528
[pid=2402] ppid=2400 vsize=1375008 CPUtime=4800.4 cores=1,3,5,7
/proc/2402/stat : 2402 (plingeling) S 2400 2402 27712 0 -1 4202496 1754406 1255 0 0 479759 281 0 0 24 0 5 0 654596541 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/2402/statm: 343752 331539 180 55 0 341638 0
[pid=2402/tid=2409] ppid=2400 vsize=1375008 CPUtime=1202.56 cores=1,3,5,7
/proc/2402/task/2409/stat : 2409 (plingeling) R 2400 2402 27712 0 -1 4202560 521061 1255 0 0 120174 82 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=2402/tid=2410] ppid=2400 vsize=1375008 CPUtime=1192.6 cores=1,3,5,7
/proc/2402/task/2410/stat : 2410 (plingeling) R 2400 2402 27712 0 -1 4202560 346340 1255 0 0 119204 56 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4278085 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2402/tid=2411] ppid=2400 vsize=1375008 CPUtime=1202.6 cores=1,3,5,7
/proc/2402/task/2411/stat : 2411 (plingeling) R 2400 2402 27712 0 -1 4202560 467788 1255 0 0 120183 77 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=2402/tid=2412] ppid=2400 vsize=1375008 CPUtime=1202.59 cores=1,3,5,7
/proc/2402/task/2412/stat : 2412 (plingeling) R 2400 2402 27712 0 -1 4202560 417754 1255 0 0 120195 64 0 0 25 0 5 0 654596545 1408008192 331539 33554432000 4194304 4417452 140734342002256 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.4
Current children cumulated vsize (KiB) 1375008

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.68
CPU time (s): 4800.51
CPU user time (s): 4797.63
CPU system time (s): 2.87456
CPU usage (%): 399.151
Max. virtual memory (cumulated for all children) (KiB): 1407776

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.63
system time used= 2.87456
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1755661
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 101
involuntary context switches= 35265

runsolver used 2.67559 second user time and 7.32689 second system time

The end

Launcher Data 

Begin job on node121 at 2011-05-15 05:26:08
IDJOB=3371029
IDBENCH=83988
IDSOLVER=1848
FILE ID=node121/3371029-1305429968
RUNJOBID= node121-1305429968-2377
PBS_JOBID= 13324576
Free space on /tmp= 73492 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_60_20_50_3-dir.shuffled-as.sat03-114.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3371029-1305429968/watcher-3371029-1305429968 -o /tmp/evaluation-result-3371029-1305429968/solver-3371029-1305429968 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3371029-1305429968.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9ffa0325c9e00b41c025f41d20ec5a0c
RANDOM SEED=1877196460

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      20972264 kB
Buffers:        304964 kB
Cached:        5951280 kB
SwapCached:       7656 kB
Active:        6397192 kB
Inactive:      5230472 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      20972264 kB
SwapTotal:    67111528 kB
SwapFree:     67057400 kB
Dirty:            1272 kB
Writeback:           0 kB
AnonPages:     5369468 kB
Mapped:          21688 kB
Slab:           264024 kB
PageTables:      24404 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS: 30665128 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73492 MiB
End job on node121 at 2011-05-15 05:46:13