Trace number 3309882

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 40000.2 5003.64

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
32/aes_32_5_keyfind_1.cnf
MD5SUM3f4ea4cf17e734bbe947db7c1dc9fde7
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark7602.69
Satisfiable
(Un)Satisfiability was proved
Number of variables1116
Number of clauses4312
Sum of the clauses size13688
Maximum clause length5
Minimum clause length1
Number of clauses of size 184
Number of clauses of size 2200
Number of clauses of size 33008
Number of clauses of size 4920
Number of clauses of size 5100
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3309882-1304013849/watcher-3309882-1304013849 -o /tmp/evaluation-result-3309882-1304013849/solver-3309882-1304013849 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3309882-1304013849.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.47 6.90 6.96 2/170 25504
/proc/meminfo: memFree=13755008/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=9988 CPUtime=0 cores=0-7
/proc/25504/stat : 25504 (plingeling) R 25502 25504 25032 0 -1 4202496 559 1251 0 0 0 0 0 0 25 0 1 0 512983640 10227712 487 33554432000 4194304 4417332 140736143676096 18446744073709551615 235594485279 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/25504/statm: 2497 487 147 55 0 383 0

[startup+0.011806 s]
/proc/loadavg: 6.47 6.90 6.96 2/170 25504
/proc/meminfo: memFree=13755008/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=10808 CPUtime=0 cores=0-7
/proc/25504/stat : 25504 (plingeling) R 25502 25504 25032 0 -1 4202496 784 1251 0 0 0 0 0 0 25 0 1 0 512983640 11067392 712 33554432000 4194304 4417332 140736143676096 18446744073709551615 4235968 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/25504/statm: 2702 712 147 55 0 588 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 10808

[startup+0.124787 s]
/proc/loadavg: 6.47 6.90 6.96 2/170 25504
/proc/meminfo: memFree=13755008/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=227068 CPUtime=0.87 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 1874 1251 0 0 87 0 0 0 25 0 9 0 512983640 232517632 1633 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 56767 1633 181 55 0 54653 0
Current children cumulated CPU time (s) 0.87
Current children cumulated vsize (KiB) 227068

[startup+0.300757 s]
/proc/loadavg: 6.47 6.90 6.96 2/170 25504
/proc/meminfo: memFree=13755008/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=230276 CPUtime=2.28 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 2993 1251 0 0 228 0 0 0 25 0 9 0 512983640 235802624 2523 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 57569 2523 181 55 0 55455 0
Current children cumulated CPU time (s) 2.28
Current children cumulated vsize (KiB) 230276

[startup+0.700687 s]
/proc/loadavg: 6.47 6.90 6.96 2/170 25504
/proc/meminfo: memFree=13755008/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=233764 CPUtime=5.48 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 5377 1251 0 0 548 0 0 0 25 0 9 0 512983640 239374336 3395 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 58441 3395 181 55 0 56327 0
Current children cumulated CPU time (s) 5.48
Current children cumulated vsize (KiB) 233764

[startup+1.50055 s]
/proc/loadavg: 6.47 6.90 6.96 9/179 25518
/proc/meminfo: memFree=13742172/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=235308 CPUtime=11.88 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 9625 1251 0 0 1187 1 0 0 25 0 9 0 512983640 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 58827 3781 181 55 0 56713 0
[pid=25504/tid=25511] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 946 1251 0 0 148 0 0 0 25 0 9 0 512983641 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4251045 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=235308 CPUtime=1.47 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 924 1251 0 0 147 0 0 0 25 0 9 0 512983641 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 1174 1251 0 0 148 0 0 0 25 0 9 0 512983641 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 1172 1251 0 0 148 0 0 0 20 0 9 0 512983641 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4236928 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 935 1251 0 0 148 0 0 0 25 0 9 0 512983641 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4264888 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 1315 1251 0 0 148 0 0 0 25 0 9 0 512983641 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4236613 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 1356 1251 0 0 148 0 0 0 25 0 9 0 512983642 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=235308 CPUtime=1.48 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 875 1251 0 0 148 0 0 0 25 0 9 0 512983642 240955392 3781 33554432000 4194304 4417332 140736143676096 18446744073709551615 4250858 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.88
Current children cumulated vsize (KiB) 235308

[startup+3.10027 s]
/proc/loadavg: 6.47 6.90 6.96 9/179 25518
/proc/meminfo: memFree=13736272/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=243056 CPUtime=24.67 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 19248 1251 0 0 2464 3 0 0 25 0 9 0 512983640 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 60764 5719 181 55 0 58650 0
[pid=25504/tid=25511] ppid=25502 vsize=243056 CPUtime=3.08 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 2293 1251 0 0 308 0 0 0 25 0 9 0 512983641 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364722 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=243056 CPUtime=3.07 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 1925 1251 0 0 307 0 0 0 25 0 9 0 512983641 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4250936 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=243056 CPUtime=3.07 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 2437 1251 0 0 307 0 0 0 25 0 9 0 512983641 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=243056 CPUtime=3.08 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 2165 1251 0 0 308 0 0 0 25 0 9 0 512983641 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4236961 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=243056 CPUtime=3.08 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 2104 1251 0 0 308 0 0 0 25 0 9 0 512983641 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4357623 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=243056 CPUtime=3.08 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 2650 1251 0 0 308 0 0 0 25 0 9 0 512983641 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4366038 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=243056 CPUtime=3.07 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 2381 1251 0 0 307 0 0 0 25 0 9 0 512983642 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4366475 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=243056 CPUtime=3.08 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 2365 1251 0 0 308 0 0 0 25 0 9 0 512983642 248889344 5719 33554432000 4194304 4417332 140736143676096 18446744073709551615 4357912 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.67
Current children cumulated vsize (KiB) 243056

[startup+6.30071 s]
/proc/loadavg: 6.67 6.94 6.97 10/180 25520
/proc/meminfo: memFree=13720268/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=248464 CPUtime=50.12 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 33547 1251 0 0 5006 6 0 0 25 0 9 0 512983640 254427136 7053 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 62116 7053 181 55 0 60002 0
[pid=25504/tid=25511] ppid=25502 vsize=248464 CPUtime=6.28 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 4107 1251 0 0 628 0 0 0 25 0 9 0 512983641 254427136 7053 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=248464 CPUtime=6.25 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 3744 1251 0 0 625 0 0 0 25 0 9 0 512983641 254427136 7053 33554432000 4194304 4417332 140736143676096 18446744073709551615 18446744073699065939 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=248464 CPUtime=6.28 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 4470 1251 0 0 627 1 0 0 25 0 9 0 512983641 254427136 7053 33554432000 4194304 4417332 140736143676096 18446744073709551615 4320808 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=248464 CPUtime=6.14 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 3550 1251 0 0 614 0 0 0 25 0 9 0 512983641 254427136 7053 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=248464 CPUtime=6.27 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 3923 1251 0 0 627 0 0 0 25 0 9 0 512983641 254427136 7053 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365018 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=248468 CPUtime=6.27 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 4184 1251 0 0 627 0 0 0 25 0 9 0 512983641 254431232 7054 33554432000 4194304 4417332 140736143676096 18446744073709551615 4319682 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=248468 CPUtime=6.27 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 4411 1251 0 0 627 0 0 0 25 0 9 0 512983642 254431232 7054 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364452 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=248468 CPUtime=6.27 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 4230 1251 0 0 627 0 0 0 25 0 9 0 512983642 254431232 7054 33554432000 4194304 4417332 140736143676096 18446744073709551615 4358169 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.12
Current children cumulated vsize (KiB) 248464

[startup+12.7006 s]
/proc/loadavg: 6.94 6.99 6.99 10/180 25520
/proc/meminfo: memFree=13699732/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=263168 CPUtime=101.04 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 60586 1251 0 0 10093 11 0 0 25 0 9 0 512983640 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 65792 10693 181 55 0 63678 0
[pid=25504/tid=25511] ppid=25502 vsize=263168 CPUtime=12.67 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 6920 1251 0 0 1267 0 0 0 25 0 9 0 512983641 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=263168 CPUtime=12.62 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 6336 1251 0 0 1261 1 0 0 25 0 9 0 512983641 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=263168 CPUtime=12.67 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 8416 1251 0 0 1266 1 0 0 25 0 9 0 512983641 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4236549 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=263168 CPUtime=12.3 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 6525 1251 0 0 1229 1 0 0 25 0 9 0 512983641 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=263168 CPUtime=12.68 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 7442 1251 0 0 1267 1 0 0 25 0 9 0 512983641 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364498 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=263168 CPUtime=12.67 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 8002 1251 0 0 1266 1 0 0 25 0 9 0 512983641 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4357293 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=263168 CPUtime=12.67 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 8401 1251 0 0 1266 1 0 0 25 0 9 0 512983642 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4236995 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=263168 CPUtime=12.67 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 7616 1251 0 0 1267 0 0 0 25 0 9 0 512983642 269484032 10693 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364671 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.04
Current children cumulated vsize (KiB) 263168

[startup+25.5003 s]
/proc/loadavg: 7.38 7.09 7.02 9/179 25520
/proc/meminfo: memFree=13703480/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=278036 CPUtime=202.96 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 105811 1251 0 0 20274 22 0 0 25 0 9 0 512983640 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 69509 14461 181 55 0 67395 0
[pid=25504/tid=25511] ppid=25502 vsize=278036 CPUtime=25.48 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 12639 1251 0 0 2546 2 0 0 25 0 9 0 512983641 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=278036 CPUtime=25.34 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 11923 1251 0 0 2532 2 0 0 25 0 9 0 512983641 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=278036 CPUtime=25.47 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 14191 1251 0 0 2545 2 0 0 25 0 9 0 512983641 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364452 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=278036 CPUtime=24.69 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 12684 1251 0 0 2466 3 0 0 25 0 9 0 512983641 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365900 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=278036 CPUtime=25.47 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 13086 1251 0 0 2545 2 0 0 25 0 9 0 512983641 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=278036 CPUtime=25.47 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 13756 1251 0 0 2545 2 0 0 25 0 9 0 512983641 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364680 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=278036 CPUtime=25.47 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 13774 1251 0 0 2545 2 0 0 25 0 9 0 512983642 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4250651 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=278036 CPUtime=25.47 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 12830 1251 0 0 2545 2 0 0 25 0 9 0 512983642 284708864 14461 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 202.96
Current children cumulated vsize (KiB) 278036

[startup+51.1009 s]
/proc/loadavg: 7.59 7.16 7.04 9/179 25521
/proc/meminfo: memFree=13688704/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=297148 CPUtime=407.63 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 188661 1251 0 0 40723 40 0 0 25 0 9 0 512983640 304279552 19233 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 74287 19233 181 55 0 72173 0
[pid=25504/tid=25511] ppid=25502 vsize=297148 CPUtime=51.08 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 23956 1251 0 0 5104 4 0 0 25 0 9 0 512983641 304279552 19233 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364590 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=297148 CPUtime=50.81 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 20467 1251 0 0 5077 4 0 0 25 0 9 0 512983641 304279552 19233 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364448 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=297148 CPUtime=51.07 cores=0-7

################
# More data... #
################

/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 530454 1251 0 0 465913 108 0 0 25 0 9 0 512983641 683073536 111793 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=667064 CPUtime=4662.38 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 519710 1251 0 0 466148 90 0 0 25 0 9 0 512983641 683073536 111793 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=667064 CPUtime=4662.37 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 644524 1251 0 0 466126 111 0 0 25 0 9 0 512983641 683073536 111793 33554432000 4194304 4417332 140736143676096 18446744073709551615 4366324 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=667064 CPUtime=4662.37 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 527334 1251 0 0 466143 94 0 0 25 0 9 0 512983642 683073536 111793 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=667064 CPUtime=4662.38 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 523185 1251 0 0 466148 90 0 0 25 0 9 0 512983642 683073536 111793 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37271.5
Current children cumulated vsize (KiB) 667064

[startup+4722.31 s]
/proc/loadavg: 8.00 8.03 8.00 9/179 25657
/proc/meminfo: memFree=13330128/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=653044 CPUtime=37751.2 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 4219649 1251 0 0 3774350 771 0 0 25 0 9 0 512983640 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 163261 108288 181 55 0 161147 0
[pid=25504/tid=25511] ppid=25502 vsize=653044 CPUtime=4722.38 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 416823 1251 0 0 472157 81 0 0 25 0 9 0 512983641 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=653044 CPUtime=4696.63 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 524851 1251 0 0 469566 97 0 0 25 0 9 0 512983641 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=653044 CPUtime=4722.38 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 506491 1251 0 0 472147 91 0 0 25 0 9 0 512983641 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=653044 CPUtime=4720.2 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 538835 1251 0 0 471911 109 0 0 25 0 9 0 512983641 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=653044 CPUtime=4722.39 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 523547 1251 0 0 472148 91 0 0 25 0 9 0 512983641 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=653044 CPUtime=4722.38 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 650504 1251 0 0 472126 112 0 0 25 0 9 0 512983641 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=653044 CPUtime=4722.38 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 530622 1251 0 0 472144 94 0 0 25 0 9 0 512983642 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=653044 CPUtime=4722.39 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 527048 1251 0 0 472148 91 0 0 25 0 9 0 512983642 668717056 108288 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37751.2
Current children cumulated vsize (KiB) 653044

[startup+4782.3 s]
/proc/loadavg: 8.07 8.04 8.00 9/179 25659
/proc/meminfo: memFree=13296380/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=683140 CPUtime=38230.9 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 4253640 1251 0 0 3822309 776 0 0 25 0 9 0 512983640 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 170785 115812 181 55 0 168671 0
[pid=25504/tid=25511] ppid=25502 vsize=683140 CPUtime=4782.38 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 420279 1251 0 0 478156 82 0 0 25 0 9 0 512983641 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=683140 CPUtime=4756.3 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 526407 1251 0 0 475533 97 0 0 25 0 9 0 512983641 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364409 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=683140 CPUtime=4782.37 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 511163 1251 0 0 478145 92 0 0 25 0 9 0 512983641 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=683140 CPUtime=4780.19 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 544587 1251 0 0 477909 110 0 0 25 0 9 0 512983641 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=683140 CPUtime=4782.39 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 527482 1251 0 0 478147 92 0 0 25 0 9 0 512983641 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364818 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=683140 CPUtime=4782.37 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 657520 1251 0 0 478124 113 0 0 25 0 9 0 512983641 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=683140 CPUtime=4782.38 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 534016 1251 0 0 478143 95 0 0 25 0 9 0 512983642 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=683140 CPUtime=4782.38 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 531258 1251 0 0 478147 91 0 0 25 0 9 0 512983642 699535360 115812 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38230.9
Current children cumulated vsize (KiB) 683140

[startup+4842.3 s]
/proc/loadavg: 8.08 8.04 8.00 9/179 25660
/proc/meminfo: memFree=13363692/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=615828 CPUtime=38710.5 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 4288496 1251 0 0 3870271 781 0 0 25 0 9 0 512983640 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 153957 98984 181 55 0 151843 0
[pid=25504/tid=25511] ppid=25502 vsize=615828 CPUtime=4842.38 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 422887 1251 0 0 484156 82 0 0 25 0 9 0 512983641 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4366038 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=615828 CPUtime=4815.97 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 532987 1251 0 0 481499 98 0 0 25 0 9 0 512983641 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364887 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=615828 CPUtime=4842.37 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 516574 1251 0 0 484144 93 0 0 25 0 9 0 512983641 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365883 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=615828 CPUtime=4840.18 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 546704 1251 0 0 483908 110 0 0 25 0 9 0 512983641 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=615828 CPUtime=4842.39 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 531358 1251 0 0 484147 92 0 0 25 0 9 0 512983641 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=615828 CPUtime=4842.38 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 666089 1251 0 0 484123 115 0 0 25 0 9 0 512983641 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=615828 CPUtime=4842.38 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 535816 1251 0 0 484143 95 0 0 25 0 9 0 512983642 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=615828 CPUtime=4842.38 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 535153 1251 0 0 484146 92 0 0 25 0 9 0 512983642 630607872 98984 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38710.5
Current children cumulated vsize (KiB) 615828

[startup+4902.3 s]
/proc/loadavg: 8.09 8.05 8.00 9/179 25662
/proc/meminfo: memFree=13312332/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=667180 CPUtime=39190.2 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 4328715 1251 0 0 3918235 786 0 0 25 0 9 0 512983640 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 166795 111822 181 55 0 164681 0
[pid=25504/tid=25511] ppid=25502 vsize=667180 CPUtime=4902.38 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 427497 1251 0 0 490155 83 0 0 25 0 9 0 512983641 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=667180 CPUtime=4875.64 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 538649 1251 0 0 487466 98 0 0 25 0 9 0 512983641 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364887 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=667180 CPUtime=4902.38 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 518745 1251 0 0 490144 94 0 0 25 0 9 0 512983641 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=667180 CPUtime=4900.19 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 549551 1251 0 0 489908 111 0 0 25 0 9 0 512983641 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4199435 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=667180 CPUtime=4902.4 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 539426 1251 0 0 490147 93 0 0 25 0 9 0 512983641 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=667180 CPUtime=4902.38 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 673198 1251 0 0 490122 116 0 0 25 0 9 0 512983641 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=667180 CPUtime=4902.38 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 539065 1251 0 0 490143 95 0 0 25 0 9 0 512983642 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=667180 CPUtime=4902.39 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 541656 1251 0 0 490146 93 0 0 25 0 9 0 512983642 683192320 111822 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364479 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39190.2
Current children cumulated vsize (KiB) 667180

[startup+4962.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/179 25664
/proc/meminfo: memFree=13266308/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=680332 CPUtime=39669.9 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 4365972 1251 0 0 3966196 792 0 0 25 0 9 0 512983640 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 170083 115110 181 55 0 167969 0
[pid=25504/tid=25511] ppid=25502 vsize=680332 CPUtime=4962.38 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 430123 1251 0 0 496154 84 0 0 25 0 9 0 512983641 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364531 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=680332 CPUtime=4935.32 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 546227 1251 0 0 493433 99 0 0 25 0 9 0 512983641 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4365963 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=680332 CPUtime=4962.38 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 524955 1251 0 0 496143 95 0 0 25 0 9 0 512983641 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=680332 CPUtime=4960.17 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 556577 1251 0 0 495906 111 0 0 25 0 9 0 512983641 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=680332 CPUtime=4962.39 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 543338 1251 0 0 496146 93 0 0 25 0 9 0 512983641 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364541 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=680332 CPUtime=4962.37 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 677754 1251 0 0 496121 116 0 0 25 0 9 0 512983641 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=680332 CPUtime=4962.38 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 540848 1251 0 0 496143 95 0 0 25 0 9 0 512983642 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=680332 CPUtime=4962.39 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 545222 1251 0 0 496145 94 0 0 25 0 9 0 512983642 696659968 115110 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39669.9
Current children cumulated vsize (KiB) 680332



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.6 s]
/proc/loadavg: 8.04 8.04 8.00 9/179 25665
/proc/meminfo: memFree=13322464/32950928 swapFree=67111528/67111528
[pid=25504] ppid=25502 vsize=657012 CPUtime=40000.1 cores=0-7
/proc/25504/stat : 25504 (plingeling) S 25502 25504 25032 0 -1 4202496 4390545 1251 0 0 3999214 796 0 0 25 0 9 0 512983640 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 235606669285 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/25504/statm: 164253 109280 181 55 0 162139 0
[pid=25504/tid=25511] ppid=25502 vsize=657012 CPUtime=5003.69 cores=0-7
/proc/25504/task/25511/stat : 25511 (plingeling) R 25502 25504 25032 0 -1 4202560 431257 1251 0 0 500285 84 0 0 25 0 9 0 512983641 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364796 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=25504/tid=25512] ppid=25502 vsize=657012 CPUtime=4976.4 cores=0-7
/proc/25504/task/25512/stat : 25512 (plingeling) R 25502 25504 25032 0 -1 4202560 547801 1251 0 0 497540 100 0 0 25 0 9 0 512983641 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=25504/tid=25513] ppid=25502 vsize=657012 CPUtime=5003.69 cores=0-7
/proc/25504/task/25513/stat : 25513 (plingeling) R 25502 25504 25032 0 -1 4202560 526421 1251 0 0 500274 95 0 0 25 0 9 0 512983641 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=25504/tid=25514] ppid=25502 vsize=657012 CPUtime=5001.47 cores=0-7
/proc/25504/task/25514/stat : 25514 (plingeling) R 25502 25504 25032 0 -1 4202560 559665 1251 0 0 500035 112 0 0 25 0 9 0 512983641 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364556 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=25504/tid=25515] ppid=25502 vsize=657012 CPUtime=5003.7 cores=0-7
/proc/25504/task/25515/stat : 25515 (plingeling) R 25502 25504 25032 0 -1 4202560 546326 1251 0 0 500276 94 0 0 25 0 9 0 512983641 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=25504/tid=25516] ppid=25502 vsize=657012 CPUtime=5003.69 cores=0-7
/proc/25504/task/25516/stat : 25516 (plingeling) R 25502 25504 25032 0 -1 4202560 684907 1251 0 0 500251 118 0 0 25 0 9 0 512983641 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=25504/tid=25517] ppid=25502 vsize=657012 CPUtime=5003.69 cores=0-7
/proc/25504/task/25517/stat : 25517 (plingeling) R 25502 25504 25032 0 -1 4202560 544214 1251 0 0 500273 96 0 0 25 0 9 0 512983642 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364549 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=25504/tid=25518] ppid=25502 vsize=657012 CPUtime=5003.7 cores=0-7
/proc/25504/task/25518/stat : 25518 (plingeling) R 25502 25504 25032 0 -1 4202560 549026 1251 0 0 500275 95 0 0 25 0 9 0 512983642 672780288 109280 33554432000 4194304 4417332 140736143676096 18446744073709551615 4364656 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 657012

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5003.64
CPU time (s): 40000.2
CPU user time (s): 39992.2
CPU system time (s): 7.99378
CPU usage (%): 799.422
Max. virtual memory (cumulated for all children) (KiB): 728724

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.2
system time used= 7.99378
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4391796
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 292
involuntary context switches= 191749

runsolver used 8.23575 second user time and 19.0471 second system time

The end

Launcher Data 

Begin job on node120 at 2011-04-28 20:04:09
IDJOB=3309882
IDBENCH=83430
IDSOLVER=1695
FILE ID=node120/3309882-1304013849
RUNJOBID= node120-1304007130-25046
PBS_JOBID= 13168827
Free space on /tmp= 72076 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/32/aes_32_5_keyfind_1.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3309882-1304013849/watcher-3309882-1304013849 -o /tmp/evaluation-result-3309882-1304013849/solver-3309882-1304013849 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3309882-1304013849.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3f4ea4cf17e734bbe947db7c1dc9fde7
RANDOM SEED=1540574383

node120.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13755288 kB
Buffers:       2478312 kB
Cached:       16076108 kB
SwapCached:          0 kB
Active:       10824624 kB
Inactive:      7797652 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13755288 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             416 kB
Writeback:           0 kB
AnonPages:       67368 kB
Mapped:          14692 kB
Slab:           507048 kB
PageTables:       3832 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   189988 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72076 MiB
End job on node120 at 2011-04-28 21:27:34