Trace number 3317737

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-industrial? (TO) 3979.12 1300.05

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k50-eijkbs6669-tseitin.cnf
MD5SUM7886cd26d6e0801ea382a1ffd6224023
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables192503
Number of clauses564953
Sum of the clauses size1318221
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2376634
Number of clauses of size 3188317
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.37/0.64	c INFO: loaded portfolio from HOME/etc/borg-mix+class.industrial.pickle
0.37/0.64	c INFO: solving HOME/instance-3317737-1304269722.cnf
0.37/0.64	c INFO: computing task features
2.15/2.42	c INFO: computed variable-clause graph statistics
2.15/2.49	c INFO: computed clause balance statistics
10.85/11.14	c INFO: computed variable balance statistics
17.35/17.62	c INFO: computed small-clause counts
17.46/17.74	c INFO: computed Horn-clause counts
60.05/60.36	c INFO: computed variable graph statistics
60.05/60.36	c INFO: collected features for HOME/instance-3317737-1304269722.cnf in 59.67 s
60.05/60.38	c INFO: running lingeling-276@0 for 97 with 4740 remaining (b = 0.56)
60.15/60.41	c INFO: running lingeling-276@0 for 1067 with 4740 remaining (b = 0.53)
60.15/60.43	c INFO: running cryptominisat-2.9.0@0 for 582 with 4740 remaining (b = 0.45)
60.15/60.46	c INFO: running lingeling-276@0 for 1067 with 4740 remaining (b = 0.44)
449.50/190.59	c INFO: running SApperloT@0 for 97 with 4610 remaining (b = 0.04)
1037.15/386.89	c INFO: running cryptominisat-2.9.0@0 for 485 with 4413 remaining (b = 0.31)
1914.30/679.81	c INFO: running precosat-570@0 for 388 with 4120 remaining (b = 0.24)
3109.14/1021.93	c INFO: running lingeling-276@0 for 97 with 3778 remaining (b = 0.08)
3725.44/1215.21	c INFO: running lingeling-276@96 for 97 with 3585 remaining (b = 0.08)
3804.14/1241.60	c INFO: running precosat-570@487 for 194 with 3559 remaining (b = 0.11)
3961.48/1294.17	c INFO: running lingeling-276@0 for 97 with 3506 remaining (b = 0.08)
3979.12/1300.04	c INFO: running lingeling-276@0 for 291 with 3500 remaining (b = 0.16)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317737-1304269722/watcher-3317737-1304269722 -o /tmp/evaluation-result-3317737-1304269722/solver-3317737-1304269722 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3317737-1304269722.cnf -seed 1760787075 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.94 8.05 7.30 7/219 5706
/proc/meminfo: memFree=21289596/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=15772 CPUtime=0 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 881 0 0 0 0 0 0 0 25 0 1 0 538572023 16150528 762 33554432000 4194304 5683644 140736319844496 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 3943 762 343 364 0 430 0

[startup+0.102532 s]
/proc/loadavg: 7.94 8.05 7.30 7/219 5706
/proc/meminfo: memFree=21289596/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=93956 CPUtime=0.06 cores=1,3,5,7
/proc/5706/stat : 5706 (python) S 5704 5706 5278 0 -1 4202496 2696 0 0 0 4 2 0 0 25 0 1 0 538572023 96210944 2406 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/5706/statm: 23489 2406 764 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 93956

[startup+0.200518 s]
/proc/loadavg: 7.94 8.05 7.30 7/219 5706
/proc/meminfo: memFree=21289596/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=124228 CPUtime=0.11 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 4488 467 1 0 7 4 0 0 25 0 1 0 538572023 127209472 3889 33554432000 4194304 5683644 140736319844496 18446744073709551615 4620004 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 31057 3889 997 364 0 2868 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 124228

[startup+0.300501 s]
/proc/loadavg: 7.94 8.05 7.30 7/219 5706
/proc/meminfo: memFree=21289596/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=135744 CPUtime=0.17 cores=1,3,5,7
/proc/5706/stat : 5706 (python) S 5704 5706 5278 0 -1 4202496 5958 936 1 0 11 6 0 0 17 0 1 0 538572023 139001856 4149 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089694213 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/5706/statm: 33936 4149 1032 364 0 3122 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135744

[startup+0.704475 s]
/proc/loadavg: 7.94 8.05 7.30 7/219 5706
/proc/meminfo: memFree=21289596/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=291864 CPUtime=0.45 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 12592 936 1 0 32 13 0 0 16 0 1 0 538572023 298868736 10275 33554432000 4194304 5683644 140736319844496 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 72966 10275 1761 364 0 8890 0
Current children cumulated CPU time (s) 0.45
Current children cumulated vsize (KiB) 291864

[startup+1.50029 s]
/proc/loadavg: 7.94 8.05 7.30 7/220 5711
/proc/meminfo: memFree=21203612/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=343728 CPUtime=1.25 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 25262 936 1 0 110 15 0 0 16 0 1 0 538572023 351977472 22945 33554432000 4194304 5683644 140736319844496 18446744073709551615 4603727 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 85932 22946 1761 364 0 21856 0
Current children cumulated CPU time (s) 1.25
Current children cumulated vsize (KiB) 343728

[startup+3.101 s]
/proc/loadavg: 7.94 8.05 7.30 7/220 5711
/proc/meminfo: memFree=21131196/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=384592 CPUtime=2.85 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 54117 936 1 0 268 17 0 0 19 0 1 0 538572023 393822208 33533 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089059076 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 96148 33533 1826 364 0 32072 0
Current children cumulated CPU time (s) 2.85
Current children cumulated vsize (KiB) 384592

[startup+6.30044 s]
/proc/loadavg: 7.87 8.04 7.30 7/220 5711
/proc/meminfo: memFree=21085168/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=384592 CPUtime=6.05 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 54117 936 1 0 588 17 0 0 25 0 1 0 538572023 393822208 33533 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089357442 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 96148 33533 1826 364 0 32072 0
Current children cumulated CPU time (s) 6.05
Current children cumulated vsize (KiB) 384592

[startup+12.7003 s]
/proc/loadavg: 7.72 8.00 7.30 7/220 5711
/proc/meminfo: memFree=21132328/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=384592 CPUtime=12.45 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 54118 936 1 0 1228 17 0 0 25 0 1 0 538572023 393822208 33534 33554432000 4194304 5683644 140736319844496 18446744073709551615 4423431 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 96148 33534 1827 364 0 32072 0
Current children cumulated CPU time (s) 12.45
Current children cumulated vsize (KiB) 384592

[startup+25.501 s]
/proc/loadavg: 7.34 7.90 7.27 7/220 5712
/proc/meminfo: memFree=21048736/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=389004 CPUtime=25.26 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 55227 936 1 0 2508 18 0 0 25 0 1 0 538572023 398340096 34643 33554432000 4194304 5683644 140736319844496 18446744073709551615 47555817367680 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 97251 34643 1833 364 0 33175 0
Current children cumulated CPU time (s) 25.26
Current children cumulated vsize (KiB) 389004

[startup+51.1005 s]
/proc/loadavg: 6.88 7.75 7.24 7/221 5713
/proc/meminfo: memFree=21051872/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=389004 CPUtime=50.85 cores=1,3,5,7
/proc/5706/stat : 5706 (python) R 5704 5706 5278 0 -1 4202496 55227 936 1 0 5067 18 0 0 25 0 1 0 538572023 398340096 34643 33554432000 4194304 5683644 140736319844496 18446744073709551615 47555817367692 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5706/statm: 97251 34643 1833 364 0 33175 0
Current children cumulated CPU time (s) 50.85
Current children cumulated vsize (KiB) 389004

[startup+102.301 s]
/proc/loadavg: 7.91 7.87 7.30 10/233 5727
/proc/meminfo: memFree=20648160/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=421392 CPUtime=60.21 cores=1,3,5,7
/proc/5706/stat : 5706 (python) S 5704 5706 5278 0 -1 4202496 57932 936 1 0 6002 19 0 0 25 0 5 0 538572023 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/5706/statm: 105348 32545 1881 364 0 41257 0
[pid=5715] ppid=5706 vsize=380380 CPUtime=0.06 cores=1,3,5,7
/proc/5715/stat : 5715 (python) S 5706 5706 5278 0 -1 4202560 996 0 0 0 2 4 0 0 18 0 1 0 538578062 389509120 31030 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718287 0 0 16781312 514 8111106074196541704 0 0 17 5 0 0 0
/proc/5715/statm: 95095 31030 382 364 0 31013 0
[pid=5717] ppid=5715 vsize=53872 CPUtime=41.9 cores=1,3,5,7
/proc/5717/stat : 5717 (lingeling) R 5715 5717 5717 0 -1 4202496 27423 0 0 0 4187 3 0 0 25 0 1 0 538578063 55164928 11615 33554432000 4194304 4352236 140736172580544 18446744073709551615 4328114 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/5717/statm: 13468 11615 129 39 0 12548 0
[pid=5718] ppid=5706 vsize=390636 CPUtime=0.06 cores=1,3,5,7
/proc/5718/stat : 5718 (python) S 5706 5706 5278 0 -1 4202560 920 0 0 0 3 3 0 0 18 0 1 0 538578065 400011264 31022 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541704 0 0 17 7 0 0 0
/proc/5718/statm: 97659 31022 370 364 0 33574 0
[pid=5720] ppid=5718 vsize=53888 CPUtime=39.72 cores=1,3,5,7
/proc/5720/stat : 5720 (lingeling) R 5718 5720 5720 0 -1 4202496 26564 0 0 0 3968 4 0 0 25 0 1 0 538578065 55181312 11619 33554432000 4194304 4352236 140736844172336 18446744073709551615 4231316 0 0 16781312 17506 0 0 0 17 7 0 0 0
/proc/5720/statm: 13472 11619 129 39 0 12552 0
[pid=5721] ppid=5706 vsize=400892 CPUtime=0.06 cores=1,3,5,7
/proc/5721/stat : 5721 (python) S 5706 5706 5278 0 -1 4202560 927 0 0 0 3 3 0 0 15 0 1 0 538578067 410513408 31026 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 4609346261978415365 0 0 17 5 0 0 0
/proc/5721/statm: 100223 31026 370 364 0 36135 0
[pid=5725] ppid=5721 vsize=252648 CPUtime=20.63 cores=1,3,5,7
/proc/5725/stat : 5725 (cryptominisat-2) R 5721 5725 5725 0 -1 4202496 63969 0 0 0 2054 9 0 0 25 0 1 0 538578070 258711552 53816 33554432000 4194304 4677096 140734952162496 18446744073709551615 4450695 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5725/statm: 63162 53816 392 118 0 58121 0
[pid=5723] ppid=5706 vsize=411148 CPUtime=0.06 cores=1,3,5,7
/proc/5723/stat : 5723 (python) S 5706 5706 5278 0 -1 4202560 950 0 0 0 3 3 0 0 18 0 1 0 538578070 421015552 31030 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541704 0 0 17 5 0 0 0
/proc/5723/statm: 102787 31030 370 364 0 38696 0
[pid=5726] ppid=5723 vsize=53892 CPUtime=22.81 cores=1,3,5,7
/proc/5726/stat : 5726 (lingeling) R 5723 5726 5726 0 -1 4202496 27424 0 0 0 2278 3 0 0 25 0 1 0 538578071 55185408 11620 33554432000 4194304 4352236 140734486261216 18446744073709551615 4328979 0 0 16781312 17506 0 0 0 17 5 0 0 0
/proc/5726/statm: 13473 11620 129 39 0 12553 0
[pid=5706/tid=5716] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5716/stat : 5716 (python) S 5704 5706 5278 0 -1 4202560 23 936 0 0 0 0 0 0 25 0 5 0 538578062 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5719] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5719/stat : 5719 (python) S 5704 5706 5278 0 -1 4202560 21 936 0 0 0 0 0 0 25 0 5 0 538578065 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 7 0 0 0
[pid=5706/tid=5722] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5722/stat : 5722 (python) S 5704 5706 5278 0 -1 4202560 30 936 0 0 0 0 0 0 25 0 5 0 538578068 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5724] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5724/stat : 5724 (python) S 5704 5706 5278 0 -1 4202560 22 936 0 0 0 0 0 0 25 0 5 0 538578070 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 185.51
Current children cumulated vsize (KiB) 2418748

[startup+162.301 s]
/proc/loadavg: 8.73 8.12 7.42 10/233 5728
/proc/meminfo: memFree=20620068/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=421392 CPUtime=60.21 cores=1,3,5,7
/proc/5706/stat : 5706 (python) S 5704 5706 5278 0 -1 4202496 57932 936 1 0 6002 19 0 0 25 0 5 0 538572023 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/5706/statm: 105348 32545 1881 364 0 41257 0
[pid=5715] ppid=5706 vsize=380380 CPUtime=0.15 cores=1,3,5,7
/proc/5715/stat : 5715 (python) S 5706 5706 5278 0 -1 4202560 1019 0 0 0 7 8 0 0 18 0 1 0 538578062 389509120 31030 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718287 0 0 16781312 514 8111106074196541704 0 0 17 5 0 0 0
/proc/5715/statm: 95095 31030 382 364 0 31013 0
[pid=5717] ppid=5715 vsize=53872 CPUtime=83.99 cores=1,3,5,7
/proc/5717/stat : 5717 (lingeling) R 5715 5717 5717 0 -1 4202496 27423 0 0 0 8396 3 0 0 25 0 1 0 538578063 55164928 11615 33554432000 4194304 4352236 140736172580544 18446744073709551615 4328105 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/5717/statm: 13468 11615 129 39 0 12548 0
[pid=5718] ppid=5706 vsize=390636 CPUtime=0.15 cores=1,3,5,7
/proc/5718/stat : 5718 (python) S 5706 5706 5278 0 -1 4202560 941 0 0 0 7 8 0 0 18 0 1 0 538578065 400011264 31022 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541704 0 0 17 7 0 0 0
/proc/5718/statm: 97659 31022 370 364 0 33574 0
[pid=5720] ppid=5718 vsize=53884 CPUtime=99.24 cores=1,3,5,7
/proc/5720/stat : 5720 (lingeling) R 5718 5720 5720 0 -1 4202496 26564 0 0 0 9920 4 0 0 25 0 1 0 538578065 55177216 11618 33554432000 4194304 4352236 140736844172336 18446744073709551615 4328194 0 0 16781312 17506 0 0 0 17 7 0 0 0
/proc/5720/statm: 13471 11618 129 39 0 12551 0
[pid=5721] ppid=5706 vsize=400892 CPUtime=0.18 cores=1,3,5,7
/proc/5721/stat : 5721 (python) S 5706 5706 5278 0 -1 4202560 951 0 0 0 10 8 0 0 15 0 1 0 538578067 410513408 31026 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541701 0 0 17 5 0 0 0
/proc/5721/statm: 100223 31026 370 364 0 36135 0
[pid=5725] ppid=5721 vsize=265756 CPUtime=68.33 cores=1,3,5,7
/proc/5725/stat : 5725 (cryptominisat-2) R 5721 5725 5725 0 -1 4202496 80542 0 0 0 6819 14 0 0 25 0 1 0 538578070 272134144 57552 33554432000 4194304 4677096 140734952162496 18446744073709551615 4358608 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/5725/statm: 66439 57552 392 118 0 61398 0
[pid=5723] ppid=5706 vsize=411148 CPUtime=0.15 cores=1,3,5,7
/proc/5723/stat : 5723 (python) S 5706 5706 5278 0 -1 4202560 969 0 0 0 7 8 0 0 18 0 1 0 538578070 421015552 31030 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541704 0 0 17 5 0 0 0
/proc/5723/statm: 102787 31030 370 364 0 38696 0
[pid=5726] ppid=5723 vsize=53884 CPUtime=52.7 cores=1,3,5,7
/proc/5726/stat : 5726 (lingeling) R 5723 5726 5726 0 -1 4202496 27424 0 0 0 5267 3 0 0 25 0 1 0 538578071 55177216 11618 33554432000 4194304 4352236 140734486261216 18446744073709551615 4247822 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/5726/statm: 13471 11618 129 39 0 12551 0
[pid=5706/tid=5716] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5716/stat : 5716 (python) S 5704 5706 5278 0 -1 4202560 23 936 0 0 0 0 0 0 25 0 5 0 538578062 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5719] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5719/stat : 5719 (python) S 5704 5706 5278 0 -1 4202560 21 936 0 0 0 0 0 0 25 0 5 0 538578065 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 7 0 0 0
[pid=5706/tid=5722] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5722/stat : 5722 (python) S 5704 5706 5278 0 -1 4202560 30 936 0 0 0 0 0 0 25 0 5 0 538578068 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5724] ppid=5704 vsize=421392 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5724/stat : 5724 (python) S 5704 5706 5278 0 -1 4202560 22 936 0 0 0 0 0 0 25 0 5 0 538578070 431505408 32545 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 365.1
Current children cumulated vsize (KiB) 2431844


################
# More data... #
################

/proc/meminfo: memFree=27059512/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=452160 CPUtime=60.38 cores=1,3,5,7
/proc/5706/stat : 5706 (python) S 5704 5706 5278 0 -1 4202496 61021 936 1 0 6017 21 0 0 18 0 8 0 538572023 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/5706/statm: 113040 32564 1888 364 0 48940 0
[pid=5715] ppid=5706 vsize=390624 CPUtime=0.61 cores=1,3,5,7
/proc/5715/stat : 5715 (python) S 5706 5706 5278 0 -1 4202560 1209 0 0 0 32 29 0 0 18 0 2 0 538578062 399998976 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5715/statm: 97656 31069 416 364 0 33574 0
[pid=5717] ppid=5715 vsize=58016 CPUtime=240.16 cores=1,3,5,7
/proc/5717/stat : 5717 (lingeling) R 5715 5717 5717 0 -1 4202496 54061 0 0 0 24010 6 0 0 25 0 1 0 538578063 59408384 13674 33554432000 4194304 4352236 140736172580544 18446744073709551615 4328832 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/5717/statm: 14504 13674 129 39 0 13584 0
[pid=5715/tid=5730] ppid=5706 vsize=390624 CPUtime=0 cores=1,3,5,7
/proc/5715/task/5730/stat : 5730 (python) S 5706 5706 5278 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 538591081 399998976 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604454224046016 0 0 -1 5 0 0 0
[pid=5718] ppid=5706 vsize=390636 CPUtime=1.88 cores=1,3,5,7
/proc/5718/stat : 5718 (python) S 5706 5706 5278 0 -1 4202560 1128 0 0 0 97 91 0 0 18 0 2 0 538578065 400011264 31052 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5718/statm: 97659 31052 399 364 0 33574 0
[pid=5720] ppid=5718 vsize=94388 CPUtime=1070.32 cores=1,3,5,7
/proc/5720/stat : 5720 (lingeling) R 5718 5720 5720 0 -1 4202496 140115 0 0 0 107014 18 0 0 25 0 1 0 538578065 96653312 22767 33554432000 4194304 4352236 140736844172336 18446744073709551615 4231337 0 0 16781312 17506 0 0 0 17 7 0 0 0
/proc/5720/statm: 23597 22767 129 39 0 22677 0
[pid=5718/tid=5973] ppid=5706 vsize=390636 CPUtime=0 cores=1,3,5,7
/proc/5718/task/5973/stat : 5973 (python) S 5706 5706 5278 0 -1 4202560 42 0 0 0 0 0 0 0 19 0 2 0 538701439 400011264 31052 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604464849333248 0 0 -1 7 0 0 0
[pid=5721] ppid=5706 vsize=400892 CPUtime=1.04 cores=1,3,5,7
/proc/5721/stat : 5721 (python) S 5706 5706 5278 0 -1 4202560 1159 0 0 0 56 48 0 0 16 0 2 0 538578067 410513408 31057 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5721/statm: 100223 31057 400 364 0 36135 0
[pid=5725] ppid=5721 vsize=343552 CPUtime=583.28 cores=1,3,5,7
/proc/5725/stat : 5725 (cryptominisat-2) T 5721 5725 5725 0 -1 4202496 207785 0 0 0 58283 45 0 0 25 0 1 0 538578070 351797248 76231 33554432000 4194304 4677096 140734952162496 18446744073709551615 4319991 0 0 16781312 2 18446744071562691360 0 0 17 5 0 0 0
/proc/5725/statm: 85888 76231 392 118 0 80847 0
[pid=5721/tid=5803] ppid=5706 vsize=400892 CPUtime=0 cores=1,3,5,7
/proc/5721/task/5803/stat : 5803 (python) S 5706 5706 5278 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 538640003 410513408 31057 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604454224046912 0 0 -1 5 0 0 0
[pid=5723] ppid=5706 vsize=411148 CPUtime=1.85 cores=1,3,5,7
/proc/5723/stat : 5723 (python) S 5706 5706 5278 0 -1 4202560 1005 0 0 0 97 88 0 0 18 0 1 0 538578070 421015552 31030 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541704 0 0 17 5 0 0 0
/proc/5723/statm: 102787 31030 370 364 0 38696 0
[pid=5726] ppid=5723 vsize=109204 CPUtime=1014.3 cores=1,3,5,7
/proc/5726/stat : 5726 (lingeling) R 5723 5726 5726 0 -1 4202496 116007 0 0 0 101416 14 0 0 25 0 1 0 538578071 111824896 26471 33554432000 4194304 4352236 140734486261216 18446744073709551615 4328000 0 0 16781312 17506 0 0 0 17 5 0 0 0
/proc/5726/statm: 27301 26471 129 39 0 26381 0
[pid=5731] ppid=5706 vsize=421404 CPUtime=0.29 cores=1,3,5,7
/proc/5731/stat : 5731 (python) S 5706 5706 5278 0 -1 4202560 1151 0 0 0 16 13 0 0 19 0 2 0 538591083 431517696 31064 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5731/statm: 105351 31064 399 364 0 41257 0
[pid=5733] ppid=5731 vsize=134380 CPUtime=97.37 cores=1,3,5,7
/proc/5733/stat : 5733 (SApperloT-base) T 5731 5733 5733 0 -1 4202496 33093 0 0 0 9728 9 0 0 25 0 1 0 538591084 137605120 32155 33554432000 134512640 135674940 4291825904 18446744073709551615 134565290 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5733/statm: 33595 32155 100 284 0 33309 0
[pid=5731/tid=5788] ppid=5706 vsize=421404 CPUtime=0 cores=1,3,5,7
/proc/5731/task/5788/stat : 5788 (python) S 5706 5706 5278 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538610710 431517696 31064 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604471125857344 0 0 -1 5 0 0 0
[pid=5789] ppid=5706 vsize=431660 CPUtime=1.07 cores=1,3,5,7
/proc/5789/stat : 5789 (python) S 5706 5706 5278 0 -1 4202560 1198 0 0 0 56 51 0 0 17 0 2 0 538610713 442019840 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5789/statm: 107915 31069 400 364 0 43818 0
[pid=5791] ppid=5789 vsize=330552 CPUtime=488.2 cores=1,3,5,7
/proc/5791/stat : 5791 (cryptominisat-2) T 5789 5791 5791 0 -1 4202496 188504 0 0 0 48773 47 0 0 25 0 1 0 538610714 338485248 73775 33554432000 4194304 4677096 140736254435728 18446744073709551615 4336710 0 0 16781312 2 18446744071562691360 0 0 17 1 0 0 0
/proc/5791/statm: 82638 73775 399 118 0 77597 0
[pid=5789/tid=5868] ppid=5706 vsize=431660 CPUtime=0 cores=1,3,5,7
/proc/5789/task/5868/stat : 5868 (python) S 5706 5706 5278 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 538674214 442019840 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604471134453952 0 0 -1 5 0 0 0
[pid=5804] ppid=5706 vsize=441916 CPUtime=0.92 cores=1,3,5,7
/proc/5804/stat : 5804 (python) S 5706 5706 5278 0 -1 4202560 1175 0 0 0 51 41 0 0 18 0 2 0 538640005 452521984 31072 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5804/statm: 110479 31072 399 364 0 46379 0
[pid=5806] ppid=5804 vsize=193832 CPUtime=417.45 cores=1,3,5,7
/proc/5806/stat : 5806 (precosat) R 5804 5806 5806 0 -1 4202496 78452 0 0 0 41733 12 0 0 25 0 1 0 538640006 198483968 47027 33554432000 4194304 4890832 140734785201664 18446744073709551615 4213276 0 0 16781312 17442 0 0 0 17 7 0 0 0
/proc/5806/statm: 48458 47027 86 171 0 48285 0
[pid=5804/tid=5967] ppid=5706 vsize=441916 CPUtime=0 cores=1,3,5,7
/proc/5804/task/5967/stat : 5967 (python) S 5706 5706 5278 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538696181 452521984 31072 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604463509610624 0 0 -1 5 0 0 0
[pid=5706/tid=5716] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5716/stat : 5716 (python) S 5704 5706 5278 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 8 0 538578062 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5719] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5719/stat : 5719 (python) S 5704 5706 5278 0 -1 4202560 25 936 0 0 0 0 0 0 18 0 8 0 538578065 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 7 0 0 0
[pid=5706/tid=5722] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5722/stat : 5722 (python) S 5704 5706 5278 0 -1 4202560 30 936 0 0 0 0 0 0 25 0 8 0 538578068 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5724] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5724/stat : 5724 (python) S 5704 5706 5278 0 -1 4202560 22 936 0 0 0 0 0 0 25 0 8 0 538578070 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5732] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5732/stat : 5732 (python) S 5704 5706 5278 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 8 0 538591083 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5790] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5790/stat : 5790 (python) S 5704 5706 5278 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 8 0 538610713 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5805] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5805/stat : 5805 (python) S 5704 5706 5278 0 -1 4202560 33 936 0 0 0 0 0 0 18 0 8 0 538640006 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3979.12
Current children cumulated vsize (KiB) 4604364

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1300 s]
/proc/loadavg: 9.14 9.05 8.55 10/243 5974
/proc/meminfo: memFree=27059512/32950928 swapFree=67111428/67111528
[pid=5706] ppid=5704 vsize=452160 CPUtime=60.38 cores=1,3,5,7
/proc/5706/stat : 5706 (python) S 5704 5706 5278 0 -1 4202496 61021 936 1 0 6017 21 0 0 18 0 8 0 538572023 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/5706/statm: 113040 32564 1888 364 0 48940 0
[pid=5715] ppid=5706 vsize=390624 CPUtime=0.61 cores=1,3,5,7
/proc/5715/stat : 5715 (python) S 5706 5706 5278 0 -1 4202560 1209 0 0 0 32 29 0 0 18 0 2 0 538578062 399998976 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5715/statm: 97656 31069 416 364 0 33574 0
[pid=5717] ppid=5715 vsize=58016 CPUtime=240.16 cores=1,3,5,7
/proc/5717/stat : 5717 (lingeling) R 5715 5717 5717 0 -1 4202496 54061 0 0 0 24010 6 0 0 25 0 1 0 538578063 59408384 13674 33554432000 4194304 4352236 140736172580544 18446744073709551615 4328832 0 0 16781312 17506 0 0 0 17 3 0 0 0
/proc/5717/statm: 14504 13674 129 39 0 13584 0
[pid=5715/tid=5730] ppid=5706 vsize=390624 CPUtime=0 cores=1,3,5,7
/proc/5715/task/5730/stat : 5730 (python) S 5706 5706 5278 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 538591081 399998976 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604454224046016 0 0 -1 5 0 0 0
[pid=5718] ppid=5706 vsize=390636 CPUtime=1.88 cores=1,3,5,7
/proc/5718/stat : 5718 (python) S 5706 5706 5278 0 -1 4202560 1128 0 0 0 97 91 0 0 18 0 2 0 538578065 400011264 31052 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5718/statm: 97659 31052 399 364 0 33574 0
[pid=5720] ppid=5718 vsize=94388 CPUtime=1070.32 cores=1,3,5,7
/proc/5720/stat : 5720 (lingeling) R 5718 5720 5720 0 -1 4202496 140115 0 0 0 107014 18 0 0 25 0 1 0 538578065 96653312 22767 33554432000 4194304 4352236 140736844172336 18446744073709551615 4231337 0 0 16781312 17506 0 0 0 17 7 0 0 0
/proc/5720/statm: 23597 22767 129 39 0 22677 0
[pid=5718/tid=5973] ppid=5706 vsize=390636 CPUtime=0 cores=1,3,5,7
/proc/5718/task/5973/stat : 5973 (python) S 5706 5706 5278 0 -1 4202560 42 0 0 0 0 0 0 0 19 0 2 0 538701439 400011264 31052 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604464849333248 0 0 -1 7 0 0 0
[pid=5721] ppid=5706 vsize=400892 CPUtime=1.04 cores=1,3,5,7
/proc/5721/stat : 5721 (python) S 5706 5706 5278 0 -1 4202560 1159 0 0 0 56 48 0 0 16 0 2 0 538578067 410513408 31057 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5721/statm: 100223 31057 400 364 0 36135 0
[pid=5725] ppid=5721 vsize=343552 CPUtime=583.28 cores=1,3,5,7
/proc/5725/stat : 5725 (cryptominisat-2) T 5721 5725 5725 0 -1 4202496 207785 0 0 0 58283 45 0 0 25 0 1 0 538578070 351797248 76231 33554432000 4194304 4677096 140734952162496 18446744073709551615 4319991 0 0 16781312 2 18446744071562691360 0 0 17 5 0 0 0
/proc/5725/statm: 85888 76231 392 118 0 80847 0
[pid=5721/tid=5803] ppid=5706 vsize=400892 CPUtime=0 cores=1,3,5,7
/proc/5721/task/5803/stat : 5803 (python) S 5706 5706 5278 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 538640003 410513408 31057 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604454224046912 0 0 -1 5 0 0 0
[pid=5723] ppid=5706 vsize=411148 CPUtime=1.85 cores=1,3,5,7
/proc/5723/stat : 5723 (python) S 5706 5706 5278 0 -1 4202560 1005 0 0 0 97 88 0 0 18 0 1 0 538578070 421015552 31030 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 8111106074196541704 0 0 17 5 0 0 0
/proc/5723/statm: 102787 31030 370 364 0 38696 0
[pid=5726] ppid=5723 vsize=109204 CPUtime=1014.3 cores=1,3,5,7
/proc/5726/stat : 5726 (lingeling) R 5723 5726 5726 0 -1 4202496 116007 0 0 0 101416 14 0 0 25 0 1 0 538578071 111824896 26471 33554432000 4194304 4352236 140734486261216 18446744073709551615 4328000 0 0 16781312 17506 0 0 0 17 5 0 0 0
/proc/5726/statm: 27301 26471 129 39 0 26381 0
[pid=5731] ppid=5706 vsize=421404 CPUtime=0.29 cores=1,3,5,7
/proc/5731/stat : 5731 (python) S 5706 5706 5278 0 -1 4202560 1151 0 0 0 16 13 0 0 19 0 2 0 538591083 431517696 31064 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5731/statm: 105351 31064 399 364 0 41257 0
[pid=5733] ppid=5731 vsize=134380 CPUtime=97.37 cores=1,3,5,7
/proc/5733/stat : 5733 (SApperloT-base) T 5731 5733 5733 0 -1 4202496 33093 0 0 0 9728 9 0 0 25 0 1 0 538591084 137605120 32155 33554432000 134512640 135674940 4291825904 18446744073709551615 134565290 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5733/statm: 33595 32155 100 284 0 33309 0
[pid=5731/tid=5788] ppid=5706 vsize=421404 CPUtime=0 cores=1,3,5,7
/proc/5731/task/5788/stat : 5788 (python) S 5706 5706 5278 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538610710 431517696 31064 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604471125857344 0 0 -1 5 0 0 0
[pid=5789] ppid=5706 vsize=431660 CPUtime=1.07 cores=1,3,5,7
/proc/5789/stat : 5789 (python) S 5706 5706 5278 0 -1 4202560 1198 0 0 0 56 51 0 0 17 0 2 0 538610713 442019840 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5789/statm: 107915 31069 400 364 0 43818 0
[pid=5791] ppid=5789 vsize=330552 CPUtime=488.2 cores=1,3,5,7
/proc/5791/stat : 5791 (cryptominisat-2) T 5789 5791 5791 0 -1 4202496 188504 0 0 0 48773 47 0 0 25 0 1 0 538610714 338485248 73775 33554432000 4194304 4677096 140736254435728 18446744073709551615 4336710 0 0 16781312 2 18446744071562691360 0 0 17 1 0 0 0
/proc/5791/statm: 82638 73775 399 118 0 77597 0
[pid=5789/tid=5868] ppid=5706 vsize=431660 CPUtime=0 cores=1,3,5,7
/proc/5789/task/5868/stat : 5868 (python) S 5706 5706 5278 0 -1 4202560 45 0 0 0 0 0 0 0 17 0 2 0 538674214 442019840 31069 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604471134453952 0 0 -1 5 0 0 0
[pid=5804] ppid=5706 vsize=441916 CPUtime=0.92 cores=1,3,5,7
/proc/5804/stat : 5804 (python) S 5706 5706 5278 0 -1 4202560 1175 0 0 0 51 41 0 0 18 0 2 0 538640005 452521984 31072 33554432000 4194304 5683644 140736319844496 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5804/statm: 110479 31072 399 364 0 46379 0
[pid=5806] ppid=5804 vsize=193832 CPUtime=417.45 cores=1,3,5,7
/proc/5806/stat : 5806 (precosat) R 5804 5806 5806 0 -1 4202496 78452 0 0 0 41733 12 0 0 25 0 1 0 538640006 198483968 47027 33554432000 4194304 4890832 140734785201664 18446744073709551615 4213276 0 0 16781312 17442 0 0 0 17 7 0 0 0
/proc/5806/statm: 48458 47027 86 171 0 48285 0
[pid=5804/tid=5967] ppid=5706 vsize=441916 CPUtime=0 cores=1,3,5,7
/proc/5804/task/5967/stat : 5967 (python) S 5706 5706 5278 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538696181 452521984 31072 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 514 18446604463509610624 0 0 -1 5 0 0 0
[pid=5706/tid=5716] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5716/stat : 5716 (python) S 5704 5706 5278 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 8 0 538578062 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5719] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5719/stat : 5719 (python) S 5704 5706 5278 0 -1 4202560 25 936 0 0 0 0 0 0 18 0 8 0 538578065 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 7 0 0 0
[pid=5706/tid=5722] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5722/stat : 5722 (python) S 5704 5706 5278 0 -1 4202560 30 936 0 0 0 0 0 0 25 0 8 0 538578068 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5724] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5724/stat : 5724 (python) S 5704 5706 5278 0 -1 4202560 22 936 0 0 0 0 0 0 25 0 8 0 538578070 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5732] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5732/stat : 5732 (python) S 5704 5706 5278 0 -1 4202560 27 936 0 0 0 0 0 0 20 0 8 0 538591083 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5790] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5790/stat : 5790 (python) S 5704 5706 5278 0 -1 4202560 25 936 0 0 0 0 0 0 20 0 8 0 538610713 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
[pid=5706/tid=5805] ppid=5704 vsize=452160 CPUtime=0 cores=1,3,5,7
/proc/5706/task/5805/stat : 5805 (python) S 5704 5706 5278 0 -1 4202560 33 936 0 0 0 0 0 0 18 0 8 0 538640006 463011840 32564 33554432000 4194304 5683644 140736319844496 18446744073709551615 248101521841 0 0 16781312 2 18446604471125858240 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3979.12
Current children cumulated vsize (KiB) 4604364

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5706 and gives
#  childrusage.ru_utime.tv_sec=303
#  childrusage.ru_utime.tv_usec=306890
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=32538
# CPU time returned by wait4() is 306.339
# while last known CPU time is 3979.12
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1300.05
CPU time (s): 3979.12
CPU user time (s): 3973.79
CPU system time (s): 5.33
CPU usage (%): 306.074
Max. virtual memory (cumulated for all children) (KiB): 4661328

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 303.307
system time used= 3.03254
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 122045
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7241
involuntary context switches= 2462

runsolver used 2.83357 second user time and 6.23605 second system time

The end

Launcher Data 

Begin job on node123 at 2011-05-01 19:08:42
IDJOB=3317737
IDBENCH=82493
IDSOLVER=1706
FILE ID=node123/3317737-1304269722
RUNJOBID= node123-1304267283-5297
PBS_JOBID= 13173956
Free space on /tmp= 73416 MiB

SOLVER NAME= borg-sat 11.03.02-industrial
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k50-eijkbs6669-tseitin.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.industrial.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317737-1304269722/watcher-3317737-1304269722 -o /tmp/evaluation-result-3317737-1304269722/solver-3317737-1304269722 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3317737-1304269722.cnf -seed 1760787075 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7886cd26d6e0801ea382a1ffd6224023
RANDOM SEED=1760787075

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      21289628 kB
Buffers:        352168 kB
Cached:        1712644 kB
SwapCached:        100 kB
Active:        9404024 kB
Inactive:      1874872 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      21289628 kB
SwapTotal:    67111528 kB
SwapFree:     67111428 kB
Dirty:           42012 kB
Writeback:           0 kB
AnonPages:     9213436 kB
Mapped:          24576 kB
Slab:           289916 kB
PageTables:      30336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 11248416 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73384 MiB
End job on node123 at 2011-05-01 19:30:24