Trace number 3317501

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.3 1204.76

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v650-c2769-S29470018-050.UNKNOWN.cnf
MD5SUM27350cd58dd0500df065558763d7d963
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables650
Number of clauses2769
Sum of the clauses size8307
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32769
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.32/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.46	c INFO: solving HOME/instance-3317501-1304269881.cnf
0.32/0.46	c INFO: computing task features
0.32/0.48	c INFO: computed variable-clause graph statistics
0.32/0.48	c INFO: computed clause balance statistics
0.40/0.52	c INFO: computed variable balance statistics
0.40/0.56	c INFO: computed small-clause counts
0.40/0.56	c INFO: computed Horn-clause counts
0.40/0.57	c INFO: computed variable graph statistics
0.40/0.57	c INFO: collected features for HOME/instance-3317501-1304269881.cnf in 0.10 s
0.40/0.59	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.58)
0.51/0.62	c INFO: running march_hi@0 for 194 with 4799 remaining (b = 0.16)
0.51/0.65	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.07)
0.51/0.68	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.26)
399.83/100.78	c INFO: running march_hi@0 for 485 with 4699 remaining (b = 0.27)
400.21/100.86	c INFO: running march_hi@0 for 485 with 4699 remaining (b = 0.24)
787.91/197.96	c INFO: running march_hi@0 for 97 with 4602 remaining (b = 0.06)
1179.90/296.16	c INFO: running march_hi@97 for 97 with 4504 remaining (b = 0.06)
1558.02/392.22	c INFO: running march_hi@99 for 97 with 4408 remaining (b = 0.06)
1595.57/401.67	c INFO: running march_hi@486 for 97 with 4398 remaining (b = 0.06)
1953.27/491.25	c INFO: running march_hi@200 for 194 with 4309 remaining (b = 0.12)
1986.39/499.56	c INFO: running march_hi@584 for 97 with 4300 remaining (b = 0.07)
2340.48/588.27	c INFO: running march_hi@196 for 194 with 4212 remaining (b = 0.12)
2349.26/590.45	c INFO: running march_hi@100 for 97 with 4210 remaining (b = 0.06)
2375.63/597.07	c INFO: running march_hi@0 for 485 with 4203 remaining (b = 0.21)
2724.88/684.58	c INFO: running march_hi@296 for 97 with 4115 remaining (b = 0.08)
2746.05/689.82	c INFO: running march_hi@196 for 194 with 4110 remaining (b = 0.11)
3113.74/781.90	c INFO: running march_hi@396 for 97 with 4018 remaining (b = 0.06)
3138.48/788.10	c INFO: running march_hi@1070 for 291 with 4012 remaining (b = 0.17)
3510.12/881.20	c INFO: running march_hi@595 for 97 with 3919 remaining (b = 0.06)
3510.92/881.42	c INFO: running march_hi@293 for 97 with 3919 remaining (b = 0.08)
3901.42/979.23	c INFO: running march_hi@0 for 388 with 3821 remaining (b = 0.18)
3903.21/979.84	c INFO: running march_hi@391 for 97 with 3820 remaining (b = 0.05)
4297.71/1078.53	c INFO: running march_hi@1169 for 970 with 3721 remaining (b = 0.37)
4301.68/1079.51	c INFO: running kcnfs-2006@0 for 582 with 3720 remaining (b = 0.21)
4320.00/1084.15	c INFO: running march_hi@193 for 291 with 3716 remaining (b = 0.12)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317501-1304269881/watcher-3317501-1304269881 -o /tmp/evaluation-result-3317501-1304269881/solver-3317501-1304269881 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317501-1304269881.cnf -seed 1401861513 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.93 9.06 8.25 6/251 29518
/proc/meminfo: memFree=28751408/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/29518/stat : 29518 (python) R 29516 29518 28639 0 -1 4202496 879 0 0 0 0 0 0 0 25 0 1 0 538587765 16154624 760 33554432000 4194304 5683644 140736365752832 18446744073709551615 210646807237 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/29518/statm: 3944 760 343 364 0 431 0

[startup+0.0635821 s]
/proc/loadavg: 8.93 9.06 8.25 6/251 29518
/proc/meminfo: memFree=28751408/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=45816 CPUtime=0.03 cores=1,3,5,7
/proc/29518/stat : 29518 (python) R 29516 29518 28639 0 -1 4202496 1927 0 0 0 3 0 0 0 25 0 1 0 538587765 46915584 1699 33554432000 4194304 5683644 140736365752832 18446744073709551615 4870104 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/29518/statm: 11454 1699 470 364 0 1203 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 45816

[startup+0.100578 s]
/proc/loadavg: 8.93 9.06 8.25 6/251 29518
/proc/meminfo: memFree=28751408/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=104228 CPUtime=0.07 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 3060 0 0 0 5 2 0 0 17 0 1 0 538587765 106729472 2742 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647144448 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/29518/statm: 26058 2746 847 364 0 1806 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104228

[startup+0.300525 s]
/proc/loadavg: 8.93 9.06 8.25 6/251 29518
/proc/meminfo: memFree=28751408/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=182428 CPUtime=0.23 cores=1,3,5,7
/proc/29518/stat : 29518 (python) R 29516 29518 28639 0 -1 4202496 6753 933 1 0 18 5 0 0 17 0 1 0 538587765 186806272 4817 33554432000 4194304 5683644 140736365752832 18446744073709551615 210642222727 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/29518/statm: 46259 4820 1342 364 0 3723 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 182428

[startup+0.700428 s]
/proc/loadavg: 8.93 9.06 8.25 6/251 29518
/proc/meminfo: memFree=28751408/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=320280 CPUtime=0.58 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 12349 933 1 0 50 8 0 0 25 0 5 0 538587765 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29518/statm: 80070 7267 1881 364 0 15979 0
Current children cumulated CPU time (s) 0.58
Current children cumulated vsize (KiB) 320280

[startup+1.50129 s]
/proc/loadavg: 8.93 9.06 8.25 10/264 29535
/proc/meminfo: memFree=28710136/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=320280 CPUtime=0.58 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 12349 933 1 0 50 8 0 0 25 0 5 0 538587765 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29518/statm: 80070 7267 1881 364 0 15979 0
[pid=29524] ppid=29518 vsize=278572 CPUtime=0 cores=1,3,5,7
/proc/29524/stat : 29524 (python) S 29518 29518 28639 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 538587825 285257728 5572 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166479 0 0 16781312 514 5637727733774548991 0 0 17 3 0 0 0
/proc/29524/statm: 69643 5572 354 364 0 5561 0
[pid=29526] ppid=29524 vsize=1017940 CPUtime=0.89 cores=1,3,5,7
/proc/29526/stat : 29526 (gnovelty+2) R 29524 29526 29526 0 -1 4202496 453 0 0 0 89 0 0 0 21 0 1 0 538587825 1042370560 191 33554432000 134512640 135663704 4291717696 18446744073709551615 134525544 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29526/statm: 254485 191 113 282 0 254201 0
[pid=29527] ppid=29518 vsize=289424 CPUtime=0 cores=1,3,5,7
/proc/29527/stat : 29527 (python) S 29518 29518 28639 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 538587828 296370176 5714 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 5777361862211371271 0 0 17 3 0 0 0
/proc/29527/statm: 72356 5714 348 364 0 8271 0
[pid=29529] ppid=29527 vsize=2744 CPUtime=0.86 cores=1,3,5,7
/proc/29529/stat : 29529 (march_hi) R 29527 29529 29529 0 -1 4202496 742 0 0 0 86 0 0 0 25 0 1 0 538587828 2809856 481 33554432000 4194304 4955518 140733462883936 18446744073709551615 4310190 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29529/statm: 686 481 104 186 0 499 0
[pid=29530] ppid=29518 vsize=299320 CPUtime=0 cores=1,3,5,7
/proc/29530/stat : 29530 (python) S 29518 29518 28639 0 -1 4202560 736 0 0 0 0 0 0 0 22 0 1 0 538587831 306503680 5627 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 4656906736949526527 0 0 17 7 0 0 0
/proc/29530/statm: 74830 5627 348 364 0 10742 0
[pid=29532] ppid=29530 vsize=1017940 CPUtime=0.83 cores=1,3,5,7
/proc/29532/stat : 29532 (gnovelty+2) R 29530 29532 29532 0 -1 4202496 454 0 0 0 83 0 0 0 25 0 1 0 538587831 1042370560 192 33554432000 134512640 135663704 4286982224 18446744073709551615 134525342 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29532/statm: 254485 192 113 282 0 254201 0
[pid=29533] ppid=29518 vsize=310036 CPUtime=0 cores=1,3,5,7
/proc/29533/stat : 29533 (python) S 29518 29518 28639 0 -1 4202560 766 0 0 0 0 0 0 0 20 0 1 0 538587834 317476864 5730 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 3460418183603650822 0 0 17 7 0 0 0
/proc/29533/statm: 77509 5730 348 364 0 13418 0
[pid=29535] ppid=29533 vsize=2676 CPUtime=0.8 cores=1,3,5,7
/proc/29535/stat : 29535 (march_hi) R 29533 29535 29535 0 -1 4202496 742 0 0 0 80 0 0 0 25 0 1 0 538587834 2740224 478 33554432000 4194304 4955518 140736003407664 18446744073709551615 4238422 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29535/statm: 669 478 104 186 0 482 0
[pid=29518/tid=29525] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29525/stat : 29525 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 5 0 538587825 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 1 0 0 0
[pid=29518/tid=29528] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29528/stat : 29528 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 21 0 5 0 538587828 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 3 0 0 0
[pid=29518/tid=29531] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29531/stat : 29531 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 23 0 5 0 538587831 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29534] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29534/stat : 29534 (python) S 29516 29518 28639 0 -1 4202560 27 933 0 0 0 0 0 0 25 0 5 0 538587834 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3.96
Current children cumulated vsize (KiB) 3538932

[startup+3.10102 s]
/proc/loadavg: 8.93 9.06 8.25 10/264 29535
/proc/meminfo: memFree=28708648/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=320280 CPUtime=0.58 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 12349 933 1 0 50 8 0 0 25 0 5 0 538587765 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29518/statm: 80070 7267 1881 364 0 15979 0
[pid=29524] ppid=29518 vsize=278572 CPUtime=0 cores=1,3,5,7
/proc/29524/stat : 29524 (python) S 29518 29518 28639 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 538587825 285257728 5572 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166479 0 0 16781312 514 5637727733774516486 0 0 17 3 0 0 0
/proc/29524/statm: 69643 5572 354 364 0 5561 0
[pid=29526] ppid=29524 vsize=1017940 CPUtime=2.49 cores=1,3,5,7
/proc/29526/stat : 29526 (gnovelty+2) R 29524 29526 29526 0 -1 4202496 453 0 0 0 249 0 0 0 25 0 1 0 538587825 1042370560 191 33554432000 134512640 135663704 4291717696 18446744073709551615 134544183 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29526/statm: 254485 191 113 282 0 254201 0
[pid=29527] ppid=29518 vsize=289424 CPUtime=0 cores=1,3,5,7
/proc/29527/stat : 29527 (python) S 29518 29518 28639 0 -1 4202560 753 0 0 0 0 0 0 0 15 0 1 0 538587828 296370176 5714 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 5777361862211371271 0 0 17 3 0 0 0
/proc/29527/statm: 72356 5714 348 364 0 8271 0
[pid=29529] ppid=29527 vsize=3176 CPUtime=2.46 cores=1,3,5,7
/proc/29529/stat : 29529 (march_hi) R 29527 29529 29529 0 -1 4202496 810 0 0 0 246 0 0 0 25 0 1 0 538587828 3252224 549 33554432000 4194304 4955518 140733462883936 18446744073709551615 4227970 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29529/statm: 794 549 104 186 0 607 0
[pid=29530] ppid=29518 vsize=299320 CPUtime=0 cores=1,3,5,7
/proc/29530/stat : 29530 (python) S 29518 29518 28639 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 538587831 306503680 5627 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 4656906736949494023 0 0 17 7 0 0 0
/proc/29530/statm: 74830 5627 348 364 0 10742 0
[pid=29532] ppid=29530 vsize=1017940 CPUtime=2.43 cores=1,3,5,7
/proc/29532/stat : 29532 (gnovelty+2) R 29530 29532 29532 0 -1 4202496 454 0 0 0 243 0 0 0 25 0 1 0 538587831 1042370560 192 33554432000 134512640 135663704 4286982224 18446744073709551615 134544183 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29532/statm: 254485 192 113 282 0 254201 0
[pid=29533] ppid=29518 vsize=310036 CPUtime=0 cores=1,3,5,7
/proc/29533/stat : 29533 (python) S 29518 29518 28639 0 -1 4202560 767 0 0 0 0 0 0 0 15 0 1 0 538587834 317476864 5730 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 3460418183603650822 0 0 17 7 0 0 0
/proc/29533/statm: 77509 5730 348 364 0 13418 0
[pid=29535] ppid=29533 vsize=3176 CPUtime=2.39 cores=1,3,5,7
/proc/29535/stat : 29535 (march_hi) R 29533 29535 29535 0 -1 4202496 810 0 0 0 239 0 0 0 25 0 1 0 538587834 3252224 546 33554432000 4194304 4955518 140736003407664 18446744073709551615 4289863 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29535/statm: 794 546 104 186 0 607 0
[pid=29518/tid=29525] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29525/stat : 29525 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 5 0 538587825 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 1 0 0 0
[pid=29518/tid=29528] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29528/stat : 29528 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 21 0 5 0 538587828 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 3 0 0 0
[pid=29518/tid=29531] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29531/stat : 29531 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 23 0 5 0 538587831 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29534] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29534/stat : 29534 (python) S 29516 29518 28639 0 -1 4202560 27 933 0 0 0 0 0 0 25 0 5 0 538587834 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 10.35
Current children cumulated vsize (KiB) 3539864

[startup+6.30048 s]
/proc/loadavg: 9.18 9.11 8.27 11/264 29535
/proc/meminfo: memFree=28703092/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=320280 CPUtime=0.58 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 12349 933 1 0 50 8 0 0 25 0 5 0 538587765 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29518/statm: 80070 7267 1881 364 0 15979 0
[pid=29524] ppid=29518 vsize=278572 CPUtime=0 cores=1,3,5,7
/proc/29524/stat : 29524 (python) S 29518 29518 28639 0 -1 4202560 896 0 0 0 0 0 0 0 18 0 1 0 538587825 285257728 5594 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166479 0 0 16781312 514 5637727733774516486 0 0 17 3 0 0 0
/proc/29524/statm: 69643 5594 376 364 0 5561 0
[pid=29526] ppid=29524 vsize=1017940 CPUtime=5.69 cores=1,3,5,7
/proc/29526/stat : 29526 (gnovelty+2) R 29524 29526 29526 0 -1 4202496 453 0 0 0 569 0 0 0 25 0 1 0 538587825 1042370560 191 33554432000 134512640 135663704 4291717696 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29526/statm: 254485 191 113 282 0 254201 0
[pid=29527] ppid=29518 vsize=289424 CPUtime=0 cores=1,3,5,7
/proc/29527/stat : 29527 (python) S 29518 29518 28639 0 -1 4202560 905 0 0 0 0 0 0 0 15 0 1 0 538587828 296370176 5736 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 5777361862211371271 0 0 17 3 0 0 0
/proc/29527/statm: 72356 5736 370 364 0 8271 0
[pid=29529] ppid=29527 vsize=3892 CPUtime=5.64 cores=1,3,5,7
/proc/29529/stat : 29529 (march_hi) R 29527 29529 29529 0 -1 4202496 958 0 0 0 564 0 0 0 25 0 1 0 538587828 3985408 697 33554432000 4194304 4955518 140733462883936 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29529/statm: 973 697 104 186 0 786 0
[pid=29530] ppid=29518 vsize=299320 CPUtime=0 cores=1,3,5,7
/proc/29530/stat : 29530 (python) S 29518 29518 28639 0 -1 4202560 892 0 0 0 0 0 0 0 18 0 1 0 538587831 306503680 5650 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 4656906736949494023 0 0 17 7 0 0 0
/proc/29530/statm: 74830 5650 371 364 0 10742 0
[pid=29532] ppid=29530 vsize=1017940 CPUtime=5.63 cores=1,3,5,7
/proc/29532/stat : 29532 (gnovelty+2) R 29530 29532 29532 0 -1 4202496 454 0 0 0 563 0 0 0 25 0 1 0 538587831 1042370560 192 33554432000 134512640 135663704 4286982224 18446744073709551615 134545772 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29532/statm: 254485 192 113 282 0 254201 0
[pid=29533] ppid=29518 vsize=310036 CPUtime=0 cores=1,3,5,7
/proc/29533/stat : 29533 (python) S 29518 29518 28639 0 -1 4202560 925 0 0 0 0 0 0 0 15 0 1 0 538587834 317476864 5753 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 3460418183603650822 0 0 17 7 0 0 0
/proc/29533/statm: 77509 5753 371 364 0 13418 0
[pid=29535] ppid=29533 vsize=3892 CPUtime=5.54 cores=1,3,5,7
/proc/29535/stat : 29535 (march_hi) R 29533 29535 29535 0 -1 4202496 955 0 0 0 554 0 0 0 25 0 1 0 538587834 3985408 691 33554432000 4194304 4955518 140736003407664 18446744073709551615 4209628 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29535/statm: 973 691 104 186 0 786 0
[pid=29518/tid=29525] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29525/stat : 29525 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 5 0 538587825 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 1 0 0 0
[pid=29518/tid=29528] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29528/stat : 29528 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 21 0 5 0 538587828 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 3 0 0 0
[pid=29518/tid=29531] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29531/stat : 29531 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 23 0 5 0 538587831 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29534] ppid=29516 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29534/stat : 29534 (python) S 29516 29518 28639 0 -1 4202560 27 933 0 0 0 0 0 0 25 0 5 0 538587834 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.08
Current children cumulated vsize (KiB) 3541296

[startup+12.7014 s]
/proc/loadavg: 9.41 9.16 8.29 10/264 29535
/proc/meminfo: memFree=28698544/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=320280 CPUtime=0.58 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 12349 933 1 0 50 8 0 0 25 0 5 0 538587765 327966720 7267 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29518/statm: 80070 7267 1881 364 0 15979 0
[pid=29524] ppid=29518 vsize=278572 CPUtime=0.01 cores=1,3,5,7
/proc/29524/stat : 29524 (python) S 29518 29518 28639 0 -1 4202560 900 0 0 0 1 0 0 0 18 0 1 0 538587825 285257728 5595 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166479 0 0 16781312 514 10812949191270039814 0 0 17 3 0 0 0
/proc/29524/statm: 69643 5595 377 364 0 5561 0
[pid=29526] ppid=29524 vsize=1017940 CPUtime=12.08 cores=1,3,5,7
/proc/29526/stat : 29526 (gnovelty+2) R 29524 29526 29526 0 -1 4202496 453 0 0 0 1208 0 0 0 25 0 1 0 538587825 1042370560 191 33554432000 134512640 135663704 4291717696 18446744073709551615 134544242 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29526/statm: 254485 191 113 282 0 254201 0
[pid=29527] ppid=29518 vsize=289424 CPUtime=0.02 cores=1,3,5,7

################
# More data... #
################

[pid=29541] ppid=29518 vsize=320292 CPUtime=1.7 cores=1,3,5,7
/proc/29541/stat : 29541 (python) S 29518 29518 28639 0 -1 4202560 1167 0 0 0 90 80 0 0 15 0 2 0 538597844 327979008 5787 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29541/statm: 80073 5787 400 364 0 15979 0
[pid=29543] ppid=29541 vsize=173412 CPUtime=973.35 cores=1,3,5,7
/proc/29543/stat : 29543 (march_hi) T 29541 29543 29543 0 -1 4202496 33074 0 0 0 97328 7 0 0 25 0 1 0 538597844 177573888 32811 33554432000 4194304 4955518 140735893641040 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/29543/statm: 43353 32811 104 186 0 43166 0
[pid=29541/tid=29616] ppid=29518 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/29541/task/29616/stat : 29616 (python) S 29518 29518 28639 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538646808 327979008 5787 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604470982011776 0 0 -1 5 0 0 0
[pid=29545] ppid=29518 vsize=330156 CPUtime=1.2 cores=1,3,5,7
/proc/29545/stat : 29545 (python) S 29518 29518 28639 0 -1 4202560 1177 0 0 0 69 51 0 0 16 0 2 0 538597851 338079744 5697 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29545/statm: 82539 5697 400 364 0 18442 0
[pid=29547] ppid=29545 vsize=110236 CPUtime=684.35 cores=1,3,5,7
/proc/29547/stat : 29547 (march_hi) T 29545 29547 29547 0 -1 4202496 23771 0 0 0 68421 14 0 0 25 0 1 0 538597852 112881664 23506 33554432000 4194304 4955518 140736866864256 18446744073709551615 4238717 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29547/statm: 27559 23506 104 186 0 27372 0
[pid=29545/tid=29615] ppid=29518 vsize=330156 CPUtime=0 cores=1,3,5,7
/proc/29545/task/29615/stat : 29615 (python) S 29518 29518 28639 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538646590 338079744 5697 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604466881863936 0 0 -1 7 0 0 0
[pid=29617] ppid=29518 vsize=340524 CPUtime=1.08 cores=1,3,5,7
/proc/29617/stat : 29617 (python) S 29518 29518 28639 0 -1 4202560 1233 0 0 0 55 53 0 0 15 0 2 0 538647472 348696576 5734 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/29617/statm: 85131 5734 406 364 0 21031 0
[pid=29619] ppid=29617 vsize=110236 CPUtime=603.76 cores=1,3,5,7
/proc/29619/stat : 29619 (march_hi) R 29617 29619 29619 0 -1 4202496 21335 0 0 0 60373 3 0 0 25 0 1 0 538647473 112881664 21068 33554432000 4194304 4955518 140735186184528 18446744073709551615 4314026 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29619/statm: 27559 21068 104 186 0 27372 0
[pid=29617/tid=29646] ppid=29518 vsize=340524 CPUtime=0 cores=1,3,5,7
/proc/29617/task/29646/stat : 29646 (python) S 29518 29518 28639 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 538696177 348696576 5734 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604471130308800 0 0 -1 7 0 0 0
[pid=29637] ppid=29518 vsize=351268 CPUtime=0.41 cores=1,3,5,7
/proc/29637/stat : 29637 (python) S 29518 29518 28639 0 -1 4202560 1045 0 0 0 22 19 0 0 15 0 1 0 538685689 359698432 5852 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 10812949191270039812 0 0 17 1 0 0 0
/proc/29637/statm: 87817 5852 377 364 0 23714 0
[pid=29639] ppid=29637 vsize=51568 CPUtime=225.07 cores=1,3,5,7
/proc/29639/stat : 29639 (march_hi) R 29637 29639 29639 0 -1 4202496 8778 0 0 0 22505 2 0 0 25 0 1 0 538685690 52805632 8506 33554432000 4194304 4955518 140734792743248 18446744073709551615 4263703 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29639/statm: 12892 8506 104 186 0 12705 0
[pid=29643] ppid=29518 vsize=361524 CPUtime=0.21 cores=1,3,5,7
/proc/29643/stat : 29643 (python) S 29518 29518 28639 0 -1 4202560 993 0 0 0 12 9 0 0 18 0 1 0 538695717 370200576 5850 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 10812949191270039812 0 0 17 7 0 0 0
/proc/29643/statm: 90381 5850 371 364 0 26275 0
[pid=29645] ppid=29643 vsize=182580 CPUtime=124.99 cores=1,3,5,7
/proc/29645/stat : 29645 (kcnfs-2006) R 29643 29645 29645 0 -1 4202496 653 0 0 0 12499 0 0 0 25 0 1 0 538695717 186961920 388 33554432000 134512640 135163436 4294907840 18446744073709551615 134675333 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29645/statm: 45645 388 77 159 0 45485 0
[pid=29518/tid=29525] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29525/stat : 29525 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 10 0 538587825 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 1 0 0 0
[pid=29518/tid=29528] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29528/stat : 29528 (python) S 29516 29518 28639 0 -1 4202560 35 933 0 0 0 0 0 0 18 0 10 0 538587828 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29531] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29531/stat : 29531 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 23 0 10 0 538587831 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29534] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29534/stat : 29534 (python) S 29516 29518 28639 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 10 0 538587834 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29542] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29542/stat : 29542 (python) S 29516 29518 28639 0 -1 4202560 43 933 0 0 0 0 0 0 18 0 10 0 538597844 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29546] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29546/stat : 29546 (python) S 29516 29518 28639 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 10 0 538597852 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29618] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29618/stat : 29618 (python) S 29516 29518 28639 0 -1 4202560 35 933 0 0 0 0 0 0 18 0 10 0 538647472 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29638] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29638/stat : 29638 (python) S 29516 29518 28639 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 10 0 538685689 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29644] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29644/stat : 29644 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 10 0 538695717 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 6273572

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.7 s]
/proc/loadavg: 9.20 9.16 8.89 10/294 29653
/proc/meminfo: memFree=28141596/32951132 swapFree=67080200/67111528
[pid=29518] ppid=29516 vsize=371476 CPUtime=1.23 cores=1,3,5,7
/proc/29518/stat : 29518 (python) S 29516 29518 28639 0 -1 4202496 18166 933 1 0 114 9 0 0 18 0 10 0 538587765 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/29518/statm: 92869 7277 1892 364 0 28763 0
[pid=29524] ppid=29518 vsize=288816 CPUtime=0.16 cores=1,3,5,7
/proc/29524/stat : 29524 (python) S 29518 29518 28639 0 -1 4202560 1080 0 0 0 8 8 0 0 19 0 2 0 538587825 295747584 5633 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29524/statm: 72204 5633 410 364 0 8122 0
[pid=29526] ppid=29524 vsize=1017940 CPUtime=100.02 cores=1,3,5,7
/proc/29526/stat : 29526 (gnovelty+2) T 29524 29526 29526 0 -1 4202496 453 0 0 0 9988 14 0 0 25 0 1 0 538587825 1042370560 191 33554432000 134512640 135663704 4291717696 18446744073709551615 134524551 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/29526/statm: 254485 191 113 282 0 254201 0
[pid=29524/tid=29540] ppid=29518 vsize=288816 CPUtime=0 cores=1,3,5,7
/proc/29524/task/29540/stat : 29540 (python) S 29518 29518 28639 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538597841 295747584 5633 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604454225329024 0 0 -1 3 0 0 0
[pid=29527] ppid=29518 vsize=289424 CPUtime=1.77 cores=1,3,5,7
/proc/29527/stat : 29527 (python) S 29518 29518 28639 0 -1 4202560 1133 0 0 0 99 78 0 0 15 0 2 0 538587828 296370176 5767 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/29527/statm: 72356 5767 400 364 0 8271 0
[pid=29529] ppid=29527 vsize=173412 CPUtime=1003.9 cores=1,3,5,7
/proc/29529/stat : 29529 (march_hi) R 29527 29529 29529 0 -1 4202496 34019 0 0 0 100344 46 0 0 25 0 1 0 538587828 177573888 33758 33554432000 4194304 4955518 140733462883936 18446744073709551615 4227679 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29529/statm: 43353 33758 104 186 0 43166 0
[pid=29527/tid=29550] ppid=29518 vsize=289424 CPUtime=0 cores=1,3,5,7
/proc/29527/task/29550/stat : 29550 (python) S 29518 29518 28639 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538607559 296370176 5767 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604471113726976 0 0 -1 3 0 0 0
[pid=29530] ppid=29518 vsize=299320 CPUtime=0.16 cores=1,3,5,7
/proc/29530/stat : 29530 (python) S 29518 29518 28639 0 -1 4202560 1084 0 0 0 10 6 0 0 19 0 2 0 538587831 306503680 5680 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/29530/statm: 74830 5680 400 364 0 10742 0
[pid=29532] ppid=29530 vsize=1017940 CPUtime=100.16 cores=1,3,5,7
/proc/29532/stat : 29532 (gnovelty+2) T 29530 29532 29532 0 -1 4202496 454 0 0 0 10016 0 0 0 25 0 1 0 538587831 1042370560 192 33554432000 134512640 135663704 4286982224 18446744073709551615 134525187 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29532/statm: 254485 192 113 282 0 254201 0
[pid=29530/tid=29544] ppid=29518 vsize=299320 CPUtime=0 cores=1,3,5,7
/proc/29530/task/29544/stat : 29544 (python) S 29518 29518 28639 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538597848 306503680 5680 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604466881865728 0 0 -1 7 0 0 0
[pid=29533] ppid=29518 vsize=310036 CPUtime=1.7 cores=1,3,5,7
/proc/29533/stat : 29533 (python) S 29518 29518 28639 0 -1 4202560 1161 0 0 0 90 80 0 0 15 0 2 0 538587834 317476864 5783 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/29533/statm: 77509 5783 400 364 0 13418 0
[pid=29535] ppid=29533 vsize=173412 CPUtime=975.08 cores=1,3,5,7
/proc/29535/stat : 29535 (march_hi) T 29533 29535 29535 0 -1 4202496 33165 0 0 0 97502 6 0 0 25 0 1 0 538587834 177573888 32901 33554432000 4194304 4955518 140736003407664 18446744073709551615 4238735 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29535/statm: 43353 32901 104 186 0 43166 0
[pid=29533/tid=29600] ppid=29518 vsize=310036 CPUtime=0 cores=1,3,5,7
/proc/29533/task/29600/stat : 29600 (python) S 29518 29518 28639 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538626985 317476864 5783 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604470979873536 0 0 -1 7 0 0 0
[pid=29541] ppid=29518 vsize=320292 CPUtime=1.7 cores=1,3,5,7
/proc/29541/stat : 29541 (python) S 29518 29518 28639 0 -1 4202560 1167 0 0 0 90 80 0 0 15 0 2 0 538597844 327979008 5787 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29541/statm: 80073 5787 400 364 0 15979 0
[pid=29543] ppid=29541 vsize=173412 CPUtime=973.35 cores=1,3,5,7
/proc/29543/stat : 29543 (march_hi) T 29541 29543 29543 0 -1 4202496 33074 0 0 0 97328 7 0 0 25 0 1 0 538597844 177573888 32811 33554432000 4194304 4955518 140735893641040 18446744073709551615 4227502 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/29543/statm: 43353 32811 104 186 0 43166 0
[pid=29541/tid=29616] ppid=29518 vsize=320292 CPUtime=0 cores=1,3,5,7
/proc/29541/task/29616/stat : 29616 (python) S 29518 29518 28639 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538646808 327979008 5787 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604470982011776 0 0 -1 5 0 0 0
[pid=29545] ppid=29518 vsize=330156 CPUtime=1.2 cores=1,3,5,7
/proc/29545/stat : 29545 (python) S 29518 29518 28639 0 -1 4202560 1177 0 0 0 69 51 0 0 16 0 2 0 538597851 338079744 5697 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/29545/statm: 82539 5697 400 364 0 18442 0
[pid=29547] ppid=29545 vsize=110236 CPUtime=684.35 cores=1,3,5,7
/proc/29547/stat : 29547 (march_hi) T 29545 29547 29547 0 -1 4202496 23771 0 0 0 68421 14 0 0 25 0 1 0 538597852 112881664 23506 33554432000 4194304 4955518 140736866864256 18446744073709551615 4238717 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29547/statm: 27559 23506 104 186 0 27372 0
[pid=29545/tid=29615] ppid=29518 vsize=330156 CPUtime=0 cores=1,3,5,7
/proc/29545/task/29615/stat : 29615 (python) S 29518 29518 28639 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538646590 338079744 5697 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604466881863936 0 0 -1 7 0 0 0
[pid=29617] ppid=29518 vsize=340524 CPUtime=1.08 cores=1,3,5,7
/proc/29617/stat : 29617 (python) S 29518 29518 28639 0 -1 4202560 1233 0 0 0 55 53 0 0 15 0 2 0 538647472 348696576 5734 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/29617/statm: 85131 5734 406 364 0 21031 0
[pid=29619] ppid=29617 vsize=110236 CPUtime=603.76 cores=1,3,5,7
/proc/29619/stat : 29619 (march_hi) R 29617 29619 29619 0 -1 4202496 21335 0 0 0 60373 3 0 0 25 0 1 0 538647473 112881664 21068 33554432000 4194304 4955518 140735186184528 18446744073709551615 4314026 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29619/statm: 27559 21068 104 186 0 27372 0
[pid=29617/tid=29646] ppid=29518 vsize=340524 CPUtime=0 cores=1,3,5,7
/proc/29617/task/29646/stat : 29646 (python) S 29518 29518 28639 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 538696177 348696576 5734 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 514 18446604471130308800 0 0 -1 7 0 0 0
[pid=29637] ppid=29518 vsize=351268 CPUtime=0.41 cores=1,3,5,7
/proc/29637/stat : 29637 (python) S 29518 29518 28639 0 -1 4202560 1045 0 0 0 22 19 0 0 15 0 1 0 538685689 359698432 5852 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 10812949191270039812 0 0 17 1 0 0 0
/proc/29637/statm: 87817 5852 377 364 0 23714 0
[pid=29639] ppid=29637 vsize=51568 CPUtime=225.07 cores=1,3,5,7
/proc/29639/stat : 29639 (march_hi) R 29637 29639 29639 0 -1 4202496 8778 0 0 0 22505 2 0 0 25 0 1 0 538685690 52805632 8506 33554432000 4194304 4955518 140734792743248 18446744073709551615 4263703 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29639/statm: 12892 8506 104 186 0 12705 0
[pid=29643] ppid=29518 vsize=361524 CPUtime=0.21 cores=1,3,5,7
/proc/29643/stat : 29643 (python) S 29518 29518 28639 0 -1 4202560 993 0 0 0 12 9 0 0 18 0 1 0 538695717 370200576 5850 33554432000 4194304 5683644 140736365752832 18446744073709551615 210647166534 0 0 16781312 514 10812949191270039812 0 0 17 7 0 0 0
/proc/29643/statm: 90381 5850 371 364 0 26275 0
[pid=29645] ppid=29643 vsize=182580 CPUtime=124.99 cores=1,3,5,7
/proc/29645/stat : 29645 (kcnfs-2006) R 29643 29645 29645 0 -1 4202496 653 0 0 0 12499 0 0 0 25 0 1 0 538695717 186961920 388 33554432000 134512640 135163436 4294907840 18446744073709551615 134675333 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29645/statm: 45645 388 77 159 0 45485 0
[pid=29518/tid=29525] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29525/stat : 29525 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 10 0 538587825 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 1 0 0 0
[pid=29518/tid=29528] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29528/stat : 29528 (python) S 29516 29518 28639 0 -1 4202560 35 933 0 0 0 0 0 0 18 0 10 0 538587828 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29531] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29531/stat : 29531 (python) S 29516 29518 28639 0 -1 4202560 23 933 0 0 0 0 0 0 23 0 10 0 538587831 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29534] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29534/stat : 29534 (python) S 29516 29518 28639 0 -1 4202560 36 933 0 0 0 0 0 0 18 0 10 0 538587834 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29542] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29542/stat : 29542 (python) S 29516 29518 28639 0 -1 4202560 43 933 0 0 0 0 0 0 18 0 10 0 538597844 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29546] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29546/stat : 29546 (python) S 29516 29518 28639 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 10 0 538597852 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29618] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29618/stat : 29618 (python) S 29516 29518 28639 0 -1 4202560 35 933 0 0 0 0 0 0 18 0 10 0 538647472 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29638] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29638/stat : 29638 (python) S 29516 29518 28639 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 10 0 538685689 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
[pid=29518/tid=29644] ppid=29516 vsize=371476 CPUtime=0 cores=1,3,5,7
/proc/29518/task/29644/stat : 29644 (python) S 29516 29518 28639 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 10 0 538695717 380391424 7277 33554432000 4194304 5683644 140736365752832 18446744073709551615 210658970033 0 0 16781312 2 18446604462124546752 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 6273572

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29518 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=160823
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=101984
# CPU time returned by wait4() is 1.26281
# while last known CPU time is 4800.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1204.76
CPU time (s): 4800.3
CPU user time (s): 4795.45
CPU system time (s): 4.85
CPU usage (%): 398.443
Max. virtual memory (cumulated for all children) (KiB): 6273572

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.16082
system time used= 0.101984
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19243
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1146
involuntary context switches= 63

runsolver used 3.14952 second user time and 7.03793 second system time

The end

Launcher Data 

Begin job on node129 at 2011-05-01 19:11:21
IDJOB=3317501
IDBENCH=82966
IDSOLVER=1705
FILE ID=node129/3317501-1304269881
RUNJOBID= node129-1304267282-28658
PBS_JOBID= 13173950
Free space on /tmp= 67300 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v650-c2769-S29470018-050.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317501-1304269881/watcher-3317501-1304269881 -o /tmp/evaluation-result-3317501-1304269881/solver-3317501-1304269881 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317501-1304269881.cnf -seed 1401861513 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 27350cd58dd0500df065558763d7d963
RANDOM SEED=1401861513

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      28751688 kB
Buffers:        315044 kB
Cached:        1733312 kB
SwapCached:       5128 kB
Active:        2082192 kB
Inactive:      1746712 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      28751688 kB
SwapTotal:    67111528 kB
SwapFree:     67080200 kB
Dirty:           31976 kB
Writeback:           0 kB
AnonPages:     1778128 kB
Mapped:          22728 kB
Slab:           293524 kB
PageTables:      18392 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 26451088 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 67268 MiB
End job on node129 at 2011-05-01 19:31:28