Trace number 3317490

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.33 1203.26

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S1647930270-030.UNKNOWN.cnf
MD5SUMc50e43fe8603bb92c9367487557e99ee
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.32/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.32/0.46	c INFO: solving HOME/instance-3317490-1304269839.cnf
0.32/0.46	c INFO: computing task features
0.32/0.48	c INFO: computed variable-clause graph statistics
0.32/0.48	c INFO: computed clause balance statistics
0.40/0.55	c INFO: computed variable balance statistics
0.40/0.59	c INFO: computed small-clause counts
0.40/0.59	c INFO: computed Horn-clause counts
0.40/0.59	c INFO: computed variable graph statistics
0.40/0.59	c INFO: collected features for HOME/instance-3317490-1304269839.cnf in 0.13 s
0.50/0.62	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.63)
0.50/0.65	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.07)
0.50/0.68	c INFO: running march_hi@0 for 485 with 4799 remaining (b = 0.34)
0.60/0.71	c INFO: running march_hi@0 for 485 with 4799 remaining (b = 0.31)
400.31/100.82	c INFO: running march_hi@0 for 485 with 4699 remaining (b = 0.28)
400.31/100.85	c INFO: running march_hi@0 for 485 with 4699 remaining (b = 0.25)
1941.90/486.94	c INFO: running march_hi@100 for 97 with 4313 remaining (b = 0.06)
1957.46/490.81	c INFO: running march_hi@0 for 291 with 4309 remaining (b = 0.16)
2347.16/588.44	c INFO: running march_hi@585 for 97 with 4212 remaining (b = 0.09)
2361.52/592.00	c INFO: running march_hi@485 for 97 with 4208 remaining (b = 0.06)
2365.14/592.91	c INFO: running march_hi@99 for 97 with 4207 remaining (b = 0.06)
2741.19/687.17	c INFO: running march_hi@0 for 485 with 4113 remaining (b = 0.22)
2742.38/687.47	c INFO: running march_hi@1075 for 291 with 4113 remaining (b = 0.14)
2745.96/688.34	c INFO: running march_hi@198 for 194 with 4112 remaining (b = 0.13)
3133.96/785.59	c INFO: running march_hi@100 for 291 with 4014 remaining (b = 0.15)
3527.20/884.21	c INFO: running march_hi@490 for 194 with 3916 remaining (b = 0.10)
3895.65/976.59	c INFO: running march_hi@0 for 388 with 3823 remaining (b = 0.17)
4299.57/1077.74	c INFO: running march_hi@1171 for 970 with 3722 remaining (b = 0.35)
4305.56/1079.22	c INFO: running march_hi@294 for 97 with 3721 remaining (b = 0.07)
4680.77/1173.29	c INFO: running march_hi@95 for 291 with 3627 remaining (b = 0.13)
4707.90/1180.07	c INFO: running march_hi@0 for 485 with 3620 remaining (b = 0.18)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317490-1304269839/watcher-3317490-1304269839 -o /tmp/evaluation-result-3317490-1304269839/solver-3317490-1304269839 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317490-1304269839.cnf -seed 631078968 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.43 8.25 7.62 6/254 787
/proc/meminfo: memFree=23995856/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=15080 CPUtime=0 cores=1,3,5,7
/proc/787/stat : 787 (python) S 785 787 31940 0 -1 4202496 618 0 0 0 0 0 0 0 25 0 1 0 519400050 15441920 514 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106766848 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/787/statm: 3770 514 300 364 0 257 0

[startup+0.0698961 s]
/proc/loadavg: 8.43 8.25 7.62 6/254 787
/proc/meminfo: memFree=23995856/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=54536 CPUtime=0.04 cores=1,3,5,7
/proc/787/stat : 787 (python) R 785 787 31940 0 -1 4202496 2057 0 0 0 3 1 0 0 25 0 1 0 519400050 55844864 1814 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106764645 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/787/statm: 13634 1814 492 364 0 1284 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 54536

[startup+0.100852 s]
/proc/loadavg: 8.43 8.25 7.62 6/254 787
/proc/meminfo: memFree=23995856/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=104228 CPUtime=0.07 cores=1,3,5,7
/proc/787/stat : 787 (python) R 785 787 31940 0 -1 4202496 3072 0 0 0 5 2 0 0 17 0 1 0 519400050 106729472 2752 33554432000 4194304 5683644 140736872090928 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/787/statm: 26057 2752 847 364 0 1805 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104228

[startup+0.300854 s]
/proc/loadavg: 8.43 8.25 7.62 6/254 787
/proc/meminfo: memFree=23995856/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=187692 CPUtime=0.23 cores=1,3,5,7
/proc/787/stat : 787 (python) R 785 787 31940 0 -1 4202496 6829 936 1 0 16 7 0 0 17 0 1 0 519400050 192196608 4883 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106766848 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/787/statm: 46923 4883 1399 364 0 3724 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 187692

[startup+0.700747 s]
/proc/loadavg: 8.43 8.25 7.62 6/254 787
/proc/meminfo: memFree=23995856/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=309980 CPUtime=0.6 cores=1,3,5,7
/proc/787/stat : 787 (python) R 785 787 31940 0 -1 4202496 11979 936 1 0 50 10 0 0 23 0 4 0 519400050 317419520 7249 33554432000 4194304 5683644 140736872090928 18446744073709551615 255110207318 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/787/statm: 77495 7249 1878 364 0 13407 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 309980

[startup+1.5006 s]
/proc/loadavg: 8.43 8.25 7.62 12/267 804
/proc/meminfo: memFree=23956560/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=320236 CPUtime=0.61 cores=1,3,5,7
/proc/787/stat : 787 (python) S 785 787 31940 0 -1 4202496 12406 936 1 0 51 10 0 0 25 0 5 0 519400050 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/787/statm: 80059 7256 1881 364 0 15968 0
[pid=793] ppid=787 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/793/stat : 793 (python) S 787 787 31940 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 519400112 285237248 5572 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788879 0 0 16781312 514 7306962306598961151 0 0 17 1 0 0 0
/proc/793/statm: 69638 5572 354 364 0 5556 0
[pid=795] ppid=793 vsize=1017800 CPUtime=0.87 cores=1,3,5,7
/proc/795/stat : 795 (gnovelty+2) R 793 795 795 0 -1 4202496 442 0 0 0 87 0 0 0 21 0 1 0 519400113 1042227200 180 33554432000 134512640 135663704 4294837360 18446744073709551615 134547151 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/795/statm: 254450 180 113 282 0 254166 0
[pid=796] ppid=787 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/796/stat : 796 (python) S 787 787 31940 0 -1 4202560 741 0 0 0 0 0 0 0 20 0 1 0 519400115 296288256 5694 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 4478239945726623743 0 0 17 5 0 0 0
/proc/796/statm: 72336 5694 348 364 0 8251 0
[pid=798] ppid=796 vsize=1017800 CPUtime=0.84 cores=1,3,5,7
/proc/798/stat : 798 (gnovelty+2) R 796 798 798 0 -1 4202496 444 0 0 0 84 0 0 0 25 0 1 0 519400116 1042227200 181 33554432000 134512640 135663704 4289547904 18446744073709551615 134524894 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/798/statm: 254450 181 113 282 0 254166 0
[pid=799] ppid=787 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/799/stat : 799 (python) S 787 787 31940 0 -1 4202560 759 0 0 0 0 0 0 0 15 0 1 0 519400118 306307072 5595 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 12854900618019438856 0 0 17 7 0 0 0
/proc/799/statm: 74782 5595 348 364 0 10694 0
[pid=801] ppid=799 vsize=1992 CPUtime=0.8 cores=1,3,5,7
/proc/801/stat : 801 (march_hi) R 799 801 801 0 -1 4202496 604 0 0 0 80 0 0 0 25 0 1 0 519400119 2039808 340 33554432000 4194304 4955518 140734078280576 18446744073709551615 4208669 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/801/statm: 498 340 106 186 0 311 0
[pid=802] ppid=787 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/802/stat : 802 (python) S 787 787 31940 0 -1 4202560 763 0 0 0 0 0 0 0 20 0 1 0 519400121 317431808 5719 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 17300270109582917895 0 0 17 5 0 0 0
/proc/802/statm: 77498 5719 348 364 0 13407 0
[pid=804] ppid=802 vsize=1992 CPUtime=0.78 cores=1,3,5,7
/proc/804/stat : 804 (march_hi) R 802 804 804 0 -1 4202496 603 0 0 0 78 0 0 0 25 0 1 0 519400122 2039808 338 33554432000 4194304 4955518 140734674823296 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/804/statm: 498 338 106 186 0 311 0
[pid=787/tid=794] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/794/stat : 794 (python) S 785 787 31940 0 -1 4202560 25 936 0 0 0 0 0 0 19 0 5 0 519400113 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=797] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/797/stat : 797 (python) S 785 787 31940 0 -1 4202560 21 936 0 0 0 0 0 0 21 0 5 0 519400115 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=800] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/800/stat : 800 (python) S 785 787 31940 0 -1 4202560 26 936 0 0 0 0 0 0 23 0 5 0 519400118 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=803] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/803/stat : 803 (python) S 785 787 31940 0 -1 4202560 27 936 0 0 0 0 0 0 25 0 5 0 519400122 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3.9
Current children cumulated vsize (KiB) 3536836

[startup+3.10132 s]
/proc/loadavg: 8.43 8.25 7.62 11/267 804
/proc/meminfo: memFree=23954948/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=320236 CPUtime=0.61 cores=1,3,5,7
/proc/787/stat : 787 (python) S 785 787 31940 0 -1 4202496 12406 936 1 0 51 10 0 0 25 0 5 0 519400050 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/787/statm: 80059 7256 1881 364 0 15968 0
[pid=793] ppid=787 vsize=278552 CPUtime=0 cores=1,3,5,7
/proc/793/stat : 793 (python) S 787 787 31940 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 519400112 285237248 5572 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788879 0 0 16781312 514 7306962306598928646 0 0 17 1 0 0 0
/proc/793/statm: 69638 5572 354 364 0 5556 0
[pid=795] ppid=793 vsize=1017800 CPUtime=2.47 cores=1,3,5,7
/proc/795/stat : 795 (gnovelty+2) R 793 795 795 0 -1 4202496 442 0 0 0 247 0 0 0 25 0 1 0 519400113 1042227200 180 33554432000 134512640 135663704 4294837360 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/795/statm: 254450 180 113 282 0 254166 0
[pid=796] ppid=787 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/796/stat : 796 (python) S 787 787 31940 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 519400115 296288256 5694 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 4478239945726591238 0 0 17 5 0 0 0
/proc/796/statm: 72336 5694 348 364 0 8251 0
[pid=798] ppid=796 vsize=1017800 CPUtime=2.44 cores=1,3,5,7
/proc/798/stat : 798 (gnovelty+2) R 796 798 798 0 -1 4202496 444 0 0 0 244 0 0 0 25 0 1 0 519400116 1042227200 181 33554432000 134512640 135663704 4289547904 18446744073709551615 134525258 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/798/statm: 254450 181 113 282 0 254166 0
[pid=799] ppid=787 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/799/stat : 799 (python) S 787 787 31940 0 -1 4202560 765 0 0 0 0 0 0 0 15 0 1 0 519400118 306307072 5595 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 12854900618019438856 0 0 17 7 0 0 0
/proc/799/statm: 74782 5595 348 364 0 10694 0
[pid=801] ppid=799 vsize=2472 CPUtime=2.39 cores=1,3,5,7
/proc/801/stat : 801 (march_hi) R 799 801 801 0 -1 4202496 699 0 0 0 239 0 0 0 25 0 1 0 519400119 2531328 435 33554432000 4194304 4955518 140734078280576 18446744073709551615 4346565 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/801/statm: 618 435 106 186 0 431 0
[pid=802] ppid=787 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/802/stat : 802 (python) S 787 787 31940 0 -1 4202560 766 0 0 0 0 0 0 0 15 0 1 0 519400121 317431808 5719 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 10767077020698378503 0 0 17 5 0 0 0
/proc/802/statm: 77498 5719 348 364 0 13407 0
[pid=804] ppid=802 vsize=2472 CPUtime=2.38 cores=1,3,5,7
/proc/804/stat : 804 (march_hi) R 802 804 804 0 -1 4202496 699 0 0 0 238 0 0 0 25 0 1 0 519400122 2531328 434 33554432000 4194304 4955518 140734674823296 18446744073709551615 4224471 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/804/statm: 618 434 106 186 0 431 0
[pid=787/tid=794] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/794/stat : 794 (python) S 785 787 31940 0 -1 4202560 25 936 0 0 0 0 0 0 19 0 5 0 519400113 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=797] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/797/stat : 797 (python) S 785 787 31940 0 -1 4202560 21 936 0 0 0 0 0 0 21 0 5 0 519400115 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=800] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/800/stat : 800 (python) S 785 787 31940 0 -1 4202560 26 936 0 0 0 0 0 0 23 0 5 0 519400118 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=803] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/803/stat : 803 (python) S 785 787 31940 0 -1 4202560 27 936 0 0 0 0 0 0 25 0 5 0 519400122 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.29
Current children cumulated vsize (KiB) 3537796

[startup+6.30076 s]
/proc/loadavg: 8.48 8.26 7.63 10/267 805
/proc/meminfo: memFree=23951188/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=320236 CPUtime=0.61 cores=1,3,5,7
/proc/787/stat : 787 (python) S 785 787 31940 0 -1 4202496 12406 936 1 0 51 10 0 0 25 0 5 0 519400050 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/787/statm: 80059 7256 1881 364 0 15968 0
[pid=793] ppid=787 vsize=278552 CPUtime=0.01 cores=1,3,5,7
/proc/793/stat : 793 (python) S 787 787 31940 0 -1 4202560 915 0 0 0 1 0 0 0 18 0 1 0 519400112 285237248 5594 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788879 0 0 16781312 514 7306962306598928646 0 0 17 1 0 0 0
/proc/793/statm: 69638 5594 376 364 0 5556 0
[pid=795] ppid=793 vsize=1017800 CPUtime=5.67 cores=1,3,5,7
/proc/795/stat : 795 (gnovelty+2) R 793 795 795 0 -1 4202496 442 0 0 0 567 0 0 0 25 0 1 0 519400113 1042227200 180 33554432000 134512640 135663704 4294837360 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/795/statm: 254450 180 113 282 0 254166 0
[pid=796] ppid=787 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/796/stat : 796 (python) S 787 787 31940 0 -1 4202560 901 0 0 0 0 0 0 0 18 0 1 0 519400115 296288256 5717 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 4478239945726591238 0 0 17 5 0 0 0
/proc/796/statm: 72336 5717 371 364 0 8251 0
[pid=798] ppid=796 vsize=1017800 CPUtime=5.63 cores=1,3,5,7
/proc/798/stat : 798 (gnovelty+2) R 796 798 798 0 -1 4202496 444 0 0 0 563 0 0 0 25 0 1 0 519400116 1042227200 181 33554432000 134512640 135663704 4289547904 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/798/statm: 254450 181 113 282 0 254166 0
[pid=799] ppid=787 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/799/stat : 799 (python) S 787 787 31940 0 -1 4202560 926 0 0 0 0 0 0 0 15 0 1 0 519400118 306307072 5618 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 12854900618019438856 0 0 17 7 0 0 0
/proc/799/statm: 74782 5618 371 364 0 10694 0
[pid=801] ppid=799 vsize=3324 CPUtime=5.55 cores=1,3,5,7
/proc/801/stat : 801 (march_hi) R 799 801 801 0 -1 4202496 870 0 0 0 555 0 0 0 25 0 1 0 519400119 3403776 606 33554432000 4194304 4955518 140734078280576 18446744073709551615 4235627 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/801/statm: 831 606 106 186 0 644 0
[pid=802] ppid=787 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/802/stat : 802 (python) S 787 787 31940 0 -1 4202560 945 0 0 0 0 0 0 0 15 0 1 0 519400121 317431808 5742 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 17300270109582917895 0 0 17 5 0 0 0
/proc/802/statm: 77498 5742 371 364 0 13407 0
[pid=804] ppid=802 vsize=3324 CPUtime=5.56 cores=1,3,5,7
/proc/804/stat : 804 (march_hi) R 802 804 804 0 -1 4202496 871 0 0 0 556 0 0 0 25 0 1 0 519400122 3403776 606 33554432000 4194304 4955518 140734674823296 18446744073709551615 4210606 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/804/statm: 831 606 106 186 0 644 0
[pid=787/tid=794] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/794/stat : 794 (python) S 785 787 31940 0 -1 4202560 25 936 0 0 0 0 0 0 19 0 5 0 519400113 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=797] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/797/stat : 797 (python) S 785 787 31940 0 -1 4202560 21 936 0 0 0 0 0 0 21 0 5 0 519400115 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=800] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/800/stat : 800 (python) S 785 787 31940 0 -1 4202560 26 936 0 0 0 0 0 0 23 0 5 0 519400118 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=803] ppid=785 vsize=320236 CPUtime=0 cores=1,3,5,7
/proc/787/task/803/stat : 803 (python) S 785 787 31940 0 -1 4202560 27 936 0 0 0 0 0 0 25 0 5 0 519400122 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.03
Current children cumulated vsize (KiB) 3539500

[startup+12.7006 s]
/proc/loadavg: 8.52 8.27 7.63 11/267 805
/proc/meminfo: memFree=23946624/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=320236 CPUtime=0.61 cores=1,3,5,7
/proc/787/stat : 787 (python) S 785 787 31940 0 -1 4202496 12406 936 1 0 51 10 0 0 25 0 5 0 519400050 327921664 7256 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/787/statm: 80059 7256 1881 364 0 15968 0
[pid=793] ppid=787 vsize=278552 CPUtime=0.01 cores=1,3,5,7
/proc/793/stat : 793 (python) S 787 787 31940 0 -1 4202560 917 0 0 0 1 0 0 0 18 0 1 0 519400112 285237248 5595 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788879 0 0 16781312 514 17454896624525345030 0 0 17 1 0 0 0
/proc/793/statm: 69638 5595 377 364 0 5556 0
[pid=795] ppid=793 vsize=1017800 CPUtime=12.07 cores=1,3,5,7
/proc/795/stat : 795 (gnovelty+2) R 793 795 795 0 -1 4202496 442 0 0 0 1207 0 0 0 25 0 1 0 519400113 1042227200 180 33554432000 134512640 135663704 4294837360 18446744073709551615 134525408 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/795/statm: 254450 180 113 282 0 254166 0
[pid=796] ppid=787 vsize=289344 CPUtime=0.01 cores=1,3,5,7

################
# More data... #
################

/proc/820/stat : 820 (march_hi) T 818 820 820 0 -1 4202496 41450 0 0 0 97312 8 0 0 25 0 1 0 519410133 219492352 41184 33554432000 4194304 4955518 140734070619008 18446744073709551615 4235624 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/820/statm: 53587 41184 106 186 0 53400 0
[pid=818/tid=860] ppid=787 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/818/task/860/stat : 860 (python) S 787 787 31940 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519459339 327933952 5776 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604471112710912 0 0 -1 5 0 0 0
[pid=821] ppid=787 vsize=330116 CPUtime=1.14 cores=1,3,5,7
/proc/821/stat : 821 (python) S 787 787 31940 0 -1 4202560 1181 0 0 0 62 52 0 0 15 0 2 0 519410136 338038784 5702 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/821/statm: 82529 5702 400 364 0 18432 0
[pid=823] ppid=821 vsize=141220 CPUtime=585.62 cores=1,3,5,7
/proc/823/stat : 823 (march_hi) T 821 823 823 0 -1 4202496 26120 0 0 0 58556 6 0 0 25 0 1 0 519410136 144609280 25852 33554432000 4194304 4955518 140735369712112 18446744073709551615 4235369 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/823/statm: 35305 25852 106 186 0 35118 0
[pid=821/tid=859] ppid=787 vsize=330116 CPUtime=0 cores=1,3,5,7
/proc/821/task/859/stat : 859 (python) S 787 787 31940 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519459248 338038784 5702 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604467498877184 0 0 -1 3 0 0 0
[pid=864] ppid=787 vsize=340940 CPUtime=0.95 cores=1,3,5,7
/proc/864/stat : 864 (python) S 787 787 31940 0 -1 4202560 1232 0 0 0 49 46 0 0 15 0 2 0 519468767 349122560 5835 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/864/statm: 85235 5835 406 364 0 21135 0
[pid=866] ppid=864 vsize=97020 CPUtime=494.31 cores=1,3,5,7
/proc/866/stat : 866 (march_hi) R 864 866 866 0 -1 4202496 22262 0 0 0 49426 5 0 0 25 0 1 0 519468768 99348480 21992 33554432000 4194304 4955518 140737170993008 18446744073709551615 4224648 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/866/statm: 24255 21992 106 186 0 24068 0
[pid=864/tid=894] ppid=787 vsize=340940 CPUtime=0 cores=1,3,5,7
/proc/864/task/894/stat : 894 (python) S 787 787 31940 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 519517377 349122560 5835 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604458925867968 0 0 -1 3 0 0 0
[pid=883] ppid=787 vsize=351196 CPUtime=0.41 cores=1,3,5,7
/proc/883/stat : 883 (python) S 787 787 31940 0 -1 4202560 1052 0 0 0 22 19 0 0 15 0 1 0 519497709 359624704 5822 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 17454896624525345032 0 0 17 3 0 0 0
/proc/883/statm: 87799 5822 377 364 0 23696 0
[pid=885] ppid=883 vsize=42788 CPUtime=204.04 cores=1,3,5,7
/proc/885/stat : 885 (march_hi) R 883 885 885 0 -1 4202496 10100 0 0 0 20402 2 0 0 25 0 1 0 519497709 43814912 9830 33554432000 4194304 4955518 140734072030144 18446744073709551615 4208614 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/885/statm: 10697 9830 106 186 0 10510 0
[pid=896] ppid=787 vsize=361452 CPUtime=0 cores=1,3,5,7
/proc/896/stat : 896 (python) S 787 787 31940 0 -1 4202560 803 0 0 0 0 0 0 0 15 0 1 0 519518058 370126848 5805 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 7486024922007830790 0 0 17 3 0 0 0
/proc/896/statm: 90363 5805 348 364 0 26257 0
[pid=898] ppid=896 vsize=2472 CPUtime=2.2 cores=1,3,5,7
/proc/898/stat : 898 (march_hi) R 896 898 898 0 -1 4202496 697 0 0 0 220 0 0 0 25 0 1 0 519518058 2531328 425 33554432000 4194304 4955518 140735801452080 18446744073709551615 4224450 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/898/statm: 618 425 106 186 0 431 0
[pid=787/tid=794] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/794/stat : 794 (python) S 785 787 31940 0 -1 4202560 25 936 0 0 0 0 0 0 19 0 10 0 519400113 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=797] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/797/stat : 797 (python) S 785 787 31940 0 -1 4202560 21 936 0 0 0 0 0 0 21 0 10 0 519400115 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=800] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/800/stat : 800 (python) S 785 787 31940 0 -1 4202560 38 936 0 0 0 0 0 0 18 0 10 0 519400118 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=803] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/803/stat : 803 (python) S 785 787 31940 0 -1 4202560 42 936 0 0 0 0 0 0 18 0 10 0 519400122 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=819] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/819/stat : 819 (python) S 785 787 31940 0 -1 4202560 32 936 0 0 0 0 0 0 18 0 10 0 519410133 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=822] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/822/stat : 822 (python) S 785 787 31940 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 10 0 519410136 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
[pid=787/tid=865] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/865/stat : 865 (python) S 785 787 31940 0 -1 4202560 30 936 0 0 0 0 0 0 18 0 10 0 519468767 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
[pid=787/tid=884] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/884/stat : 884 (python) S 785 787 31940 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 10 0 519497709 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
[pid=787/tid=897] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/897/stat : 897 (python) S 785 787 31940 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 10 0 519518058 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4717.05
Current children cumulated vsize (KiB) 6225052



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.2 s]
/proc/loadavg: 7.98 8.94 8.67 6/275 919
/proc/meminfo: memFree=22752744/32951124 swapFree=67111448/67111528
[pid=787] ppid=785 vsize=371696 CPUtime=1.09 cores=1,3,5,7
/proc/787/stat : 787 (python) S 785 787 31940 0 -1 4202496 17066 936 1 0 98 11 0 0 19 0 10 0 519400050 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/787/statm: 92924 7353 1892 364 0 28818 0
[pid=793] ppid=787 vsize=288796 CPUtime=0.17 cores=1,3,5,7
/proc/793/stat : 793 (python) S 787 787 31940 0 -1 4202560 1100 0 0 0 9 8 0 0 18 0 2 0 519400112 295727104 5633 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/793/statm: 72199 5633 410 364 0 8117 0
[pid=795] ppid=793 vsize=1017800 CPUtime=100.15 cores=1,3,5,7
/proc/795/stat : 795 (gnovelty+2) T 793 795 795 0 -1 4202496 442 0 0 0 10015 0 0 0 25 0 1 0 519400113 1042227200 180 33554432000 134512640 135663704 4294837360 18446744073709551615 134525255 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/795/statm: 254450 180 113 282 0 254166 0
[pid=793/tid=816] ppid=787 vsize=288796 CPUtime=0 cores=1,3,5,7
/proc/793/task/816/stat : 816 (python) S 787 787 31940 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 519410129 295727104 5633 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604467498879872 0 0 -1 3 0 0 0
[pid=796] ppid=787 vsize=289344 CPUtime=0.16 cores=1,3,5,7
/proc/796/stat : 796 (python) S 787 787 31940 0 -1 4202560 1083 0 0 0 7 9 0 0 19 0 2 0 519400115 296288256 5747 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/796/statm: 72336 5747 400 364 0 8251 0
[pid=798] ppid=796 vsize=1017800 CPUtime=99.99 cores=1,3,5,7
/proc/798/stat : 798 (gnovelty+2) T 796 798 798 0 -1 4202496 444 0 0 0 9999 0 0 0 25 0 1 0 519400116 1042227200 181 33554432000 134512640 135663704 4289547904 18446744073709551615 134544370 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/798/statm: 254450 181 113 282 0 254166 0
[pid=796/tid=817] ppid=787 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/796/task/817/stat : 817 (python) S 787 787 31940 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 519410132 296288256 5747 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604471135233920 0 0 -1 3 0 0 0
[pid=799] ppid=787 vsize=299128 CPUtime=2.25 cores=1,3,5,7
/proc/799/stat : 799 (python) S 787 787 31940 0 -1 4202560 1155 0 0 0 119 106 0 0 15 0 2 0 519400118 306307072 5648 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/799/statm: 74782 5648 400 364 0 10694 0
[pid=801] ppid=799 vsize=214348 CPUtime=1169.5 cores=1,3,5,7
/proc/801/stat : 801 (march_hi) T 799 801 801 0 -1 4202496 48512 0 0 0 116941 9 0 0 25 0 1 0 519400119 219492352 48248 33554432000 4194304 4955518 140734078280576 18446744073709551615 4208656 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/801/statm: 53587 48248 106 186 0 53400 0
[pid=799/tid=845] ppid=787 vsize=299128 CPUtime=0 cores=1,3,5,7
/proc/799/task/845/stat : 845 (python) S 787 787 31940 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 519449129 306307072 5648 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604466724152512 0 0 -1 7 0 0 0
[pid=802] ppid=787 vsize=309992 CPUtime=2.08 cores=1,3,5,7
/proc/802/stat : 802 (python) S 787 787 31940 0 -1 4202560 1183 0 0 0 105 103 0 0 15 0 2 0 519400121 317431808 5772 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/802/statm: 77498 5772 400 364 0 13407 0
[pid=804] ppid=802 vsize=214348 CPUtime=1098.85 cores=1,3,5,7
/proc/804/stat : 804 (march_hi) R 802 804 804 0 -1 4202496 46053 0 0 0 109876 9 0 0 25 0 1 0 519400122 219492352 45788 33554432000 4194304 4955518 140734674823296 18446744073709551615 4224738 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/804/statm: 53587 45788 106 186 0 53400 0
[pid=802/tid=844] ppid=787 vsize=309992 CPUtime=0 cores=1,3,5,7
/proc/802/task/844/stat : 844 (python) S 787 787 31940 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 519448742 317431808 5772 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604468304500608 0 0 -1 5 0 0 0
[pid=818] ppid=787 vsize=320248 CPUtime=1.89 cores=1,3,5,7
/proc/818/stat : 818 (python) S 787 787 31940 0 -1 4202560 1191 0 0 0 101 88 0 0 15 0 2 0 519410133 327933952 5776 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/818/statm: 80062 5776 400 364 0 15968 0
[pid=820] ppid=818 vsize=214348 CPUtime=973.2 cores=1,3,5,7
/proc/820/stat : 820 (march_hi) T 818 820 820 0 -1 4202496 41450 0 0 0 97312 8 0 0 25 0 1 0 519410133 219492352 41184 33554432000 4194304 4955518 140734070619008 18446744073709551615 4235624 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/820/statm: 53587 41184 106 186 0 53400 0
[pid=818/tid=860] ppid=787 vsize=320248 CPUtime=0 cores=1,3,5,7
/proc/818/task/860/stat : 860 (python) S 787 787 31940 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519459339 327933952 5776 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604471112710912 0 0 -1 5 0 0 0
[pid=821] ppid=787 vsize=330116 CPUtime=1.14 cores=1,3,5,7
/proc/821/stat : 821 (python) S 787 787 31940 0 -1 4202560 1181 0 0 0 62 52 0 0 15 0 2 0 519410136 338038784 5702 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/821/statm: 82529 5702 400 364 0 18432 0
[pid=823] ppid=821 vsize=141220 CPUtime=585.62 cores=1,3,5,7
/proc/823/stat : 823 (march_hi) T 821 823 823 0 -1 4202496 26120 0 0 0 58556 6 0 0 25 0 1 0 519410136 144609280 25852 33554432000 4194304 4955518 140735369712112 18446744073709551615 4235369 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/823/statm: 35305 25852 106 186 0 35118 0
[pid=821/tid=859] ppid=787 vsize=330116 CPUtime=0 cores=1,3,5,7
/proc/821/task/859/stat : 859 (python) S 787 787 31940 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519459248 338038784 5702 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604467498877184 0 0 -1 3 0 0 0
[pid=864] ppid=787 vsize=340940 CPUtime=1 cores=1,3,5,7
/proc/864/stat : 864 (python) S 787 787 31940 0 -1 4202560 1232 0 0 0 52 48 0 0 15 0 2 0 519468767 349122560 5835 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/864/statm: 85235 5835 406 364 0 21135 0
[pid=866] ppid=864 vsize=97020 CPUtime=515.21 cores=1,3,5,7
/proc/866/stat : 866 (march_hi) R 864 866 866 0 -1 4202496 23075 0 0 0 51516 5 0 0 25 0 1 0 519468768 99348480 22805 33554432000 4194304 4955518 140737170993008 18446744073709551615 4235379 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/866/statm: 24255 22805 106 186 0 24068 0
[pid=864/tid=894] ppid=787 vsize=340940 CPUtime=0 cores=1,3,5,7
/proc/864/task/894/stat : 894 (python) S 787 787 31940 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 519517377 349122560 5835 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 514 18446604458925867968 0 0 -1 3 0 0 0
[pid=883] ppid=787 vsize=351196 CPUtime=0.44 cores=1,3,5,7
/proc/883/stat : 883 (python) S 787 787 31940 0 -1 4202560 1055 0 0 0 24 20 0 0 15 0 1 0 519497709 359624704 5822 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 12854905016065949960 0 0 17 3 0 0 0
/proc/883/statm: 87799 5822 377 364 0 23696 0
[pid=885] ppid=883 vsize=62432 CPUtime=224.55 cores=1,3,5,7
/proc/885/stat : 885 (march_hi) R 883 885 885 0 -1 4202496 11059 0 0 0 22453 2 0 0 25 0 1 0 519497709 63930368 10789 33554432000 4194304 4955518 140734072030144 18446744073709551615 4285265 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/885/statm: 15608 10789 106 186 0 15421 0
[pid=896] ppid=787 vsize=361452 CPUtime=0.05 cores=1,3,5,7
/proc/896/stat : 896 (python) S 787 787 31940 0 -1 4202560 986 0 0 0 3 2 0 0 15 0 1 0 519518058 370126848 5828 33554432000 4194304 5683644 140736872090928 18446744073709551615 255106788934 0 0 16781312 514 7486024922007830790 0 0 17 3 0 0 0
/proc/896/statm: 90363 5828 371 364 0 26257 0
[pid=898] ppid=896 vsize=7180 CPUtime=22.99 cores=1,3,5,7
/proc/898/stat : 898 (march_hi) R 896 898 898 0 -1 4202496 1816 0 0 0 2299 0 0 0 25 0 1 0 519518058 7352320 1544 33554432000 4194304 4955518 140735801452080 18446744073709551615 4208656 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/898/statm: 1795 1544 106 186 0 1608 0
[pid=787/tid=794] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/794/stat : 794 (python) S 785 787 31940 0 -1 4202560 25 936 0 0 0 0 0 0 19 0 10 0 519400113 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=797] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/797/stat : 797 (python) S 785 787 31940 0 -1 4202560 21 936 0 0 0 0 0 0 21 0 10 0 519400115 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 1 0 0 0
[pid=787/tid=800] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/800/stat : 800 (python) S 785 787 31940 0 -1 4202560 38 936 0 0 0 0 0 0 18 0 10 0 519400118 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=803] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/803/stat : 803 (python) S 785 787 31940 0 -1 4202560 42 936 0 0 0 0 0 0 18 0 10 0 519400122 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=819] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/819/stat : 819 (python) S 785 787 31940 0 -1 4202560 32 936 0 0 0 0 0 0 18 0 10 0 519410133 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 5 0 0 0
[pid=787/tid=822] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/822/stat : 822 (python) S 785 787 31940 0 -1 4202560 28 936 0 0 0 0 0 0 18 0 10 0 519410136 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
[pid=787/tid=865] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/865/stat : 865 (python) S 785 787 31940 0 -1 4202560 30 936 0 0 0 0 0 0 18 0 10 0 519468767 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
[pid=787/tid=884] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/884/stat : 884 (python) S 785 787 31940 0 -1 4202560 30 936 0 0 0 0 0 0 20 0 10 0 519497709 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
[pid=787/tid=897] ppid=785 vsize=371696 CPUtime=0 cores=1,3,5,7
/proc/787/task/897/stat : 897 (python) S 785 787 31940 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 10 0 519518058 380616704 7353 33554432000 4194304 5683644 140736872090928 18446744073709551615 255118592433 0 0 16781312 2 18446604458925869760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.33
Current children cumulated vsize (KiB) 6249404

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 787 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=3847
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=118981
# CPU time returned by wait4() is 1.12283
# while last known CPU time is 4800.33
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1203.26
CPU time (s): 4800.33
CPU user time (s): 4795.47
CPU system time (s): 4.86
CPU usage (%): 398.943
Max. virtual memory (cumulated for all children) (KiB): 6249404

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.00385
system time used= 0.118981
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18504
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1209
involuntary context switches= 45

runsolver used 2.92556 second user time and 6.63299 second system time

The end

Launcher Data 

Begin job on node144 at 2011-05-01 19:10:39
IDJOB=3317490
IDBENCH=82955
IDSOLVER=1705
FILE ID=node144/3317490-1304269839
RUNJOBID= node144-1304267279-31959
PBS_JOBID= 13173935
Free space on /tmp= 73524 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S1647930270-030.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317490-1304269839/watcher-3317490-1304269839 -o /tmp/evaluation-result-3317490-1304269839/solver-3317490-1304269839 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317490-1304269839.cnf -seed 631078968 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c50e43fe8603bb92c9367487557e99ee
RANDOM SEED=631078968

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23996012 kB
Buffers:       2139796 kB
Cached:        4914560 kB
SwapCached:         80 kB
Active:        1741940 kB
Inactive:      6646136 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23996012 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           32284 kB
Writeback:           0 kB
AnonPages:     1332828 kB
Mapped:          25948 kB
Slab:           488672 kB
PageTables:      17100 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 33069072 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73520 MiB
End job on node144 at 2011-05-01 19:30:45