Trace number 3317349

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.23 1205.06

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v550-c2343-S1694067460-072.UNKNOWN.cnf
MD5SUMf2d2b125797e776fb2e7099729714d99
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables550
Number of clauses2343
Sum of the clauses size7029
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32343
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.39/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.39/0.67	c INFO: solving HOME/instance-3317349-1304268962.cnf
0.39/0.67	c INFO: computing task features
0.39/0.68	c INFO: computed variable-clause graph statistics
0.39/0.68	c INFO: computed clause balance statistics
0.46/0.72	c INFO: computed variable balance statistics
0.46/0.75	c INFO: computed small-clause counts
0.46/0.75	c INFO: computed Horn-clause counts
0.46/0.75	c INFO: computed variable graph statistics
0.46/0.75	c INFO: collected features for HOME/instance-3317349-1304268962.cnf in 0.09 s
0.46/0.78	c INFO: running gnovelty+2@0 for 97 with 4799 remaining (b = 0.57)
0.56/0.81	c INFO: running march_hi@0 for 388 with 4799 remaining (b = 0.33)
0.56/0.84	c INFO: running march_hi@0 for 485 with 4799 remaining (b = 0.33)
0.56/0.87	c INFO: running march_hi@0 for 97 with 4799 remaining (b = 0.09)
393.82/100.98	c INFO: running gnovelty+2@0 for 97 with 4699 remaining (b = 0.07)
404.97/103.74	c INFO: running march_hi@98 for 97 with 4696 remaining (b = 0.07)
787.86/199.65	c INFO: running march_hi@193 for 97 with 4601 remaining (b = 0.08)
793.85/201.18	c INFO: running march_hi@0 for 485 with 4599 remaining (b = 0.27)
1167.92/294.81	c INFO: running march_hi@293 for 97 with 4505 remaining (b = 0.10)
1576.26/397.13	c INFO: running march_hi@394 for 97 with 4403 remaining (b = 0.07)
1597.00/402.30	c INFO: running march_hi@0 for 97 with 4398 remaining (b = 0.07)
1939.87/488.24	c INFO: running march_hi@0 for 97 with 4312 remaining (b = 0.06)
1957.88/492.77	c INFO: running march_hi@788 for 194 with 4307 remaining (b = 0.12)
1979.41/498.10	c INFO: running march_hi@486 for 97 with 4302 remaining (b = 0.08)
2326.73/585.20	c INFO: running march_hi@95 for 291 with 4215 remaining (b = 0.19)
2359.82/593.40	c INFO: running march_hi@582 for 97 with 4207 remaining (b = 0.08)
2727.49/685.79	c INFO: running march_hi@0 for 388 with 4114 remaining (b = 0.23)
2737.46/688.30	c INFO: running march_hi@95 for 291 with 4112 remaining (b = 0.15)
2766.21/695.45	c INFO: running march_hi@883 for 97 with 4105 remaining (b = 0.05)
3166.59/795.80	c INFO: running march_hi@774 for 97 with 4004 remaining (b = 0.05)
3505.49/880.69	c INFO: running march_hi@189 for 97 with 3919 remaining (b = 0.06)
3543.00/890.05	c INFO: running march_hi@0 for 388 with 3910 remaining (b = 0.19)
3883.44/975.36	c INFO: running march_hi@283 for 194 with 3825 remaining (b = 0.11)
3922.15/985.04	c INFO: running march_hi@1069 for 291 with 3815 remaining (b = 0.19)
4301.37/1080.03	c INFO: running march_hi@196 for 194 with 3720 remaining (b = 0.11)
4670.13/1172.48	c INFO: running march_hi@485 for 97 with 3628 remaining (b = 0.04)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317349-1304268962/watcher-3317349-1304268962 -o /tmp/evaluation-result-3317349-1304268962/solver-3317349-1304268962 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317349-1304268962.cnf -seed 966247080 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.96 8.14 6.80 5/280 30927
/proc/meminfo: memFree=30119780/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=15384 CPUtime=0 cores=1,3,5,7
/proc/30927/stat : 30927 (python) R 30925 30927 30218 0 -1 4202496 820 0 0 0 0 0 0 0 25 0 1 0 538494833 15753216 704 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970593415 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/30927/statm: 3846 704 334 364 0 333 0

[startup+0.0796599 s]
/proc/loadavg: 7.96 8.14 6.80 5/280 30927
/proc/meminfo: memFree=30119780/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=50016 CPUtime=0.04 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 1949 0 0 0 3 1 0 0 25 0 1 0 538494833 51216384 1713 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970548224 0 2147483391 16781312 2 18446744071699109054 0 0 17 3 0 0 0
/proc/30927/statm: 12504 1713 476 364 0 1204 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 50016

[startup+0.100665 s]
/proc/loadavg: 7.96 8.14 6.80 5/280 30927
/proc/meminfo: memFree=30119780/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=89772 CPUtime=0.06 cores=1,3,5,7
/proc/30927/stat : 30927 (python) R 30925 30927 30218 0 -1 4202496 2673 0 0 0 5 1 0 0 17 0 1 0 538494833 91926528 2388 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970548224 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/30927/statm: 22443 2388 750 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89772

[startup+0.300626 s]
/proc/loadavg: 7.96 8.14 6.80 5/280 30927
/proc/meminfo: memFree=30119780/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=135752 CPUtime=0.19 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 5964 937 1 0 14 5 0 0 18 0 1 0 538494833 139010048 4151 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970546181 0 2147483391 16781312 2 18446744071699109054 0 0 17 3 0 0 0
/proc/30927/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135752

[startup+0.700561 s]
/proc/loadavg: 7.96 8.14 6.80 5/280 30927
/proc/meminfo: memFree=30119780/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=278452 CPUtime=0.46 cores=1,3,5,7
/proc/30927/stat : 30927 (python) R 30925 30927 30218 0 -1 4202496 9337 937 1 0 34 12 0 0 16 0 1 0 538494833 285134848 7006 33554432000 4194304 5683644 140733731625088 18446744073709551615 242973958837 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/30927/statm: 69613 7006 1826 364 0 5537 0
Current children cumulated CPU time (s) 0.46
Current children cumulated vsize (KiB) 278452

[startup+1.50139 s]
/proc/loadavg: 7.96 8.14 6.80 9/293 30944
/proc/meminfo: memFree=30079624/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=320268 CPUtime=0.63 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 12353 937 1 0 50 13 0 0 22 0 5 0 538494833 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/30927/statm: 80067 7263 1881 364 0 15976 0
[pid=30933] ppid=30927 vsize=278612 CPUtime=0 cores=1,3,5,7
/proc/30933/stat : 30933 (python) S 30927 30927 30218 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 538494912 285298688 5575 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570255 0 0 16781312 514 5588809636821073919 0 0 17 3 0 0 0
/proc/30933/statm: 69653 5575 354 364 0 5571 0
[pid=30935] ppid=30933 vsize=1017800 CPUtime=0.7 cores=1,3,5,7
/proc/30935/stat : 30935 (gnovelty+2) R 30933 30935 30935 0 -1 4202496 446 0 0 0 70 0 0 0 21 0 1 0 538494912 1042227200 182 33554432000 134512640 135663704 4287615424 18446744073709551615 134545670 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/30935/statm: 254450 182 113 282 0 254166 0
[pid=30936] ppid=30927 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/30936/stat : 30936 (python) S 30927 30927 30218 0 -1 4202560 761 0 0 0 0 0 0 0 15 0 1 0 538494914 296288256 5693 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 12025257522211291398 0 0 17 1 0 0 0
/proc/30936/statm: 72336 5693 348 364 0 8251 0
[pid=30938] ppid=30936 vsize=2500 CPUtime=0.67 cores=1,3,5,7
/proc/30938/stat : 30938 (march_hi) R 30936 30938 30938 0 -1 4202496 693 0 0 0 67 0 0 0 22 0 1 0 538494915 2560000 430 33554432000 4194304 4955518 140733213634112 18446744073709551615 4290789 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/30938/statm: 625 430 104 186 0 438 0
[pid=30939] ppid=30927 vsize=299228 CPUtime=0 cores=1,3,5,7
/proc/30939/stat : 30939 (python) S 30927 30927 30218 0 -1 4202560 756 0 0 0 0 0 0 0 15 0 1 0 538494917 306409472 5601 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 4363310744054235397 0 0 17 1 0 0 0
/proc/30939/statm: 74807 5601 348 364 0 10719 0
[pid=30941] ppid=30939 vsize=2500 CPUtime=0.65 cores=1,3,5,7
/proc/30941/stat : 30941 (march_hi) R 30939 30941 30941 0 -1 4202496 694 0 0 0 65 0 0 0 22 0 1 0 538494918 2560000 429 33554432000 4194304 4955518 140733957091312 18446744073709551615 4261424 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/30941/statm: 625 429 104 186 0 438 0
[pid=30942] ppid=30927 vsize=310024 CPUtime=0 cores=1,3,5,7
/proc/30942/stat : 30942 (python) S 30927 30927 30218 0 -1 4202560 767 0 0 0 0 0 0 0 17 0 1 0 538494920 317464576 5726 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 8481892185703153926 0 0 17 7 0 0 0
/proc/30942/statm: 77506 5726 348 364 0 13415 0
[pid=30944] ppid=30942 vsize=2500 CPUtime=0.61 cores=1,3,5,7
/proc/30944/stat : 30944 (march_hi) R 30942 30944 30944 0 -1 4202496 691 0 0 0 61 0 0 0 25 0 1 0 538494921 2560000 429 33554432000 4194304 4955518 140733612855248 18446744073709551615 4224651 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/30944/statm: 625 429 104 186 0 438 0
[pid=30927/tid=30934] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30934/stat : 30934 (python) S 30925 30927 30218 0 -1 4202560 26 937 0 0 0 0 0 0 18 0 5 0 538494912 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 3 0 0 0
[pid=30927/tid=30937] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30937/stat : 30937 (python) S 30925 30927 30218 0 -1 4202560 21 937 0 0 0 0 0 0 20 0 5 0 538494915 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=30940] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30940/stat : 30940 (python) S 30925 30927 30218 0 -1 4202560 20 937 0 0 0 0 0 0 20 0 5 0 538494918 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 1 0 0 0
[pid=30927/tid=30943] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30943/stat : 30943 (python) S 30925 30927 30218 0 -1 4202560 27 937 0 0 0 0 0 0 22 0 5 0 538494921 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3.26
Current children cumulated vsize (KiB) 2522776

[startup+3.10111 s]
/proc/loadavg: 7.96 8.14 6.80 9/293 30944
/proc/meminfo: memFree=30076772/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=320268 CPUtime=0.63 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 12353 937 1 0 50 13 0 0 22 0 5 0 538494833 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/30927/statm: 80067 7263 1881 364 0 15976 0
[pid=30933] ppid=30927 vsize=278612 CPUtime=0 cores=1,3,5,7
/proc/30933/stat : 30933 (python) S 30927 30927 30218 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 538494912 285298688 5575 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570255 0 0 16781312 514 5588809636821041412 0 0 17 3 0 0 0
/proc/30933/statm: 69653 5575 354 364 0 5571 0
[pid=30935] ppid=30933 vsize=1017800 CPUtime=2.3 cores=1,3,5,7
/proc/30935/stat : 30935 (gnovelty+2) R 30933 30935 30935 0 -1 4202496 446 0 0 0 230 0 0 0 25 0 1 0 538494912 1042227200 182 33554432000 134512640 135663704 4287615424 18446744073709551615 134546755 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/30935/statm: 254450 182 113 282 0 254166 0
[pid=30936] ppid=30927 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/30936/stat : 30936 (python) S 30927 30927 30218 0 -1 4202560 762 0 0 0 0 0 0 0 15 0 1 0 538494914 296288256 5693 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 12025257522211291398 0 0 17 1 0 0 0
/proc/30936/statm: 72336 5693 348 364 0 8251 0
[pid=30938] ppid=30936 vsize=2820 CPUtime=2.27 cores=1,3,5,7
/proc/30938/stat : 30938 (march_hi) R 30936 30938 30938 0 -1 4202496 749 0 0 0 227 0 0 0 25 0 1 0 538494915 2887680 486 33554432000 4194304 4955518 140733213634112 18446744073709551615 4227621 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/30938/statm: 705 486 104 186 0 518 0
[pid=30939] ppid=30927 vsize=299228 CPUtime=0 cores=1,3,5,7
/proc/30939/stat : 30939 (python) S 30927 30927 30218 0 -1 4202560 760 0 0 0 0 0 0 0 15 0 1 0 538494917 306409472 5601 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 4363310744054235397 0 0 17 1 0 0 0
/proc/30939/statm: 74807 5601 348 364 0 10719 0
[pid=30941] ppid=30939 vsize=2820 CPUtime=2.25 cores=1,3,5,7
/proc/30941/stat : 30941 (march_hi) R 30939 30941 30941 0 -1 4202496 751 0 0 0 225 0 0 0 25 0 1 0 538494918 2887680 486 33554432000 4194304 4955518 140733957091312 18446744073709551615 4261387 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/30941/statm: 705 486 104 186 0 518 0
[pid=30942] ppid=30927 vsize=310024 CPUtime=0 cores=1,3,5,7
/proc/30942/stat : 30942 (python) S 30927 30927 30218 0 -1 4202560 769 0 0 0 0 0 0 0 15 0 1 0 538494920 317464576 5726 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 8481892185703153926 0 0 17 7 0 0 0
/proc/30942/statm: 77506 5726 348 364 0 13415 0
[pid=30944] ppid=30942 vsize=2820 CPUtime=2.2 cores=1,3,5,7
/proc/30944/stat : 30944 (march_hi) R 30942 30944 30944 0 -1 4202496 746 0 0 0 220 0 0 0 25 0 1 0 538494921 2887680 484 33554432000 4194304 4955518 140733612855248 18446744073709551615 4261364 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/30944/statm: 705 484 104 186 0 518 0
[pid=30927/tid=30934] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30934/stat : 30934 (python) S 30925 30927 30218 0 -1 4202560 26 937 0 0 0 0 0 0 18 0 5 0 538494912 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 3 0 0 0
[pid=30927/tid=30937] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30937/stat : 30937 (python) S 30925 30927 30218 0 -1 4202560 21 937 0 0 0 0 0 0 20 0 5 0 538494915 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=30940] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30940/stat : 30940 (python) S 30925 30927 30218 0 -1 4202560 20 937 0 0 0 0 0 0 20 0 5 0 538494918 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 1 0 0 0
[pid=30927/tid=30943] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30943/stat : 30943 (python) S 30925 30927 30218 0 -1 4202560 27 937 0 0 0 0 0 0 22 0 5 0 538494921 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 9.65
Current children cumulated vsize (KiB) 2523736

[startup+6.30056 s]
/proc/loadavg: 8.12 8.17 6.81 11/293 30944
/proc/meminfo: memFree=30075040/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=320268 CPUtime=0.63 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 12353 937 1 0 50 13 0 0 22 0 5 0 538494833 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/30927/statm: 80067 7263 1881 364 0 15976 0
[pid=30933] ppid=30927 vsize=278612 CPUtime=0.01 cores=1,3,5,7
/proc/30933/stat : 30933 (python) S 30927 30927 30218 0 -1 4202560 895 0 0 0 1 0 0 0 18 0 1 0 538494912 285298688 5598 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570255 0 0 16781312 514 5588809636821041412 0 0 17 3 0 0 0
/proc/30933/statm: 69653 5598 377 364 0 5571 0
[pid=30935] ppid=30933 vsize=1017800 CPUtime=5.5 cores=1,3,5,7
/proc/30935/stat : 30935 (gnovelty+2) R 30933 30935 30935 0 -1 4202496 446 0 0 0 550 0 0 0 25 0 1 0 538494912 1042227200 182 33554432000 134512640 135663704 4287615424 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/30935/statm: 254450 182 113 282 0 254166 0
[pid=30936] ppid=30927 vsize=289344 CPUtime=0.01 cores=1,3,5,7
/proc/30936/stat : 30936 (python) S 30927 30927 30218 0 -1 4202560 911 0 0 0 1 0 0 0 15 0 1 0 538494914 296288256 5716 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 12025257522211291398 0 0 17 1 0 0 0
/proc/30936/statm: 72336 5716 371 364 0 8251 0
[pid=30938] ppid=30936 vsize=3808 CPUtime=5.44 cores=1,3,5,7
/proc/30938/stat : 30938 (march_hi) R 30936 30938 30938 0 -1 4202496 904 0 0 0 544 0 0 0 25 0 1 0 538494915 3899392 641 33554432000 4194304 4955518 140733213634112 18446744073709551615 4209701 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/30938/statm: 952 641 104 186 0 765 0
[pid=30939] ppid=30927 vsize=299228 CPUtime=0 cores=1,3,5,7
/proc/30939/stat : 30939 (python) S 30927 30927 30218 0 -1 4202560 905 0 0 0 0 0 0 0 15 0 1 0 538494917 306409472 5624 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 4363310744054235397 0 0 17 3 0 0 0
/proc/30939/statm: 74807 5624 371 364 0 10719 0
[pid=30941] ppid=30939 vsize=3808 CPUtime=5.43 cores=1,3,5,7
/proc/30941/stat : 30941 (march_hi) R 30939 30941 30941 0 -1 4202496 906 0 0 0 543 0 0 0 25 0 1 0 538494918 3899392 641 33554432000 4194304 4955518 140733957091312 18446744073709551615 4227640 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/30941/statm: 952 641 104 186 0 765 0
[pid=30942] ppid=30927 vsize=310024 CPUtime=0 cores=1,3,5,7
/proc/30942/stat : 30942 (python) S 30927 30927 30218 0 -1 4202560 906 0 0 0 0 0 0 0 16 0 1 0 538494920 317464576 5748 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 4592514937980452863 0 0 17 7 0 0 0
/proc/30942/statm: 77506 5748 370 364 0 13415 0
[pid=30944] ppid=30942 vsize=3808 CPUtime=5.36 cores=1,3,5,7
/proc/30944/stat : 30944 (march_hi) R 30942 30944 30944 0 -1 4202496 899 0 0 0 536 0 0 0 25 0 1 0 538494921 3899392 637 33554432000 4194304 4955518 140733612855248 18446744073709551615 4227502 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/30944/statm: 952 637 104 186 0 765 0
[pid=30927/tid=30934] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30934/stat : 30934 (python) S 30925 30927 30218 0 -1 4202560 26 937 0 0 0 0 0 0 18 0 5 0 538494912 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 3 0 0 0
[pid=30927/tid=30937] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30937/stat : 30937 (python) S 30925 30927 30218 0 -1 4202560 21 937 0 0 0 0 0 0 20 0 5 0 538494915 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=30940] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30940/stat : 30940 (python) S 30925 30927 30218 0 -1 4202560 20 937 0 0 0 0 0 0 20 0 5 0 538494918 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 1 0 0 0
[pid=30927/tid=30943] ppid=30925 vsize=320268 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30943/stat : 30943 (python) S 30925 30927 30218 0 -1 4202560 27 937 0 0 0 0 0 0 22 0 5 0 538494921 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 22.38
Current children cumulated vsize (KiB) 2526700

[startup+12.7005 s]
/proc/loadavg: 8.35 8.22 6.84 9/293 30945
/proc/meminfo: memFree=30071348/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=320268 CPUtime=0.63 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 12353 937 1 0 50 13 0 0 22 0 5 0 538494833 327954432 7263 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/30927/statm: 80067 7263 1881 364 0 15976 0
[pid=30933] ppid=30927 vsize=278612 CPUtime=0.01 cores=1,3,5,7
/proc/30933/stat : 30933 (python) S 30927 30927 30218 0 -1 4202560 897 0 0 0 1 0 0 0 18 0 1 0 538494912 285298688 5598 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570255 0 0 16781312 514 14559979265614381316 0 0 17 3 0 0 0
/proc/30933/statm: 69653 5598 377 364 0 5571 0
[pid=30935] ppid=30933 vsize=1017800 CPUtime=11.9 cores=1,3,5,7
/proc/30935/stat : 30935 (gnovelty+2) R 30933 30935 30935 0 -1 4202496 446 0 0 0 1190 0 0 0 25 0 1 0 538494912 1042227200 182 33554432000 134512640 135663704 4287615424 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/30935/statm: 254450 182 113 282 0 254166 0
[pid=30936] ppid=30927 vsize=289344 CPUtime=0.04 cores=1,3,5,7

################
# More data... #
################

/proc/30941/stat : 30941 (march_hi) T 30939 30941 30941 0 -1 4202496 33117 0 0 0 116811 6 0 0 25 0 1 0 538494918 171945984 32852 33554432000 4194304 4955518 140733957091312 18446744073709551615 4227892 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/30941/statm: 41979 32852 104 186 0 41792 0
[pid=30939/tid=31063] ppid=30927 vsize=299228 CPUtime=0 cores=1,3,5,7
/proc/30939/task/31063/stat : 31063 (python) S 30927 30927 30218 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538543655 306409472 5654 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604455588668736 0 0 -1 3 0 0 0
[pid=30942] ppid=30927 vsize=310024 CPUtime=1.51 cores=1,3,5,7
/proc/30942/stat : 30942 (python) S 30927 30927 30218 0 -1 4202560 1170 0 0 0 85 66 0 0 15 0 2 0 538494920 317464576 5779 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/30942/statm: 77506 5779 400 364 0 13415 0
[pid=30944] ppid=30942 vsize=123716 CPUtime=779.99 cores=1,3,5,7
/proc/30944/stat : 30944 (march_hi) T 30942 30944 30944 0 -1 4202496 23251 0 0 0 77994 5 0 0 25 0 1 0 538494921 126685184 22989 33554432000 4194304 4955518 140733612855248 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/30944/statm: 30929 22989 104 186 0 30742 0
[pid=30942/tid=31005] ppid=30927 vsize=310024 CPUtime=0 cores=1,3,5,7
/proc/30942/task/31005/stat : 31005 (python) S 30927 30927 30218 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538505204 317464576 5779 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604471161840640 0 0 -1 7 0 0 0
[pid=31002] ppid=30927 vsize=320280 CPUtime=0.16 cores=1,3,5,7
/proc/31002/stat : 31002 (python) S 30927 30927 30218 0 -1 4202560 1092 0 0 0 9 7 0 0 19 0 2 0 538504931 327966720 5783 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31002/statm: 80070 5783 400 364 0 15976 0
[pid=31004] ppid=31002 vsize=1017800 CPUtime=100.13 cores=1,3,5,7
/proc/31004/stat : 31004 (gnovelty+2) T 31002 31004 31004 0 -1 4202496 446 0 0 0 10013 0 0 0 25 0 1 0 538504932 1042227200 182 33554432000 134512640 135663704 4294465680 18446744073709551615 134524978 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31004/statm: 254450 182 113 282 0 254166 0
[pid=31002/tid=31025] ppid=30927 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/31002/task/31025/stat : 31025 (python) S 30927 30927 30218 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538514948 327966720 5783 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604471161841536 0 0 -1 7 0 0 0
[pid=31026] ppid=30927 vsize=330632 CPUtime=1.75 cores=1,3,5,7
/proc/31026/stat : 31026 (python) S 30927 30927 30218 0 -1 4202560 1180 0 0 0 95 80 0 0 15 0 2 0 538514951 338567168 5831 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31026/statm: 82658 5831 400 364 0 18561 0
[pid=31028] ppid=31026 vsize=123716 CPUtime=882.34 cores=1,3,5,7
/proc/31028/stat : 31028 (march_hi) R 31026 31028 31028 0 -1 4202496 25929 0 0 0 88229 5 0 0 25 0 1 0 538514951 126685184 25662 33554432000 4194304 4955518 140734513666128 18446744073709551615 4290075 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/31028/statm: 30929 25662 104 186 0 30742 0
[pid=31026/tid=31080] ppid=30927 vsize=330632 CPUtime=0 cores=1,3,5,7
/proc/31026/task/31080/stat : 31080 (python) S 30927 30927 30218 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538563660 338567168 5831 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604471124310784 0 0 -1 5 0 0 0
[pid=31076] ppid=30927 vsize=341036 CPUtime=0.8 cores=1,3,5,7
/proc/31076/stat : 31076 (python) S 30927 30927 30218 0 -1 4202560 1228 0 0 0 42 38 0 0 15 0 2 0 538563412 349220864 5858 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31076/statm: 85259 5858 406 364 0 21159 0
[pid=31078] ppid=31076 vsize=58212 CPUtime=402.61 cores=1,3,5,7
/proc/31078/stat : 31078 (march_hi) R 31076 31078 31078 0 -1 4202496 13097 0 0 0 40259 2 0 0 25 0 1 0 538563412 59609088 12832 33554432000 4194304 4955518 140733974670992 18446744073709551615 4209701 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31078/statm: 14553 12832 104 186 0 14366 0
[pid=31076/tid=31097] ppid=30927 vsize=341036 CPUtime=0 cores=1,3,5,7
/proc/31076/task/31097/stat : 31097 (python) S 30927 30927 30218 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538602834 349220864 5858 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604467303416768 0 0 -1 5 0 0 0
[pid=31088] ppid=30927 vsize=351292 CPUtime=0.58 cores=1,3,5,7
/proc/31088/stat : 31088 (python) S 30927 30927 30218 0 -1 4202560 1069 0 0 0 32 26 0 0 15 0 1 0 538583839 359723008 5846 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 14559979265614381320 0 0 17 7 0 0 0
/proc/31088/statm: 87823 5846 377 364 0 23720 0
[pid=31090] ppid=31088 vsize=58212 CPUtime=290.02 cores=1,3,5,7
/proc/31090/stat : 31090 (march_hi) R 31088 31090 31090 0 -1 4202496 9724 0 0 0 28999 3 0 0 25 0 1 0 538583839 59609088 9456 33554432000 4194304 4955518 140735828080128 18446744073709551615 4227664 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/31090/statm: 14553 9456 104 186 0 14366 0
[pid=30927/tid=30934] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30934/stat : 30934 (python) S 30925 30927 30218 0 -1 4202560 26 937 0 0 0 0 0 0 18 0 9 0 538494912 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 3 0 0 0
[pid=30927/tid=30937] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30937/stat : 30937 (python) S 30925 30927 30218 0 -1 4202560 33 937 0 0 0 0 0 0 18 0 9 0 538494915 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=30940] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30940/stat : 30940 (python) S 30925 30927 30218 0 -1 4202560 29 937 0 0 0 0 0 0 18 0 9 0 538494918 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 1 0 0 0
[pid=30927/tid=30943] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30943/stat : 30943 (python) S 30925 30927 30218 0 -1 4202560 40 937 0 0 0 0 0 0 18 0 9 0 538494921 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31003] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31003/stat : 31003 (python) S 30925 30927 30218 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 9 0 538504931 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31027] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31027/stat : 31027 (python) S 30925 30927 30218 0 -1 4202560 35 937 0 0 0 0 0 0 18 0 9 0 538514951 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31077] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31077/stat : 31077 (python) S 30925 30927 30218 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 9 0 538563412 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31089] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31089/stat : 31089 (python) S 30925 30927 30218 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 9 0 538583839 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4709.63
Current children cumulated vsize (KiB) 5583316



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205 s]
/proc/loadavg: 8.22 8.18 7.79 9/318 31103
/proc/meminfo: memFree=29203696/32951132 swapFree=67111364/67111528
[pid=30927] ppid=30925 vsize=361536 CPUtime=1.27 cores=1,3,5,7
/proc/30927/stat : 30927 (python) S 30925 30927 30218 0 -1 4202496 16903 937 1 0 113 14 0 0 18 0 9 0 538494833 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/30927/statm: 90384 7376 1890 364 0 26281 0
[pid=30933] ppid=30927 vsize=288856 CPUtime=0.16 cores=1,3,5,7
/proc/30933/stat : 30933 (python) S 30927 30927 30218 0 -1 4202560 1080 0 0 0 8 8 0 0 19 0 2 0 538494912 295788544 5636 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/30933/statm: 72214 5636 410 364 0 8132 0
[pid=30935] ppid=30933 vsize=1017800 CPUtime=98.4 cores=1,3,5,7
/proc/30935/stat : 30935 (gnovelty+2) T 30933 30935 30935 0 -1 4202496 446 0 0 0 9840 0 0 0 25 0 1 0 538494912 1042227200 182 33554432000 134512640 135663704 4287615424 18446744073709551615 134524697 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/30935/statm: 254450 182 113 282 0 254166 0
[pid=30933/tid=31001] ppid=30927 vsize=288856 CPUtime=0 cores=1,3,5,7
/proc/30933/task/31001/stat : 31001 (python) S 30927 30927 30218 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538504928 295788544 5636 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604462803873984 0 0 -1 3 0 0 0
[pid=30936] ppid=30927 vsize=289344 CPUtime=4.01 cores=1,3,5,7
/proc/30936/stat : 30936 (python) S 30927 30927 30218 0 -1 4202560 1142 0 0 0 218 183 0 0 15 0 2 0 538494914 296288256 5746 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/30936/statm: 72336 5746 400 364 0 8251 0
[pid=30938] ppid=30936 vsize=123716 CPUtime=998.21 cores=1,3,5,7
/proc/30938/stat : 30938 (march_hi) R 30936 30938 30938 0 -1 4202496 28763 0 0 0 99768 53 0 0 25 0 1 0 538494915 126685184 28500 33554432000 4194304 4955518 140733213634112 18446744073709551615 4261370 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/30938/statm: 30929 28500 104 186 0 30742 0
[pid=30936/tid=31059] ppid=30927 vsize=289344 CPUtime=0 cores=1,3,5,7
/proc/30936/task/31059/stat : 31059 (python) S 30927 30927 30218 0 -1 4202560 43 0 0 0 0 0 0 0 15 0 2 0 538535060 296288256 5746 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604455588669632 0 0 -1 1 0 0 0
[pid=30939] ppid=30927 vsize=299228 CPUtime=2.27 cores=1,3,5,7
/proc/30939/stat : 30939 (python) S 30927 30927 30218 0 -1 4202560 1154 0 0 0 117 110 0 0 15 0 2 0 538494917 306409472 5654 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/30939/statm: 74807 5654 400 364 0 10719 0
[pid=30941] ppid=30939 vsize=167916 CPUtime=1168.17 cores=1,3,5,7
/proc/30941/stat : 30941 (march_hi) T 30939 30941 30941 0 -1 4202496 33117 0 0 0 116811 6 0 0 25 0 1 0 538494918 171945984 32852 33554432000 4194304 4955518 140733957091312 18446744073709551615 4227892 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/30941/statm: 41979 32852 104 186 0 41792 0
[pid=30939/tid=31063] ppid=30927 vsize=299228 CPUtime=0 cores=1,3,5,7
/proc/30939/task/31063/stat : 31063 (python) S 30927 30927 30218 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538543655 306409472 5654 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604455588668736 0 0 -1 3 0 0 0
[pid=30942] ppid=30927 vsize=310024 CPUtime=1.51 cores=1,3,5,7
/proc/30942/stat : 30942 (python) S 30927 30927 30218 0 -1 4202560 1170 0 0 0 85 66 0 0 15 0 2 0 538494920 317464576 5779 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/30942/statm: 77506 5779 400 364 0 13415 0
[pid=30944] ppid=30942 vsize=123716 CPUtime=779.99 cores=1,3,5,7
/proc/30944/stat : 30944 (march_hi) T 30942 30944 30944 0 -1 4202496 23251 0 0 0 77994 5 0 0 25 0 1 0 538494921 126685184 22989 33554432000 4194304 4955518 140733612855248 18446744073709551615 4209701 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/30944/statm: 30929 22989 104 186 0 30742 0
[pid=30942/tid=31005] ppid=30927 vsize=310024 CPUtime=0 cores=1,3,5,7
/proc/30942/task/31005/stat : 31005 (python) S 30927 30927 30218 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538505204 317464576 5779 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604471161840640 0 0 -1 7 0 0 0
[pid=31002] ppid=30927 vsize=320280 CPUtime=0.16 cores=1,3,5,7
/proc/31002/stat : 31002 (python) S 30927 30927 30218 0 -1 4202560 1092 0 0 0 9 7 0 0 19 0 2 0 538504931 327966720 5783 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982376907 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31002/statm: 80070 5783 400 364 0 15976 0
[pid=31004] ppid=31002 vsize=1017800 CPUtime=100.13 cores=1,3,5,7
/proc/31004/stat : 31004 (gnovelty+2) T 31002 31004 31004 0 -1 4202496 446 0 0 0 10013 0 0 0 25 0 1 0 538504932 1042227200 182 33554432000 134512640 135663704 4294465680 18446744073709551615 134524978 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31004/statm: 254450 182 113 282 0 254166 0
[pid=31002/tid=31025] ppid=30927 vsize=320280 CPUtime=0 cores=1,3,5,7
/proc/31002/task/31025/stat : 31025 (python) S 30927 30927 30218 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538514948 327966720 5783 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604471161841536 0 0 -1 7 0 0 0
[pid=31026] ppid=30927 vsize=330632 CPUtime=1.79 cores=1,3,5,7
/proc/31026/stat : 31026 (python) S 30927 30927 30218 0 -1 4202560 1180 0 0 0 97 82 0 0 15 0 2 0 538514951 338567168 5831 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31026/statm: 82658 5831 400 364 0 18561 0
[pid=31028] ppid=31026 vsize=123716 CPUtime=905 cores=1,3,5,7
/proc/31028/stat : 31028 (march_hi) R 31026 31028 31028 0 -1 4202496 26471 0 0 0 90495 5 0 0 25 0 1 0 538514951 126685184 26204 33554432000 4194304 4955518 140734513666128 18446744073709551615 4290403 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/31028/statm: 30929 26204 104 186 0 30742 0
[pid=31026/tid=31080] ppid=30927 vsize=330632 CPUtime=0 cores=1,3,5,7
/proc/31026/task/31080/stat : 31080 (python) S 30927 30927 30218 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 538563660 338567168 5831 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604471124310784 0 0 -1 5 0 0 0
[pid=31076] ppid=30927 vsize=341036 CPUtime=0.85 cores=1,3,5,7
/proc/31076/stat : 31076 (python) S 30927 30927 30218 0 -1 4202560 1228 0 0 0 45 40 0 0 15 0 2 0 538563412 349220864 5858 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/31076/statm: 85259 5858 406 364 0 21159 0
[pid=31078] ppid=31076 vsize=77856 CPUtime=425.23 cores=1,3,5,7
/proc/31078/stat : 31078 (march_hi) R 31076 31078 31078 0 -1 4202496 13720 0 0 0 42521 2 0 0 25 0 1 0 538563412 79724544 13455 33554432000 4194304 4955518 140733974670992 18446744073709551615 4224658 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31078/statm: 19464 13455 104 186 0 19277 0
[pid=31076/tid=31097] ppid=30927 vsize=341036 CPUtime=0 cores=1,3,5,7
/proc/31076/task/31097/stat : 31097 (python) S 30927 30927 30218 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 538602834 349220864 5858 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 514 18446604467303416768 0 0 -1 5 0 0 0
[pid=31088] ppid=30927 vsize=351292 CPUtime=0.63 cores=1,3,5,7
/proc/31088/stat : 31088 (python) S 30927 30927 30218 0 -1 4202560 1072 0 0 0 35 28 0 0 15 0 1 0 538583839 359723008 5846 33554432000 4194304 5683644 140733731625088 18446744073709551615 242970570310 0 0 16781312 514 16496443546794950920 0 0 17 7 0 0 0
/proc/31088/statm: 87823 5846 377 364 0 23720 0
[pid=31090] ppid=31088 vsize=58212 CPUtime=312.45 cores=1,3,5,7
/proc/31090/stat : 31090 (march_hi) R 31088 31090 31090 0 -1 4202496 10370 0 0 0 31242 3 0 0 25 0 1 0 538583839 59609088 10102 33554432000 4194304 4955518 140735828080128 18446744073709551615 4210584 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/31090/statm: 14553 10102 104 186 0 14366 0
[pid=30927/tid=30934] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30934/stat : 30934 (python) S 30925 30927 30218 0 -1 4202560 26 937 0 0 0 0 0 0 18 0 9 0 538494912 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 3 0 0 0
[pid=30927/tid=30937] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30937/stat : 30937 (python) S 30925 30927 30218 0 -1 4202560 33 937 0 0 0 0 0 0 18 0 9 0 538494915 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=30940] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30940/stat : 30940 (python) S 30925 30927 30218 0 -1 4202560 29 937 0 0 0 0 0 0 18 0 9 0 538494918 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 1 0 0 0
[pid=30927/tid=30943] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/30943/stat : 30943 (python) S 30925 30927 30218 0 -1 4202560 40 937 0 0 0 0 0 0 18 0 9 0 538494921 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31003] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31003/stat : 31003 (python) S 30925 30927 30218 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 9 0 538504931 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31027] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31027/stat : 31027 (python) S 30925 30927 30218 0 -1 4202560 35 937 0 0 0 0 0 0 18 0 9 0 538514951 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31077] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31077/stat : 31077 (python) S 30925 30927 30218 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 9 0 538563412 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
[pid=30927/tid=31089] ppid=30925 vsize=361536 CPUtime=0 cores=1,3,5,7
/proc/30927/task/31089/stat : 31089 (python) S 30925 30927 30218 0 -1 4202560 28 937 0 0 0 0 0 0 20 0 9 0 538583839 370212864 7376 33554432000 4194304 5683644 140733731625088 18446744073709551615 242982373809 0 0 16781312 2 18446604470995360448 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 5602960

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 30927 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=147825
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=146977
# CPU time returned by wait4() is 1.2948
# while last known CPU time is 4800.23
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1205.06
CPU time (s): 4800.23
CPU user time (s): 4794.11
CPU system time (s): 6.12
CPU usage (%): 398.34
Max. virtual memory (cumulated for all children) (KiB): 5602960

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.14783
system time used= 0.146977
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 17845
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3089
involuntary context switches= 47

runsolver used 3.02154 second user time and 6.84796 second system time

The end

Launcher Data 

Begin job on node128 at 2011-05-01 18:56:02
IDJOB=3317349
IDBENCH=82814
IDSOLVER=1705
FILE ID=node128/3317349-1304268962
RUNJOBID= node128-1304267283-30237
PBS_JOBID= 13173951
Free space on /tmp= 67584 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v550-c2343-S1694067460-072.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317349-1304268962/watcher-3317349-1304268962 -o /tmp/evaluation-result-3317349-1304268962/solver-3317349-1304268962 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317349-1304268962.cnf -seed 966247080 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f2d2b125797e776fb2e7099729714d99
RANDOM SEED=966247080

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30120060 kB
Buffers:        490712 kB
Cached:        1284932 kB
SwapCached:          0 kB
Active:         902084 kB
Inactive:      1484240 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30120060 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           31840 kB
Writeback:           4 kB
AnonPages:      610576 kB
Mapped:          25564 kB
Slab:           362248 kB
PageTables:      22232 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 33200112 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 67544 MiB
End job on node128 at 2011-05-01 19:16:09