Trace number 3386779

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (TO) 5000.06 4980.89

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S121099030-067.UNKNOWN.cnf
MD5SUMb936bd7aa1624f527379bba2a9d6f8ed
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.34/0.69	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.34/0.69	c INFO: solving HOME/instance-3386779-1305592212.cnf
0.34/0.69	c INFO: computing task features
0.42/0.70	c INFO: computed variable-clause graph statistics
0.42/0.70	c INFO: computed clause balance statistics
0.42/0.77	c INFO: computed variable balance statistics
0.52/0.81	c INFO: computed small-clause counts
0.52/0.81	c INFO: computed Horn-clause counts
0.52/0.81	c INFO: computed variable graph statistics
0.52/0.81	c INFO: collected features for HOME/instance-3386779-1305592212.cnf in 0.13 s
0.52/0.84	c INFO: running gnovelty+2@0 for 97 with 4999 remaining (b = 0.63)
101.19/101.22	c INFO: running gnovelty+2@0 for 97 with 4899 remaining (b = 0.08)
200.93/200.63	c INFO: running march_hi@0 for 485 with 4800 remaining (b = 0.34)
692.64/690.46	c INFO: running march_hi@489 for 194 with 4310 remaining (b = 0.17)
888.70/885.79	c INFO: running march_hi@685 for 97 with 4115 remaining (b = 0.07)
985.60/982.28	c INFO: running march_hi@0 for 485 with 4018 remaining (b = 0.31)
1475.81/1470.51	c INFO: running march_hi@488 for 194 with 3530 remaining (b = 0.14)
1670.26/1664.26	c INFO: running march_hi@681 for 97 with 3336 remaining (b = 0.05)
1767.86/1761.40	c INFO: running march_hi@0 for 388 with 3239 remaining (b = 0.24)
2162.00/2154.05	c INFO: running march_hi@392 for 97 with 2846 remaining (b = 0.07)
2258.17/2249.87	c INFO: running march_hi@488 for 97 with 2751 remaining (b = 0.06)
2354.06/2345.39	c INFO: running march_hi@583 for 97 with 2655 remaining (b = 0.06)
2454.54/2445.45	c INFO: running march_hi@0 for 388 with 2555 remaining (b = 0.22)
2850.24/2839.52	c INFO: running march_hi@0 for 485 with 2161 remaining (b = 0.22)
3339.13/3326.56	c INFO: running march_hi@394 for 97 with 1674 remaining (b = 0.04)
3431.83/3418.87	c INFO: running march_hi@0 for 97 with 1582 remaining (b = 0.03)
3532.36/3519.07	c INFO: running adaptg2wsat2009++@0 for 97 with 1482 remaining (b = 0.03)
3630.13/3616.49	c INFO: running march_hi@781 for 1384 with 1384 remaining (b = 0.41)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386779-1305592212/watcher-3386779-1305592212 -o /tmp/evaluation-result-3386779-1305592212/solver-3386779-1305592212 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386779-1305592212.cnf -seed 2080514635 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.22 2.10 2.02 2/515 12131
/proc/meminfo: memFree=14220768/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/12131/stat : 12131 (python) R 12129 12131 11438 0 -1 4202496 890 0 0 0 0 0 0 0 19 0 1 0 670817933 16154624 772 33554432000 4194304 5683644 140733556631664 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/12131/statm: 3944 773 343 364 0 431 0

[startup+0.0197589 s]
/proc/loadavg: 2.22 2.10 2.02 2/515 12131
/proc/meminfo: memFree=14220768/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=16240 CPUtime=0 cores=1,3,5,7
/proc/12131/stat : 12131 (python) R 12129 12131 11438 0 -1 4202496 1044 0 0 0 0 0 0 0 18 0 1 0 670817933 16629760 916 33554432000 4194304 5683644 140733556631664 18446744073709551615 5159789 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/12131/statm: 4060 916 381 364 0 547 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16240

[startup+0.100822 s]
/proc/loadavg: 2.22 2.10 2.02 2/515 12131
/proc/meminfo: memFree=14220768/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=85620 CPUtime=0.05 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 2629 0 0 0 3 2 0 0 17 0 1 0 670817933 87674880 2348 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500689920 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 0
/proc/12131/statm: 21405 2348 713 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85620

[startup+0.30078 s]
/proc/loadavg: 2.22 2.10 2.02 2/515 12131
/proc/meminfo: memFree=14220768/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=135748 CPUtime=0.19 cores=1,3,5,7
/proc/12131/stat : 12131 (python) R 12129 12131 11438 0 -1 4202496 5974 937 1 0 12 7 0 0 17 0 1 0 670817933 139005952 4149 33554432000 4194304 5683644 140733556631664 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/12131/statm: 33937 4149 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700682 s]
/proc/loadavg: 2.22 2.10 2.02 2/515 12131
/proc/meminfo: memFree=14220768/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=278284 CPUtime=0.42 cores=1,3,5,7
/proc/12131/stat : 12131 (python) R 12129 12131 11438 0 -1 4202496 9244 937 1 0 27 15 0 0 15 0 1 0 670817933 284962816 6914 33554432000 4194304 5683644 140733556631664 18446744073709551615 47427429862267 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/12131/statm: 69571 6914 1761 364 0 5495 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 278284

[startup+1.50054 s]
/proc/loadavg: 2.22 2.10 2.02 3/519 12139
/proc/meminfo: memFree=14194820/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=288764 CPUtime=0.56 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 9816 937 1 0 41 15 0 0 17 0 2 0 670817933 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 72191 7085 1875 364 0 8109 0
[pid=12137] ppid=12131 vsize=278520 CPUtime=0 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 743 0 0 0 0 0 0 0 18 0 1 0 670818018 285204480 5560 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500711951 0 0 16781312 514 14811015366716686335 0 0 17 5 0 0 0
/proc/12137/statm: 69630 5560 354 364 0 5548 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=0.64 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) R 12137 12139 12139 0 -1 4202496 446 0 0 0 64 0 0 0 21 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134524882 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 2 0 670818018 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1.2
Current children cumulated vsize (KiB) 1585084

[startup+3.10121 s]
/proc/loadavg: 2.20 2.10 2.02 3/519 12139
/proc/meminfo: memFree=14193860/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=288764 CPUtime=0.56 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 9816 937 1 0 41 15 0 0 17 0 2 0 670817933 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 72191 7085 1875 364 0 8109 0
[pid=12137] ppid=12131 vsize=278520 CPUtime=0 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 743 0 0 0 0 0 0 0 18 0 1 0 670818018 285204480 5560 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500711951 0 0 16781312 514 14811015366716653829 0 0 17 5 0 0 0
/proc/12137/statm: 69630 5560 354 364 0 5548 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=2.24 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) R 12137 12139 12139 0 -1 4202496 446 0 0 0 224 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 2 0 670818018 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2.8
Current children cumulated vsize (KiB) 1585084

[startup+6.30058 s]
/proc/loadavg: 2.20 2.10 2.02 3/519 12139
/proc/meminfo: memFree=14192496/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=288764 CPUtime=0.56 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 9816 937 1 0 41 15 0 0 17 0 2 0 670817933 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 72191 7085 1875 364 0 8109 0
[pid=12137] ppid=12131 vsize=278520 CPUtime=0.01 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 893 0 0 0 1 0 0 0 18 0 1 0 670818018 285204480 5583 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500711951 0 0 16781312 514 14811015366716653829 0 0 17 5 0 0 0
/proc/12137/statm: 69630 5583 377 364 0 5548 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=5.44 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) R 12137 12139 12139 0 -1 4202496 446 0 0 0 544 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 135008531 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 2 0 670818018 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 6.01
Current children cumulated vsize (KiB) 1585084

[startup+12.7003 s]
/proc/loadavg: 2.17 2.09 2.02 3/519 12139
/proc/meminfo: memFree=14191380/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=288764 CPUtime=0.56 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 9816 937 1 0 41 15 0 0 17 0 2 0 670817933 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 72191 7085 1875 364 0 8109 0
[pid=12137] ppid=12131 vsize=278520 CPUtime=0.03 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 897 0 0 0 2 1 0 0 18 0 1 0 670818018 285204480 5583 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500711951 0 0 16781312 514 10332260299792417029 0 0 17 5 0 0 0
/proc/12137/statm: 69630 5583 377 364 0 5548 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=11.83 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) R 12137 12139 12139 0 -1 4202496 446 0 0 0 1183 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134545607 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 2 0 670818018 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.42
Current children cumulated vsize (KiB) 1585084

[startup+25.5008 s]
/proc/loadavg: 2.14 2.09 2.02 3/519 12139
/proc/meminfo: memFree=14189892/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=288764 CPUtime=0.56 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 9816 937 1 0 41 15 0 0 17 0 2 0 670817933 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 72191 7085 1875 364 0 8109 0
[pid=12137] ppid=12131 vsize=278520 CPUtime=0.07 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 903 0 0 0 4 3 0 0 18 0 1 0 670818018 285204480 5583 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500711951 0 0 16781312 514 10332260299792417029 0 0 17 5 0 0 0
/proc/12137/statm: 69630 5583 377 364 0 5548 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=24.62 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) R 12137 12139 12139 0 -1 4202496 446 0 0 0 2462 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134524768 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 2 0 670818018 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.25
Current children cumulated vsize (KiB) 1585084

[startup+51.1007 s]
/proc/loadavg: 2.09 2.08 2.01 3/522 12143
/proc/meminfo: memFree=14182168/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=288764 CPUtime=0.56 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 9816 937 1 0 41 15 0 0 17 0 2 0 670817933 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 72191 7085 1875 364 0 8109 0
[pid=12137] ppid=12131 vsize=278520 CPUtime=0.16 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 912 0 0 0 8 8 0 0 18 0 1 0 670818018 285204480 5583 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500711951 0 0 16781312 514 10332260299792417029 0 0 17 5 0 0 0
/proc/12137/statm: 69630 5583 377 364 0 5548 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=50.21 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) R 12137 12139 12139 0 -1 4202496 446 0 0 0 5021 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 2 0 670818018 295694336 7085 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.93
Current children cumulated vsize (KiB) 1585084

[startup+102.318 s]
/proc/loadavg: 2.08 2.08 2.01 3/526 12148
/proc/meminfo: memFree=14171244/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=299556 CPUtime=0.59 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 10646 937 1 0 44 15 0 0 19 0 3 0 670817933 306745344 7220 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/12131/statm: 74889 7220 1878 364 0 10804 0
[pid=12137] ppid=12131 vsize=288764 CPUtime=0.35 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 1088 0 0 0 17 18 0 0 19 0 2 0 670818018 295694336 5621 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12137/statm: 72191 5621 410 364 0 8109 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=100.26 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) T 12137 12139 12139 0 -1 4202496 446 0 0 0 10026 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134544098 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12137/tid=12145] ppid=12131 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12137/task/12145/stat : 12145 (python) S 12131 12131 11438 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 670828053 295694336 5621 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604464951086080 0 0 -1 7 0 0 0
[pid=12146] ppid=12131 vsize=289312 CPUtime=0 cores=1,3,5,7
/proc/12146/stat : 12146 (python) S 12131 12131 11438 0 -1 4202560 736 0 0 0 0 0 0 0 15 0 1 0 670828056 296255488 5686 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500712006 0 0 16781312 514 16453639009369817349 0 0 17 7 0 0 0
/proc/12146/statm: 72328 5686 348 364 0 8243 0
[pid=12148] ppid=12146 vsize=1017800 CPUtime=1.08 cores=1,3,5,7
/proc/12148/stat : 12148 (gnovelty+2) R 12146 12148 12148 0 -1 4202496 444 0 0 0 108 0 0 0 20 0 1 0 670828056 1042227200 181 33554432000 134512640 135663704 4292556736 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/12148/statm: 254450 181 113 282 0 254166 0
[pid=12131/tid=12138] ppid=12129 vsize=299556 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 3 0 670818018 306745344 7220 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12147] ppid=12129 vsize=299556 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12147/stat : 12147 (python) S 12129 12131 11438 0 -1 4202560 25 937 0 0 0 0 0 0 20 0 3 0 670828056 306745344 7220 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 102.28
Current children cumulated vsize (KiB) 2913232

[startup+162.3 s]
/proc/loadavg: 2.03 2.06 2.01 3/526 12150
/proc/meminfo: memFree=14160164/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=299556 CPUtime=0.59 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 10646 937 1 0 44 15 0 0 19 0 3 0 670817933 306745344 7220 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/12131/statm: 74889 7220 1878 364 0 10804 0
[pid=12137] ppid=12131 vsize=288764 CPUtime=0.35 cores=1,3,5,7

################
# More data... #
################

/proc/12248/stat : 12248 (march_hi) T 12246 12248 12248 0 -1 4202496 22038 0 0 0 48632 4 0 0 25 0 1 0 671062478 99041280 21770 33554432000 4194304 4955518 140736784527568 18446744073709551615 4208667 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/12248/statm: 24180 21770 106 186 0 23993 0
[pid=12246/tid=12368] ppid=12131 vsize=330744 CPUtime=0 cores=1,3,5,7
/proc/12246/task/12368/stat : 12368 (python) S 12131 12131 11438 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 671101884 338681856 5860 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604470909537280 0 0 -1 3 0 0 0
[pid=12369] ppid=12131 vsize=341000 CPUtime=1.9 cores=1,3,5,7
/proc/12369/stat : 12369 (python) S 12131 12131 11438 0 -1 4202560 1150 0 0 0 95 95 0 0 16 0 2 0 671101885 349184000 5864 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12369/statm: 85250 5864 400 364 0 21150 0
[pid=12371] ppid=12369 vsize=96720 CPUtime=487.04 cores=1,3,5,7
/proc/12371/stat : 12371 (march_hi) T 12369 12371 12371 0 -1 4202496 22039 0 0 0 48699 5 0 0 25 0 1 0 671101886 99041280 21770 33554432000 4194304 4955518 140736847886816 18446744073709551615 4208656 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12371/statm: 24180 21770 106 186 0 23993 0
[pid=12369/tid=12390] ppid=12131 vsize=341000 CPUtime=0 cores=1,3,5,7
/proc/12369/task/12390/stat : 12390 (python) S 12131 12131 11438 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 671150588 349184000 5864 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604464924529024 0 0 -1 7 0 0 0
[pid=12394] ppid=12131 vsize=351256 CPUtime=0.41 cores=1,3,5,7
/proc/12394/stat : 12394 (python) S 12131 12131 11438 0 -1 4202560 1151 0 0 0 20 21 0 0 16 0 2 0 671159820 359686144 5876 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12394/statm: 87814 5876 406 364 0 23711 0
[pid=12396] ppid=12394 vsize=28796 CPUtime=100.12 cores=1,3,5,7
/proc/12396/stat : 12396 (march_hi) T 12394 12396 12396 0 -1 4202496 5552 0 0 0 10012 0 0 0 25 0 1 0 671159821 29487104 5280 33554432000 4194304 4955518 140735141437200 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12396/statm: 7199 5280 106 186 0 7012 0
[pid=12394/tid=12399] ppid=12131 vsize=351256 CPUtime=0 cores=1,3,5,7
/proc/12394/task/12399/stat : 12399 (python) S 12131 12131 11438 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 671169839 359686144 5876 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604466525298944 0 0 -1 7 0 0 0
[pid=12400] ppid=12131 vsize=361512 CPUtime=0.38 cores=1,3,5,7
/proc/12400/stat : 12400 (python) S 12131 12131 11438 0 -1 4202560 1101 0 0 0 20 18 0 0 19 0 2 0 671169841 370188288 5874 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12400/statm: 90378 5874 400 364 0 26272 0
[pid=12402] ppid=12400 vsize=416260 CPUtime=97.4 cores=1,3,5,7
/proc/12402/stat : 12402 (adaptg2wsat2009) T 12400 12402 12402 0 -1 4202496 497 0 0 0 9740 0 0 0 25 0 1 0 671169841 426250240 230 33554432000 134512640 135107411 4294763488 18446744073709551615 134532100 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12402/statm: 104065 230 59 146 0 102941 0
[pid=12400/tid=12409] ppid=12131 vsize=361512 CPUtime=0 cores=1,3,5,7
/proc/12400/task/12409/stat : 12409 (python) S 12131 12131 11438 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 671179582 370188288 5874 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604457116255552 0 0 -1 7 0 0 0
[pid=12131/tid=12138] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 10 0 670818018 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12147] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12147/stat : 12147 (python) S 12129 12131 11438 0 -1 4202560 25 937 0 0 0 0 0 0 20 0 10 0 670828056 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 7 0 0 0
[pid=12131/tid=12154] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12154/stat : 12154 (python) S 12129 12131 11438 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 10 0 670837997 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 7 0 0 0
[pid=12131/tid=12183] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12183/stat : 12183 (python) S 12129 12131 11438 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 10 0 670916162 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 3 0 0 0
[pid=12131/tid=12213] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12213/stat : 12213 (python) S 12129 12131 11438 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 10 0 670994074 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 3 0 0 0
[pid=12131/tid=12247] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12247/stat : 12247 (python) S 12129 12131 11438 0 -1 4202560 34 937 0 0 0 0 0 0 18 0 10 0 671062478 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 3 0 0 0
[pid=12131/tid=12370] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12370/stat : 12370 (python) S 12129 12131 11438 0 -1 4202560 29 937 0 0 0 0 0 0 20 0 10 0 671101886 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12395] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12395/stat : 12395 (python) S 12129 12131 11438 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 10 0 671159821 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12401] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12401/stat : 12401 (python) S 12129 12131 11438 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 10 0 671169841 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4981.46
Current children cumulated vsize (KiB) 6695820



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+4980.82 s]
/proc/loadavg: 2.27 2.10 2.02 3/559 12454
/proc/meminfo: memFree=13034648/32950928 swapFree=45358800/67111528
[pid=12131] ppid=12129 vsize=371756 CPUtime=0.93 cores=1,3,5,7
/proc/12131/stat : 12131 (python) S 12129 12131 11438 0 -1 4202496 17736 937 1 0 76 17 0 0 18 0 10 0 670817933 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/12131/statm: 92939 7369 1892 364 0 28833 0
[pid=12137] ppid=12131 vsize=288764 CPUtime=0.35 cores=1,3,5,7
/proc/12137/stat : 12137 (python) S 12131 12131 11438 0 -1 4202560 1088 0 0 0 17 18 0 0 19 0 2 0 670818018 295694336 5621 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12137/statm: 72191 5621 410 364 0 8109 0
[pid=12139] ppid=12137 vsize=1017800 CPUtime=100.26 cores=1,3,5,7
/proc/12139/stat : 12139 (gnovelty+2) T 12137 12139 12139 0 -1 4202496 446 0 0 0 10026 0 0 0 25 0 1 0 670818018 1042227200 181 33554432000 134512640 135663704 4290388320 18446744073709551615 134544098 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/12139/statm: 254450 181 113 282 0 254166 0
[pid=12137/tid=12145] ppid=12131 vsize=288764 CPUtime=0 cores=1,3,5,7
/proc/12137/task/12145/stat : 12145 (python) S 12131 12131 11438 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 670828053 295694336 5621 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604464951086080 0 0 -1 7 0 0 0
[pid=12146] ppid=12131 vsize=289312 CPUtime=0.36 cores=1,3,5,7
/proc/12146/stat : 12146 (python) S 12131 12131 11438 0 -1 4202560 1073 0 0 0 19 17 0 0 16 0 2 0 670828056 296255488 5739 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/12146/statm: 72328 5739 400 364 0 8243 0
[pid=12148] ppid=12146 vsize=1017800 CPUtime=99.37 cores=1,3,5,7
/proc/12148/stat : 12148 (gnovelty+2) T 12146 12148 12148 0 -1 4202496 444 0 0 0 9937 0 0 0 25 0 1 0 670828056 1042227200 181 33554432000 134512640 135663704 4292556736 18446744073709551615 134524894 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12148/statm: 254450 181 113 282 0 254166 0
[pid=12146/tid=12152] ppid=12131 vsize=289312 CPUtime=0 cores=1,3,5,7
/proc/12146/task/12152/stat : 12152 (python) S 12131 12131 11438 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 670837993 296255488 5739 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604461038571904 0 0 -1 3 0 0 0
[pid=12153] ppid=12131 vsize=299388 CPUtime=8.48 cores=1,3,5,7
/proc/12153/stat : 12153 (python) S 12131 12131 11438 0 -1 4202560 1153 0 0 0 410 438 0 0 15 0 2 0 670837996 306573312 5681 33554432000 4194304 5683644 140733556631664 18446744073709551615 254500712006 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/12153/statm: 74847 5681 400 364 0 10759 0
[pid=12155] ppid=12153 vsize=475432 CPUtime=2145.97 cores=1,3,5,7
/proc/12155/stat : 12155 (march_hi) R 12153 12155 12155 0 -1 4202496 85121 0 0 0 214579 18 0 0 25 0 1 0 670837997 486842368 84854 33554432000 4194304 4955518 140736682592080 18446744073709551615 4297449 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/12155/statm: 118858 84854 106 186 0 118671 0
[pid=12153/tid=12169] ppid=12131 vsize=299388 CPUtime=0 cores=1,3,5,7
/proc/12153/task/12169/stat : 12169 (python) S 12131 12131 11438 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 670886977 306573312 5681 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604471152642176 0 0 -1 7 0 0 0
[pid=12182] ppid=12131 vsize=310232 CPUtime=3.03 cores=1,3,5,7
/proc/12182/stat : 12182 (python) S 12131 12131 11438 0 -1 4202560 1173 0 0 0 155 148 0 0 16 0 2 0 670916162 317677568 5832 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12182/statm: 77558 5832 400 364 0 13467 0
[pid=12184] ppid=12182 vsize=140920 CPUtime=779.07 cores=1,3,5,7
/proc/12184/stat : 12184 (march_hi) T 12182 12184 12184 0 -1 4202496 34059 0 0 0 77900 7 0 0 25 0 1 0 670916162 144302080 33791 33554432000 4194304 4955518 140734636938352 18446744073709551615 4297472 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/12184/statm: 35230 33791 106 186 0 35043 0
[pid=12182/tid=12203] ppid=12131 vsize=310232 CPUtime=0 cores=1,3,5,7
/proc/12182/task/12203/stat : 12203 (python) S 12131 12131 11438 0 -1 4202560 45 0 0 0 0 0 0 0 15 0 2 0 670964982 317677568 5832 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604471007435712 0 0 -1 7 0 0 0
[pid=12212] ppid=12131 vsize=320488 CPUtime=2.69 cores=1,3,5,7
/proc/12212/stat : 12212 (python) S 12131 12131 11438 0 -1 4202560 1163 0 0 0 136 133 0 0 15 0 2 0 670994074 328179712 5847 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12212/statm: 80122 5847 400 364 0 16028 0
[pid=12214] ppid=12212 vsize=140920 CPUtime=683.98 cores=1,3,5,7
/proc/12214/stat : 12214 (march_hi) T 12212 12214 12214 0 -1 4202496 30118 0 0 0 68393 5 0 0 25 0 1 0 670994074 144302080 29851 33554432000 4194304 4955518 140735843723408 18446744073709551615 4285374 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/12214/statm: 35230 29851 106 186 0 35043 0
[pid=12212/tid=12237] ppid=12131 vsize=320488 CPUtime=0 cores=1,3,5,7
/proc/12212/task/12237/stat : 12237 (python) S 12131 12131 11438 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 671033337 328179712 5847 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604456459772992 0 0 -1 7 0 0 0
[pid=12246] ppid=12131 vsize=330744 CPUtime=1.96 cores=1,3,5,7
/proc/12246/stat : 12246 (python) S 12131 12131 11438 0 -1 4202560 1158 0 0 0 103 93 0 0 15 0 2 0 671062478 338681856 5860 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12246/statm: 82686 5860 400 364 0 18589 0
[pid=12248] ppid=12246 vsize=96720 CPUtime=486.36 cores=1,3,5,7
/proc/12248/stat : 12248 (march_hi) T 12246 12248 12248 0 -1 4202496 22038 0 0 0 48632 4 0 0 25 0 1 0 671062478 99041280 21770 33554432000 4194304 4955518 140736784527568 18446744073709551615 4208667 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/12248/statm: 24180 21770 106 186 0 23993 0
[pid=12246/tid=12368] ppid=12131 vsize=330744 CPUtime=0 cores=1,3,5,7
/proc/12246/task/12368/stat : 12368 (python) S 12131 12131 11438 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 671101884 338681856 5860 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604470909537280 0 0 -1 3 0 0 0
[pid=12369] ppid=12131 vsize=341000 CPUtime=1.9 cores=1,3,5,7
/proc/12369/stat : 12369 (python) S 12131 12131 11438 0 -1 4202560 1150 0 0 0 95 95 0 0 16 0 2 0 671101885 349184000 5864 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12369/statm: 85250 5864 400 364 0 21150 0
[pid=12371] ppid=12369 vsize=96720 CPUtime=487.04 cores=1,3,5,7
/proc/12371/stat : 12371 (march_hi) T 12369 12371 12371 0 -1 4202496 22039 0 0 0 48699 5 0 0 25 0 1 0 671101886 99041280 21770 33554432000 4194304 4955518 140736847886816 18446744073709551615 4208656 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12371/statm: 24180 21770 106 186 0 23993 0
[pid=12369/tid=12390] ppid=12131 vsize=341000 CPUtime=0 cores=1,3,5,7
/proc/12369/task/12390/stat : 12390 (python) S 12131 12131 11438 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 671150588 349184000 5864 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604464924529024 0 0 -1 7 0 0 0
[pid=12394] ppid=12131 vsize=351256 CPUtime=0.41 cores=1,3,5,7
/proc/12394/stat : 12394 (python) S 12131 12131 11438 0 -1 4202560 1151 0 0 0 20 21 0 0 16 0 2 0 671159820 359686144 5876 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12394/statm: 87814 5876 406 364 0 23711 0
[pid=12396] ppid=12394 vsize=28796 CPUtime=100.12 cores=1,3,5,7
/proc/12396/stat : 12396 (march_hi) T 12394 12396 12396 0 -1 4202496 5552 0 0 0 10012 0 0 0 25 0 1 0 671159821 29487104 5280 33554432000 4194304 4955518 140735141437200 18446744073709551615 4224648 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12396/statm: 7199 5280 106 186 0 7012 0
[pid=12394/tid=12399] ppid=12131 vsize=351256 CPUtime=0 cores=1,3,5,7
/proc/12394/task/12399/stat : 12399 (python) S 12131 12131 11438 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 671169839 359686144 5876 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604466525298944 0 0 -1 7 0 0 0
[pid=12400] ppid=12131 vsize=361512 CPUtime=0.38 cores=1,3,5,7
/proc/12400/stat : 12400 (python) S 12131 12131 11438 0 -1 4202560 1101 0 0 0 20 18 0 0 19 0 2 0 671169841 370188288 5874 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/12400/statm: 90378 5874 400 364 0 26272 0
[pid=12402] ppid=12400 vsize=416260 CPUtime=97.4 cores=1,3,5,7
/proc/12402/stat : 12402 (adaptg2wsat2009) T 12400 12402 12402 0 -1 4202496 497 0 0 0 9740 0 0 0 25 0 1 0 671169841 426250240 230 33554432000 134512640 135107411 4294763488 18446744073709551615 134532100 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/12402/statm: 104065 230 59 146 0 102941 0
[pid=12400/tid=12409] ppid=12131 vsize=361512 CPUtime=0 cores=1,3,5,7
/proc/12400/task/12409/stat : 12409 (python) S 12131 12131 11438 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 671179582 370188288 5874 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 514 18446604457116255552 0 0 -1 7 0 0 0
[pid=12131/tid=12138] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12138/stat : 12138 (python) S 12129 12131 11438 0 -1 4202560 32 937 0 0 0 0 0 0 18 0 10 0 670818018 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12147] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12147/stat : 12147 (python) S 12129 12131 11438 0 -1 4202560 25 937 0 0 0 0 0 0 20 0 10 0 670828056 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 7 0 0 0
[pid=12131/tid=12154] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12154/stat : 12154 (python) S 12129 12131 11438 0 -1 4202560 31 937 0 0 0 0 0 0 18 0 10 0 670837997 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 7 0 0 0
[pid=12131/tid=12183] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12183/stat : 12183 (python) S 12129 12131 11438 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 10 0 670916162 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 3 0 0 0
[pid=12131/tid=12213] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12213/stat : 12213 (python) S 12129 12131 11438 0 -1 4202560 30 937 0 0 0 0 0 0 18 0 10 0 670994074 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 3 0 0 0
[pid=12131/tid=12247] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12247/stat : 12247 (python) S 12129 12131 11438 0 -1 4202560 34 937 0 0 0 0 0 0 18 0 10 0 671062478 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 3 0 0 0
[pid=12131/tid=12370] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12370/stat : 12370 (python) S 12129 12131 11438 0 -1 4202560 29 937 0 0 0 0 0 0 20 0 10 0 671101886 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12395] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12395/stat : 12395 (python) S 12129 12131 11438 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 10 0 671159821 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
[pid=12131/tid=12401] ppid=12129 vsize=371756 CPUtime=0 cores=1,3,5,7
/proc/12131/task/12401/stat : 12401 (python) S 12129 12131 11438 0 -1 4202560 27 937 0 0 0 0 0 0 20 0 10 0 671169841 380678144 7369 33554432000 4194304 5683644 140733556631664 18446744073709551615 254512515505 0 0 16781312 2 18446604467392936064 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5000.06
Current children cumulated vsize (KiB) 6695820

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 12131 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=765883
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=177972
# CPU time returned by wait4() is 0.943855
# while last known CPU time is 5000.06
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 4980.89
CPU time (s): 5000.06
CPU user time (s): 4989.69
CPU system time (s): 10.37
CPU usage (%): 100.385
Max. virtual memory (cumulated for all children) (KiB): 6695824

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.765883
system time used= 0.177972
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 18679
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2903
involuntary context switches= 42

runsolver used 28.6087 second user time and 67.8507 second system time

The end

Launcher Data 

Begin job on node149 at 2011-05-17 02:30:12
IDJOB=3386779
IDBENCH=82844
IDSOLVER=1872
FILE ID=node149/3386779-1305592212
RUNJOBID= node149-1305586142-11457
PBS_JOBID= 13350329
Free space on /tmp= 73488 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S121099030-067.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386779-1305592212/watcher-3386779-1305592212 -o /tmp/evaluation-result-3386779-1305592212/solver-3386779-1305592212 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386779-1305592212.cnf -seed 2080514635 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b936bd7aa1624f527379bba2a9d6f8ed
RANDOM SEED=2080514635

node149.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      14220924 kB
Buffers:         98916 kB
Cached:         640716 kB
SwapCached:      36272 kB
Active:        9270020 kB
Inactive:      9172472 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      14220924 kB
SwapTotal:    67111528 kB
SwapFree:     45358800 kB
Dirty:           31820 kB
Writeback:           0 kB
AnonPages:    17678768 kB
Mapped:          34832 kB
Slab:            99700 kB
PageTables:     118772 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 167570244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73456 MiB
End job on node149 at 2011-05-17 03:53:15