Trace number 3317151

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 1989.67 501.719

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S837431592-027-UNKNOWN.cnf
MD5SUM3b045b7b4cbac0d258a07eda037c4299
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.37/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.37/0.67	c INFO: solving HOME/instance-3317151-1304267807.cnf
0.37/0.67	c INFO: computing task features
0.54/0.88	c INFO: computed variable-clause graph statistics
0.54/0.89	c INFO: computed clause balance statistics
1.64/1.90	c INFO: computed variable balance statistics
2.14/2.49	c INFO: computed small-clause counts
2.14/2.49	c INFO: computed Horn-clause counts
2.24/2.54	c INFO: computed variable graph statistics
2.24/2.54	c INFO: collected features for HOME/instance-3317151-1304267807.cnf in 1.86 s
2.24/2.56	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.38)
2.24/2.59	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.13)
2.34/2.62	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
2.34/2.65	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.07)
401.99/102.79	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.05)
402.40/102.82	c INFO: running gnovelty+2@0 for 97 with 4697 remaining (b = 0.05)
402.40/102.86	c INFO: running FH@0 for 97 with 4697 remaining (b = 0.03)
402.59/102.92	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.03)
801.09/202.97	c INFO: running adaptg2wsat2009++@0 for 97 with 4597 remaining (b = 0.02)
801.47/203.06	c INFO: running FH@0 for 97 with 4597 remaining (b = 0.02)
801.69/203.10	c INFO: running hybridGM3@0 for 194 with 4597 remaining (b = 0.04)
801.69/203.15	c INFO: running gnovelty+2@0 for 97 with 4597 remaining (b = 0.02)
1196.11/302.18	c INFO: running hybridGM3@0 for 194 with 4498 remaining (b = 0.03)
1200.47/303.26	c INFO: running adaptg2wsat2009++@0 for 97 with 4497 remaining (b = 0.01)
1201.14/303.47	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.03)
1583.90/399.54	c INFO: running hybridGM3@98 for 97 with 4401 remaining (b = 0.01)
1599.43/403.47	c INFO: running hybridGM3@0 for 194 with 4397 remaining (b = 0.02)
1973.77/497.66	c INFO: running hybridGM3@100 for 97 with 4303 remaining (b = 0.01)
1982.87/499.91	c INFO: running TNM@0 for 1552 with 4300 remaining (b = 0.19)
1986.04/500.74	c INFO: running FH@0 for 97 with 4299 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317151-1304267807/watcher-3317151-1304267807 -o /tmp/evaluation-result-3317151-1304267807/solver-3317151-1304267807 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317151-1304267807.cnf -seed 1755707011 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 6.56 3.35 5/220 23505
/proc/meminfo: memFree=29444136/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=15380 CPUtime=0 cores=0,2,4,6
/proc/23505/stat : 23505 (python) R 23503 23505 23228 0 -1 4202496 795 0 0 0 0 0 0 0 25 0 1 0 617011384 15749120 681 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905393808 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/23505/statm: 3845 681 322 364 0 332 0

[startup+0.104253 s]
/proc/loadavg: 8.11 6.56 3.35 5/220 23505
/proc/meminfo: memFree=29444136/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=93956 CPUtime=0.06 cores=0,2,4,6
/proc/23505/stat : 23505 (python) R 23503 23505 23228 0 -1 4202496 2695 0 0 0 4 2 0 0 25 0 1 0 617011384 96210944 2406 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905437319 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/23505/statm: 23489 2406 764 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 93956

[startup+0.200234 s]
/proc/loadavg: 8.11 6.56 3.35 5/220 23505
/proc/meminfo: memFree=29444136/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=124228 CPUtime=0.12 cores=0,2,4,6
/proc/23505/stat : 23505 (python) R 23503 23505 23228 0 -1 4202496 4272 0 1 0 8 4 0 0 25 0 1 0 617011384 127209472 3878 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905392128 0 2147483391 16781312 2 18446744071699207358 0 0 17 4 0 0 0
/proc/23505/statm: 31057 3878 996 364 0 2868 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 124228

[startup+0.300218 s]
/proc/loadavg: 8.11 6.56 3.35 5/220 23505
/proc/meminfo: memFree=29444136/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=135744 CPUtime=0.18 cores=0,2,4,6
/proc/23505/stat : 23505 (python) R 23503 23505 23228 0 -1 4202496 5966 935 1 0 13 5 0 0 25 0 1 0 617011384 139001856 4150 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905390085 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/23505/statm: 33936 4150 1032 364 0 3122 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135744

[startup+0.700142 s]
/proc/loadavg: 8.11 6.56 3.35 5/220 23505
/proc/meminfo: memFree=29444136/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=280820 CPUtime=0.44 cores=0,2,4,6
/proc/23505/stat : 23505 (python) R 23503 23505 23228 0 -1 4202496 9853 935 1 0 33 11 0 0 25 0 1 0 617011384 287559680 7532 33554432000 4194304 5683644 140735070319120 18446744073709551615 47683990021357 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/23505/statm: 70205 7532 1761 364 0 6129 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 280820

[startup+1.50099 s]
/proc/loadavg: 8.11 6.56 3.35 6/221 23511
/proc/meminfo: memFree=29409160/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=290800 CPUtime=1.24 cores=0,2,4,6
/proc/23505/stat : 23505 (python) R 23503 23505 23228 0 -1 4202496 14011 935 1 0 112 12 0 0 25 0 1 0 617011384 297779200 10085 33554432000 4194304 5683644 140735070319120 18446744073709551615 47683819249537 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/23505/statm: 72700 10085 1826 364 0 8624 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 290800

[startup+3.10071 s]
/proc/loadavg: 8.11 6.56 3.35 6/221 23511
/proc/meminfo: memFree=29408912/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=325512 CPUtime=2.39 cores=0,2,4,6
/proc/23505/stat : 23505 (python) S 23503 23505 23228 0 -1 4202496 16987 935 1 0 226 13 0 0 25 0 5 0 617011384 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/23505/statm: 81378 8577 1883 364 0 17287 0
Current children cumulated CPU time (s) 2.39
Current children cumulated vsize (KiB) 325512
heavy processes:

[startup+6.30112 s]
/proc/loadavg: 7.86 6.53 3.35 9/233 23523
/proc/meminfo: memFree=29375860/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=325512 CPUtime=2.39 cores=0,2,4,6
/proc/23505/stat : 23505 (python) S 23503 23505 23228 0 -1 4202496 16987 935 1 0 226 13 0 0 25 0 5 0 617011384 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/23505/statm: 81378 8577 1883 364 0 17287 0
[pid=23512] ppid=23505 vsize=284984 CPUtime=0 cores=0,2,4,6
/proc/23512/stat : 23512 (python) S 23505 23505 23228 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 617011641 291823616 6899 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414159 0 0 16781312 514 4088907551255527684 0 0 17 2 0 0 0
/proc/23512/statm: 71246 6899 355 364 0 7164 0
[pid=23514] ppid=23512 vsize=1020308 CPUtime=3.69 cores=0,2,4,6
/proc/23514/stat : 23514 (gnovelty+2) R 23512 23514 23514 0 -1 4202496 1157 0 0 0 369 0 0 0 25 0 1 0 617011642 1044795392 892 33554432000 134512640 135663704 4293228656 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23514/statm: 255077 892 113 282 0 254793 0
[pid=23515] ppid=23505 vsize=295004 CPUtime=0 cores=0,2,4,6
/proc/23515/stat : 23515 (python) S 23505 23505 23228 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 617011644 302084096 7032 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4051977704457601284 0 0 17 2 0 0 0
/proc/23515/statm: 73751 7032 349 364 0 9666 0
[pid=23517] ppid=23515 vsize=1020308 CPUtime=3.69 cores=0,2,4,6
/proc/23517/stat : 23517 (gnovelty+2) R 23515 23517 23517 0 -1 4202496 1155 0 0 0 369 0 0 0 25 0 1 0 617011645 1044795392 892 33554432000 134512640 135663704 4294756384 18446744073709551615 134544264 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23517/statm: 255077 892 113 282 0 254793 0
[pid=23518] ppid=23505 vsize=305328 CPUtime=0 cores=0,2,4,6
/proc/23518/stat : 23518 (python) S 23505 23505 23228 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 617011647 312655872 7036 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4603267337066348801 0 0 17 2 0 0 0
/proc/23518/statm: 76332 7036 349 364 0 12244 0
[pid=23520] ppid=23518 vsize=423400 CPUtime=3.66 cores=0,2,4,6
/proc/23520/stat : 23520 (adaptg2wsat2009) R 23518 23520 23520 0 -1 4202496 2622 0 0 0 366 0 0 0 25 0 1 0 617011647 433561600 2357 33554432000 134512640 135107411 4291524544 18446744073709551615 134531411 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23520/statm: 105850 2357 59 146 0 104726 0
[pid=23521] ppid=23505 vsize=315416 CPUtime=0 cores=0,2,4,6
/proc/23521/stat : 23521 (python) S 23505 23505 23228 0 -1 4202560 754 0 0 0 0 0 0 0 18 0 1 0 617011650 322985984 7040 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4049656635411366147 0 0 17 0 0 0 0
/proc/23521/statm: 78854 7040 349 364 0 14763 0
[pid=23523] ppid=23521 vsize=423400 CPUtime=3.63 cores=0,2,4,6
/proc/23523/stat : 23523 (adaptg2wsat2009) R 23521 23523 23523 0 -1 4202496 2620 0 0 0 363 0 0 0 25 0 1 0 617011650 433561600 2356 33554432000 134512640 135107411 4292701776 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23523/statm: 105850 2356 59 146 0 104726 0
[pid=23505/tid=23513] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23513/stat : 23513 (python) S 23503 23505 23228 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 617011641 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23516] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23516/stat : 23516 (python) S 23503 23505 23228 0 -1 4202560 22 935 0 0 0 0 0 0 25 0 5 0 617011644 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23519] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23519/stat : 23519 (python) S 23503 23505 23228 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 617011647 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23522] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23522/stat : 23522 (python) S 23503 23505 23228 0 -1 4202560 24 935 0 0 0 0 0 0 25 0 5 0 617011650 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 17.06
Current children cumulated vsize (KiB) 4413660

[startup+12.7009 s]
/proc/loadavg: 7.87 6.56 3.38 9/233 23523
/proc/meminfo: memFree=29374768/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=325512 CPUtime=2.39 cores=0,2,4,6
/proc/23505/stat : 23505 (python) S 23503 23505 23228 0 -1 4202496 16987 935 1 0 226 13 0 0 25 0 5 0 617011384 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/23505/statm: 81378 8577 1883 364 0 17287 0
[pid=23512] ppid=23505 vsize=284984 CPUtime=0.01 cores=0,2,4,6
/proc/23512/stat : 23512 (python) S 23505 23505 23228 0 -1 4202560 901 0 0 0 0 1 0 0 18 0 1 0 617011641 291823616 6922 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414159 0 0 16781312 514 4472338312769536260 0 0 17 2 0 0 0
/proc/23512/statm: 71246 6922 378 364 0 7164 0
[pid=23514] ppid=23512 vsize=1020308 CPUtime=10.05 cores=0,2,4,6
/proc/23514/stat : 23514 (gnovelty+2) R 23512 23514 23514 0 -1 4202496 1157 0 0 0 1005 0 0 0 25 0 1 0 617011642 1044795392 892 33554432000 134512640 135663704 4293228656 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23514/statm: 255077 892 113 282 0 254793 0
[pid=23515] ppid=23505 vsize=295004 CPUtime=0 cores=0,2,4,6
/proc/23515/stat : 23515 (python) S 23505 23505 23228 0 -1 4202560 916 0 0 0 0 0 0 0 18 0 1 0 617011644 302084096 7055 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4051977704457601284 0 0 17 2 0 0 0
/proc/23515/statm: 73751 7055 372 364 0 9666 0
[pid=23517] ppid=23515 vsize=1020308 CPUtime=10.09 cores=0,2,4,6
/proc/23517/stat : 23517 (gnovelty+2) R 23515 23517 23517 0 -1 4202496 1155 0 0 0 1009 0 0 0 25 0 1 0 617011645 1044795392 892 33554432000 134512640 135663704 4294756384 18446744073709551615 134525213 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23517/statm: 255077 892 113 282 0 254793 0
[pid=23518] ppid=23505 vsize=305328 CPUtime=0.01 cores=0,2,4,6
/proc/23518/stat : 23518 (python) S 23505 23505 23228 0 -1 4202560 910 0 0 0 0 1 0 0 18 0 1 0 617011647 312655872 7059 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4472338312769536257 0 0 17 2 0 0 0
/proc/23518/statm: 76332 7059 372 364 0 12244 0
[pid=23520] ppid=23518 vsize=423400 CPUtime=10.01 cores=0,2,4,6
/proc/23520/stat : 23520 (adaptg2wsat2009) R 23518 23520 23520 0 -1 4202496 2622 0 0 0 1001 0 0 0 25 0 1 0 617011647 433561600 2357 33554432000 134512640 135107411 4291524544 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23520/statm: 105850 2357 59 146 0 104726 0
[pid=23521] ppid=23505 vsize=315416 CPUtime=0.01 cores=0,2,4,6
/proc/23521/stat : 23521 (python) S 23505 23505 23228 0 -1 4202560 925 0 0 0 1 0 0 0 18 0 1 0 617011650 322985984 7063 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4472338312769536259 0 0 17 0 0 0 0
/proc/23521/statm: 78854 7063 372 364 0 14763 0
[pid=23523] ppid=23521 vsize=423400 CPUtime=10.01 cores=0,2,4,6
/proc/23523/stat : 23523 (adaptg2wsat2009) R 23521 23523 23523 0 -1 4202496 2620 0 0 0 1001 0 0 0 25 0 1 0 617011650 433561600 2356 33554432000 134512640 135107411 4292701776 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23523/statm: 105850 2356 59 146 0 104726 0
[pid=23505/tid=23513] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23513/stat : 23513 (python) S 23503 23505 23228 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 617011641 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23516] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23516/stat : 23516 (python) S 23503 23505 23228 0 -1 4202560 22 935 0 0 0 0 0 0 25 0 5 0 617011644 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23519] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23519/stat : 23519 (python) S 23503 23505 23228 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 617011647 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23522] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23522/stat : 23522 (python) S 23503 23505 23228 0 -1 4202560 24 935 0 0 0 0 0 0 25 0 5 0 617011650 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 42.58
Current children cumulated vsize (KiB) 4413660

[startup+25.5006 s]
/proc/loadavg: 7.97 6.64 3.46 9/233 23523
/proc/meminfo: memFree=29372592/32950928 swapFree=67054472/67111528
[pid=23505] ppid=23503 vsize=325512 CPUtime=2.39 cores=0,2,4,6
/proc/23505/stat : 23505 (python) S 23503 23505 23228 0 -1 4202496 16987 935 1 0 226 13 0 0 25 0 5 0 617011384 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/23505/statm: 81378 8577 1883 364 0 17287 0
[pid=23512] ppid=23505 vsize=284984 CPUtime=0.02 cores=0,2,4,6
/proc/23512/stat : 23512 (python) S 23505 23505 23228 0 -1 4202560 906 0 0 0 1 1 0 0 18 0 1 0 617011641 291823616 6922 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414159 0 0 16781312 514 4472338312769536260 0 0 17 2 0 0 0
/proc/23512/statm: 71246 6922 378 364 0 7164 0
[pid=23514] ppid=23512 vsize=1020308 CPUtime=22.77 cores=0,2,4,6
/proc/23514/stat : 23514 (gnovelty+2) R 23512 23514 23514 0 -1 4202496 1157 0 0 0 2277 0 0 0 25 0 1 0 617011642 1044795392 892 33554432000 134512640 135663704 4293228656 18446744073709551615 134524888 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23514/statm: 255077 892 113 282 0 254793 0
[pid=23515] ppid=23505 vsize=295004 CPUtime=0.03 cores=0,2,4,6
/proc/23515/stat : 23515 (python) S 23505 23505 23228 0 -1 4202560 923 0 0 0 2 1 0 0 18 0 1 0 617011644 302084096 7055 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4472338312769536260 0 0 17 2 0 0 0
/proc/23515/statm: 73751 7055 372 364 0 9666 0
[pid=23517] ppid=23515 vsize=1020308 CPUtime=22.89 cores=0,2,4,6
/proc/23517/stat : 23517 (gnovelty+2) R 23515 23517 23517 0 -1 4202496 1155 0 0 0 2289 0 0 0 25 0 1 0 617011645 1044795392 892 33554432000 134512640 135663704 4294756384 18446744073709551615 134525302 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23517/statm: 255077 892 113 282 0 254793 0
[pid=23518] ppid=23505 vsize=305328 CPUtime=0.03 cores=0,2,4,6
/proc/23518/stat : 23518 (python) S 23505 23505 23228 0 -1 4202560 915 0 0 0 2 1 0 0 18 0 1 0 617011647 312655872 7059 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4472338312769536257 0 0 17 2 0 0 0
/proc/23518/statm: 76332 7059 372 364 0 12244 0
[pid=23520] ppid=23518 vsize=423400 CPUtime=22.75 cores=0,2,4,6
/proc/23520/stat : 23520 (adaptg2wsat2009) R 23518 23520 23520 0 -1 4202496 2622 0 0 0 2275 0 0 0 25 0 1 0 617011647 433561600 2357 33554432000 134512640 135107411 4291524544 18446744073709551615 134531709 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23520/statm: 105850 2357 59 146 0 104726 0
[pid=23521] ppid=23505 vsize=315416 CPUtime=0.03 cores=0,2,4,6
/proc/23521/stat : 23521 (python) S 23505 23505 23228 0 -1 4202560 931 0 0 0 2 1 0 0 18 0 1 0 617011650 322985984 7063 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4472338312769536259 0 0 17 0 0 0 0
/proc/23521/statm: 78854 7063 372 364 0 14763 0
[pid=23523] ppid=23521 vsize=423400 CPUtime=22.8 cores=0,2,4,6
/proc/23523/stat : 23523 (adaptg2wsat2009) R 23521 23523 23523 0 -1 4202496 2620 0 0 0 2279 1 0 0 25 0 1 0 617011650 433561600 2356 33554432000 134512640 135107411 4292701776 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23523/statm: 105850 2356 59 146 0 104726 0
[pid=23505/tid=23513] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23513/stat : 23513 (python) S 23503 23505 23228 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 5 0 617011641 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23516] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23516/stat : 23516 (python) S 23503 23505 23228 0 -1 4202560 22 935 0 0 0 0 0 0 25 0 5 0 617011644 333324288 8577 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23519] ppid=23503 vsize=325512 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

/proc/23520/stat : 23520 (adaptg2wsat2009) T 23518 23520 23520 0 -1 4202496 2622 0 0 0 9968 0 0 0 25 0 1 0 617011647 433561600 2357 33554432000 134512640 135107411 4291524544 18446744073709551615 134532318 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/23520/statm: 105850 2357 59 146 0 104726 0
[pid=23518/tid=23527] ppid=23505 vsize=305328 CPUtime=0 cores=0,2,4,6
/proc/23518/task/23527/stat : 23527 (python) S 23505 23505 23228 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 617021663 312655872 7088 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453984474944 0 0 -1 2 0 0 0
[pid=23521] ppid=23505 vsize=315416 CPUtime=0.15 cores=0,2,4,6
/proc/23521/stat : 23521 (python) S 23505 23505 23228 0 -1 4202560 1113 0 0 0 9 6 0 0 19 0 2 0 617011650 322985984 7092 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23521/statm: 78854 7092 400 364 0 14763 0
[pid=23523] ppid=23521 vsize=423400 CPUtime=100.01 cores=0,2,4,6
/proc/23523/stat : 23523 (adaptg2wsat2009) T 23521 23523 23523 0 -1 4202496 2620 0 0 0 10000 1 0 0 25 0 1 0 617011650 433561600 2356 33554432000 134512640 135107411 4292701776 18446744073709551615 134531535 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/23523/statm: 105850 2356 59 146 0 104726 0
[pid=23521/tid=23531] ppid=23505 vsize=315416 CPUtime=0 cores=0,2,4,6
/proc/23521/task/23531/stat : 23531 (python) S 23505 23505 23228 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617021666 322985984 7092 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453855577408 0 0 -1 0 0 0 0
[pid=23528] ppid=23505 vsize=326276 CPUtime=0.15 cores=0,2,4,6
/proc/23528/stat : 23528 (python) S 23505 23505 23228 0 -1 4202560 1119 0 0 0 9 6 0 0 19 0 2 0 617021664 334106624 7096 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23528/statm: 81569 7096 400 364 0 17475 0
[pid=23530] ppid=23528 vsize=423400 CPUtime=99.97 cores=0,2,4,6
/proc/23530/stat : 23530 (adaptg2wsat2009) T 23528 23530 23530 0 -1 4202496 2619 0 0 0 9997 0 0 0 25 0 1 0 617021664 433561600 2356 33554432000 134512640 135107411 4287633680 18446744073709551615 134531709 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/23530/statm: 105850 2356 59 146 0 104726 0
[pid=23528/tid=23554] ppid=23505 vsize=326276 CPUtime=0 cores=0,2,4,6
/proc/23528/task/23554/stat : 23554 (python) S 23505 23505 23228 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617031678 334106624 7096 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453876008768 0 0 -1 0 0 0 0
[pid=23532] ppid=23505 vsize=336048 CPUtime=0.14 cores=0,2,4,6
/proc/23532/stat : 23532 (python) S 23505 23505 23228 0 -1 4202560 1120 0 0 0 7 7 0 0 19 0 2 0 617021667 344113152 7166 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/23532/statm: 84012 7166 399 364 0 19915 0
[pid=23534] ppid=23532 vsize=1020308 CPUtime=100.03 cores=0,2,4,6
/proc/23534/stat : 23534 (gnovelty+2) T 23532 23534 23534 0 -1 4202496 1156 0 0 0 10003 0 0 0 25 0 1 0 617021667 1044795392 892 33554432000 134512640 135663704 4287215600 18446744073709551615 134544225 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/23534/statm: 255077 892 113 282 0 254793 0
[pid=23532/tid=23558] ppid=23505 vsize=336048 CPUtime=0 cores=0,2,4,6
/proc/23532/task/23558/stat : 23558 (python) S 23505 23505 23228 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617031687 344113152 7166 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604448508650432 0 0 -1 4 0 0 0
[pid=23535] ppid=23505 vsize=346496 CPUtime=0.15 cores=0,2,4,6
/proc/23535/stat : 23535 (python) S 23505 23505 23228 0 -1 4202560 1176 0 0 0 8 7 0 0 19 0 2 0 617021671 354811904 7177 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/23535/statm: 86624 7177 406 364 0 22524 0
[pid=23537] ppid=23535 vsize=253032 CPUtime=100.14 cores=0,2,4,6
/proc/23537/stat : 23537 (FH) T 23535 23537 23537 0 -1 4202496 2495 0 0 0 10014 0 0 0 25 0 1 0 617021671 259104768 2233 33554432000 134512640 134936988 4290252464 18446744073709551615 134554976 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/23537/statm: 63258 2233 41 104 0 62762 0
[pid=23535/tid=23559] ppid=23505 vsize=346496 CPUtime=0 cores=0,2,4,6
/proc/23535/task/23559/stat : 23559 (python) S 23505 23505 23228 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 617031687 354811904 7177 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604452939946880 0 0 -1 6 0 0 0
[pid=23539] ppid=23505 vsize=356732 CPUtime=0.15 cores=0,2,4,6
/proc/23539/stat : 23539 (python) S 23505 23505 23228 0 -1 4202560 1165 0 0 0 8 7 0 0 19 0 2 0 617021677 365293568 7181 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/23539/statm: 89183 7181 406 364 0 25080 0
[pid=23541] ppid=23539 vsize=423400 CPUtime=98.23 cores=0,2,4,6
/proc/23541/stat : 23541 (adaptg2wsat2009) T 23539 23541 23541 0 -1 4202496 2629 0 0 0 9823 0 0 0 25 0 1 0 617021677 433561600 2357 33554432000 134512640 135107411 4291864128 18446744073709551615 134532371 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/23541/statm: 105850 2357 59 146 0 104726 0
[pid=23539/tid=23563] ppid=23505 vsize=356732 CPUtime=0 cores=0,2,4,6
/proc/23539/task/23563/stat : 23563 (python) S 23505 23505 23228 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617031694 365293568 7181 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604452939945984 0 0 -1 6 0 0 0
[pid=23555] ppid=23505 vsize=366816 CPUtime=0.16 cores=0,2,4,6
/proc/23555/stat : 23555 (python) S 23505 23505 23228 0 -1 4202560 1143 0 0 0 9 7 0 0 19 0 2 0 617031682 375619584 7178 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/23555/statm: 91704 7178 399 364 0 27598 0
[pid=23557] ppid=23555 vsize=423400 CPUtime=99.13 cores=0,2,4,6
/proc/23557/stat : 23557 (adaptg2wsat2009) T 23555 23557 23557 0 -1 4202496 2622 0 0 0 9913 0 0 0 25 0 1 0 617031682 433561600 2356 33554432000 134512640 135107411 4288895424 18446744073709551615 134531741 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/23557/statm: 105850 2356 59 146 0 104726 0
[pid=23555/tid=23577] ppid=23505 vsize=366816 CPUtime=0 cores=0,2,4,6
/proc/23555/task/23577/stat : 23577 (python) S 23505 23505 23228 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617041598 375619584 7178 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604449067857088 0 0 -1 6 0 0 0
[pid=23560] ppid=23505 vsize=377752 CPUtime=0.16 cores=0,2,4,6
/proc/23560/stat : 23560 (python) S 23505 23505 23228 0 -1 4202560 1161 0 0 0 9 7 0 0 19 0 2 0 617031690 386818048 7183 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/23560/statm: 94438 7183 400 364 0 30329 0
[pid=23562] ppid=23560 vsize=253028 CPUtime=100.14 cores=0,2,4,6
/proc/23562/stat : 23562 (FH) T 23560 23562 23562 0 -1 4202496 2499 0 0 0 10014 0 0 0 25 0 1 0 617031691 259100672 2232 33554432000 134512640 134936988 4288991024 18446744073709551615 134545336 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/23562/statm: 63257 2232 41 104 0 62761 0
[pid=23560/tid=23581] ppid=23505 vsize=377752 CPUtime=0 cores=0,2,4,6
/proc/23560/task/23581/stat : 23581 (python) S 23505 23505 23228 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 617041706 386818048 7183 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453856116928 0 0 -1 0 0 0 0
[pid=23564] ppid=23505 vsize=387748 CPUtime=0.5 cores=0,2,4,6
/proc/23564/stat : 23564 (python) S 23505 23505 23228 0 -1 4202560 1201 0 0 0 27 23 0 0 18 0 2 0 617031695 397053952 7236 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/23564/statm: 96937 7236 400 364 0 32825 0
[pid=23566] ppid=23564 vsize=268424 CPUtime=295.64 cores=0,2,4,6
/proc/23566/stat : 23566 (hybridGM3) T 23564 23566 23566 0 -1 4202496 6190 0 0 0 29564 0 0 0 25 0 1 0 617031695 274866176 5919 33554432000 134512640 135204385 4288055408 18446744073709551615 134538397 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/23566/statm: 67106 5919 63 169 0 66936 0
[pid=23564/tid=23594] ppid=23505 vsize=387748 CPUtime=0 cores=0,2,4,6
/proc/23564/task/23594/stat : 23594 (python) S 23505 23505 23228 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 617051334 397053952 7236 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453851829376 0 0 -1 4 0 0 0
[pid=23567] ppid=23505 vsize=398132 CPUtime=0.16 cores=0,2,4,6
/proc/23567/stat : 23567 (python) S 23505 23505 23228 0 -1 4202560 1168 0 0 0 9 7 0 0 19 0 2 0 617031700 407687168 7240 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/23567/statm: 99533 7240 400 364 0 35418 0
[pid=23569] ppid=23567 vsize=1020308 CPUtime=99.78 cores=0,2,4,6
/proc/23569/stat : 23569 (gnovelty+2) T 23567 23569 23569 0 -1 4202496 1161 0 0 0 9978 0 0 0 25 0 1 0 617031700 1044795392 891 33554432000 134512640 135663704 4286705776 18446744073709551615 134524534 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/23569/statm: 255077 891 113 282 0 254793 0
[pid=23567/tid=23585] ppid=23505 vsize=398132 CPUtime=0 cores=0,2,4,6
/proc/23567/task/23585/stat : 23585 (python) S 23505 23505 23228 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 617041727 407687168 7240 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453855578304 0 0 -1 2 0 0 0
[pid=23578] ppid=23505 vsize=408192 CPUtime=0.33 cores=0,2,4,6
/proc/23578/stat : 23578 (python) S 23505 23505 23228 0 -1 4202560 1196 0 0 0 17 16 0 0 18 0 2 0 617041602 417988608 7244 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/23578/statm: 102048 7244 400 364 0 37930 0
[pid=23580] ppid=23578 vsize=268420 CPUtime=199.35 cores=0,2,4,6
/proc/23580/stat : 23580 (hybridGM3) R 23578 23580 23580 0 -1 4202496 6190 0 0 0 19935 0 0 0 25 0 1 0 617041603 274862080 5918 33554432000 134512640 135204385 4294021120 18446744073709551615 134538295 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/23580/statm: 67105 5918 63 169 0 66935 0
[pid=23578/tid=23606] ppid=23505 vsize=408192 CPUtime=0 cores=0,2,4,6
/proc/23578/task/23606/stat : 23606 (python) S 23505 23505 23228 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 617061146 417988608 7244 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453873586432 0 0 -1 4 0 0 0
[pid=23582] ppid=23505 vsize=418592 CPUtime=0.17 cores=0,2,4,6
/proc/23582/stat : 23582 (python) S 23505 23505 23228 0 -1 4202560 1170 0 0 0 9 8 0 0 19 0 2 0 617041711 428638208 7248 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/23582/statm: 104648 7248 400 364 0 40527 0
[pid=23584] ppid=23582 vsize=423400 CPUtime=99.5 cores=0,2,4,6
/proc/23584/stat : 23584 (adaptg2wsat2009) T 23582 23584 23584 0 -1 4202496 2628 0 0 0 9950 0 0 0 25 0 1 0 617041711 433561600 2356 33554432000 134512640 135107411 4293516304 18446744073709551615 134531435 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/23584/statm: 105850 2356 59 146 0 104726 0
[pid=23582/tid=23595] ppid=23505 vsize=418592 CPUtime=0 cores=0,2,4,6
/proc/23582/task/23595/stat : 23595 (python) S 23505 23505 23228 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 617051727 428638208 7248 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453889905920 0 0 -1 4 0 0 0
[pid=23586] ppid=23505 vsize=428816 CPUtime=0.32 cores=0,2,4,6
/proc/23586/stat : 23586 (python) S 23505 23505 23228 0 -1 4202560 1231 0 0 0 17 15 0 0 19 0 2 0 617041732 439107584 7259 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917220811 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/23586/statm: 107204 7259 407 364 0 43080 0
[pid=23588] ppid=23586 vsize=268420 CPUtime=194.13 cores=0,2,4,6
/proc/23588/stat : 23588 (hybridGM3) T 23586 23588 23588 0 -1 4202496 6190 0 0 0 19413 0 0 0 25 0 1 0 617041732 274862080 5918 33554432000 134512640 135204385 4293738048 18446744073709551615 134538368 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/23588/statm: 67105 5918 63 169 0 66935 0
[pid=23586/tid=23607] ppid=23505 vsize=428816 CPUtime=0 cores=0,2,4,6
/proc/23586/task/23607/stat : 23607 (python) S 23505 23505 23228 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 617061371 439107584 7259 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 514 18446604453860911872 0 0 -1 4 0 0 0
[pid=23596] ppid=23505 vsize=439084 CPUtime=0.15 cores=0,2,4,6
/proc/23596/stat : 23596 (python) S 23505 23505 23228 0 -1 4202560 1084 0 0 0 7 8 0 0 18 0 1 0 617051732 449622016 7258 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4472338312769536260 0 0 17 6 0 0 0
/proc/23596/statm: 109771 7258 379 364 0 45644 0
[pid=23598] ppid=23596 vsize=268420 CPUtime=95.42 cores=0,2,4,6
/proc/23598/stat : 23598 (hybridGM3) R 23596 23598 23598 0 -1 4202496 6193 0 0 0 9542 0 0 0 25 0 1 0 617051732 274862080 5919 33554432000 134512640 135204385 4291408336 18446744073709551615 134538432 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/23598/statm: 67105 5919 63 169 0 66935 0
[pid=23608] ppid=23505 vsize=449612 CPUtime=0 cores=0,2,4,6
/proc/23608/stat : 23608 (python) S 23505 23505 23228 0 -1 4202560 852 0 0 0 0 0 0 0 18 0 1 0 617061376 460402688 7242 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4137404810133503999 0 0 17 6 0 0 0
/proc/23608/statm: 112403 7242 351 364 0 48273 0
[pid=23610] ppid=23608 vsize=461508 CPUtime=1.67 cores=0,2,4,6
/proc/23610/stat : 23610 (TNM) R 23608 23610 23610 0 -1 4202496 2467 0 0 0 167 0 0 0 25 0 1 0 617061376 472584192 2195 33554432000 134512640 135008748 4291682976 18446744073709551615 134527477 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/23610/statm: 115377 2195 46 122 0 114277 0
[pid=23611] ppid=23505 vsize=459956 CPUtime=0 cores=0,2,4,6
/proc/23611/stat : 23611 (python) S 23505 23505 23228 0 -1 4202560 828 0 0 0 0 0 0 0 19 0 1 0 617061459 470994944 7242 33554432000 4194304 5683644 140735070319120 18446744073709551615 260905414214 0 0 16781312 514 4137400961842806783 0 0 17 4 0 0 0
/proc/23611/statm: 114989 7242 347 364 0 50856 0
[pid=23613] ppid=23611 vsize=249960 CPUtime=0.82 cores=0,2,4,6
/proc/23613/stat : 23613 (FH) R 23611 23613 23613 0 -1 4202496 1818 0 0 0 82 0 0 0 23 0 1 0 617061459 255959040 1547 33554432000 134512640 134936988 4288533920 18446744073709551615 134554762 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/23613/statm: 62490 1547 39 104 0 61994 0
[pid=23505/tid=23513] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23513/stat : 23513 (python) S 23503 23505 23228 0 -1 4202560 26 935 0 0 0 0 0 0 25 0 19 0 617011641 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23516] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23516/stat : 23516 (python) S 23503 23505 23228 0 -1 4202560 22 935 0 0 0 0 0 0 25 0 19 0 617011644 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23519] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23519/stat : 23519 (python) S 23503 23505 23228 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 19 0 617011647 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23522] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23522/stat : 23522 (python) S 23503 23505 23228 0 -1 4202560 24 935 0 0 0 0 0 0 25 0 19 0 617011650 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 0 0 0 0
[pid=23505/tid=23529] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23529/stat : 23529 (python) S 23503 23505 23228 0 -1 4202560 25 935 0 0 0 0 0 0 20 0 19 0 617021664 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 0 0 0 0
[pid=23505/tid=23533] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23533/stat : 23533 (python) S 23503 23505 23228 0 -1 4202560 27 935 0 0 0 0 0 0 22 0 19 0 617021667 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 2 0 0 0
[pid=23505/tid=23536] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23536/stat : 23536 (python) S 23503 23505 23228 0 -1 4202560 30 935 0 0 0 0 0 0 24 0 19 0 617021671 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 6 0 0 0
[pid=23505/tid=23540] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23540/stat : 23540 (python) S 23503 23505 23228 0 -1 4202560 30 935 0 0 0 0 0 0 25 0 19 0 617021677 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 6 0 0 0
[pid=23505/tid=23556] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23556/stat : 23556 (python) S 23503 23505 23228 0 -1 4202560 32 935 0 0 0 0 0 0 20 0 19 0 617031682 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 6 0 0 0
[pid=23505/tid=23561] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23561/stat : 23561 (python) S 23503 23505 23228 0 -1 4202560 21 935 0 0 0 0 0 0 21 0 19 0 617031691 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23565] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23565/stat : 23565 (python) S 23503 23505 23228 0 -1 4202560 25 935 0 0 0 0 0 0 18 0 19 0 617031695 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23568] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23568/stat : 23568 (python) S 23503 23505 23228 0 -1 4202560 29 935 0 0 0 0 0 0 25 0 19 0 617031700 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23579] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23579/stat : 23579 (python) S 23503 23505 23228 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 19 0 617041603 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23583] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23583/stat : 23583 (python) S 23503 23505 23228 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 19 0 617041711 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23587] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23587/stat : 23587 (python) S 23503 23505 23228 0 -1 4202560 26 935 0 0 0 0 0 0 20 0 19 0 617041732 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23597] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23597/stat : 23597 (python) S 23503 23505 23228 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 19 0 617051732 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23609] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23609/stat : 23609 (python) S 23503 23505 23228 0 -1 4202560 30 935 0 0 0 0 0 0 20 0 19 0 617061376 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
[pid=23505/tid=23612] ppid=23503 vsize=469560 CPUtime=0 cores=0,2,4,6
/proc/23505/task/23612/stat : 23612 (python) S 23503 23505 23228 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 19 0 617061459 480829440 8810 33554432000 4194304 5683644 140735070319120 18446744073709551615 260917217713 0 0 16781312 2 18446604445737730432 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1989.67
Current children cumulated vsize (KiB) 16093632

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 23505 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=941552
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=177972
# CPU time returned by wait4() is 3.11952
# while last known CPU time is 1989.67
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 501.719
CPU time (s): 1989.67
CPU user time (s): 1988.04
CPU system time (s): 1.63
CPU usage (%): 396.57
Max. virtual memory (cumulated for all children) (KiB): 16093632

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.94155
system time used= 0.177972
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 32017
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3154
involuntary context switches= 49

runsolver used 2.35164 second user time and 5.97309 second system time

The end

Launcher Data 

Begin job on node116 at 2011-05-01 18:36:47
IDJOB=3317151
IDBENCH=82616
IDSOLVER=1705
FILE ID=node116/3317151-1304267807
RUNJOBID= node116-1304267284-23246
PBS_JOBID= 13173963
Free space on /tmp= 73256 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S837431592-027-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317151-1304267807/watcher-3317151-1304267807 -o /tmp/evaluation-result-3317151-1304267807/solver-3317151-1304267807 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317151-1304267807.cnf -seed 1755707011 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3b045b7b4cbac0d258a07eda037c4299
RANDOM SEED=1755707011

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      29444292 kB
Buffers:       1041588 kB
Cached:        1673128 kB
SwapCached:       9656 kB
Active:         759700 kB
Inactive:      2391448 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      29444292 kB
SwapTotal:    67111528 kB
SwapFree:     67054472 kB
Dirty:           32660 kB
Writeback:          20 kB
AnonPages:      435204 kB
Mapped:          20268 kB
Slab:           283596 kB
PageTables:      13240 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 10996788 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73224 MiB
End job on node116 at 2011-05-01 18:45:11