Trace number 3317056

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 4344.82 1118.74

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S1969540674-065-UNKNOWN.cnf
MD5SUMfcb14103324f734a2e69e02c062e67ed
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables50000
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.51/19.76	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.51/19.77	c INFO: solving HOME/instance-3317056-1304267281.cnf
0.51/19.77	c INFO: computing task features
1.15/20.49	c INFO: computed variable-clause graph statistics
1.25/20.52	c INFO: computed clause balance statistics
4.96/24.26	c INFO: computed variable balance statistics
7.85/27.18	c INFO: computed small-clause counts
7.95/27.20	c INFO: computed Horn-clause counts
10.95/30.27	c INFO: computed variable graph statistics
10.95/30.27	c INFO: collected features for HOME/instance-3317056-1304267281.cnf in 10.49 s
11.05/30.30	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.35)
11.05/30.33	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.21)
11.05/30.37	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.14)
11.15/30.40	c INFO: running hybridGM3@0 for 97 with 4789 remaining (b = 0.09)
411.09/130.59	c INFO: running hybridGM3@0 for 97 with 4689 remaining (b = 0.08)
411.46/130.62	c INFO: running hybridGM3@0 for 97 with 4689 remaining (b = 0.08)
411.46/130.66	c INFO: running hybridGM3@0 for 97 with 4689 remaining (b = 0.08)
411.46/130.69	c INFO: running hybridGM3@100 for 97 with 4689 remaining (b = 0.08)
787.23/224.76	c INFO: running hybridGM3@99 for 97 with 4595 remaining (b = 0.08)
791.20/225.73	c INFO: running TNM@0 for 97 with 4594 remaining (b = 0.08)
791.57/225.83	c INFO: running hybridGM3@100 for 97 with 4593 remaining (b = 0.07)
811.58/230.82	c INFO: running hybridGM3@100 for 97 with 4588 remaining (b = 0.07)
1187.36/324.91	c INFO: running hybridGM3@0 for 194 with 4494 remaining (b = 0.13)
1187.36/324.94	c INFO: running hybridGM3@0 for 194 with 4494 remaining (b = 0.09)
1191.32/325.97	c INFO: running hybridGM3@0 for 194 with 4493 remaining (b = 0.07)
1195.28/326.99	c INFO: running TNM@0 for 97 with 4492 remaining (b = 0.03)
1590.42/426.18	c INFO: running hybridGM3@0 for 194 with 4393 remaining (b = 0.05)
1966.07/520.46	c INFO: running hybridGM3@94 for 97 with 4299 remaining (b = 0.02)
1970.07/521.44	c INFO: running hybridGM3@100 for 97 with 4298 remaining (b = 0.02)
1986.39/525.50	c INFO: running hybridGM3@101 for 97 with 4294 remaining (b = 0.02)
2349.30/616.62	c INFO: running hybridGM3@297 for 97 with 4203 remaining (b = 0.02)
2365.23/620.66	c INFO: running hybridGM3@0 for 194 with 4199 remaining (b = 0.04)
2385.11/625.69	c INFO: running TNM@0 for 97 with 4194 remaining (b = 0.02)
2385.90/625.81	c INFO: running TNM@0 for 97 with 4193 remaining (b = 0.01)
2752.27/717.81	c INFO: running hybridGM3@397 for 97 with 4101 remaining (b = 0.01)
2780.18/724.90	c INFO: running TNM@0 for 97 with 4094 remaining (b = 0.01)
2780.86/725.02	c INFO: running TNM@0 for 97 with 4094 remaining (b = 0.01)
3130.75/813.00	c INFO: running hybridGM3@0 for 194 with 4006 remaining (b = 0.03)
3143.48/816.27	c INFO: running hybridGM3@98 for 97 with 4003 remaining (b = 0.01)
3175.24/824.24	c INFO: running TNM@0 for 97 with 3995 remaining (b = 0.01)
3178.80/825.12	c INFO: running TNM@197 for 1843 with 3994 remaining (b = 0.18)
3541.57/916.48	c INFO: running hybridGM3@0 for 194 with 3903 remaining (b = 0.02)
3569.34/923.46	c INFO: running hybridGM3@0 for 194 with 3896 remaining (b = 0.02)
3931.65/1014.68	c INFO: running hybridGM3@0 for 194 with 3805 remaining (b = 0.02)
4342.39/1118.06	c INFO: running adaptg2wsat2009++@0 for 194 with 3701 remaining (b = 0.01)
4342.78/1118.18	c INFO: running hybridGM3@99 for 97 with 3701 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317056-1304267281/watcher-3317056-1304267281 -o /tmp/evaluation-result-3317056-1304267281/solver-3317056-1304267281 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317056-1304267281.cnf -seed 489172721 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 1/177 31978
/proc/meminfo: memFree=25351312/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=11800 CPUtime=0 cores=1,3,5,7
/proc/31978/stat : 31978 (runsolver) S 31976 31978 31940 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 1 0 519144261 12083200 54 33554432000 4194304 5434984 140734816438992 18446744073709551615 5055495 0 2147483391 4096 24578 18446744071699199166 0 0 17 5 0 0 0
/proc/31978/statm: 2950 54 34 303 0 2644 0

[startup+0.0775161 s]
/proc/loadavg: 0.00 0.00 0.00 1/177 31978
/proc/meminfo: memFree=25351312/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=16400 CPUtime=0.01 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 1122 0 0 0 1 0 0 0 17 0 1 0 519144261 16793600 989 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106764645 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/31978/statm: 4100 989 413 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.100513 s]
/proc/loadavg: 0.00 0.00 0.00 1/177 31978
/proc/meminfo: memFree=25351312/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=16540 CPUtime=0.01 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 1186 0 0 0 1 0 0 0 17 0 1 0 519144261 16936960 1046 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106766848 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/31978/statm: 4135 1046 417 364 0 622 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16540

[startup+0.300488 s]
/proc/loadavg: 0.00 0.00 0.00 1/177 31978
/proc/meminfo: memFree=25351312/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=18888 CPUtime=0.01 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 1250 0 1 0 1 0 0 0 16 0 1 0 519144261 19341312 1105 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106766848 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 0
/proc/31978/statm: 4722 1105 421 364 0 691 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 18888

[startup+0.700447 s]
/proc/loadavg: 0.00 0.00 0.00 1/177 31978
/proc/meminfo: memFree=25351312/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=21108 CPUtime=0.02 cores=1,3,5,7
/proc/31978/stat : 31978 (python) D 31976 31978 31940 0 -1 4202496 1314 0 2 0 2 0 0 0 16 0 1 0 519144261 21614592 1166 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106766848 0 0 16781312 2 0 0 0 17 7 0 0 3
/proc/31978/statm: 5277 1166 428 364 0 724 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 21108

[startup+1.50031 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 31979
/proc/meminfo: memFree=25346744/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=36512 CPUtime=0.05 cores=1,3,5,7
/proc/31978/stat : 31978 (python) D 31976 31978 31940 0 -1 4202496 1604 0 7 0 3 2 0 0 16 0 1 0 519144261 37388288 1444 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106764645 0 0 16781312 2 0 0 0 17 1 0 0 6
/proc/31978/statm: 9128 1444 452 364 0 945 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 36512

[startup+3.10103 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 31979
/proc/meminfo: memFree=25343752/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=84608 CPUtime=0.08 cores=1,3,5,7
/proc/31978/stat : 31978 (python) D 31976 31978 31940 0 -1 4202496 2395 0 15 0 4 4 0 0 16 0 1 0 519144261 86638592 2136 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106766848 0 0 16781312 2 0 0 0 17 1 0 0 40
/proc/31978/statm: 21152 2136 707 364 0 1369 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 84608
heavy processes:

[startup+6.3005 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 31979
/proc/meminfo: memFree=25331488/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=104228 CPUtime=0.12 cores=1,3,5,7
/proc/31978/stat : 31978 (python) D 31976 31978 31940 0 -1 4202496 3035 0 24 0 6 6 0 0 16 0 1 0 519144261 106729472 2741 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106764645 0 0 16781312 2 0 0 0 17 7 0 0 89
/proc/31978/statm: 26057 2741 847 364 0 1805 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 104228
heavy processes:

[startup+12.7003 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 31987
/proc/meminfo: memFree=25301596/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=202008 CPUtime=0.37 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 7189 930 75 8 23 14 0 0 16 0 1 0 519144261 206856192 5287 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106764645 0 2147483391 16781312 2 18446744071699199166 0 0 17 7 0 0 163
/proc/31978/statm: 50502 5287 1490 364 0 4011 0
Current children cumulated CPU time (s) 0.37
Current children cumulated vsize (KiB) 202008
heavy processes:

[startup+25.5011 s]
/proc/loadavg: 0.40 0.08 0.03 6/190 32000
/proc/meminfo: memFree=25197848/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=328000 CPUtime=6.25 cores=1,3,5,7
/proc/31978/stat : 31978 (python) R 31976 31978 31940 0 -1 4202496 27349 930 108 8 605 20 0 0 25 0 1 0 519144261 335872000 19385 33554432000 4194304 5683644 140734228530640 18446744073709551615 47741738369866 0 0 16781312 2 0 0 0 17 3 0 0 298
/proc/31978/statm: 82000 19385 1827 364 0 17924 0
Current children cumulated CPU time (s) 6.25
Current children cumulated vsize (KiB) 328000
heavy processes:

[startup+51.1006 s]
/proc/loadavg: 2.82 0.67 0.22 9/202 32012
/proc/meminfo: memFree=25035900/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=365132 CPUtime=11.15 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 30336 930 108 8 1094 21 0 0 20 0 5 0 519144261 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 298
/proc/31978/statm: 91283 18479 1881 364 0 27192 0
[pid=32001] ppid=31978 vsize=324120 CPUtime=0.02 cores=1,3,5,7
/proc/32001/stat : 32001 (python) S 31978 31978 31940 0 -1 4202560 911 0 0 0 1 1 0 0 15 0 1 0 519147292 331898880 16958 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788879 0 0 16781312 514 13424737907982172424 0 0 17 3 0 0 0
/proc/32001/statm: 81030 16958 376 364 0 16948 0
[pid=32003] ppid=32001 vsize=276236 CPUtime=20.76 cores=1,3,5,7
/proc/32003/stat : 32003 (hybridGM3) R 32001 32003 32003 0 -1 4202496 9513 0 0 0 2075 1 0 0 25 0 1 0 519147292 282865664 9252 33554432000 134512640 135204385 4292984848 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/32003/statm: 69059 9252 64 169 0 68889 0
[pid=32004] ppid=31978 vsize=334376 CPUtime=0.02 cores=1,3,5,7
/proc/32004/stat : 32004 (python) S 31978 31978 31940 0 -1 4202560 905 0 0 0 1 1 0 0 15 0 1 0 519147295 342401024 16956 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172424 0 0 17 7 0 0 0
/proc/32004/statm: 83594 16956 370 364 0 19509 0
[pid=32006] ppid=32004 vsize=276240 CPUtime=20.72 cores=1,3,5,7
/proc/32006/stat : 32006 (hybridGM3) R 32004 32006 32006 0 -1 4202496 9519 0 0 0 2071 1 0 0 25 0 1 0 519147295 282869760 9257 33554432000 134512640 135204385 4287121040 18446744073709551615 134547764 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/32006/statm: 69060 9257 64 169 0 68890 0
[pid=32007] ppid=31978 vsize=344632 CPUtime=0.02 cores=1,3,5,7
/proc/32007/stat : 32007 (python) S 31978 31978 31940 0 -1 4202560 924 0 0 0 1 1 0 0 15 0 1 0 519147298 352903168 16960 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172423 0 0 17 7 0 0 0
/proc/32007/statm: 86158 16960 370 364 0 22070 0
[pid=32009] ppid=32007 vsize=276240 CPUtime=20.55 cores=1,3,5,7
/proc/32009/stat : 32009 (hybridGM3) R 32007 32009 32009 0 -1 4202496 9520 0 0 0 2054 1 0 0 25 0 1 0 519147299 282869760 9254 33554432000 134512640 135204385 4290009088 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/32009/statm: 69060 9254 64 169 0 68890 0
[pid=32010] ppid=31978 vsize=354888 CPUtime=0.01 cores=1,3,5,7
/proc/32010/stat : 32010 (python) S 31978 31978 31940 0 -1 4202560 935 0 0 0 0 1 0 0 15 0 1 0 519147301 363405312 16963 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172421 0 0 17 1 0 0 0
/proc/32010/statm: 88722 16963 369 364 0 24631 0
[pid=32012] ppid=32010 vsize=276244 CPUtime=20.66 cores=1,3,5,7
/proc/32012/stat : 32012 (hybridGM3) R 32010 32012 32012 0 -1 4202496 9511 0 0 0 2065 1 0 0 25 0 1 0 519147302 282873856 9248 33554432000 134512640 135204385 4292306896 18446744073709551615 134545975 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/32012/statm: 69061 9248 64 169 0 68891 0
[pid=31978/tid=32002] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32002/stat : 32002 (python) S 31976 31978 31940 0 -1 4202560 26 930 0 8 0 0 0 0 25 0 5 0 519147292 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 5 0 0 0
[pid=31978/tid=32005] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32005/stat : 32005 (python) S 31976 31978 31940 0 -1 4202560 22 930 0 8 0 0 0 0 20 0 5 0 519147295 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32008] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32008/stat : 32008 (python) S 31976 31978 31940 0 -1 4202560 20 930 0 8 0 0 0 0 22 0 5 0 519147298 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32011] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32011/stat : 32011 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 5 0 519147302 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 93.91
Current children cumulated vsize (KiB) 2828108

[startup+102.308 s]
/proc/loadavg: 6.04 1.93 0.68 9/202 32014
/proc/meminfo: memFree=25035232/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=365132 CPUtime=11.15 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 30336 930 108 8 1094 21 0 0 20 0 5 0 519144261 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 298
/proc/31978/statm: 91283 18479 1881 364 0 27192 0
[pid=32001] ppid=31978 vsize=324120 CPUtime=0.09 cores=1,3,5,7
/proc/32001/stat : 32001 (python) S 31978 31978 31940 0 -1 4202560 927 0 0 0 5 4 0 0 15 0 1 0 519147292 331898880 16958 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788879 0 0 16781312 514 13424737907982172424 0 0 17 3 0 0 0
/proc/32001/statm: 81030 16958 376 364 0 16948 0
[pid=32003] ppid=32001 vsize=276236 CPUtime=71.96 cores=1,3,5,7
/proc/32003/stat : 32003 (hybridGM3) R 32001 32003 32003 0 -1 4202496 9519 0 0 0 7195 1 0 0 25 0 1 0 519147292 282865664 9258 33554432000 134512640 135204385 4292984848 18446744073709551615 134546153 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/32003/statm: 69059 9258 64 169 0 68889 0
[pid=32004] ppid=31978 vsize=334376 CPUtime=0.09 cores=1,3,5,7
/proc/32004/stat : 32004 (python) S 31978 31978 31940 0 -1 4202560 927 0 0 0 5 4 0 0 15 0 1 0 519147295 342401024 16956 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172424 0 0 17 7 0 0 0
/proc/32004/statm: 83594 16956 370 364 0 19509 0
[pid=32006] ppid=32004 vsize=276240 CPUtime=71.86 cores=1,3,5,7
/proc/32006/stat : 32006 (hybridGM3) R 32004 32006 32006 0 -1 4202496 9519 0 0 0 7185 1 0 0 25 0 1 0 519147295 282869760 9257 33554432000 134512640 135204385 4287121040 18446744073709551615 134547435 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/32006/statm: 69060 9257 64 169 0 68890 0
[pid=32007] ppid=31978 vsize=344632 CPUtime=0.08 cores=1,3,5,7
/proc/32007/stat : 32007 (python) S 31978 31978 31940 0 -1 4202560 942 0 0 0 5 3 0 0 15 0 1 0 519147298 352903168 16960 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172423 0 0 17 7 0 0 0
/proc/32007/statm: 86158 16960 370 364 0 22070 0
[pid=32009] ppid=32007 vsize=276240 CPUtime=71.33 cores=1,3,5,7
/proc/32009/stat : 32009 (hybridGM3) R 32007 32009 32009 0 -1 4202496 9520 0 0 0 7132 1 0 0 25 0 1 0 519147299 282869760 9254 33554432000 134512640 135204385 4290009088 18446744073709551615 134537483 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/32009/statm: 69060 9254 64 169 0 68890 0
[pid=32010] ppid=31978 vsize=354888 CPUtime=0.09 cores=1,3,5,7
/proc/32010/stat : 32010 (python) S 31978 31978 31940 0 -1 4202560 955 0 0 0 4 5 0 0 15 0 1 0 519147301 363405312 16964 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172421 0 0 17 1 0 0 0
/proc/32010/statm: 88722 16964 370 364 0 24631 0
[pid=32012] ppid=32010 vsize=276244 CPUtime=71.8 cores=1,3,5,7
/proc/32012/stat : 32012 (hybridGM3) R 32010 32012 32012 0 -1 4202496 9517 0 0 0 7179 1 0 0 25 0 1 0 519147302 282873856 9254 33554432000 134512640 135204385 4292306896 18446744073709551615 134546059 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/32012/statm: 69061 9254 64 169 0 68891 0
[pid=31978/tid=32002] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32002/stat : 32002 (python) S 31976 31978 31940 0 -1 4202560 26 930 0 8 0 0 0 0 25 0 5 0 519147292 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 5 0 0 0
[pid=31978/tid=32005] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32005/stat : 32005 (python) S 31976 31978 31940 0 -1 4202560 22 930 0 8 0 0 0 0 20 0 5 0 519147295 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32008] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32008/stat : 32008 (python) S 31976 31978 31940 0 -1 4202560 20 930 0 8 0 0 0 0 22 0 5 0 519147298 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32011] ppid=31976 vsize=365132 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32011/stat : 32011 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 5 0 519147302 373895168 18479 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 298.45
Current children cumulated vsize (KiB) 2828108

[startup+162.301 s]
/proc/loadavg: 7.33 3.05 1.14 9/206 32063
/proc/meminfo: memFree=25029356/32951124 swapFree=67111448/67111528
[pid=31978] ppid=31976 vsize=365132 CPUtime=11.26 cores=1,3,5,7
/proc/31978/stat : 31978 (python) S 31976 31978 31940 0 -1 4202496 31103 930 108 8 1105 21 0 0 18 0 5 0 519144261 373895168 18480 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 298

################
# More data... #
################

/proc/32111/stat : 32111 (python) S 31978 31978 31940 0 -1 4202560 1210 0 0 0 24 25 0 0 15 0 2 0 519176859 405413888 17016 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32111/statm: 98978 17016 405 364 0 34875 0
[pid=32113] ppid=32111 vsize=276240 CPUtime=295.41 cores=1,3,5,7
/proc/32113/stat : 32113 (hybridGM3) T 32111 32113 32113 0 -1 4202496 9526 0 0 0 29540 1 0 0 25 0 1 0 519176859 282869760 9254 33554432000 134512640 135204385 4289719248 18446744073709551615 134537100 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/32113/statm: 69060 9254 64 169 0 68890 0
[pid=32111/tid=32141] ppid=31978 vsize=395912 CPUtime=0 cores=1,3,5,7
/proc/32111/task/32141/stat : 32141 (python) S 31978 31978 31940 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 519196807 405413888 17016 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471135233920 0 0 -1 7 0 0 0
[pid=32114] ppid=31978 vsize=406168 CPUtime=0.15 cores=1,3,5,7
/proc/32114/stat : 32114 (python) S 31978 31978 31940 0 -1 4202560 1144 0 0 0 8 7 0 0 19 0 2 0 519176960 415916032 17014 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32114/statm: 101542 17014 399 364 0 37436 0
[pid=32116] ppid=32114 vsize=471116 CPUtime=98.99 cores=1,3,5,7
/proc/32116/stat : 32116 (TNM) T 32114 32116 32116 0 -1 4202496 8443 0 0 0 9898 1 0 0 25 0 1 0 519176961 482422784 8172 33554432000 134512640 135008748 4287245904 18446744073709551615 134527858 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/32116/statm: 117779 8172 46 122 0 116679 0
[pid=32114/tid=32131] ppid=31978 vsize=406168 CPUtime=0 cores=1,3,5,7
/proc/32114/task/32131/stat : 32131 (python) S 31978 31978 31940 0 -1 4202560 54 0 0 0 0 0 0 0 19 0 2 0 519186876 415916032 17014 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471135233024 0 0 -1 7 0 0 0
[pid=32132] ppid=31978 vsize=416424 CPUtime=0.33 cores=1,3,5,7
/proc/32132/stat : 32132 (python) S 31978 31978 31940 0 -1 4202560 1181 0 0 0 16 17 0 0 16 0 2 0 519186880 426418176 17018 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32132/statm: 104106 17018 399 364 0 39997 0
[pid=32134] ppid=32132 vsize=276236 CPUtime=197.53 cores=1,3,5,7
/proc/32134/stat : 32134 (hybridGM3) T 32132 32134 32134 0 -1 4202496 9519 0 0 0 19751 2 0 0 25 0 1 0 519186881 282865664 9253 33554432000 134512640 135204385 4293762112 18446744073709551615 134537403 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/32134/statm: 69059 9253 64 169 0 68889 0
[pid=32132/tid=32156] ppid=31978 vsize=416424 CPUtime=0 cores=1,3,5,7
/proc/32132/task/32156/stat : 32156 (python) S 31978 31978 31940 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 519206838 426418176 17018 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604463151009856 0 0 -1 7 0 0 0
[pid=32150] ppid=31978 vsize=426680 CPUtime=0.49 cores=1,3,5,7
/proc/32150/stat : 32150 (python) S 31978 31978 31940 0 -1 4202560 1201 0 0 0 23 26 0 0 15 0 2 0 519206328 436920320 17022 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32150/statm: 106670 17022 399 364 0 42558 0
[pid=32152] ppid=32150 vsize=276240 CPUtime=295.15 cores=1,3,5,7
/proc/32152/stat : 32152 (hybridGM3) T 32150 32152 32152 0 -1 4202496 9527 0 0 0 29513 2 0 0 25 0 1 0 519206328 282869760 9254 33554432000 134512640 135204385 4287957504 18446744073709551615 134537403 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/32152/statm: 69060 9254 64 169 0 68890 0
[pid=32150/tid=32230] ppid=31978 vsize=426680 CPUtime=0 cores=1,3,5,7
/proc/32150/task/32230/stat : 32230 (python) S 31978 31978 31940 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 519225884 436920320 17022 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471161707264 0 0 -1 7 0 0 0
[pid=32153] ppid=31978 vsize=436936 CPUtime=0.7 cores=1,3,5,7
/proc/32153/stat : 32153 (python) R 31978 31978 31940 0 -1 4202560 1213 0 0 0 37 33 0 0 18 0 2 0 519206831 447422464 17026 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32153/statm: 109234 17026 399 364 0 45119 0
[pid=32155] ppid=32153 vsize=471116 CPUtime=390.61 cores=1,3,5,7
/proc/32155/stat : 32155 (TNM) R 32153 32155 32155 0 -1 4202496 8446 0 0 0 39060 1 0 0 25 0 1 0 519206831 482422784 8170 33554432000 134512640 135008748 4291762816 18446744073709551615 134529367 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/32155/statm: 117779 8170 46 122 0 116679 0
[pid=32153/tid=32216] ppid=31978 vsize=436936 CPUtime=0 cores=1,3,5,7
/proc/32153/task/32216/stat : 32216 (python) S 31978 31978 31940 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519216748 447422464 17026 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471161704576 0 0 -1 3 0 0 0
[pid=32157] ppid=31978 vsize=447192 CPUtime=0.16 cores=1,3,5,7
/proc/32157/stat : 32157 (python) S 31978 31978 31940 0 -1 4202560 1180 0 0 0 7 9 0 0 19 0 2 0 519206843 457924608 17030 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32157/statm: 111798 17030 399 364 0 47680 0
[pid=32159] ppid=32157 vsize=471116 CPUtime=98.98 cores=1,3,5,7
/proc/32159/stat : 32159 (TNM) T 32157 32159 32159 0 -1 4202496 8443 0 0 0 9897 1 0 0 25 0 1 0 519206843 482422784 8171 33554432000 134512640 135008748 4286985232 18446744073709551615 134529448 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/32159/statm: 117779 8171 46 122 0 116679 0
[pid=32157/tid=32220] ppid=31978 vsize=447192 CPUtime=0 cores=1,3,5,7
/proc/32157/task/32220/stat : 32220 (python) S 31978 31978 31940 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 519216759 457924608 17030 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604467399738432 0 0 -1 7 0 0 0
[pid=32217] ppid=31978 vsize=457448 CPUtime=0.17 cores=1,3,5,7
/proc/32217/stat : 32217 (python) S 31978 31978 31940 0 -1 4202560 1184 0 0 0 9 8 0 0 19 0 2 0 519216752 468426752 17034 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/32217/statm: 114362 17034 399 364 0 50241 0
[pid=32219] ppid=32217 vsize=471116 CPUtime=97.68 cores=1,3,5,7
/proc/32219/stat : 32219 (TNM) T 32217 32219 32219 0 -1 4202496 8444 0 0 0 9763 5 0 0 25 0 1 0 519216752 482422784 8172 33554432000 134512640 135008748 4289464256 18446744073709551615 134527475 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/32219/statm: 117779 8172 46 122 0 116679 0
[pid=32217/tid=32251] ppid=31978 vsize=457448 CPUtime=0 cores=1,3,5,7
/proc/32217/task/32251/stat : 32251 (python) S 31978 31978 31940 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519226768 468426752 17034 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471159079360 0 0 -1 5 0 0 0
[pid=32221] ppid=31978 vsize=467704 CPUtime=0.16 cores=1,3,5,7
/proc/32221/stat : 32221 (python) S 31978 31978 31940 0 -1 4202560 1240 0 0 0 8 8 0 0 19 0 2 0 519216764 478928896 17046 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/32221/statm: 116926 17046 407 364 0 52802 0
[pid=32223] ppid=32221 vsize=471116 CPUtime=99 cores=1,3,5,7
/proc/32223/stat : 32223 (TNM) T 32221 32223 32223 0 -1 4202496 8447 0 0 0 9899 1 0 0 25 0 1 0 519216764 482422784 8170 33554432000 134512640 135008748 4287027616 18446744073709551615 134527111 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/32223/statm: 117779 8170 46 122 0 116679 0
[pid=32221/tid=32247] ppid=31978 vsize=467704 CPUtime=0 cores=1,3,5,7
/proc/32221/task/32247/stat : 32247 (python) S 31978 31978 31940 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 519226680 478928896 17046 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471159082048 0 0 -1 7 0 0 0
[pid=32227] ppid=31978 vsize=477960 CPUtime=0.35 cores=1,3,5,7
/proc/32227/stat : 32227 (python) S 31978 31978 31940 0 -1 4202560 1239 0 0 0 20 15 0 0 16 0 2 0 519225562 489431040 17050 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32227/statm: 119490 17050 407 364 0 55363 0
[pid=32229] ppid=32227 vsize=276240 CPUtime=198.46 cores=1,3,5,7
/proc/32229/stat : 32229 (hybridGM3) T 32227 32229 32229 0 -1 4202496 9526 0 0 0 19844 2 0 0 25 0 1 0 519225563 282869760 9255 33554432000 134512640 135204385 4294713008 18446744073709551615 134536907 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/32229/statm: 69060 9255 64 169 0 68890 0
[pid=32227/tid=32280] ppid=31978 vsize=477960 CPUtime=0 cores=1,3,5,7
/proc/32227/task/32280/stat : 32280 (python) S 31978 31978 31940 0 -1 4202560 49 0 0 0 0 0 0 0 16 0 2 0 519245725 489431040 17050 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604466724152512 0 0 -1 7 0 0 0
[pid=32248] ppid=31978 vsize=488216 CPUtime=0.17 cores=1,3,5,7
/proc/32248/stat : 32248 (python) S 31978 31978 31940 0 -1 4202560 1200 0 0 0 8 9 0 0 19 0 2 0 519226685 499933184 17048 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32248/statm: 122054 17048 401 364 0 57924 0
[pid=32250] ppid=32248 vsize=471116 CPUtime=99.16 cores=1,3,5,7
/proc/32250/stat : 32250 (TNM) T 32248 32250 32250 0 -1 4202496 8449 0 0 0 9914 2 0 0 25 0 1 0 519226686 482422784 8172 33554432000 134512640 135008748 4293796592 18446744073709551615 134524689 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/32250/statm: 117779 8172 46 122 0 116679 0
[pid=32248/tid=32274] ppid=31978 vsize=488216 CPUtime=0 cores=1,3,5,7
/proc/32248/task/32274/stat : 32274 (python) S 31978 31978 31940 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 519236603 499933184 17048 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604466724153408 0 0 -1 7 0 0 0
[pid=32255] ppid=31978 vsize=498472 CPUtime=0.37 cores=1,3,5,7
/proc/32255/stat : 32255 (python) S 31978 31978 31940 0 -1 4202560 1211 0 0 0 20 17 0 0 16 0 2 0 519235909 510435328 17051 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32255/statm: 124618 17051 400 364 0 60485 0
[pid=32257] ppid=32255 vsize=276240 CPUtime=198.64 cores=1,3,5,7
/proc/32257/stat : 32257 (hybridGM3) R 32255 32257 32257 0 -1 4202496 9526 0 0 0 19862 2 0 0 25 0 1 0 519235910 282869760 9254 33554432000 134512640 135204385 4292708224 18446744073709551615 134536887 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/32257/statm: 69060 9254 64 169 0 68890 0
[pid=32255/tid=32303] ppid=31978 vsize=498472 CPUtime=0 cores=1,3,5,7
/proc/32255/task/32303/stat : 32303 (python) S 31978 31978 31940 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 519256075 510435328 17051 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471161973824 0 0 -1 7 0 0 0
[pid=32275] ppid=31978 vsize=508728 CPUtime=0.36 cores=1,3,5,7
/proc/32275/stat : 32275 (python) S 31978 31978 31940 0 -1 4202560 1227 0 0 0 18 18 0 0 16 0 2 0 519236608 520937472 17055 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118595531 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/32275/statm: 127182 17055 400 364 0 63046 0
[pid=32277] ppid=32275 vsize=276240 CPUtime=194.54 cores=1,3,5,7
/proc/32277/stat : 32277 (hybridGM3) T 32275 32277 32277 0 -1 4202496 9534 0 0 0 19452 2 0 0 25 0 1 0 519236608 282869760 9254 33554432000 134512640 135204385 4291987456 18446744073709551615 134536922 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/32277/statm: 69060 9254 64 169 0 68890 0
[pid=32275/tid=32299] ppid=31978 vsize=508728 CPUtime=0 cores=1,3,5,7
/proc/32275/task/32299/stat : 32299 (python) S 31978 31978 31940 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 519256063 520937472 17055 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 514 18446604471161971136 0 0 -1 7 0 0 0
[pid=32281] ppid=31978 vsize=518984 CPUtime=0.18 cores=1,3,5,7
/proc/32281/stat : 32281 (python) S 31978 31978 31940 0 -1 4202560 1058 0 0 0 9 9 0 0 15 0 1 0 519245729 531439616 17030 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 13424737907982172422 0 0 17 7 0 0 0
/proc/32281/statm: 129746 17030 372 364 0 65607 0
[pid=32283] ppid=32281 vsize=276240 CPUtime=100.32 cores=1,3,5,7
/proc/32283/stat : 32283 (hybridGM3) R 32281 32283 32283 0 -1 4202496 9530 0 0 0 10031 1 0 0 25 0 1 0 519245730 282869760 9255 33554432000 134512640 135204385 4292404224 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/32283/statm: 69060 9255 64 169 0 68890 0
[pid=32300] ppid=31978 vsize=529240 CPUtime=0 cores=1,3,5,7
/proc/32300/stat : 32300 (python) S 31978 31978 31940 0 -1 4202560 850 0 0 0 0 0 0 0 19 0 1 0 519256068 541941760 17006 33554432000 4194304 5683644 140734228530640 18446744073709551615 255106788934 0 0 16781312 514 4357879431490961407 0 0 17 7 0 0 0
/proc/32300/statm: 132310 17006 344 364 0 68168 0
[pid=32302] ppid=32300 vsize=433152 CPUtime=0.52 cores=1,3,5,7
/proc/32302/stat : 32302 (adaptg2wsat2009) R 32300 32302 32302 0 -1 4202496 5762 0 0 0 52 0 0 0 22 0 1 0 519256068 443547648 5484 33554432000 134512640 135107411 4289250288 18446744073709551615 134523440 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/32302/statm: 108288 5484 49 146 0 107164 0
[pid=31978/tid=32002] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32002/stat : 32002 (python) S 31976 31978 31940 0 -1 4202560 33 930 0 8 0 0 0 0 15 0 22 0 519147292 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 5 0 0 0
[pid=31978/tid=32005] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32005/stat : 32005 (python) S 31976 31978 31940 0 -1 4202560 26 930 0 8 0 0 0 0 18 0 22 0 519147295 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32008] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32008/stat : 32008 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 15 0 22 0 519147298 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 1 0 0 0
[pid=31978/tid=32011] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32011/stat : 32011 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 18 0 22 0 519147302 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 1 0 0 0
[pid=31978/tid=32083] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32083/stat : 32083 (python) S 31976 31978 31940 0 -1 4202560 29 930 0 8 0 0 0 0 20 0 22 0 519166835 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32106] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32106/stat : 32106 (python) S 31976 31978 31940 0 -1 4202560 34 930 0 8 0 0 0 0 18 0 22 0 519176753 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 1 0 0 0
[pid=31978/tid=32109] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32109/stat : 32109 (python) S 31976 31978 31940 0 -1 4202560 33 930 0 8 0 0 0 0 18 0 22 0 519176756 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32112] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32112/stat : 32112 (python) S 31976 31978 31940 0 -1 4202560 85 930 0 8 0 0 0 0 18 0 22 0 519176859 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32115] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32115/stat : 32115 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 22 0 519176961 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32133] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32133/stat : 32133 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 22 0 519186880 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32151] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32151/stat : 32151 (python) S 31976 31978 31940 0 -1 4202560 36 930 0 8 0 0 0 0 18 0 22 0 519206328 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32154] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32154/stat : 32154 (python) S 31976 31978 31940 0 -1 4202560 34 930 0 8 0 0 0 0 18 0 22 0 519206831 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32158] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32158/stat : 32158 (python) S 31976 31978 31940 0 -1 4202560 29 930 0 8 0 0 0 0 21 0 22 0 519206843 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32218] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32218/stat : 32218 (python) S 31976 31978 31940 0 -1 4202560 28 930 0 8 0 0 0 0 20 0 22 0 519216752 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32222] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32222/stat : 32222 (python) S 31976 31978 31940 0 -1 4202560 25 930 0 8 0 0 0 0 21 0 22 0 519216764 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32228] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32228/stat : 32228 (python) S 31976 31978 31940 0 -1 4202560 25 930 0 8 0 0 0 0 20 0 22 0 519225562 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32249] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32249/stat : 32249 (python) S 31976 31978 31940 0 -1 4202560 27 930 0 8 0 0 0 0 20 0 22 0 519226686 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32256] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32256/stat : 32256 (python) S 31976 31978 31940 0 -1 4202560 34 930 0 8 0 0 0 0 18 0 22 0 519235909 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32276] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32276/stat : 32276 (python) S 31976 31978 31940 0 -1 4202560 23 930 0 8 0 0 0 0 20 0 22 0 519236608 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32282] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32282/stat : 32282 (python) S 31976 31978 31940 0 -1 4202560 28 930 0 8 0 0 0 0 20 0 22 0 519245730 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
[pid=31978/tid=32301] ppid=31976 vsize=539484 CPUtime=0 cores=1,3,5,7
/proc/31978/task/32301/stat : 32301 (python) S 31976 31978 31940 0 -1 4202560 30 930 0 8 0 0 0 0 20 0 22 0 519256068 552431616 18582 33554432000 4194304 5683644 140734228530640 18446744073709551615 255118592433 0 0 16781312 2 18446604463151012544 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4344.82
Current children cumulated vsize (KiB) 16832084

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31978 and gives
#  childrusage.ru_utime.tv_sec=12
#  childrusage.ru_utime.tv_usec=167150
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=300954
# CPU time returned by wait4() is 12.4681
# while last known CPU time is 4344.82
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1118.74
CPU time (s): 4344.82
CPU user time (s): 4340.9
CPU system time (s): 3.92
CPU usage (%): 388.368
Max. virtual memory (cumulated for all children) (KiB): 16832084

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 12.1671
system time used= 0.300954
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 50369
page faults= 116
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 16574
involuntary context switches= 172

runsolver used 5.82811 second user time and 13.144 second system time

The end

Launcher Data 

Begin job on node144 at 2011-05-01 18:28:01
IDJOB=3317056
IDBENCH=82521
IDSOLVER=1705
FILE ID=node144/3317056-1304267281
RUNJOBID= node144-1304267279-31959
PBS_JOBID= 13173935
Free space on /tmp= 73552 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S1969540674-065-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317056-1304267281/watcher-3317056-1304267281 -o /tmp/evaluation-result-3317056-1304267281/solver-3317056-1304267281 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317056-1304267281.cnf -seed 489172721 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= fcb14103324f734a2e69e02c062e67ed
RANDOM SEED=489172721

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25351592 kB
Buffers:       2134176 kB
Cached:        4851712 kB
SwapCached:         80 kB
Active:         422140 kB
Inactive:      6630336 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25351592 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           68280 kB
Writeback:           0 kB
AnonPages:       66028 kB
Mapped:          15692 kB
Slab:           483216 kB
PageTables:       4572 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   207804 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73512 MiB
End job on node144 at 2011-05-01 18:46:42