Trace number 3378154

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2384.65 601.726

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S926861876-013-UNKNOWN.cnf
MD5SUMaa5d2fab702f701bf63e2d26e76ab2b3
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.51/1.16	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.51/1.16	c INFO: solving HOME/instance-3378154-1305463229.cnf
0.51/1.16	c INFO: computing task features
0.68/1.34	c INFO: computed variable-clause graph statistics
0.68/1.34	c INFO: computed clause balance statistics
1.59/2.24	c INFO: computed variable balance statistics
1.98/2.60	c INFO: computed small-clause counts
6.13/3.70	c INFO: computed Horn-clause counts
6.13/3.70	c INFO: computed variable graph statistics
6.13/3.70	c INFO: collected features for HOME/instance-3378154-1305463229.cnf in 1.47 s
6.13/3.70	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.37)
6.13/3.70	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
6.13/3.70	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
6.13/3.70	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
401.56/102.87	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.05)
401.56/102.88	c INFO: running gnovelty+2@0 for 97 with 1098 remaining (b = 0.04)
401.56/102.89	c INFO: running FH@0 for 97 with 1098 remaining (b = 0.03)
401.93/102.90	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.03)
800.85/203.12	c INFO: running adaptg2wsat2009++@0 for 97 with 998 remaining (b = 0.02)
800.85/203.14	c INFO: running FH@0 for 97 with 998 remaining (b = 0.02)
800.85/203.16	c INFO: running hybridGM3@0 for 194 with 998 remaining (b = 0.04)
816.74/207.13	c INFO: running gnovelty+2@0 for 97 with 994 remaining (b = 0.02)
1199.55/303.38	c INFO: running hybridGM3@0 for 194 with 897 remaining (b = 0.03)
1199.55/303.39	c INFO: running adaptg2wsat2009++@0 for 97 with 897 remaining (b = 0.01)
1215.84/307.49	c INFO: running hybridGM3@0 for 194 with 893 remaining (b = 0.03)
1598.98/403.71	c INFO: running hybridGM3@0 for 194 with 797 remaining (b = 0.02)
1606.49/405.68	c INFO: running hybridGM3@102 for 97 with 795 remaining (b = 0.01)
1980.84/499.94	c INFO: running hybridGM3@99 for 97 with 701 remaining (b = 0.01)
1984.81/500.92	c INFO: running hybridGM3@298 for 97 with 700 remaining (b = 0.01)
2020.96/510.05	c INFO: running hybridGM3@100 for 97 with 691 remaining (b = 0.01)
2379.10/600.24	c INFO: running TNM@0 for 600 with 600 remaining (b = 0.07)
2382.64/601.17	c INFO: running FH@0 for 97 with 599 remaining (b = 0.01)
2384.65/601.71	c INFO: running adaptg2wsat2009++@0 for 97 with 599 remaining (b = 0.01)
2384.65/601.71	Traceback (most recent call last):
2384.65/601.71	  File "HOME/solve", line 33, in <module>
2384.65/601.71	    plac.call(main)
2384.65/601.71	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
2384.65/601.71	    cmd, result = parser_from(obj).consume(arglist)
2384.65/601.71	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
2384.65/601.71	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
2384.65/601.71	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
2384.65/601.71	    (_, answer, _) = solver(input_path, budget / cores, cores)
2384.65/601.71	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
2384.65/601.71	    solver.go(max_cost * borg.defaults.machine_speed)
2384.65/601.71	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
2384.65/601.71	    self._process.start()
2384.65/601.71	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
2384.65/601.71	    assert self._popen is None, 'cannot start a process twice'
2384.65/601.71	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378154-1305463229/watcher-3378154-1305463229 -o /tmp/evaluation-result-3378154-1305463229/solver-3378154-1305463229 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378154-1305463229.cnf -seed 2034263685 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.54 3.23 3.64 2/278 1257
/proc/meminfo: memFree=24408320/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=15080 CPUtime=0 cores=0,2,4,6
/proc/1257/stat : 1257 (python) R 1255 1257 1220 0 -1 4202496 616 0 0 0 0 0 0 0 25 0 1 0 638738275 15441920 513 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261612468 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1257/statm: 3770 513 300 364 0 257 0

[startup+0.0629359 s]
/proc/loadavg: 2.54 3.23 3.64 2/278 1257
/proc/meminfo: memFree=24408320/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=16812 CPUtime=0.02 cores=0,2,4,6
/proc/1257/stat : 1257 (python) D 1255 1257 1220 0 -1 4202496 1228 0 0 0 2 0 0 0 17 0 1 0 638738275 17215488 1086 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261612389 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1257/statm: 4203 1086 417 364 0 690 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 16812

[startup+0.100913 s]
/proc/loadavg: 2.54 3.23 3.64 2/278 1257
/proc/meminfo: memFree=24408320/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=23184 CPUtime=0.03 cores=0,2,4,6
/proc/1257/stat : 1257 (python) D 1255 1257 1220 0 -1 4202496 1324 0 0 0 3 0 0 0 17 0 1 0 638738275 23740416 1173 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261612389 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1257/statm: 5796 1173 432 364 0 723 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 23184

[startup+0.300889 s]
/proc/loadavg: 2.54 3.23 3.64 2/278 1257
/proc/meminfo: memFree=24408320/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=89768 CPUtime=0.1 cores=0,2,4,6
/proc/1257/stat : 1257 (python) D 1255 1257 1220 0 -1 4202496 2672 0 0 0 9 1 0 0 17 0 1 0 638738275 91922432 2388 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261612389 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1257/statm: 22442 2388 750 364 0 1622 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 89768

[startup+0.700802 s]
/proc/loadavg: 2.54 3.23 3.64 2/278 1257
/proc/meminfo: memFree=24408320/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=135748 CPUtime=0.26 cores=0,2,4,6
/proc/1257/stat : 1257 (python) R 1255 1257 1220 0 -1 4202496 5963 933 1 0 22 4 0 0 17 0 1 0 638738275 139005952 4151 33554432000 4194304 5683644 140735644431152 18446744073709551615 5150659 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1257/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.26
Current children cumulated vsize (KiB) 135748

[startup+1.50067 s]
/proc/loadavg: 2.54 3.23 3.64 4/280 1267
/proc/meminfo: memFree=24373212/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=285632 CPUtime=0.88 cores=0,2,4,6
/proc/1257/stat : 1257 (python) R 1255 1257 1220 0 -1 4202496 12501 933 1 0 74 14 0 0 16 0 1 0 638738275 292487168 8793 33554432000 4194304 5683644 140735644431152 18446744073709551615 4566996 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/1257/statm: 71408 8793 1826 364 0 7332 0
Current children cumulated CPU time (s) 0.88
Current children cumulated vsize (KiB) 285632

[startup+3.10028 s]
/proc/loadavg: 2.54 3.23 3.64 4/280 1267
/proc/meminfo: memFree=24352388/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=321284 CPUtime=2.06 cores=0,2,4,6
/proc/1257/stat : 1257 (python) S 1255 1257 1220 0 -1 4202496 15263 933 1 0 191 15 0 0 22 0 5 0 638738275 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/1257/statm: 80321 7529 1875 364 0 16230 0
Current children cumulated CPU time (s) 2.06
Current children cumulated vsize (KiB) 321284
heavy processes:

[startup+6.30071 s]
/proc/loadavg: 2.98 3.31 3.67 11/304 1291
/proc/meminfo: memFree=24306680/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=321284 CPUtime=2.06 cores=0,2,4,6
/proc/1257/stat : 1257 (python) S 1255 1257 1220 0 -1 4202496 15263 933 1 0 191 15 0 0 22 0 5 0 638738275 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/1257/statm: 80321 7529 1875 364 0 16230 0
[pid=1280] ppid=1257 vsize=280756 CPUtime=0 cores=0,2,4,6
/proc/1280/stat : 1280 (python) S 1257 1257 1220 0 -1 4202560 721 0 0 0 0 0 0 0 18 0 1 0 638738540 287494144 5977 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636623 0 0 16781312 514 1327964584677441793 0 0 17 4 0 0 0
/proc/1280/statm: 70189 5977 355 364 0 6107 0
[pid=1282] ppid=1280 vsize=1019708 CPUtime=3.64 cores=0,2,4,6
/proc/1282/stat : 1282 (gnovelty+2) R 1280 1282 1282 0 -1 4202496 948 0 0 0 364 0 0 0 25 0 1 0 638738540 1044180992 690 33554432000 134512640 135663704 4288872672 18446744073709551615 134525181 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1282/statm: 254927 690 113 282 0 254643 0
[pid=1283] ppid=1257 vsize=291080 CPUtime=0 cores=0,2,4,6
/proc/1283/stat : 1283 (python) S 1257 1257 1220 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 638738541 298065920 5975 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 10740214577242276098 0 0 17 0 0 0 0
/proc/1283/statm: 72770 5975 349 364 0 8685 0
[pid=1285] ppid=1283 vsize=1019708 CPUtime=3.63 cores=0,2,4,6
/proc/1285/stat : 1285 (gnovelty+2) R 1283 1285 1285 0 -1 4202496 953 0 0 0 363 0 0 0 25 0 1 0 638738541 1044180992 690 33554432000 134512640 135663704 4294932352 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1285/statm: 254927 690 113 282 0 254643 0
[pid=1286] ppid=1257 vsize=301268 CPUtime=0 cores=0,2,4,6
/proc/1286/stat : 1286 (python) S 1257 1257 1220 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 638738542 308498432 5996 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 4465935860855963908 0 0 17 4 0 0 0
/proc/1286/statm: 75317 5996 349 364 0 11229 0
[pid=1288] ppid=1286 vsize=419820 CPUtime=3.58 cores=0,2,4,6
/proc/1288/stat : 1288 (adaptg2wsat2009) R 1286 1288 1288 0 -1 4202496 1440 0 0 0 358 0 0 0 25 0 1 0 638738542 429895680 1173 33554432000 134512640 135107411 4290511712 18446744073709551615 134523424 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1288/statm: 104955 1173 49 146 0 103831 0
[pid=1289] ppid=1257 vsize=311592 CPUtime=0 cores=0,2,4,6
/proc/1289/stat : 1289 (python) S 1257 1257 1220 0 -1 4202560 741 0 0 0 0 0 0 0 18 0 1 0 638738543 319070208 6000 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 9975760151455432964 0 0 17 0 0 0 0
/proc/1289/statm: 77898 6000 349 364 0 13807 0
[pid=1291] ppid=1289 vsize=419820 CPUtime=3.61 cores=0,2,4,6
/proc/1291/stat : 1291 (adaptg2wsat2009) R 1289 1291 1291 0 -1 4202496 1434 0 0 0 360 1 0 0 25 0 1 0 638738543 429895680 1172 33554432000 134512640 135107411 4286974064 18446744073709551615 134523300 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1291/statm: 104955 1172 49 146 0 103831 0
[pid=1257/tid=1281] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1281/stat : 1281 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 5 0 638738540 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1284] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1284/stat : 1284 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 5 0 638738541 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1287] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1287/stat : 1287 (python) S 1255 1257 1220 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 5 0 638738542 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1290] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1290/stat : 1290 (python) S 1255 1257 1220 0 -1 4202560 23 933 0 0 0 0 0 0 22 0 5 0 638738543 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 16.52
Current children cumulated vsize (KiB) 4385036

[startup+12.7006 s]
/proc/loadavg: 3.38 3.39 3.69 10/304 1291
/proc/meminfo: memFree=24302816/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=321284 CPUtime=2.06 cores=0,2,4,6
/proc/1257/stat : 1257 (python) S 1255 1257 1220 0 -1 4202496 15263 933 1 0 191 15 0 0 22 0 5 0 638738275 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/1257/statm: 80321 7529 1875 364 0 16230 0
[pid=1280] ppid=1257 vsize=280756 CPUtime=0 cores=0,2,4,6
/proc/1280/stat : 1280 (python) S 1257 1257 1220 0 -1 4202560 881 0 0 0 0 0 0 0 18 0 1 0 638738540 287494144 6000 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636623 0 0 16781312 514 1327964584677441793 0 0 17 4 0 0 0
/proc/1280/statm: 70189 6000 378 364 0 6107 0
[pid=1282] ppid=1280 vsize=1019708 CPUtime=10 cores=0,2,4,6
/proc/1282/stat : 1282 (gnovelty+2) R 1280 1282 1282 0 -1 4202496 948 0 0 0 1000 0 0 0 25 0 1 0 638738540 1044180992 690 33554432000 134512640 135663704 4288872672 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1282/statm: 254927 690 113 282 0 254643 0
[pid=1283] ppid=1257 vsize=291080 CPUtime=0 cores=0,2,4,6
/proc/1283/stat : 1283 (python) S 1257 1257 1220 0 -1 4202560 891 0 0 0 0 0 0 0 18 0 1 0 638738541 298065920 5998 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 10740214577242276098 0 0 17 0 0 0 0
/proc/1283/statm: 72770 5998 372 364 0 8685 0
[pid=1285] ppid=1283 vsize=1019708 CPUtime=10 cores=0,2,4,6
/proc/1285/stat : 1285 (gnovelty+2) R 1283 1285 1285 0 -1 4202496 953 0 0 0 1000 0 0 0 25 0 1 0 638738541 1044180992 690 33554432000 134512640 135663704 4294932352 18446744073709551615 134524610 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1285/statm: 254927 690 113 282 0 254643 0
[pid=1286] ppid=1257 vsize=301268 CPUtime=0.02 cores=0,2,4,6
/proc/1286/stat : 1286 (python) S 1257 1257 1220 0 -1 4202560 901 0 0 0 1 1 0 0 18 0 1 0 638738542 308498432 6019 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 14327938331688534276 0 0 17 4 0 0 0
/proc/1286/statm: 75317 6019 372 364 0 11229 0
[pid=1288] ppid=1286 vsize=420220 CPUtime=9.92 cores=0,2,4,6
/proc/1288/stat : 1288 (adaptg2wsat2009) R 1286 1288 1288 0 -1 4202496 1667 0 0 0 992 0 0 0 25 0 1 0 638738542 430305280 1400 33554432000 134512640 135107411 4290511712 18446744073709551615 134532408 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1288/statm: 105055 1400 59 146 0 103931 0
[pid=1289] ppid=1257 vsize=311596 CPUtime=0 cores=0,2,4,6
/proc/1289/stat : 1289 (python) R 1257 1257 1220 0 -1 4202560 917 0 0 0 0 0 0 0 18 0 1 0 638738543 319074304 6023 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261616443 0 0 16781312 514 0 0 0 17 0 0 0 0
/proc/1289/statm: 77898 6023 372 364 0 13807 0
[pid=1291] ppid=1289 vsize=420220 CPUtime=10 cores=0,2,4,6
/proc/1291/stat : 1291 (adaptg2wsat2009) R 1289 1291 1291 0 -1 4202496 1662 0 0 0 999 1 0 0 25 0 1 0 638738543 430305280 1400 33554432000 134512640 135107411 4286974064 18446744073709551615 134532135 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1291/statm: 105055 1400 59 146 0 103931 0
[pid=1257/tid=1281] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1281/stat : 1281 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 5 0 638738540 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1284] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1284/stat : 1284 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 5 0 638738541 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1287] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1287/stat : 1287 (python) S 1255 1257 1220 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 5 0 638738542 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1290] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1290/stat : 1290 (python) S 1255 1257 1220 0 -1 4202560 23 933 0 0 0 0 0 0 22 0 5 0 638738543 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 42
Current children cumulated vsize (KiB) 4385840

[startup+25.5013 s]
/proc/loadavg: 4.70 3.68 3.78 11/304 1291
/proc/meminfo: memFree=24303740/32951124 swapFree=67057896/67111528
[pid=1257] ppid=1255 vsize=321284 CPUtime=2.06 cores=0,2,4,6
/proc/1257/stat : 1257 (python) S 1255 1257 1220 0 -1 4202496 15263 933 1 0 191 15 0 0 22 0 5 0 638738275 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/1257/statm: 80321 7529 1875 364 0 16230 0
[pid=1280] ppid=1257 vsize=280756 CPUtime=0.03 cores=0,2,4,6
/proc/1280/stat : 1280 (python) S 1257 1257 1220 0 -1 4202560 890 0 0 0 2 1 0 0 18 0 1 0 638738540 287494144 6000 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636623 0 0 16781312 514 14327938331688534273 0 0 17 4 0 0 0
/proc/1280/statm: 70189 6000 378 364 0 6107 0
[pid=1282] ppid=1280 vsize=1019708 CPUtime=22.74 cores=0,2,4,6
/proc/1282/stat : 1282 (gnovelty+2) R 1280 1282 1282 0 -1 4202496 948 0 0 0 2274 0 0 0 25 0 1 0 638738540 1044180992 690 33554432000 134512640 135663704 4288872672 18446744073709551615 134524882 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1282/statm: 254927 690 113 282 0 254643 0
[pid=1283] ppid=1257 vsize=291080 CPUtime=0.04 cores=0,2,4,6
/proc/1283/stat : 1283 (python) S 1257 1257 1220 0 -1 4202560 903 0 0 0 3 1 0 0 18 0 1 0 638738541 298065920 5998 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 14327938331688534274 0 0 17 0 0 0 0
/proc/1283/statm: 72770 5998 372 364 0 8685 0
[pid=1285] ppid=1283 vsize=1019708 CPUtime=22.73 cores=0,2,4,6
/proc/1285/stat : 1285 (gnovelty+2) R 1283 1285 1285 0 -1 4202496 953 0 0 0 2273 0 0 0 25 0 1 0 638738541 1044180992 690 33554432000 134512640 135663704 4294932352 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1285/statm: 254927 690 113 282 0 254643 0
[pid=1286] ppid=1257 vsize=301268 CPUtime=0.05 cores=0,2,4,6
/proc/1286/stat : 1286 (python) S 1257 1257 1220 0 -1 4202560 907 0 0 0 3 2 0 0 18 0 1 0 638738542 308498432 6019 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 14327938331688534276 0 0 17 4 0 0 0
/proc/1286/statm: 75317 6019 372 364 0 11229 0
[pid=1288] ppid=1286 vsize=420220 CPUtime=22.6 cores=0,2,4,6
/proc/1288/stat : 1288 (adaptg2wsat2009) R 1286 1288 1288 0 -1 4202496 1667 0 0 0 2260 0 0 0 25 0 1 0 638738542 430305280 1400 33554432000 134512640 135107411 4290511712 18446744073709551615 134532318 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1288/statm: 105055 1400 59 146 0 103931 0
[pid=1289] ppid=1257 vsize=311592 CPUtime=0.04 cores=0,2,4,6
/proc/1289/stat : 1289 (python) S 1257 1257 1220 0 -1 4202560 921 0 0 0 2 2 0 0 18 0 1 0 638738543 319070208 6023 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 14327938331688534276 0 0 17 0 0 0 0
/proc/1289/statm: 77898 6023 372 364 0 13807 0
[pid=1291] ppid=1289 vsize=420220 CPUtime=22.81 cores=0,2,4,6
/proc/1291/stat : 1291 (adaptg2wsat2009) R 1289 1291 1291 0 -1 4202496 1662 0 0 0 2280 1 0 0 25 0 1 0 638738543 430305280 1400 33554432000 134512640 135107411 4286974064 18446744073709551615 134532135 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1291/statm: 105055 1400 59 146 0 103931 0
[pid=1257/tid=1281] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1281/stat : 1281 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 5 0 638738540 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1284] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1284/stat : 1284 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 5 0 638738541 328994816 7529 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1287] ppid=1255 vsize=321284 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

[pid=1291] ppid=1289 vsize=420220 CPUtime=100.16 cores=0,2,4,6
/proc/1291/stat : 1291 (adaptg2wsat2009) T 1289 1291 1291 0 -1 4202496 1662 0 0 0 10015 1 0 0 25 0 1 0 638738543 430305280 1400 33554432000 134512640 135107411 4286974064 18446744073709551615 134531935 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1291/statm: 105055 1400 59 146 0 103931 0
[pid=1289/tid=1313] ppid=1257 vsize=311592 CPUtime=0 cores=0,2,4,6
/proc/1289/task/1313/stat : 1313 (python) S 1257 1257 1220 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 638748562 319070208 6052 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604450940388608 0 0 -1 0 0 0 0
[pid=1312] ppid=1257 vsize=321924 CPUtime=0.22 cores=0,2,4,6
/proc/1312/stat : 1312 (python) S 1257 1257 1220 0 -1 4202560 1094 0 0 0 10 12 0 0 19 0 2 0 638748562 329650176 6056 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/1312/statm: 80481 6056 400 364 0 16387 0
[pid=1318] ppid=1312 vsize=420220 CPUtime=102.09 cores=0,2,4,6
/proc/1318/stat : 1318 (adaptg2wsat2009) T 1312 1318 1318 0 -1 4202496 1663 0 0 0 10208 1 0 0 25 0 1 0 638748564 430305280 1400 33554432000 134512640 135107411 4294490080 18446744073709551615 134532277 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/1318/statm: 105055 1400 59 146 0 103931 0
[pid=1312/tid=1357] ppid=1257 vsize=321924 CPUtime=0 cores=0,2,4,6
/proc/1312/task/1357/stat : 1357 (python) S 1257 1257 1220 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 638758987 329650176 6056 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604453924298816 0 0 -1 6 0 0 0
[pid=1315] ppid=1257 vsize=332260 CPUtime=0.21 cores=0,2,4,6
/proc/1315/stat : 1315 (python) S 1257 1257 1220 0 -1 4202560 1095 0 0 0 12 9 0 0 19 0 2 0 638748563 340234240 6079 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1315/statm: 83065 6079 400 364 0 18968 0
[pid=1321] ppid=1315 vsize=1019708 CPUtime=100.03 cores=0,2,4,6
/proc/1321/stat : 1321 (gnovelty+2) T 1315 1321 1321 0 -1 4202496 949 0 0 0 10003 0 0 0 25 0 1 0 638748565 1044180992 689 33554432000 134512640 135663704 4294660144 18446744073709551615 134544183 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1321/statm: 254927 689 113 282 0 254643 0
[pid=1315/tid=1351] ppid=1257 vsize=332260 CPUtime=0 cores=0,2,4,6
/proc/1315/task/1351/stat : 1351 (python) S 1257 1257 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638758590 340234240 6079 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604453924300608 0 0 -1 4 0 0 0
[pid=1320] ppid=1257 vsize=342464 CPUtime=0.21 cores=0,2,4,6
/proc/1320/stat : 1320 (python) S 1257 1257 1220 0 -1 4202560 1095 0 0 0 14 7 0 0 19 0 2 0 638748565 350683136 6103 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1320/statm: 85616 6103 400 364 0 21516 0
[pid=1323] ppid=1320 vsize=249876 CPUtime=99.99 cores=0,2,4,6
/proc/1323/stat : 1323 (FH) T 1320 1323 1323 0 -1 4202496 1524 0 0 0 9999 0 0 0 25 0 1 0 638748565 255873024 1263 33554432000 134512640 134936988 4290423280 18446744073709551615 134553251 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1323/statm: 62469 1263 41 104 0 61973 0
[pid=1320/tid=1345] ppid=1257 vsize=342464 CPUtime=0 cores=0,2,4,6
/proc/1320/task/1345/stat : 1345 (python) S 1257 1257 1220 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 638758587 350683136 6103 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604442335202112 0 0 -1 0 0 0 0
[pid=1324] ppid=1257 vsize=352816 CPUtime=0.22 cores=0,2,4,6
/proc/1324/stat : 1324 (python) S 1257 1257 1220 0 -1 4202560 1138 0 0 0 10 12 0 0 19 0 2 0 638748566 361283584 6113 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1324/statm: 88204 6113 406 364 0 24101 0
[pid=1326] ppid=1324 vsize=420220 CPUtime=99.98 cores=0,2,4,6
/proc/1326/stat : 1326 (adaptg2wsat2009) T 1324 1326 1326 0 -1 4202496 1666 0 0 0 9998 0 0 0 25 0 1 0 638748566 430305280 1399 33554432000 134512640 135107411 4287823136 18446744073709551615 134531696 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1326/statm: 105055 1399 59 146 0 103931 0
[pid=1324/tid=1347] ppid=1257 vsize=352816 CPUtime=0 cores=0,2,4,6
/proc/1324/task/1347/stat : 1347 (python) S 1257 1257 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638758588 361283584 6113 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604448187902656 0 0 -1 2 0 0 0
[pid=1346] ppid=1257 vsize=363180 CPUtime=0.23 cores=0,2,4,6
/proc/1346/stat : 1346 (python) S 1257 1257 1220 0 -1 4202560 1145 0 0 0 11 12 0 0 19 0 2 0 638758588 371896320 6128 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1346/statm: 90795 6128 406 364 0 26689 0
[pid=1349] ppid=1346 vsize=420220 CPUtime=100.2 cores=0,2,4,6
/proc/1349/stat : 1349 (adaptg2wsat2009) T 1346 1349 1349 0 -1 4202496 1672 0 0 0 10020 0 0 0 25 0 1 0 638758588 430305280 1401 33554432000 134512640 135107411 4291920912 18446744073709551615 134531848 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1349/statm: 105055 1401 59 146 0 103931 0
[pid=1346/tid=1376] ppid=1257 vsize=363180 CPUtime=0 cores=0,2,4,6
/proc/1346/task/1376/stat : 1376 (python) S 1257 1257 1220 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 638768612 371896320 6128 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604446823407936 0 0 -1 2 0 0 0
[pid=1350] ppid=1257 vsize=373548 CPUtime=0.24 cores=0,2,4,6
/proc/1350/stat : 1350 (python) S 1257 1257 1220 0 -1 4202560 1112 0 0 0 12 12 0 0 19 0 2 0 638758590 382513152 6153 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1350/statm: 93387 6153 400 364 0 29278 0
[pid=1353] ppid=1350 vsize=249872 CPUtime=99.76 cores=0,2,4,6
/proc/1353/stat : 1353 (FH) T 1350 1353 1353 0 -1 4202496 1529 0 0 0 9976 0 0 0 25 0 1 0 638758590 255868928 1264 33554432000 134512640 134936988 4286621040 18446744073709551615 134553311 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1353/statm: 62468 1264 41 104 0 61972 0
[pid=1350/tid=1378] ppid=1257 vsize=373548 CPUtime=0 cores=0,2,4,6
/proc/1350/task/1378/stat : 1378 (python) S 1257 1257 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638768614 382513152 6153 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604453895373888 0 0 -1 0 0 0 0
[pid=1354] ppid=1257 vsize=383924 CPUtime=0.93 cores=0,2,4,6
/proc/1354/stat : 1354 (python) S 1257 1257 1220 0 -1 4202560 1171 0 0 0 44 49 0 0 18 0 2 0 638758591 393138176 6185 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1354/statm: 95981 6185 400 364 0 31869 0
[pid=1356] ppid=1354 vsize=268420 CPUtime=392.68 cores=0,2,4,6
/proc/1356/stat : 1356 (hybridGM3) R 1354 1356 1356 0 -1 4202496 5392 0 0 0 39265 3 0 0 25 0 1 0 638758591 274862080 5125 33554432000 134512640 135204385 4286881920 18446744073709551615 134537939 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1356/statm: 67105 5125 63 169 0 66935 0
[pid=1354/tid=1400] ppid=1257 vsize=383924 CPUtime=0 cores=0,2,4,6
/proc/1354/task/1400/stat : 1400 (python) S 1257 1257 1220 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 638778843 393138176 6185 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604442335199424 0 0 -1 4 0 0 0
[pid=1358] ppid=1257 vsize=394028 CPUtime=0.23 cores=0,2,4,6
/proc/1358/stat : 1358 (python) S 1257 1257 1220 0 -1 4202560 1131 0 0 0 12 11 0 0 19 0 2 0 638758989 403484672 6219 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1358/statm: 98507 6219 400 364 0 34392 0
[pid=1360] ppid=1358 vsize=1019708 CPUtime=99.87 cores=0,2,4,6
/proc/1360/stat : 1360 (gnovelty+2) T 1358 1360 1360 0 -1 4202496 957 0 0 0 9987 0 0 0 25 0 1 0 638758989 1044180992 689 33554432000 134512640 135663704 4290425072 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1360/statm: 254927 689 113 282 0 254643 0
[pid=1358/tid=1384] ppid=1257 vsize=394028 CPUtime=0 cores=0,2,4,6
/proc/1358/task/1384/stat : 1384 (python) S 1257 1257 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638769023 403484672 6219 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604450109097280 0 0 -1 4 0 0 0
[pid=1377] ppid=1257 vsize=404472 CPUtime=0.72 cores=0,2,4,6
/proc/1377/stat : 1377 (python) S 1257 1257 1220 0 -1 4202560 1186 0 0 0 37 35 0 0 18 0 2 0 638768613 414179328 6223 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1377/statm: 101118 6223 400 364 0 37000 0
[pid=1380] ppid=1377 vsize=268428 CPUtime=297.03 cores=0,2,4,6
/proc/1380/stat : 1380 (hybridGM3) T 1377 1380 1380 0 -1 4202496 5394 0 0 0 29702 1 0 0 25 0 1 0 638768614 274870272 5127 33554432000 134512640 135204385 4288678512 18446744073709551615 134538320 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1380/statm: 67107 5127 63 169 0 66937 0
[pid=1377/tid=1406] ppid=1257 vsize=404472 CPUtime=0 cores=0,2,4,6
/proc/1377/task/1406/stat : 1406 (python) S 1257 1257 1220 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 638788268 414179328 6223 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604453930566784 0 0 -1 4 0 0 0
[pid=1381] ppid=1257 vsize=414872 CPUtime=0.24 cores=0,2,4,6
/proc/1381/stat : 1381 (python) S 1257 1257 1220 0 -1 4202560 1151 0 0 0 14 10 0 0 19 0 2 0 638768615 424828928 6236 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1381/statm: 103718 6236 400 364 0 39597 0
[pid=1383] ppid=1381 vsize=420220 CPUtime=100.02 cores=0,2,4,6
/proc/1383/stat : 1383 (adaptg2wsat2009) T 1381 1383 1383 0 -1 4202496 1671 0 0 0 10002 0 0 0 25 0 1 0 638768615 430305280 1399 33554432000 134512640 135107411 4291493184 18446744073709551615 134532381 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1383/statm: 105055 1399 59 146 0 103931 0
[pid=1381/tid=1396] ppid=1257 vsize=414872 CPUtime=0 cores=0,2,4,6
/proc/1381/task/1396/stat : 1396 (python) S 1257 1257 1220 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 638778644 424828928 6236 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604450940387712 0 0 -1 2 0 0 0
[pid=1385] ppid=1257 vsize=425276 CPUtime=0.71 cores=0,2,4,6
/proc/1385/stat : 1385 (python) S 1257 1257 1220 0 -1 4202560 1240 0 0 0 36 35 0 0 18 0 2 0 638769024 435482624 6284 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1385/statm: 106319 6284 408 364 0 42195 0
[pid=1387] ppid=1385 vsize=268420 CPUtime=288.47 cores=0,2,4,6
/proc/1387/stat : 1387 (hybridGM3) R 1385 1387 1387 0 -1 4202496 5400 0 0 0 28845 2 0 0 25 0 1 0 638769024 274862080 5125 33554432000 134512640 135204385 4292771968 18446744073709551615 134537792 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1387/statm: 67105 5125 63 169 0 66935 0
[pid=1385/tid=1447] ppid=1257 vsize=425276 CPUtime=0 cores=0,2,4,6
/proc/1385/task/1447/stat : 1447 (python) S 1257 1257 1220 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 638789279 435482624 6284 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604453924301504 0 0 -1 2 0 0 0
[pid=1397] ppid=1257 vsize=435048 CPUtime=0.46 cores=0,2,4,6
/proc/1397/stat : 1397 (python) S 1257 1257 1220 0 -1 4202560 1211 0 0 0 25 21 0 0 19 0 2 0 638778647 445489152 6323 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/1397/statm: 108762 6323 406 364 0 44635 0
[pid=1399] ppid=1397 vsize=268420 CPUtime=195.99 cores=0,2,4,6
/proc/1399/stat : 1399 (hybridGM3) T 1397 1399 1399 0 -1 4202496 5399 0 0 0 19596 3 0 0 25 0 1 0 638778647 274862080 5125 33554432000 134512640 135204385 4289908864 18446744073709551615 134538559 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1399/statm: 67105 5125 63 169 0 66935 0
[pid=1397/tid=1463] ppid=1257 vsize=435048 CPUtime=0 cores=0,2,4,6
/proc/1397/task/1463/stat : 1463 (python) S 1257 1257 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 638798298 445489152 6323 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 514 18446604450896360512 0 0 -1 0 0 0 0
[pid=1464] ppid=1257 vsize=445976 CPUtime=0 cores=0,2,4,6
/proc/1464/stat : 1464 (python) S 1257 1257 1220 0 -1 4202560 860 0 0 0 0 0 0 0 18 0 1 0 638798300 456679424 6299 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 4237853443668049919 0 0 17 2 0 0 0
/proc/1464/statm: 111494 6299 356 364 0 47364 0
[pid=1466] ppid=1464 vsize=454088 CPUtime=1.36 cores=0,2,4,6
/proc/1466/stat : 1466 (TNM) R 1464 1466 1466 0 -1 4202496 1272 0 0 0 136 0 0 0 25 0 1 0 638798300 464986112 995 33554432000 134512640 135008748 4290977984 18446744073709551615 134523316 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1466/statm: 113522 995 45 122 0 112422 0
[pid=1467] ppid=1257 vsize=456408 CPUtime=0 cores=0,2,4,6
/proc/1467/stat : 1467 (python) S 1257 1257 1220 0 -1 4202560 826 0 0 0 0 0 0 0 19 0 1 0 638798393 467361792 6293 33554432000 4194304 5683644 140735644431152 18446744073709551615 254261636678 0 0 16781312 514 4533132788865433599 0 0 17 0 0 0 0
/proc/1467/statm: 114102 6293 346 364 0 49969 0
[pid=1469] ppid=1467 vsize=247956 CPUtime=0.43 cores=0,2,4,6
/proc/1469/stat : 1469 (FH) R 1467 1469 1469 0 -1 4202496 1259 0 0 0 43 0 0 0 21 0 1 0 638798393 253906944 987 33554432000 134512640 134936988 4290209456 18446744073709551615 134523648 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1469/statm: 61989 987 37 104 0 61493 0
[pid=1257/tid=1281] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1281/stat : 1281 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 20 0 19 0 638738540 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1284] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1284/stat : 1284 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 22 0 19 0 638738541 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1287] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1287/stat : 1287 (python) S 1255 1257 1220 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 19 0 638738542 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1290] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1290/stat : 1290 (python) S 1255 1257 1220 0 -1 4202560 23 933 0 0 0 0 0 0 22 0 19 0 638738543 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1314] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1314/stat : 1314 (python) S 1255 1257 1220 0 -1 4202560 40 933 0 0 0 0 0 0 20 0 19 0 638748563 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 2 0 0 0
[pid=1257/tid=1317] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1317/stat : 1317 (python) S 1255 1257 1220 0 -1 4202560 19 933 0 0 0 0 0 0 22 0 19 0 638748564 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1322] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1322/stat : 1322 (python) S 1255 1257 1220 0 -1 4202560 25 933 0 0 0 0 0 0 24 0 19 0 638748565 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 2 0 0 0
[pid=1257/tid=1325] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1325/stat : 1325 (python) S 1255 1257 1220 0 -1 4202560 26 933 0 0 0 0 0 0 25 0 19 0 638748566 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1348] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1348/stat : 1348 (python) S 1255 1257 1220 0 -1 4202560 18 933 0 0 0 0 0 0 20 0 19 0 638758588 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 2 0 0 0
[pid=1257/tid=1352] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1352/stat : 1352 (python) S 1255 1257 1220 0 -1 4202560 23 933 0 0 0 0 0 0 21 0 19 0 638758590 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1355] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1355/stat : 1355 (python) S 1255 1257 1220 0 -1 4202560 28 933 0 0 0 0 0 0 18 0 19 0 638758591 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1359] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1359/stat : 1359 (python) S 1255 1257 1220 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 638758989 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 4 0 0 0
[pid=1257/tid=1379] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1379/stat : 1379 (python) S 1255 1257 1220 0 -1 4202560 29 933 0 0 0 0 0 0 18 0 19 0 638768614 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1382] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1382/stat : 1382 (python) S 1255 1257 1220 0 -1 4202560 30 933 0 0 0 0 0 0 22 0 19 0 638768615 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 2 0 0 0
[pid=1257/tid=1386] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1386/stat : 1386 (python) S 1255 1257 1220 0 -1 4202560 30 933 0 0 0 0 0 0 18 0 19 0 638769024 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1398] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1398/stat : 1398 (python) S 1255 1257 1220 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 638778647 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1465] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1465/stat : 1465 (python) S 1255 1257 1220 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 19 0 638798300 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
[pid=1257/tid=1468] ppid=1255 vsize=465804 CPUtime=0 cores=0,2,4,6
/proc/1257/task/1468/stat : 1468 (python) S 1255 1257 1220 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 638798393 476983296 7862 33554432000 4194304 5683644 140735644431152 18446744073709551615 254273440177 0 0 16781312 2 18446604440928578624 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2384.65
Current children cumulated vsize (KiB) 15982572

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1257 and gives
#  childrusage.ru_utime.tv_sec=295
#  childrusage.ru_utime.tv_usec=553069
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=932554
# CPU time returned by wait4() is 298.486
# while last known CPU time is 2384.65
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 601.726
CPU time (s): 2384.65
CPU user time (s): 2381.7
CPU system time (s): 2.95
CPU usage (%): 396.302
Max. virtual memory (cumulated for all children) (KiB): 15982572

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 295.553
system time used= 2.93255
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 59826
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 24119
involuntary context switches= 3940

runsolver used 4.54831 second user time and 8.22775 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-15 14:40:29
IDJOB=3378154
IDBENCH=82563
IDSOLVER=1859
FILE ID=node140/3378154-1305463229
RUNJOBID= node140-1305463229-1238
PBS_JOBID= 13324987
Free space on /tmp= 73508 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S926861876-013-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378154-1305463229/watcher-3378154-1305463229 -o /tmp/evaluation-result-3378154-1305463229/solver-3378154-1305463229 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378154-1305463229.cnf -seed 2034263685 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= aa5d2fab702f701bf63e2d26e76ab2b3
RANDOM SEED=2034263685

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24408980 kB
Buffers:        316600 kB
Cached:        4381432 kB
SwapCached:       8088 kB
Active:        5041040 kB
Inactive:      3187112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24408980 kB
SwapTotal:    67111528 kB
SwapFree:     67057896 kB
Dirty:           65056 kB
Writeback:           0 kB
AnonPages:     3527476 kB
Mapped:          23276 kB
Slab:           227684 kB
PageTables:      21996 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 45573216 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73432 MiB
End job on node140 at 2011-05-15 14:50:33