Trace number 3385307

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 11763.7 1485.13

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S950666065-093-UNKNOWN.cnf
MD5SUMe27af9a147b2d7c0283d090128805575
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark262.183
Satisfiable
(Un)Satisfiability was proved
Number of variables34999
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.47	c INFO: solving HOME/instance-3385307-1305565023.cnf
0.31/0.47	c INFO: computing task features
0.80/0.97	c INFO: computed variable-clause graph statistics
0.80/0.99	c INFO: computed clause balance statistics
3.40/3.53	c INFO: computed variable balance statistics
5.42/5.53	c INFO: computed small-clause counts
5.42/5.54	c INFO: computed Horn-clause counts
7.00/7.10	c INFO: computed variable graph statistics
7.00/7.10	c INFO: collected features for HOME/instance-3385307-1305565023.cnf in 6.62 s
7.00/7.13	c INFO: running hybridGM3@0 for 97 with 4993 remaining (b = 0.34)
7.00/7.16	c INFO: running hybridGM3@0 for 97 with 4993 remaining (b = 0.21)
7.00/7.19	c INFO: running hybridGM3@0 for 97 with 4993 remaining (b = 0.14)
7.10/7.22	c INFO: running hybridGM3@0 for 97 with 4993 remaining (b = 0.10)
7.10/7.25	c INFO: running hybridGM3@0 for 97 with 4993 remaining (b = 0.07)
7.10/7.29	c INFO: running hybridGM3@0 for 194 with 4993 remaining (b = 0.10)
7.20/7.32	c INFO: running hybridGM3@0 for 194 with 4993 remaining (b = 0.08)
7.20/7.35	c INFO: running hybridGM3@0 for 194 with 4993 remaining (b = 0.06)
799.40/106.60	c INFO: running hybridGM3@0 for 97 with 4893 remaining (b = 0.03)
800.23/106.63	c INFO: running hybridGM3@0 for 97 with 4893 remaining (b = 0.03)
808.15/107.68	c INFO: running hybridGM3@0 for 97 with 4892 remaining (b = 0.03)
808.15/107.69	c INFO: running hybridGM3@0 for 97 with 4892 remaining (b = 0.03)
808.94/107.72	c INFO: running hybridGM3@0 for 97 with 4892 remaining (b = 0.03)
1560.15/201.93	c INFO: running hybridGM3@99 for 97 with 4798 remaining (b = 0.03)
1560.15/201.96	c INFO: running hybridGM3@99 for 97 with 4798 remaining (b = 0.03)
1560.92/202.08	c INFO: running TNM@0 for 97 with 4798 remaining (b = 0.03)
1568.91/203.03	c INFO: running hybridGM3@100 for 97 with 4797 remaining (b = 0.03)
1568.91/203.07	c INFO: running hybridGM3@100 for 97 with 4797 remaining (b = 0.03)
1577.57/204.10	c INFO: running hybridGM3@95 for 97 with 4796 remaining (b = 0.02)
1577.57/204.14	c INFO: running hybridGM3@0 for 97 with 4796 remaining (b = 0.02)
1624.63/210.09	c INFO: running hybridGM3@99 for 97 with 4790 remaining (b = 0.03)
2337.51/299.47	c INFO: running hybridGM3@0 for 97 with 4701 remaining (b = 0.03)
2361.47/302.46	c INFO: running hybridGM3@0 for 194 with 4698 remaining (b = 0.04)
2367.82/303.30	c INFO: running TNM@0 for 97 with 4697 remaining (b = 0.02)
2367.82/303.34	c INFO: running hybridGM3@0 for 194 with 4697 remaining (b = 0.03)
2377.31/304.41	c INFO: running hybridGM3@0 for 194 with 4696 remaining (b = 0.03)
2377.31/304.46	c INFO: running hybridGM3@296 for 97 with 4696 remaining (b = 0.01)
2377.31/304.50	c INFO: running TNM@0 for 97 with 4696 remaining (b = 0.01)
2401.13/307.45	c INFO: running TNM@0 for 97 with 4693 remaining (b = 0.01)
3113.87/396.85	c INFO: running hybridGM3@98 for 97 with 4603 remaining (b = 0.01)
3137.78/399.82	c INFO: running hybridGM3@396 for 97 with 4600 remaining (b = 0.01)
3168.13/403.71	c INFO: running TNM@0 for 97 with 4596 remaining (b = 0.01)
3170.28/403.95	c INFO: running TNM@0 for 97 with 4596 remaining (b = 0.01)
3201.49/407.85	c INFO: running TNM@0 for 97 with 4592 remaining (b = 0.01)
3913.66/497.27	c INFO: running hybridGM3@192 for 97 with 4503 remaining (b = 0.01)
3930.29/499.33	c INFO: running hybridGM3@100 for 97 with 4501 remaining (b = 0.01)
3937.41/500.29	c INFO: running hybridGM3@100 for 97 with 4500 remaining (b = 0.01)
3945.38/501.21	c INFO: running TNM@0 for 97 with 4499 remaining (b = 0.01)
3954.92/502.40	c INFO: running TNM@0 for 2037 with 4498 remaining (b = 0.19)
3968.41/504.11	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.02)
3969.26/504.28	c INFO: running hybridGM3@101 for 97 with 4496 remaining (b = 0.01)
4001.12/508.25	c INFO: running hybridGM3@0 for 194 with 4492 remaining (b = 0.02)
4657.18/590.69	c INFO: running hybridGM3@389 for 97 with 4409 remaining (b = 0.01)
4736.86/600.63	c INFO: running adaptg2wsat2009++@0 for 97 with 4399 remaining (b = 0.01)
4737.46/600.73	c INFO: running adaptg2wsat2009++@0 for 194 with 4399 remaining (b = 0.01)
4737.46/600.79	c INFO: running hybridGM3@296 for 97 with 4399 remaining (b = 0.01)
4768.47/604.70	c INFO: running TNM@0 for 97 with 4395 remaining (b = 0.01)
5464.50/692.11	c INFO: running hybridGM3@0 for 194 with 4308 remaining (b = 0.01)
5496.29/696.19	c INFO: running hybridGM3@393 for 97 with 4304 remaining (b = 0.01)
5534.45/701.00	c INFO: running TNM@0 for 97 with 4299 remaining (b = 0.01)
5535.24/701.06	c INFO: running adaptg2wsat2009++@100 for 97 with 4299 remaining (b = 0.01)
5567.93/705.13	c INFO: running TNM@0 for 97 with 4295 remaining (b = 0.01)
5568.52/705.21	c INFO: running hybridGM3@93 for 97 with 4295 remaining (b = 0.01)
6255.07/791.59	c INFO: running TNM@0 for 97 with 4208 remaining (b = 0.01)
6301.96/797.49	c INFO: running TNM@0 for 97 with 4203 remaining (b = 0.01)
6327.35/800.61	c INFO: running hybridGM3@188 for 97 with 4199 remaining (b = 0.01)
6333.69/801.43	c INFO: running adaptg2wsat2009++@100 for 97 with 4199 remaining (b = 0.01)
6341.65/802.49	c INFO: running TNM@0 for 97 with 4198 remaining (b = 0.01)
6366.25/805.58	c INFO: running hybridGM3@198 for 291 with 4194 remaining (b = 0.02)
7029.63/889.06	c INFO: running hybridGM3@94 for 97 with 4111 remaining (b = 0.01)
7052.69/892.02	c INFO: running TNM@196 for 1843 with 4108 remaining (b = 0.10)
7092.41/897.00	c INFO: running hybridGM3@0 for 194 with 4103 remaining (b = 0.01)
7107.47/898.86	c INFO: running adaptg2wsat2009++@0 for 194 with 4101 remaining (b = 0.01)
7132.13/901.96	c INFO: running FH@0 for 97 with 4098 remaining (b = 0.01)
7180.47/908.06	c INFO: running hybridGM3@194 for 291 with 4092 remaining (b = 0.01)
7835.88/990.51	c INFO: running hybridGM3@194 for 291 with 4010 remaining (b = 0.01)
7930.40/1002.41	c INFO: running hybridGM3@195 for 291 with 3998 remaining (b = 0.01)
8663.68/1094.72	c INFO: running adaptg2wsat2009++@98 for 97 with 3905 remaining (b = 0.00)
8719.30/1101.74	c INFO: running hybridGM3@394 for 97 with 3898 remaining (b = 0.01)
8752.65/1105.99	c INFO: running hybridGM3@96 for 97 with 3894 remaining (b = 0.01)
9469.21/1196.16	c INFO: running adaptg2wsat2009++@393 for 194 with 3804 remaining (b = 0.01)
9485.10/1198.20	c INFO: running hybridGM3@295 for 194 with 3802 remaining (b = 0.01)
9511.30/1201.41	c INFO: running hybridGM3@289 for 291 with 3799 remaining (b = 0.01)
9524.81/1203.18	c INFO: running hybridGM3@194 for 291 with 3797 remaining (b = 0.01)
10188.16/1286.66	c INFO: running hybridGM3@294 for 194 with 3713 remaining (b = 0.01)
10235.13/1292.55	c INFO: running hybridGM3@389 for 97 with 3707 remaining (b = 0.01)
10992.90/1387.96	c INFO: running hybridGM3@484 for 97 with 3612 remaining (b = 0.01)
11017.53/1391.03	c INFO: running adaptg2wsat2009++@682 for 97 with 3609 remaining (b = 0.00)
11056.57/1395.98	c INFO: running hybridGM3@390 for 97 with 3604 remaining (b = 0.01)
11759.53/1484.44	c INFO: running FH@0 for 97 with 3516 remaining (b = 0.00)
11763.68/1485.02	c INFO: running TNM@194 for 1358 with 3515 remaining (b = 0.06)
11763.68/1485.07	c INFO: running hybridGM3@201 for 291 with 3515 remaining (b = 0.01)
11763.68/1485.07	Traceback (most recent call last):
11763.68/1485.07	  File "HOME/solve", line 33, in <module>
11763.68/1485.07	    plac.call(main)
11763.68/1485.07	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
11763.68/1485.08	    cmd, result = parser_from(obj).consume(arglist)
11763.68/1485.08	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
11763.68/1485.08	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
11763.68/1485.08	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
11763.68/1485.08	    (_, answer, _) = solver(input_path, budget / cores, cores)
11763.68/1485.08	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
11763.68/1485.08	    solver.go(max_cost * borg.defaults.machine_speed)
11763.68/1485.08	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
11763.68/1485.08	    self._process.start()
11763.68/1485.08	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
11763.68/1485.08	    assert self._popen is None, 'cannot start a process twice'
11763.68/1485.08	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385307-1305565023/watcher-3385307-1305565023 -o /tmp/evaluation-result-3385307-1305565023/solver-3385307-1305565023 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385307-1305565023.cnf -seed 1794402536 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.61 1.26 0.43 2/434 3998
/proc/meminfo: memFree=20251092/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=15380 CPUtime=0 cores=0-7
/proc/3998/stat : 3998 (python) S 3996 3998 3915 0 -1 4202496 794 0 0 0 0 0 0 0 18 0 1 0 746733023 15749120 681 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367544832 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/3998/statm: 3845 681 322 364 0 332 0

[startup+0.0616759 s]
/proc/loadavg: 4.61 1.26 0.43 2/434 3998
/proc/meminfo: memFree=20251092/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=40968 CPUtime=0.03 cores=0-7
/proc/3998/stat : 3998 (python) R 3996 3998 3915 0 -1 4202496 1676 0 0 0 3 0 0 0 17 0 1 0 746733023 41951232 1501 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367544832 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/3998/statm: 10242 1501 463 364 0 1025 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 40968

[startup+0.100694 s]
/proc/loadavg: 4.61 1.26 0.43 2/434 3998
/proc/meminfo: memFree=20251092/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=104220 CPUtime=0.06 cores=0-7
/proc/3998/stat : 3998 (python) S 3996 3998 3915 0 -1 4202496 3049 0 0 0 5 1 0 0 17 0 1 0 746733023 106721280 2733 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367544832 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/3998/statm: 26055 2733 847 364 0 1803 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104220

[startup+0.300629 s]
/proc/loadavg: 4.61 1.26 0.43 2/434 3998
/proc/meminfo: memFree=20251092/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=170752 CPUtime=0.23 cores=0-7
/proc/3998/stat : 3998 (python) R 3996 3998 3915 0 -1 4202496 6729 934 1 0 17 6 0 0 17 0 1 0 746733023 174850048 4805 33554432000 4194304 5683644 140734438846528 18446744073709551615 5291040 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/3998/statm: 42688 4805 1336 364 0 3453 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 170752

[startup+0.700565 s]
/proc/loadavg: 4.61 1.26 0.43 2/434 3998
/proc/meminfo: memFree=20251092/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=295584 CPUtime=0.6 cores=0-7
/proc/3998/stat : 3998 (python) R 3996 3998 3915 0 -1 4202496 13524 934 1 0 51 9 0 0 17 0 1 0 746733023 302678016 11207 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367207621 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/3998/statm: 73896 11207 1761 364 0 9820 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 295584

[startup+1.50041 s]
/proc/loadavg: 4.61 1.26 0.43 2/435 4003
/proc/meminfo: memFree=20195808/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=312972 CPUtime=1.4 cores=0-7
/proc/3998/stat : 3998 (python) R 3996 3998 3915 0 -1 4202496 21966 934 1 0 130 10 0 0 18 0 1 0 746733023 320483328 15628 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367214077 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/3998/statm: 78243 15628 1826 364 0 14167 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 312972

[startup+3.10111 s]
/proc/loadavg: 4.32 1.26 0.43 2/435 4003
/proc/meminfo: memFree=20194940/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=312972 CPUtime=3 cores=0-7
/proc/3998/stat : 3998 (python) R 3996 3998 3915 0 -1 4202496 21966 934 1 0 290 10 0 0 22 0 1 0 746733023 320483328 15628 33554432000 4194304 5683644 140734438846528 18446744073709551615 251366907648 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/3998/statm: 78243 15628 1826 364 0 14167 0
Current children cumulated CPU time (s) 3
Current children cumulated vsize (KiB) 312972

[startup+6.30052 s]
/proc/loadavg: 4.32 1.26 0.43 2/435 4003
/proc/meminfo: memFree=20195524/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=312972 CPUtime=6.2 cores=0-7
/proc/3998/stat : 3998 (python) R 3996 3998 3915 0 -1 4202496 21973 934 1 0 610 10 0 0 25 0 1 0 746733023 320483328 15635 33554432000 4194304 5683644 140734438846528 18446744073709551615 47347151829120 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/3998/statm: 78243 15635 1833 364 0 14167 0
Current children cumulated CPU time (s) 6.2
Current children cumulated vsize (KiB) 312972

[startup+12.7013 s]
/proc/loadavg: 4.37 1.36 0.47 9/459 4027
/proc/meminfo: memFree=19914516/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=457836 CPUtime=7.26 cores=0-7
/proc/3998/stat : 3998 (python) S 3996 3998 3915 0 -1 4202496 28063 934 1 0 713 13 0 0 19 0 9 0 746733023 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/3998/statm: 114459 15291 1889 364 0 50356 0
[pid=4004] ppid=3998 vsize=310264 CPUtime=0.02 cores=0-7
/proc/4004/stat : 4004 (python) S 3998 3998 3915 0 -1 4202560 898 0 0 0 1 1 0 0 15 0 1 0 746733736 317710336 13495 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566863 0 0 16781312 514 17827676150613115137 0 0 17 7 0 0 0
/proc/4004/statm: 77566 13495 376 364 0 13484 0
[pid=4006] ppid=4004 vsize=268424 CPUtime=5.55 cores=0-7
/proc/4006/stat : 4006 (hybridGM3) R 4004 4006 4006 0 -1 4202496 8089 0 0 0 555 0 0 0 25 0 1 0 746733737 274866176 7824 33554432000 134512640 135204385 4293678384 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4006/statm: 67106 7824 64 169 0 66936 0
[pid=4007] ppid=3998 vsize=320520 CPUtime=0.02 cores=0-7
/proc/4007/stat : 4007 (python) S 3998 3998 3915 0 -1 4202560 918 0 0 0 1 1 0 0 15 0 1 0 746733739 328212480 13493 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 9451340109128368387 0 0 17 4 0 0 0
/proc/4007/statm: 80130 13493 370 364 0 16045 0
[pid=4009] ppid=4007 vsize=268428 CPUtime=5.53 cores=0-7
/proc/4009/stat : 4009 (hybridGM3) R 4007 4009 4009 0 -1 4202496 8084 0 0 0 552 1 0 0 25 0 1 0 746733740 274870272 7820 33554432000 134512640 135204385 4294633056 18446744073709551615 134536777 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4009/statm: 67107 7820 64 169 0 66937 0
[pid=4010] ppid=3998 vsize=330776 CPUtime=0.02 cores=0-7
/proc/4010/stat : 4010 (python) S 3998 3998 3915 0 -1 4202560 920 0 0 0 1 1 0 0 15 0 1 0 746733743 338714624 13497 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 4143919416528044289 0 0 17 1 0 0 0
/proc/4010/statm: 82694 13497 370 364 0 18606 0
[pid=4012] ppid=4010 vsize=268424 CPUtime=5.47 cores=0-7
/proc/4012/stat : 4012 (hybridGM3) R 4010 4012 4012 0 -1 4202496 8085 0 0 0 546 1 0 0 25 0 1 0 746733743 274866176 7823 33554432000 134512640 135204385 4292821232 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4012/statm: 67106 7823 64 169 0 66936 0
[pid=4013] ppid=3998 vsize=341032 CPUtime=0.02 cores=0-7
/proc/4013/stat : 4013 (python) S 3998 3998 3915 0 -1 4202560 948 0 0 0 1 1 0 0 15 0 1 0 746733746 349216768 13501 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 8497108327119945986 0 0 17 1 0 0 0
/proc/4013/statm: 85258 13501 370 364 0 21167 0
[pid=4015] ppid=4013 vsize=268428 CPUtime=5.46 cores=0-7
/proc/4015/stat : 4015 (hybridGM3) R 4013 4015 4015 0 -1 4202496 8084 0 0 0 545 1 0 0 25 0 1 0 746733746 274870272 7824 33554432000 134512640 135204385 4287674128 18446744073709551615 134537100 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4015/statm: 67107 7824 64 169 0 66937 0
[pid=4016] ppid=3998 vsize=416824 CPUtime=0.02 cores=0-7
/proc/4016/stat : 4016 (python) S 3998 3998 3915 0 -1 4202560 799 0 0 0 1 1 0 0 15 0 1 0 746733749 426827776 13679 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 8497096782247854338 0 0 17 1 0 0 0
/proc/4016/statm: 104206 13679 371 364 0 40112 0
[pid=4018] ppid=4016 vsize=268424 CPUtime=5.43 cores=0-7
/proc/4018/stat : 4018 (hybridGM3) R 4016 4018 4018 0 -1 4202496 8079 0 0 0 543 0 0 0 25 0 1 0 746733749 274866176 7827 33554432000 134512640 135204385 4287140304 18446744073709551615 134537249 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/4018/statm: 67106 7827 64 169 0 66936 0
[pid=4019] ppid=3998 vsize=427080 CPUtime=0.01 cores=0-7
/proc/4019/stat : 4019 (python) S 3998 3998 3915 0 -1 4202560 815 0 0 0 0 1 0 0 15 0 1 0 746733752 437329920 13641 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 9451332412546973953 0 0 17 1 0 0 0
/proc/4019/statm: 106770 13641 371 364 0 42673 0
[pid=4021] ppid=4019 vsize=268428 CPUtime=5.28 cores=0-7
/proc/4021/stat : 4021 (hybridGM3) R 4019 4021 4021 0 -1 4202496 8081 0 0 0 527 1 0 0 25 0 1 0 746733752 274870272 7822 33554432000 134512640 135204385 4290480704 18446744073709551615 134537544 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/4021/statm: 67107 7822 64 169 0 66937 0
[pid=4022] ppid=3998 vsize=437336 CPUtime=0.01 cores=0-7
/proc/4022/stat : 4022 (python) S 3998 3998 3915 0 -1 4202560 822 0 0 0 0 1 0 0 15 0 1 0 746733755 447832064 13645 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 9451336260837671171 0 0 17 1 0 0 0
/proc/4022/statm: 109334 13645 371 364 0 45234 0
[pid=4024] ppid=4022 vsize=268420 CPUtime=5.33 cores=0-7
/proc/4024/stat : 4024 (hybridGM3) R 4022 4024 4024 0 -1 4202496 8083 0 0 0 533 0 0 0 25 0 1 0 746733756 274862080 7825 33554432000 134512640 135204385 4291349648 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4024/statm: 67105 7825 64 169 0 66935 0
[pid=4025] ppid=3998 vsize=447592 CPUtime=0.02 cores=0-7
/proc/4025/stat : 4025 (python) S 3998 3998 3915 0 -1 4202560 863 0 0 0 1 1 0 0 15 0 1 0 746733758 458334208 13775 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 5402111462913048833 0 0 17 1 0 0 0
/proc/4025/statm: 111898 13775 377 364 0 47795 0
[pid=4027] ppid=4025 vsize=268420 CPUtime=5.18 cores=0-7
/proc/4027/stat : 4027 (hybridGM3) R 4025 4027 4027 0 -1 4202496 8090 0 0 0 517 1 0 0 25 0 1 0 746733759 274862080 7824 33554432000 134512640 135204385 4292189568 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4027/statm: 67105 7824 64 169 0 66935 0
[pid=3998/tid=4005] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4005/stat : 4005 (python) S 3996 3998 3915 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 9 0 746733737 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4008] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4008/stat : 4008 (python) S 3996 3998 3915 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 9 0 746733740 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 4 0 0 0
[pid=3998/tid=4011] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4011/stat : 4011 (python) S 3996 3998 3915 0 -1 4202560 26 934 0 0 0 0 0 0 21 0 9 0 746733743 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 4 0 0 0
[pid=3998/tid=4014] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4014/stat : 4014 (python) S 3996 3998 3915 0 -1 4202560 24 934 0 0 0 0 0 0 18 0 9 0 746733746 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 5 0 0 0
[pid=3998/tid=4017] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4017/stat : 4017 (python) S 3996 3998 3915 0 -1 4202560 96 934 0 0 0 0 0 0 19 0 9 0 746733749 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 0 0 0 0
[pid=3998/tid=4020] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4020/stat : 4020 (python) S 3996 3998 3915 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 9 0 746733752 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 7 0 0 0
[pid=3998/tid=4023] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4023/stat : 4023 (python) S 3996 3998 3915 0 -1 4202560 34 934 0 0 0 0 0 0 21 0 9 0 746733755 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4026] ppid=3996 vsize=457836 CPUtime=0 cores=0-7
/proc/3998/task/4026/stat : 4026 (python) S 3996 3998 3915 0 -1 4202560 34 934 0 0 0 0 0 0 19 0 9 0 746733759 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.63
Current children cumulated vsize (KiB) 5636656

[startup+25.5019 s]
/proc/loadavg: 4.93 1.58 0.55 9/459 4028
/proc/meminfo: memFree=19917164/32950928 swapFree=38710912/67111528
[pid=3998] ppid=3996 vsize=457836 CPUtime=7.26 cores=0-7
/proc/3998/stat : 3998 (python) S 3996 3998 3915 0 -1 4202496 28063 934 1 0 713 13 0 0 19 0 9 0 746733023 468824064 15291 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/3998/statm: 114459 15291 1889 364 0 50356 0
[pid=4004] ppid=3998 vsize=310264 CPUtime=0.05 cores=0-7
/proc/4004/stat : 4004 (python) S 3998 3998 3915 0 -1 4202560 905 0 0 0 3 2 0 0 15 0 1 0 746733736 317710336 13495 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566863 0 0 16781312 514 15789901768537637121 0 0 17 7 0 0 0
/proc/4004/statm: 77566 13495 376 364 0 13484 0
[pid=4006] ppid=4004 vsize=268424 CPUtime=18.35 cores=0-7
/proc/4006/stat : 4006 (hybridGM3) R 4004 4006 4006 0 -1 4202496 8092 0 0 0 1835 0 0 0 25 0 1 0 746733737 274866176 7827 33554432000 134512640 135204385 4293678384 18446744073709551615 134536843 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4006/statm: 67106 7827 64 169 0 66936 0
[pid=4007] ppid=3998 vsize=320520 CPUtime=0.05 cores=0-7
/proc/4007/stat : 4007 (python) S 3998 3998 3915 0 -1 4202560 924 0 0 0 2 3 0 0 15 0 1 0 746733739 328212480 13493 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 15789901768537637123 0 0 17 4 0 0 0
/proc/4007/statm: 80130 13493 370 364 0 16045 0
[pid=4009] ppid=4007 vsize=268428 CPUtime=18.33 cores=0-7
/proc/4009/stat : 4009 (hybridGM3) R 4007 4009 4009 0 -1 4202496 8086 0 0 0 1832 1 0 0 25 0 1 0 746733740 274870272 7822 33554432000 134512640 135204385 4294633056 18446744073709551615 134537224 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4009/statm: 67107 7822 64 169 0 66937 0
[pid=4010] ppid=3998 vsize=330776 CPUtime=0.05 cores=0-7
/proc/4010/stat : 4010 (python) S 3998 3998 3915 0 -1 4202560 923 0 0 0 3 2 0 0 15 0 1 0 746733743 338714624 13497 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 15789901768537637121 0 0 17 1 0 0 0
/proc/4010/statm: 82694 13497 370 364 0 18606 0
[pid=4012] ppid=4010 vsize=268424 CPUtime=18.23 cores=0-7
/proc/4012/stat : 4012 (hybridGM3) R 4010 4012 4012 0 -1 4202496 8085 0 0 0 1822 1 0 0 25 0 1 0 746733743 274866176 7823 33554432000 134512640 135204385 4292821232 18446744073709551615 134547471 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4012/statm: 67106 7823 64 169 0 66936 0
[pid=4013] ppid=3998 vsize=341032 CPUtime=0.05 cores=0-7
/proc/4013/stat : 4013 (python) S 3998 3998 3915 0 -1 4202560 953 0 0 0 2 3 0 0 15 0 1 0 746733746 349216768 13501 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 15789901768537637122 0 0 17 1 0 0 0
/proc/4013/statm: 85258 13501 370 364 0 21167 0
[pid=4015] ppid=4013 vsize=268428 CPUtime=18.26 cores=0-7
/proc/4015/stat : 4015 (hybridGM3) R 4013 4015 4015 0 -1 4202496 8084 0 0 0 1825 1 0 0 25 0 1 0 746733746 274870272 7824 33554432000 134512640 135204385 4287674128 18446744073709551615 134537401 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4015/statm: 67107 7824 64 169 0 66937 0
[pid=4016] ppid=3998 vsize=416824 CPUtime=0.05 cores=0-7
/proc/4016/stat : 4016 (python) S 3998 3998 3915 0 -1 4202560 799 0 0 0 3 2 0 0 15 0 1 0 746733749 426827776 13679 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 15789901768537637122 0 0 17 1 0 0 0
/proc/4016/statm: 104206 13679 371 364 0 40112 0
[pid=4018] ppid=4016 vsize=268424 CPUtime=18.23 cores=0-7
/proc/4018/stat : 4018 (hybridGM3) R 4016 4018 4018 0 -1 4202496 8079 0 0 0 1823 0 0 0 25 0 1 0 746733749 274866176 7827 33554432000 134512640 135204385 4287140304 18446744073709551615 134537488 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/4018/statm: 67106 7827 64 169 0 66936 0

################
# More data... #
################

/proc/4120/statm: 155486 14015 400 364 0 91332 0
[pid=4122] ppid=4120 vsize=465044 CPUtime=98.5 cores=0-7
/proc/4122/stat : 4122 (TNM) T 4120 4122 4122 0 -1 4202496 6034 0 0 0 9849 1 0 0 25 0 1 0 746793494 476205056 5749 33554432000 134512640 135008748 4288431776 18446744073709551615 134527526 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/4122/statm: 116261 5749 46 122 0 115161 0
[pid=4120/tid=4134] ppid=3998 vsize=621944 CPUtime=0 cores=0-7
/proc/4120/task/4134/stat : 4134 (python) S 3998 3998 3915 0 -1 4202560 86 0 0 0 0 0 0 0 19 0 2 0 746803530 636870656 14015 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604453104280576 0 0 -1 7 0 0 0
[pid=4126] ppid=3998 vsize=632200 CPUtime=1.11 cores=0-7
/proc/4126/stat : 4126 (python) S 3998 3998 3915 0 -1 4202560 1179 0 0 0 55 56 0 0 15 0 2 0 746802234 647372800 14055 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/4126/statm: 158050 14055 400 364 0 93893 0
[pid=4128] ppid=4126 vsize=268420 CPUtime=296.91 cores=0-7
/proc/4128/stat : 4128 (hybridGM3) T 4126 4128 4128 0 -1 4202496 8106 0 0 0 29690 1 0 0 25 0 1 0 746802235 274862080 7824 33554432000 134512640 135204385 4288721808 18446744073709551615 134547802 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/4128/statm: 67105 7824 64 169 0 66935 0
[pid=4126/tid=4156] ppid=3998 vsize=632200 CPUtime=0 cores=0-7
/proc/4126/task/4156/stat : 4156 (python) S 3998 3998 3915 0 -1 4202560 83 0 0 0 0 0 0 0 15 0 2 0 746821923 647372800 14055 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604441941903360 0 0 -1 6 0 0 0
[pid=4131] ppid=3998 vsize=642456 CPUtime=0.36 cores=0-7
/proc/4131/stat : 4131 (python) S 3998 3998 3915 0 -1 4202560 1184 0 0 0 17 19 0 0 19 0 2 0 746803123 657874944 14014 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/4131/statm: 160614 14014 400 364 0 96454 0
[pid=4133] ppid=4131 vsize=465044 CPUtime=99.55 cores=0-7
/proc/4133/stat : 4133 (TNM) T 4131 4133 4133 0 -1 4202496 6035 0 0 0 9955 0 0 0 25 0 1 0 746803124 476205056 5749 33554432000 134512640 135008748 4290545920 18446744073709551615 134527477 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/4133/statm: 116261 5749 46 122 0 115161 0
[pid=4131/tid=4148] ppid=3998 vsize=642456 CPUtime=0 cores=0-7
/proc/4131/task/4148/stat : 4148 (python) S 3998 3998 3915 0 -1 4202560 82 0 0 0 0 0 0 0 19 0 2 0 746813160 657874944 14014 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604441941902464 0 0 -1 6 0 0 0
[pid=4135] ppid=3998 vsize=652712 CPUtime=2.66 cores=0-7
/proc/4135/stat : 4135 (python) S 3998 3998 3915 0 -1 4202560 1192 0 0 0 112 154 0 0 18 0 2 0 746803536 668377088 14023 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/4135/statm: 163178 14023 401 364 0 99015 0
[pid=4137] ppid=4135 vsize=465044 CPUtime=692.23 cores=0-7
/proc/4137/stat : 4137 (TNM) R 4135 4137 4137 0 -1 4202496 6035 0 0 0 69220 3 0 0 25 0 1 0 746803537 476205056 5750 33554432000 134512640 135008748 4290811008 18446744073709551615 134527436 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/4137/statm: 116261 5750 46 122 0 115161 0
[pid=4135/tid=4152] ppid=3998 vsize=652712 CPUtime=0 cores=0-7
/proc/4135/task/4152/stat : 4152 (python) S 3998 3998 3915 0 -1 4202560 85 0 0 0 0 0 0 0 19 0 2 0 746813574 668377088 14023 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604446450323648 0 0 -1 0 0 0 0
[pid=4141] ppid=3998 vsize=662968 CPUtime=0.37 cores=0-7
/proc/4141/stat : 4141 (python) S 3998 3998 3915 0 -1 4202560 1227 0 0 0 19 18 0 0 19 0 2 0 746812182 678879232 14033 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/4141/statm: 165742 14033 406 364 0 101576 0
[pid=4143] ppid=4141 vsize=465044 CPUtime=99.58 cores=0-7
/proc/4143/stat : 4143 (TNM) T 4141 4143 4143 0 -1 4202496 6041 0 0 0 9957 1 0 0 25 0 1 0 746812183 476205056 5749 33554432000 134512640 135008748 4290212928 18446744073709551615 134527477 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/4143/statm: 116261 5749 46 122 0 115161 0
[pid=4141/tid=4157] ppid=3998 vsize=662968 CPUtime=0 cores=0-7
/proc/4141/task/4157/stat : 4157 (python) S 3998 3998 3915 0 -1 4202560 87 0 0 0 0 0 0 0 19 0 2 0 746822219 678879232 14033 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604453636423936 0 0 -1 2 0 0 0
[pid=4145] ppid=3998 vsize=673224 CPUtime=0.39 cores=0-7
/proc/4145/stat : 4145 (python) S 3998 3998 3915 0 -1 4202560 1229 0 0 0 18 21 0 0 19 0 2 0 746812772 689381376 14156 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/4145/statm: 168306 14156 407 364 0 104137 0
[pid=4147] ppid=4145 vsize=465044 CPUtime=99.42 cores=0-7
/proc/4147/stat : 4147 (TNM) T 4145 4147 4147 0 -1 4202496 6037 0 0 0 9942 0 0 0 25 0 1 0 746812773 476205056 5749 33554432000 134512640 135008748 4292536928 18446744073709551615 134527847 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/4147/statm: 116261 5749 46 122 0 115161 0
[pid=4145/tid=4158] ppid=3998 vsize=673224 CPUtime=0 cores=0-7
/proc/4145/task/4158/stat : 4158 (python) S 3998 3998 3915 0 -1 4202560 86 0 0 0 0 0 0 0 19 0 2 0 746822716 689381376 14156 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604453636424832 0 0 -1 2 0 0 0
[pid=4149] ppid=3998 vsize=683480 CPUtime=0.37 cores=0-7
/proc/4149/stat : 4149 (python) S 3998 3998 3915 0 -1 4202560 1234 0 0 0 19 18 0 0 19 0 2 0 746813272 699883520 14109 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/4149/statm: 170870 14109 406 364 0 106698 0
[pid=4151] ppid=4149 vsize=465044 CPUtime=98.93 cores=0-7
/proc/4151/stat : 4151 (TNM) T 4149 4151 4151 0 -1 4202496 6038 0 0 0 9892 1 0 0 25 0 1 0 746813273 476205056 5750 33554432000 134512640 135008748 4291831136 18446744073709551615 134529409 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/4151/statm: 116261 5750 46 122 0 115161 0
[pid=4149/tid=4165] ppid=3998 vsize=683480 CPUtime=0 cores=0-7
/proc/4149/task/4165/stat : 4165 (python) S 3998 3998 3915 0 -1 4202560 90 0 0 0 0 0 0 0 19 0 2 0 746823212 699883520 14109 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604447884794816 0 0 -1 1 0 0 0
[pid=4159] ppid=3998 vsize=693736 CPUtime=1.17 cores=0-7
/proc/4159/stat : 4159 (python) S 3998 3998 3915 0 -1 4202560 1251 0 0 0 59 58 0 0 15 0 2 0 746822723 710385664 14148 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/4159/statm: 173434 14148 406 364 0 109259 0
[pid=4161] ppid=4159 vsize=268424 CPUtime=292.77 cores=0-7
/proc/4161/stat : 4161 (hybridGM3) T 4159 4161 4161 0 -1 4202496 8113 0 0 0 29276 1 0 0 25 0 1 0 746822724 274866176 7825 33554432000 134512640 135204385 4290550800 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/4161/statm: 67106 7825 64 169 0 66936 0
[pid=4159/tid=4176] ppid=3998 vsize=693736 CPUtime=0 cores=0-7
/proc/4159/task/4176/stat : 4176 (python) S 3998 3998 3915 0 -1 4202560 94 0 0 0 0 0 0 0 15 0 2 0 746843616 710385664 14148 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604447884793024 0 0 -1 7 0 0 0
[pid=4162] ppid=3998 vsize=703992 CPUtime=2.25 cores=0-7
/proc/4162/stat : 4162 (python) S 3998 3998 3915 0 -1 4202560 1264 0 0 0 111 114 0 0 18 0 2 0 746822909 720887808 14250 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/4162/statm: 175998 14250 406 364 0 111820 0
[pid=4164] ppid=4162 vsize=431704 CPUtime=582.16 cores=0-7
/proc/4164/stat : 4164 (adaptg2wsat2009) R 4162 4164 4164 0 -1 4202496 6276 0 0 0 58215 1 0 0 25 0 1 0 746822910 442064896 5983 33554432000 134512640 135107411 4291328176 18446744073709551615 134532128 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4164/statm: 107926 5983 59 146 0 106802 0
[pid=4162/tid=4175] ppid=3998 vsize=703992 CPUtime=0 cores=0-7
/proc/4162/task/4175/stat : 4175 (python) S 3998 3998 3915 0 -1 4202560 92 0 0 0 0 0 0 0 18 0 2 0 746842489 720887808 14250 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604447884792128 0 0 -1 6 0 0 0
[pid=4166] ppid=3998 vsize=714248 CPUtime=0.39 cores=0-7
/proc/4166/stat : 4166 (python) S 3998 3998 3915 0 -1 4202560 1243 0 0 0 19 20 0 0 19 0 2 0 746823219 731389952 14195 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379373515 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/4166/statm: 178562 14195 401 364 0 114381 0
[pid=4168] ppid=4166 vsize=260404 CPUtime=100 cores=0-7
/proc/4168/stat : 4168 (FH) T 4166 4168 4168 0 -1 4202496 6188 0 0 0 9998 2 0 0 25 0 1 0 746823220 266653696 5899 33554432000 134512640 134936988 4294244128 18446744073709551615 134554976 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/4168/statm: 65101 5899 41 104 0 64605 0
[pid=4166/tid=4172] ppid=3998 vsize=714248 CPUtime=0 cores=0-7
/proc/4166/task/4172/stat : 4172 (python) S 3998 3998 3915 0 -1 4202560 91 0 0 0 0 0 0 0 19 0 2 0 746833257 731389952 14195 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 514 18446604443685229824 0 0 -1 6 0 0 0
[pid=4187] ppid=3998 vsize=724504 CPUtime=0 cores=0-7
/proc/4187/stat : 4187 (python) S 3998 3998 3915 0 -1 4202560 929 0 0 0 0 0 0 0 19 0 1 0 746881467 741892096 14145 33554432000 4194304 5683644 140734438846528 18446744073709551615 251367566918 0 0 16781312 514 15708574746016743423 0 0 17 2 0 0 0
/proc/4187/statm: 181126 14145 345 364 0 116942 0
[pid=4189] ppid=4187 vsize=255216 CPUtime=0.47 cores=0-7
/proc/4189/stat : 4189 (FH) R 4187 4189 4189 0 -1 4202496 3713 0 0 0 47 0 0 0 21 0 1 0 746881468 261341184 3421 33554432000 134512640 134936988 4294805984 18446744073709551615 134520268 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4189/statm: 63804 3421 38 104 0 63308 0
[pid=3998/tid=4005] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4005/stat : 4005 (python) S 3996 3998 3915 0 -1 4202560 34 934 0 0 0 0 0 0 15 0 36 0 746733737 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4008] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4008/stat : 4008 (python) S 3996 3998 3915 0 -1 4202560 38 934 0 0 0 0 0 0 18 0 36 0 746733740 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4011] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4011/stat : 4011 (python) S 3996 3998 3915 0 -1 4202560 42 934 0 0 0 0 0 0 15 0 36 0 746733743 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 4 0 0 0
[pid=3998/tid=4014] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4014/stat : 4014 (python) S 3996 3998 3915 0 -1 4202560 47 934 0 0 0 0 0 0 15 0 36 0 746733746 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 5 0 0 0
[pid=3998/tid=4017] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4017/stat : 4017 (python) S 3996 3998 3915 0 -1 4202560 112 934 0 0 0 0 0 0 15 0 36 0 746733749 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4020] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4020/stat : 4020 (python) S 3996 3998 3915 0 -1 4202560 57 934 0 0 0 0 0 0 18 0 36 0 746733752 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4023] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4023/stat : 4023 (python) S 3996 3998 3915 0 -1 4202560 52 934 0 0 0 0 0 0 18 0 36 0 746733755 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4026] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4026/stat : 4026 (python) S 3996 3998 3915 0 -1 4202560 45 934 0 0 0 0 0 0 18 0 36 0 746733759 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4039] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4039/stat : 4039 (python) S 3996 3998 3915 0 -1 4202560 44 934 0 0 0 0 0 0 20 0 36 0 746753231 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4058] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4058/stat : 4058 (python) S 3996 3998 3915 0 -1 4202560 71 934 0 0 0 0 0 0 18 0 36 0 746763269 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4061] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4061/stat : 4061 (python) S 3996 3998 3915 0 -1 4202560 31 934 0 0 0 0 0 0 19 0 36 0 746763354 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 7 0 0 0
[pid=3998/tid=4064] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4064/stat : 4064 (python) S 3996 3998 3915 0 -1 4202560 43 934 0 0 0 0 0 0 18 0 36 0 746763358 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4067] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4067/stat : 4067 (python) S 3996 3998 3915 0 -1 4202560 38 934 0 0 0 0 0 0 18 0 36 0 746763465 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4070] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4070/stat : 4070 (python) S 3996 3998 3915 0 -1 4202560 40 934 0 0 0 0 0 0 21 0 36 0 746763473 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4073] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4073/stat : 4073 (python) S 3996 3998 3915 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 36 0 746763769 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4080] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4080/stat : 4080 (python) S 3996 3998 3915 0 -1 4202560 39 934 0 0 0 0 0 0 18 0 36 0 746773394 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4084] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4084/stat : 4084 (python) S 3996 3998 3915 0 -1 4202560 35 934 0 0 0 0 0 0 20 0 36 0 746773419 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4088] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4088/stat : 4088 (python) S 3996 3998 3915 0 -1 4202560 35 934 0 0 0 0 0 0 20 0 36 0 746773809 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4095] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4095/stat : 4095 (python) S 3996 3998 3915 0 -1 4202560 39 934 0 0 0 0 0 0 20 0 36 0 746783144 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4099] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4099/stat : 4099 (python) S 3996 3998 3915 0 -1 4202560 34 934 0 0 0 0 0 0 20 0 36 0 746783264 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4103] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4103/stat : 4103 (python) S 3996 3998 3915 0 -1 4202560 35 934 0 0 0 0 0 0 20 0 36 0 746783435 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4108] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4108/stat : 4108 (python) S 3996 3998 3915 0 -1 4202560 48 934 0 0 0 0 0 0 18 0 36 0 746783849 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4115] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4115/stat : 4115 (python) S 3996 3998 3915 0 -1 4202560 40 934 0 0 0 0 0 0 18 0 36 0 746793087 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4118] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4118/stat : 4118 (python) S 3996 3998 3915 0 -1 4202560 39 934 0 0 0 0 0 0 18 0 36 0 746793097 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 4 0 0 0
[pid=3998/tid=4121] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4121/stat : 4121 (python) S 3996 3998 3915 0 -1 4202560 36 934 0 0 0 0 0 0 20 0 36 0 746793494 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4127] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4127/stat : 4127 (python) S 3996 3998 3915 0 -1 4202560 42 934 0 0 0 0 0 0 18 0 36 0 746802235 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4132] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4132/stat : 4132 (python) S 3996 3998 3915 0 -1 4202560 36 934 0 0 0 0 0 0 20 0 36 0 746803124 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 6 0 0 0
[pid=3998/tid=4136] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4136/stat : 4136 (python) S 3996 3998 3915 0 -1 4202560 41 934 0 0 0 0 0 0 18 0 36 0 746803537 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 3 0 0 0
[pid=3998/tid=4142] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4142/stat : 4142 (python) S 3996 3998 3915 0 -1 4202560 39 934 0 0 0 0 0 0 20 0 36 0 746812182 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 1 0 0 0
[pid=3998/tid=4146] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4146/stat : 4146 (python) S 3996 3998 3915 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 36 0 746812773 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4150] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4150/stat : 4150 (python) S 3996 3998 3915 0 -1 4202560 37 934 0 0 0 0 0 0 20 0 36 0 746813273 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4160] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4160/stat : 4160 (python) S 3996 3998 3915 0 -1 4202560 45 934 0 0 0 0 0 0 18 0 36 0 746822723 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4163] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4163/stat : 4163 (python) S 3996 3998 3915 0 -1 4202560 40 934 0 0 0 0 0 0 18 0 36 0 746822909 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4167] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4167/stat : 4167 (python) S 3996 3998 3915 0 -1 4202560 32 934 0 0 0 0 0 0 19 0 36 0 746823219 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
[pid=3998/tid=4188] ppid=3996 vsize=734748 CPUtime=0 cores=0-7
/proc/3998/task/4188/stat : 4188 (python) S 3996 3998 3915 0 -1 4202560 42 934 0 0 0 0 0 0 20 0 36 0 746881468 752381952 15740 33554432000 4194304 5683644 140734438846528 18446744073709551615 251379370417 0 0 16781312 2 18446604440570773568 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11763.7
Current children cumulated vsize (KiB) 32550924

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 3998 and gives
#  childrusage.ru_utime.tv_sec=3630
#  childrusage.ru_utime.tv_usec=587066
#  childrusage.ru_stime.tv_sec=22
#  childrusage.ru_stime.tv_usec=517576
# CPU time returned by wait4() is 3653.1
# while last known CPU time is 11763.7
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1485.13
CPU time (s): 11763.7
CPU user time (s): 11741.2
CPU system time (s): 22.48
CPU usage (%): 792.096
Max. virtual memory (cumulated for all children) (KiB): 32550924

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3630.59
system time used= 22.5176
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 160085
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 21261
involuntary context switches= 26660

runsolver used 16.7015 second user time and 46.7719 second system time

The end

Launcher Data 

Begin job on node113 at 2011-05-16 18:57:03
IDJOB=3385307
IDBENCH=82572
IDSOLVER=1870
FILE ID=node113/3385307-1305565023
RUNJOBID= node113-1305564965-3929
PBS_JOBID= 13350137
Free space on /tmp= 73436 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S950666065-093-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385307-1305565023/watcher-3385307-1305565023 -o /tmp/evaluation-result-3385307-1305565023/solver-3385307-1305565023 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385307-1305565023.cnf -seed 1794402536 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e27af9a147b2d7c0283d090128805575
RANDOM SEED=1794402536

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      20251380 kB
Buffers:        302156 kB
Cached:        3950348 kB
SwapCached:      93692 kB
Active:        4786196 kB
Inactive:      7609216 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      20251380 kB
SwapTotal:    67111528 kB
SwapFree:     38710912 kB
Dirty:           34548 kB
Writeback:           0 kB
AnonPages:     8061684 kB
Mapped:          22936 kB
Slab:           126576 kB
PageTables:     107316 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 132597804 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73404 MiB
End job on node113 at 2011-05-16 19:21:51