Trace number 3264862

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.36 1227.7

General information on the benchmark

NameSAT05/crafted/jarvisalo05/mod2c-rand3bip-sat/
mod2c-rand3bip-sat-250-3.shuffled-as.sat05-2535.cnf
MD5SUM22bc1843113c53c5f8a8f8cec42dcbe5
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark240.682
Satisfiable
(Un)Satisfiability was proved
Number of variables352
Number of clauses2492
Sum of the clauses size11032
Maximum clause length6
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 41592
Number of clauses of size 5736
Number of clauses of size over 5164

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264862-1304100568.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 352
0.00/0.00	c Clauses = 2492
0.00/0.00	c Literals = 11032
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 352
0.00/0.00	c Clauses = 2492
0.00/0.00	c Literals = 11032
0.00/0.00	c a2
0.00/0.00	c a1
0.02/0.08	c Independent sub-problems = 1
0.02/0.08	c Sub-problem with 352 variables and 2492 clauses
0.02/0.08	c using permutation method 4
0.02/0.08	c count = 2
0.02/0.08	c Ordering finished 0.080 seconds
0.02/0.08	c mapped 0.080 seconds
0.02/0.08	c permuted 0.080 seconds
0.02/0.08	c about to sort
0.02/0.08	c sorted
0.02/0.08	c clauses now 2492
0.02/0.08	c init finished 0.080 seconds
0.02/0.08	c tree built 0.080 seconds
0.02/0.08	c 
0.02/0.08	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.02/0.08	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.02/0.08	c ................................ | ..........                    |                                  0.1         0.0             
0.02/0.09	c ................................ | ..........                  1 |              65                  0.1        10.0          214
0.02/0.09	c ................................ | ..........                  2 |              65                  0.1        20.0          214
0.02/0.09	c ................................ | ..........                  3 |              65                  0.1        30.0          214
0.02/0.09	c ................................ | ..........                  4 |              65                  0.1        40.0          214
0.02/0.09	c ................................ | ..........                  5 |              65                  0.1        50.0          214
0.02/0.09	c ................................ | ..........               1000 | 113          525                 0.1     10000.0         6160
95.74/24.18	c ................................ | ..........            1524000 | 478505       720021             96.1     15861.8     19363406
191.73/48.37	c ................................ | ..........            2611000 | 830612       1230151           192.0     13598.3     31899237
287.94/72.63	c ................................ | ..........            3471000 | 1105720      1663830           288.1     12048.7     38432365
383.81/96.85	c ................................ | ..........            4300000 | 1381326      2039050           384.0     11197.3     45458119
479.87/121.13	c ................................ | *.........            5065000 | 1627535      2420745           480.0     10551.9     50314547
575.86/145.46	c ................................ | *.........            5812000 | 1872970      2778067           576.1     10088.7     52926959
671.73/169.78	c ................................ | *.........            6525000 | 2107739      3114082           672.1      9709.0     57746459
767.89/194.23	c ................................ | *.........            7188000 | 2321334      3446366           768.0      9359.4     59447847
863.82/218.55	c ................................ | *.........            7896000 | 2557433      3768685           864.0      9138.9     62143127
959.86/242.84	c ................................ | **........            8550000 | 2768796      4087904           960.0      8905.9     64615324
1055.98/267.42	c ................................ | **........            9181000 | 2973192      4396262          1056.1      8693.5     67008770
1151.91/291.94	c ................................ | **........            9844000 | 3194509      4695057          1152.1      8544.6     69190966
1247.74/316.48	c ................................ | **........           10481000 | 3402792      5020301          1248.0      8397.9     70379947
1343.83/341.05	c ................................ | **........           11116000 | 3611373      5342166          1344.0      8270.5     72568735
1440.05/365.50	c ................................ | ***.......           11734000 | 3815621      5634516          1440.1      8148.3     74821197
1536.04/390.10	c ................................ | ***.......           12338000 | 4011245      5947376          1536.0      8032.3     78757855
1632.11/414.70	c ................................ | ***.......           12927000 | 4204331      6238059          1632.1      7920.3     80493240
1728.01/439.11	c ................................ | ***.......           13540000 | 4405339      6526888          1728.1      7835.3     81893100
1824.06/463.92	c ................................ | ***.......           14113000 | 4590240      6821079          1824.2      7736.8     83767006
1919.92/488.52	c ................................ | ****......           14685000 | 4777543      7103072          1920.0      7648.4     85334331
2015.86/513.08	c ................................ | ****......           15240000 | 4965157      7353682          2016.2      7558.8     88215865
2111.86/537.54	c ................................ | ****......           15799000 | 5144171      7637884          2112.0      7480.6     91936816
2207.91/562.03	c ................................ | ****......           16329000 | 5314427      7910210          2208.1      7395.2     94696136
2304.09/586.70	c ................................ | ****......           16894000 | 5503985      8166618          2304.1      7332.1     97867890
2399.80/611.26	c ................................ | *****.....           17421000 | 5677723      8416221          2400.0      7258.8     98474173
2495.94/635.83	c ................................ | *****.....           17926000 | 5838962      8672185          2496.1      7181.7    100372019
2591.81/660.48	c ................................ | *****.....           18440000 | 6007472      8913862          2592.2      7113.8    101020439
2687.91/685.06	c ................................ | *****.....           18935000 | 6171758      9142380          2688.2      7043.9    101341307
2784.05/709.51	c ................................ | *****.....           19425000 | 6328676      9388493          2784.1      6977.1    102437620
2879.79/734.16	c ................................ | ******....           19906000 | 6483866      9628170          2880.1      6911.7    104553979
2976.11/758.80	c ................................ | ******....           20428000 | 6659008      9855964          2976.1      6864.0    106685667
3071.80/783.65	c ................................ | ******....           20931000 | 6822499      10108115         3072.0      6813.4    107778522
3167.92/808.33	c ................................ | ******....           21434000 | 6983758      10369812         3168.1      6765.7    107988964
3263.86/832.78	c ................................ | ******....           21960000 | 7156928      10624347         3264.2      6727.6    109865653
3359.85/857.47	c ................................ | *******...           22480000 | 7332029      10862299         3360.2      6690.2    112472357
3455.81/882.36	c ................................ | *******...           22973000 | 7490661      11115584         3456.1      6647.1    115286999
3551.70/906.98	c ................................ | *******...           23442000 | 7641528      11360627         3552.1      6599.6    116080150
3648.04/931.84	c ................................ | *******...           23951000 | 7810572      11602221         3648.2      6565.1    118315185
3744.03/956.32	c ................................ | *******...           24449000 | 7978119      11828658         3744.1      6530.0    118315185
3839.81/980.89	c ................................ | ********..           24928000 | 8132958      12066209         3840.2      6491.4    120556868
3935.90/1005.73	c ................................ | ********..           25395000 | 8284563      12298045         3936.0      6451.9    122445748
4031.82/1030.28	c ................................ | ********..           25883000 | 8449051      12517543         4032.2      6419.2    123819388
4127.67/1054.78	c ................................ | ********..           26362000 | 8603673      12756360         4128.0      6386.1    125008046
4223.91/1079.65	c ................................ | ********..           26816000 | 8749554      12985275         4224.1      6348.3    125854124
4319.83/1104.26	c ................................ | *********.           27303000 | 8912747      13205572         4320.1      6320.0    126256639
4415.91/1128.83	c ................................ | *********.           27769000 | 9070025      13414441         4416.0      6288.2    127565523
4512.12/1153.31	c ................................ | *********.           28203000 | 9217265      13617383         4512.2      6250.4    130182483
4607.79/1178.28	c ................................ | *********.           28718000 | 9385299      13883830         4608.1      6232.0    132017731
4704.05/1202.92	c ................................ | *********.           29203000 | 9542965      14140392         4704.1      6208.0    133447784

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264862-1304100568/watcher-3264862-1304100568 -o /tmp/evaluation-result-3264862-1304100568/solver-3264862-1304100568 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264862-1304100568.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.41 9.08 9.57 3/176 29853
/proc/meminfo: memFree=17441464/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=9640 CPUtime=0 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) R 29851 29853 28861 0 -1 4202496 496 0 0 0 0 0 0 0 24 0 1 0 600287582 9871360 438 33554432000 4194304 4338100 140734420077136 18446744073709551615 4206993 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29853/statm: 2410 438 152 36 0 315 0

[startup+0.021441 s]
/proc/loadavg: 6.41 9.08 9.57 3/176 29853
/proc/meminfo: memFree=17441464/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=9780 CPUtime=0.02 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) R 29851 29853 28861 0 -1 4202496 510 0 0 0 2 0 0 0 24 0 1 0 600287582 10014720 452 33554432000 4194304 4338100 140734420077136 18446744073709551615 4295973 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29853/statm: 2445 452 155 36 0 350 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 9780

[startup+0.10274 s]
/proc/loadavg: 6.41 9.08 9.57 3/176 29853
/proc/meminfo: memFree=17441464/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=198440 CPUtime=0.12 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1940 0 0 0 12 0 0 0 24 0 5 0 600287582 203202560 1843 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 49610 1845 175 36 0 47515 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 198440

[startup+0.300396 s]
/proc/loadavg: 6.41 9.08 9.57 3/176 29853
/proc/meminfo: memFree=17441464/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=270888 CPUtime=0.91 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 5529 0 0 0 90 1 0 0 24 0 5 0 600287582 277389312 5432 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 67722 5432 175 36 0 65627 0
Current children cumulated CPU time (s) 0.91
Current children cumulated vsize (KiB) 270888

[startup+0.851657 s]
/proc/loadavg: 6.41 9.08 9.57 3/176 29853
/proc/meminfo: memFree=17441464/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=311944 CPUtime=4.22 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 15757 0 0 0 414 8 0 0 24 0 5 0 600287582 319430656 15660 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 78768 17036 175 36 0 76673 0
Current children cumulated CPU time (s) 4.22
Current children cumulated vsize (KiB) 311944

[startup+1.57621 s]
/proc/loadavg: 6.41 9.08 9.57 3/176 29853
/proc/meminfo: memFree=17441464/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=331100 CPUtime=6.01 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 21157 0 0 0 590 11 0 0 24 0 5 0 600287582 339046400 21060 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 82775 21062 175 36 0 80680 0
Current children cumulated CPU time (s) 6.01
Current children cumulated vsize (KiB) 331100

[startup+3.10091 s]
/proc/loadavg: 6.54 9.06 9.56 9/186 29863
/proc/meminfo: memFree=17200356/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=391048 CPUtime=12.09 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 36489 0 0 0 1185 24 0 0 24 0 5 0 600287582 400433152 36392 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 97762 36392 175 36 0 95667 0
[pid=29853/tid=29856] ppid=29851 vsize=391048 CPUtime=2.98 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 5372 0 0 0 292 6 0 0 22 0 5 0 600287591 400433152 36392 33554432000 4194304 4338100 140734420077136 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=391048 CPUtime=2.99 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 5004 0 0 0 293 6 0 0 19 0 5 0 600287591 400433152 36392 33554432000 4194304 4338100 140734420077136 18446744073709551615 215987987842 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=391048 CPUtime=2.99 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 17389 0 0 0 296 3 0 0 25 0 5 0 600287591 400433152 36392 33554432000 4194304 4338100 140734420077136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=391048 CPUtime=3 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 7211 0 0 0 293 7 0 0 19 0 5 0 600287591 400433152 36392 33554432000 4194304 4338100 140734420077136 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.09
Current children cumulated vsize (KiB) 391048

[startup+6.30035 s]
/proc/loadavg: 6.54 9.06 9.56 9/186 29863
/proc/meminfo: memFree=16967744/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=509904 CPUtime=24.85 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 65669 0 0 0 2437 48 0 0 24 0 5 0 600287582 522141696 65572 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 127476 65573 176 36 0 125381 0
[pid=29853/tid=29856] ppid=29851 vsize=509904 CPUtime=6.15 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 9469 0 0 0 602 13 0 0 25 0 5 0 600287591 522141696 65573 33554432000 4194304 4338100 140734420077136 18446744073709551615 215987987477 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=509904 CPUtime=6.19 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 9627 0 0 0 606 13 0 0 25 0 5 0 600287591 522141696 65573 33554432000 4194304 4338100 140734420077136 18446744073709551615 215987987937 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=509904 CPUtime=6.19 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 32294 0 0 0 614 5 0 0 25 0 5 0 600287591 522141696 65573 33554432000 4194304 4338100 140734420077136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=509904 CPUtime=6.19 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 12767 0 0 0 605 14 0 0 25 0 5 0 600287591 522141696 65573 33554432000 4194304 4338100 140734420077136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.85
Current children cumulated vsize (KiB) 509904

[startup+12.7002 s]
/proc/loadavg: 6.76 9.03 9.54 9/186 29864
/proc/meminfo: memFree=16676140/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=648728 CPUtime=50.33 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 101687 0 0 0 4938 95 0 0 24 0 5 0 600287582 664297472 101590 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 162182 101590 176 36 0 160087 0
[pid=29853/tid=29856] ppid=29851 vsize=648728 CPUtime=12.49 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 15300 0 0 0 1221 28 0 0 25 0 5 0 600287591 664297472 101590 33554432000 4194304 4338100 140734420077136 18446744073709551615 215987987579 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=648728 CPUtime=12.57 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 14814 0 0 0 1231 26 0 0 25 0 5 0 600287591 664297472 101590 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=648728 CPUtime=12.59 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 50888 0 0 0 1249 10 0 0 25 0 5 0 600287591 664297472 101590 33554432000 4194304 4338100 140734420077136 18446744073709551615 4279542 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=648728 CPUtime=12.57 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 19172 0 0 0 1228 29 0 0 19 0 5 0 600287591 664297472 101590 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.33
Current children cumulated vsize (KiB) 648728

[startup+25.501 s]
/proc/loadavg: 6.95 8.99 9.53 9/186 29864
/proc/meminfo: memFree=16219304/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=904660 CPUtime=101.33 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 166996 0 0 0 9937 196 0 0 24 0 5 0 600287582 926371840 166899 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 226165 166899 176 36 0 224070 0
[pid=29853/tid=29856] ppid=29851 vsize=904660 CPUtime=25.19 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 26908 0 0 0 2458 61 0 0 25 0 5 0 600287591 926371840 166899 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=904660 CPUtime=25.33 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 24048 0 0 0 2476 57 0 0 25 0 5 0 600287591 926371840 166899 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=904660 CPUtime=25.37 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 82875 0 0 0 2522 15 0 0 25 0 5 0 600287591 926371840 166899 33554432000 4194304 4338100 140734420077136 18446744073709551615 215987988206 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=904660 CPUtime=25.33 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 31652 0 0 0 2471 62 0 0 25 0 5 0 600287591 926371840 166899 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.33
Current children cumulated vsize (KiB) 904660

[startup+51.1005 s]
/proc/loadavg: 7.31 8.91 9.49 9/186 29865
/proc/meminfo: memFree=15521684/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=1290644 CPUtime=202.71 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 265987 0 0 0 19886 385 0 0 24 0 5 0 600287582 1321619456 265890 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 322661 265890 176 36 0 320566 0
[pid=29853/tid=29856] ppid=29851 vsize=1290644 CPUtime=50.35 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 44772 0 0 0 4913 122 0 0 25 0 5 0 600287591 1321619456 265890 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=1290644 CPUtime=50.67 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 40173 0 0 0 4951 116 0 0 25 0 5 0 600287591 1321619456 265890 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=1290644 CPUtime=50.89 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 130117 0 0 0 5066 23 0 0 25 0 5 0 600287591 1321619456 265890 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=1290644 CPUtime=50.66 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 49412 0 0 0 4945 121 0 0 25 0 5 0 600287591 1321619456 265890 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.71
Current children cumulated vsize (KiB) 1290644

[startup+102.301 s]
/proc/loadavg: 7.70 8.77 9.41 9/186 29866
/proc/meminfo: memFree=14487648/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=1760956 CPUtime=405.59 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 389182 0 0 0 39843 716 0 0 24 0 5 0 600287582 1803218944 389085 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 440239 389086 176 36 0 438144 0
[pid=29853/tid=29856] ppid=29851 vsize=1760956 CPUtime=100.98 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 71114 0 0 0 9864 234 0 0 19 0 5 0 600287591 1803218944 389086 33554432000 4194304 4338100 140734420077136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=1760956 CPUtime=101.24 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 57834 0 0 0 9903 221 0 0 21 0 5 0 600287591 1803218944 389086 33554432000 4194304 4338100 140734420077136 18446744073709551615 215987987508 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=1760956 CPUtime=101.81 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 192358 0 0 0 10146 35 0 0 25 0 5 0 600287591 1803218944 389086 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=1760956 CPUtime=101.43 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 66364 0 0 0 9919 224 0 0 25 0 5 0 600287591 1803218944 389086 33554432000 4194304 4338100 140734420077136 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 405.59
Current children cumulated vsize (KiB) 1760956

[startup+162.301 s]
/proc/loadavg: 7.89 8.63 9.32 9/186 29868
/proc/meminfo: memFree=13869124/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=2096884 CPUtime=642.71 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 476528 0 0 0 63198 1073 0 0 24 0 5 0 600287582 2147209216 476431 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 524221 476431 176 36 0 522126 0
[pid=29853/tid=29856] ppid=29851 vsize=2096884 CPUtime=160.02 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 84886 0 0 0 15650 352 0 0 25 0 5 0 600287591 2147209216 476431 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=2096884 CPUtime=160.41 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 67113 0 0 0 15703 338 0 0 25 0 5 0 600287591 2147209216 476431 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=2096884 CPUtime=161.59 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 237443 0 0 0 16114 45 0 0 25 0 5 0 600287591 2147209216 476431 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=2096884 CPUtime=160.56 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 85573 0 0 0 15720 336 0 0 25 0 5 0 600287591 2147209216 476431 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 642.71
Current children cumulated vsize (KiB) 2096884

[startup+222.301 s]
/proc/loadavg: 7.96 8.51 9.23 9/186 29869
/proc/meminfo: memFree=13437568/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=2308444 CPUtime=878.82 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 533185 0 0 0 86494 1388 0 0 24 0 5 0 600287582 2363846656 533088 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 577111 533089 176 36 0 575016 0
[pid=29853/tid=29856] ppid=29851 vsize=2308444 CPUtime=218.6 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 88012 0 0 0 21401 459 0 0 25 0 5 0 600287591 2363846656 533089 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=10594864/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=3949892 CPUtime=2989.89 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 930112 0 0 0 295083 3906 0 0 24 0 5 0 600287582 4044689408 930015 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 987473 930015 176 36 0 985378 0
[pid=29853/tid=29856] ppid=29851 vsize=3949892 CPUtime=742.27 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 154174 0 0 0 72917 1310 0 0 25 0 5 0 600287591 4044689408 930015 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=3949892 CPUtime=745.89 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 102229 0 0 0 73340 1249 0 0 25 0 5 0 600287591 4044689408 930015 33554432000 4194304 4338100 140734420077136 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=3949892 CPUtime=755.67 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 543469 0 0 0 75444 123 0 0 25 0 5 0 600287591 4044689408 930015 33554432000 4194304 4338100 140734420077136 18446744073709551615 4235674 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=3949892 CPUtime=745.94 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 128727 0 0 0 73372 1222 0 0 25 0 5 0 600287591 4044689408 930015 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2989.89
Current children cumulated vsize (KiB) 3949892

[startup+822.301 s]
/proc/loadavg: 8.04 8.04 8.60 9/186 29885
/proc/meminfo: memFree=10435868/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4022412 CPUtime=3222.71 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 950675 0 0 0 318123 4148 0 0 24 0 5 0 600287582 4118949888 950578 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1005603 950578 176 36 0 1003508 0
[pid=29853/tid=29856] ppid=29851 vsize=4022412 CPUtime=800.35 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 157059 0 0 0 78644 1391 0 0 25 0 5 0 600287591 4118949888 950578 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4022412 CPUtime=803.53 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 103529 0 0 0 79026 1327 0 0 25 0 5 0 600287591 4118949888 950578 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4022412 CPUtime=814.95 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 558424 0 0 0 81368 127 0 0 25 0 5 0 600287591 4118949888 950578 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4022412 CPUtime=803.75 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 130150 0 0 0 79074 1301 0 0 18 0 5 0 600287591 4118949888 950579 33554432000 4194304 4338100 140734420077136 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3222.71
Current children cumulated vsize (KiB) 4022412

[startup+882.301 s]
/proc/loadavg: 7.90 8.00 8.56 9/186 29887
/proc/meminfo: memFree=10113424/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4266716 CPUtime=3455.81 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1010314 0 0 0 341167 4414 0 0 24 0 5 0 600287582 4369117184 1010217 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1066679 1010217 176 36 0 1064584 0
[pid=29853/tid=29856] ppid=29851 vsize=4266716 CPUtime=858.22 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 169464 0 0 0 84343 1479 0 0 25 0 5 0 600287591 4369117184 1010217 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4266716 CPUtime=861.44 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 105171 0 0 0 84733 1411 0 0 25 0 5 0 600287591 4369117184 1010217 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4266716 CPUtime=874.07 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 602829 0 0 0 87272 135 0 0 25 0 5 0 600287591 4369117184 1010217 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4266716 CPUtime=861.94 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 131337 0 0 0 84808 1386 0 0 25 0 5 0 600287591 4369117184 1010217 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3455.81
Current children cumulated vsize (KiB) 4266716

[startup+942.307 s]
/proc/loadavg: 8.03 8.01 8.52 9/186 29888
/proc/meminfo: memFree=9820140/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4368548 CPUtime=3689.14 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1041550 0 0 0 364252 4662 0 0 24 0 5 0 600287582 4473393152 1041453 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1092137 1041453 176 36 0 1090042 0
[pid=29853/tid=29856] ppid=29851 vsize=4368548 CPUtime=916.18 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 174899 0 0 0 90056 1562 0 0 25 0 5 0 600287591 4473393152 1041453 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4368548 CPUtime=919.82 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 109236 0 0 0 90491 1491 0 0 25 0 5 0 600287591 4473393152 1041453 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4368548 CPUtime=933.35 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 622205 0 0 0 93193 142 0 0 25 0 5 0 600287591 4473393152 1041453 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254332 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4368548 CPUtime=919.66 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 133697 0 0 0 90501 1465 0 0 25 0 5 0 600287591 4473393152 1041453 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3689.14
Current children cumulated vsize (KiB) 4368548

[startup+1002.3 s]
/proc/loadavg: 7.86 7.97 8.48 9/186 29890
/proc/meminfo: memFree=9604108/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4488892 CPUtime=3922.72 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1074918 0 0 0 387377 4895 0 0 24 0 5 0 600287582 4596625408 1074821 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1122223 1074821 176 36 0 1120128 0
[pid=29853/tid=29856] ppid=29851 vsize=4488892 CPUtime=974.44 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 178753 0 0 0 95803 1641 0 0 25 0 5 0 600287591 4596625408 1074821 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4488892 CPUtime=977.95 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 110366 0 0 0 96232 1563 0 0 19 0 5 0 600287591 4596625408 1074821 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4488892 CPUtime=991.94 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 647321 0 0 0 99046 148 0 0 24 0 5 0 600287591 4596625408 1074821 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4488892 CPUtime=978.28 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 136965 0 0 0 96286 1542 0 0 25 0 5 0 600287591 4596625408 1074821 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3922.72
Current children cumulated vsize (KiB) 4488892

[startup+1062.3 s]
/proc/loadavg: 8.02 7.99 8.45 9/186 29892
/proc/meminfo: memFree=9438676/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4587084 CPUtime=4156.38 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1103129 0 0 0 410489 5149 0 0 24 0 5 0 600287582 4697174016 1103032 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1146771 1103032 176 36 0 1144676 0
[pid=29853/tid=29856] ppid=29851 vsize=4587084 CPUtime=1032.37 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 183485 0 0 0 101509 1728 0 0 25 0 5 0 600287591 4697174016 1103032 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4587084 CPUtime=1037.01 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 111482 0 0 0 102058 1643 0 0 25 0 5 0 600287591 4697174016 1103032 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4587084 CPUtime=1050.52 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 663599 0 0 0 104900 152 0 0 25 0 5 0 600287591 4697174016 1103032 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4587084 CPUtime=1036.36 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 143050 0 0 0 102012 1624 0 0 25 0 5 0 600287591 4697174016 1103032 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4156.38
Current children cumulated vsize (KiB) 4587084

[startup+1122.3 s]
/proc/loadavg: 8.06 8.01 8.43 9/186 29893
/proc/meminfo: memFree=9270256/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4708324 CPUtime=4391.12 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1127330 0 0 0 433729 5383 0 0 24 0 5 0 600287582 4821323776 1127233 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1177081 1127233 176 36 0 1174986 0
[pid=29853/tid=29856] ppid=29851 vsize=4708324 CPUtime=1090.77 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 187158 0 0 0 107273 1804 0 0 25 0 5 0 600287591 4821323776 1127233 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4708324 CPUtime=1095.54 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 115018 0 0 0 107835 1719 0 0 25 0 5 0 600287591 4821323776 1127233 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 7 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4708324 CPUtime=1109.86 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 678075 0 0 0 110830 156 0 0 25 0 5 0 600287591 4821323776 1127233 33554432000 4194304 4338100 140734420077136 18446744073709551615 4235929 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4708324 CPUtime=1094.83 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 145566 0 0 0 107780 1703 0 0 25 0 5 0 600287591 4821323776 1127233 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4391.12
Current children cumulated vsize (KiB) 4708324

[startup+1182.3 s]
/proc/loadavg: 8.00 7.99 8.39 9/186 29895
/proc/meminfo: memFree=9029916/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=4866332 CPUtime=4623.45 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1171089 0 0 0 456710 5635 0 0 24 0 5 0 600287582 4983123968 1170992 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1216583 1170992 176 36 0 1214488 0
[pid=29853/tid=29856] ppid=29851 vsize=4866332 CPUtime=1148.09 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 188265 0 0 0 112925 1884 0 0 25 0 5 0 600287591 4983123968 1170992 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=4866332 CPUtime=1153.49 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 117158 0 0 0 113547 1802 0 0 22 0 5 0 600287591 4983123968 1170992 33554432000 4194304 4338100 140734420077136 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=4866332 CPUtime=1169 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 714501 0 0 0 116735 165 0 0 25 0 5 0 600287591 4983123968 1170992 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 5 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=4866332 CPUtime=1152.74 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) S 29851 29853 28861 0 -1 4202560 149652 0 0 0 113492 1782 0 0 25 0 5 0 600287591 4983123968 1170992 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000405254 0 0 4096 0 18446604471153407296 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4623.45
Current children cumulated vsize (KiB) 4866332



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1227.5 s]
/proc/loadavg: 7.75 7.93 8.35 9/186 29959
/proc/meminfo: memFree=12482980/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=5013804 CPUtime=4800.14 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) S 29851 29853 28861 0 -1 4202496 1211185 0 0 0 474200 5814 0 0 24 0 5 0 600287582 5134135296 1211088 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 1253451 1211088 176 36 0 1251356 0
[pid=29853/tid=29856] ppid=29851 vsize=5013804 CPUtime=1191.94 cores=1,3,5,7
/proc/29853/task/29856/stat : 29856 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 194192 0 0 0 117251 1943 0 0 25 0 5 0 600287591 5134135296 1211089 33554432000 4194304 4338100 140734420077136 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29853/tid=29857] ppid=29851 vsize=5013804 CPUtime=1197.56 cores=1,3,5,7
/proc/29853/task/29857/stat : 29857 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 118022 0 0 0 117897 1859 0 0 25 0 5 0 600287591 5134135296 1211089 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29853/tid=29858] ppid=29851 vsize=5013804 CPUtime=1213.5 cores=1,3,5,7
/proc/29853/task/29858/stat : 29858 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 746746 0 0 0 121177 173 0 0 25 0 5 0 600287591 5134135296 1211089 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=5013804 CPUtime=1197.04 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4202560 150713 0 0 0 117866 1838 0 0 25 0 5 0 600287591 5134135296 1211089 33554432000 4194304 4338100 140734420077136 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.14
Current children cumulated vsize (KiB) 5013804

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1227.61 s]
/proc/loadavg: 7.75 7.93 8.35 9/186 29959
/proc/meminfo: memFree=12482980/32950940 swapFree=67111528/67111528
[pid=29853] ppid=29851 vsize=0 CPUtime=4800.26 cores=1,3,5,7
/proc/29853/stat : 29853 (ttsth-5-4) Z 29851 29853 28861 0 -1 4203548 1211188 0 0 0 474202 5824 0 0 18 0 2 0 600287582 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29853/statm: 0 0 0 0 0 0 0
[pid=29853/tid=29859] ppid=29851 vsize=0 CPUtime=1197.14 cores=1,3,5,7
/proc/29853/task/29859/stat : 29859 (ttsth-5-4) R 29851 29853 28861 0 -1 4203588 150713 0 0 0 117866 1848 0 0 25 0 2 0 600287591 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.26
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1227.7
CPU time (s): 4800.36
CPU user time (s): 4742.02
CPU system time (s): 58.3321
CPU usage (%): 391.005
Max. virtual memory (cumulated for all children) (KiB): 5013804

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4742.03
system time used= 58.3321
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1211188
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 249030
involuntary context switches= 28443

runsolver used 2.04869 second user time and 4.65029 second system time

The end

Launcher Data 

Begin job on node117 at 2011-04-29 20:09:28
IDJOB=3264862
IDBENCH=85355
IDSOLVER=1654
FILE ID=node117/3264862-1304100568
RUNJOBID= node117-1304100568-29835
PBS_JOBID= 13170599
Free space on /tmp= 71680 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/jarvisalo05/mod2c-rand3bip-sat/mod2c-rand3bip-sat-250-3.shuffled-as.sat05-2535.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264862-1304100568/watcher-3264862-1304100568 -o /tmp/evaluation-result-3264862-1304100568/solver-3264862-1304100568 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264862-1304100568.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 22bc1843113c53c5f8a8f8cec42dcbe5
RANDOM SEED=824226131

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      17441472 kB
Buffers:       2381488 kB
Cached:       12480920 kB
SwapCached:          0 kB
Active:        8149056 kB
Inactive:      6778032 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      17441472 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:              28 kB
Writeback:           0 kB
AnonPages:       64276 kB
Mapped:          15268 kB
Slab:           518772 kB
PageTables:       4684 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS:   205964 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71680 MiB
End job on node117 at 2011-04-29 20:29:58