Trace number 3273404

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1203.4

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n160-s40283905-sat.cnf
MD5SUM8e3c134eb643f2d9a45c884774c054a8
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.19497
Satisfiable
(Un)Satisfiability was proved
Number of variables160
Number of clauses384
Sum of the clauses size960
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2320
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 564
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273404-1304056916.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 160
0.00/0.00	c Clauses = 384
0.00/0.00	c Literals = 960
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 160
0.00/0.00	c Clauses = 384
0.00/0.00	c Literals = 960
0.00/0.00	c a2
0.00/0.00	c a1
1.39/1.48	c Independent sub-problems = 1
1.39/1.48	c Sub-problem with 160 variables and 384 clauses
1.39/1.48	c using permutation method 3
1.39/1.48	c Ordering finished 1.480 seconds
1.39/1.48	c mapped 1.480 seconds
1.39/1.48	c permuted 1.480 seconds
1.39/1.48	c about to sort
1.39/1.48	c sorted
1.39/1.49	c clauses now 384
1.39/1.49	c init finished 1.480 seconds
1.39/1.49	c tree built 1.480 seconds
1.39/1.49	c 
1.39/1.49	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.39/1.49	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.39/1.49	c ................................ | ..........                    |                                  1.5         0.0             
1.39/1.49	c ................................ | ..........                  1 |                                  1.5         0.7             
1.39/1.49	c ................................ | ..........                  2 |                                  1.5         1.4             
1.39/1.49	c ................................ | ..........                  3 |                                  1.5         2.0             
1.39/1.49	c ................................ | ..........                  4 |                                  1.5         2.7             
1.39/1.49	c ................................ | ..........                  5 |                                  1.5         3.4             
1.39/1.49	c ................................ | ..........               1000 | 63           828                 1.5       666.7        14330
96.16/25.20	c ................................ | ..........             613000 | 127836       89423              96.2      6374.1      7225517
191.63/49.19	c ................................ | ..........             802000 | 170234       114044            192.0      4176.6     11138835
288.24/73.36	c ................................ | ..........             955000 | 207928       125064            288.5      3310.2     12966525
383.71/97.28	c ................................ | ..........            1102000 | 241988       136835            384.0      2869.5     15075248
479.97/121.35	c ................................ | *.........            1290000 | 286783       148063            480.2      2686.4     16112924
575.76/145.38	c ................................ | *.........            1500000 | 338023       161597            576.0      2603.9     17595790
673.49/169.83	c ................................ | *.........            1746000 | 399180       176469            673.6      2591.9     18644963
768.15/193.50	c ................................ | *.........            1818000 | 416495       182823            768.1      2366.7     19706753
863.62/217.49	c ................................ | *.........            1922000 | 441306       191174            864.0      2224.5     20932982
959.83/241.55	c ................................ | **........            2020000 | 464660       200508            960.0      2104.1     21750717
1057.68/266.03	c ................................ | **........            2118000 | 487137       208994           1057.8      2002.2     22366605
1152.27/289.76	c ................................ | **........            2202000 | 509632       221306           1152.5      1910.6     23657220
1248.11/313.78	c ................................ | **........            2320000 | 542874       236238           1248.4      1858.3     25226973
1343.95/337.80	c ................................ | **........            2470000 | 581788       244969           1344.3      1837.4     25226973
1440.52/361.96	c ................................ | ***.......            2621000 | 619217       256415           1440.7      1819.2     25226973
1536.74/386.07	c ................................ | ***.......            2727000 | 647426       263498           1537.0      1774.2     25727564
1632.63/410.07	c ................................ | ***.......            2813000 | 665630       274379           1632.9      1722.7     26975059
1728.09/433.94	c ................................ | ***.......            2904000 | 685133       284061           1728.3      1680.3     28102347
1823.96/457.96	c ................................ | ***.......            3028000 | 711353       300896           1824.2      1659.9     29646574
1919.83/481.95	c ................................ | ****......            3204000 | 751033       314521           1920.0      1668.7     30966831
2016.05/506.02	c ................................ | ****......            3354000 | 784509       326870           2016.1      1663.6     32640298
2111.90/530.06	c ................................ | ****......            3421000 | 800957       339018           2112.1      1619.7     34318388
2208.14/554.12	c ................................ | ****......            3519000 | 825054       347422           2208.2      1593.6     35080074
2303.92/578.18	c ................................ | ****......            3608000 | 848918       354490           2304.2      1565.8     35839621
2400.15/602.28	c ................................ | *****.....            3730000 | 880299       357887           2400.5      1553.8     36853011
2496.30/626.30	c ................................ | *****.....            3951000 | 930438       371640           2496.3      1582.7     38087234
2592.16/650.37	c ................................ | *****.....            4150000 | 975863       383514           2592.4      1600.8     38981035
2688.77/674.58	c ................................ | *****.....            4275000 | 1005949      392649           2689.1      1589.8     39317565
2785.00/698.65	c ................................ | *****.....            4383000 | 1030498      402314           2785.2      1573.7     39832590
2879.98/722.40	c ................................ | ******....            4512000 | 1060401      415135           2880.0      1566.7     40518648
2976.18/746.52	c ................................ | ******....            4698000 | 1104451      435084           2976.3      1578.5     42802685
3072.42/770.61	c ................................ | ******....            4811000 | 1131859      443319           3072.5      1565.8     44086310
3167.85/794.58	c ................................ | ******....            4894000 | 1151306      451802           3168.2      1544.7     45160220
3264.40/818.75	c ................................ | ******....            4985000 | 1172161      459774           3264.6      1527.0     45160220
3360.65/842.84	c ................................ | *******...            5045000 | 1186766      467617           3360.8      1501.1     45160220
3456.49/866.88	c ................................ | *******...            5131000 | 1209585      474512           3456.8      1484.3     45160220
3551.89/890.75	c ................................ | *******...            5224000 | 1233847      480126           3552.1      1470.7     46175216
3647.73/914.79	c ................................ | *******...            5382000 | 1270683      495434           3648.1      1475.3     46175216
3744.29/938.95	c ................................ | *******...            5475000 | 1293406      503757           3744.5      1462.1     46175216
3840.18/962.91	c ................................ | ********..            5557000 | 1311753      512142           3840.2      1447.0     46175216
3935.99/986.92	c ................................ | ********..            5649000 | 1330611      524917           3936.1      1435.2     46454048
4032.26/1011.06	c ................................ | ********..            5823000 | 1365493      541180           4032.5      1444.0     47874915
4128.51/1035.12	c ................................ | ********..            5902000 | 1382166      553493           4128.6      1429.6     49444553
4226.30/1059.67	c ................................ | ********..            5970000 | 1397058      562338           4226.6      1412.5     50151142
4320.97/1083.30	c ................................ | *********.            6055000 | 1417095      569046           4321.0      1401.3     50199937
4416.43/1107.22	c ................................ | *********.            6169000 | 1444733      578264           4416.5      1396.8     51290228
4511.85/1131.14	c ................................ | *********.            6268000 | 1467855      588902           4512.0      1389.2     52247043
4608.02/1155.22	c ................................ | *********.            6474000 | 1514911      601331           4608.1      1404.9     53879028
4704.30/1179.39	c ................................ | *********.            6565000 | 1536884      606755           4704.6      1395.4     54490813

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3273404-1304056916/watcher-3273404-1304056916 -o /tmp/evaluation-result-3273404-1304056916/solver-3273404-1304056916 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273404-1304056916.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.38 7.69 7.83 4/176 22219
/proc/meminfo: memFree=22615808/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=8840 CPUtime=0 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) R 22217 22219 22182 0 -1 4202496 312 0 0 0 0 0 0 0 23 0 1 0 517288692 9052160 254 33554432000 4194304 4338100 140736875573392 18446744073709551615 4290784 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/22219/statm: 2210 254 163 36 0 115 0

[startup+0.106461 s]
/proc/loadavg: 6.38 7.69 7.83 4/176 22219
/proc/meminfo: memFree=22615808/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=8840 CPUtime=0.1 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) R 22217 22219 22182 0 -1 4202496 312 0 0 0 10 0 0 0 24 0 1 0 517288692 9052160 254 33554432000 4194304 4338100 140736875573392 18446744073709551615 239476350860 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/22219/statm: 2210 254 163 36 0 115 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 8840

[startup+0.200444 s]
/proc/loadavg: 6.38 7.69 7.83 4/176 22219
/proc/meminfo: memFree=22615808/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=8840 CPUtime=0.19 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) R 22217 22219 22182 0 -1 4202496 323 0 0 0 19 0 0 0 24 0 1 0 517288692 9052160 265 33554432000 4194304 4338100 140736875573392 18446744073709551615 4245893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/22219/statm: 2210 265 173 36 0 115 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 8840

[startup+0.300426 s]
/proc/loadavg: 6.38 7.69 7.83 4/176 22219
/proc/meminfo: memFree=22615808/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=8840 CPUtime=0.29 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) R 22217 22219 22182 0 -1 4202496 323 0 0 0 29 0 0 0 25 0 1 0 517288692 9052160 265 33554432000 4194304 4338100 140736875573392 18446744073709551615 4246508 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/22219/statm: 2210 265 173 36 0 115 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 8840

[startup+0.700349 s]
/proc/loadavg: 6.38 7.69 7.83 4/176 22219
/proc/meminfo: memFree=22615808/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=8840 CPUtime=0.69 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) R 22217 22219 22182 0 -1 4202496 323 0 0 0 69 0 0 0 25 0 1 0 517288692 9052160 265 33554432000 4194304 4338100 140736875573392 18446744073709551615 239476350949 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/22219/statm: 2210 265 173 36 0 115 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 8840

[startup+1.50013 s]
/proc/loadavg: 6.38 7.69 7.83 6/182 22225
/proc/meminfo: memFree=22534080/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=195376 CPUtime=1.52 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 1076 0 0 0 152 0 0 0 25 0 5 0 517288692 200065024 1018 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 48844 1018 196 36 0 46749 0
Current children cumulated CPU time (s) 1.52
Current children cumulated vsize (KiB) 195376

[startup+3.10082 s]
/proc/loadavg: 6.38 7.69 7.83 9/186 22229
/proc/meminfo: memFree=22436212/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=230172 CPUtime=7.91 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 11479 0 0 0 790 1 0 0 25 0 5 0 517288692 235696128 11421 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 57543 11422 196 36 0 55448 0
[pid=22219/tid=22226] ppid=22217 vsize=230172 CPUtime=1.59 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 3878 0 0 0 159 0 0 0 25 0 5 0 517288841 235696128 11422 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=230172 CPUtime=1.6 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 1391 0 0 0 160 0 0 0 25 0 5 0 517288841 235696128 11422 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=230172 CPUtime=1.6 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 3983 0 0 0 160 0 0 0 20 0 5 0 517288841 235696128 11422 33554432000 4194304 4338100 140736875573392 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=230172 CPUtime=1.6 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 1462 0 0 0 160 0 0 0 20 0 5 0 517288841 235696128 11422 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 7.91
Current children cumulated vsize (KiB) 230172

[startup+6.30121 s]
/proc/loadavg: 6.51 7.70 7.83 9/186 22229
/proc/meminfo: memFree=22201716/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=286604 CPUtime=20.69 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 24887 0 0 0 2062 7 0 0 25 0 5 0 517288692 293482496 24829 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 71651 24829 196 36 0 69556 0
[pid=22219/tid=22226] ppid=22217 vsize=286604 CPUtime=4.77 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 8440 0 0 0 476 1 0 0 25 0 5 0 517288841 293482496 24829 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=286604 CPUtime=4.8 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 2452 0 0 0 479 1 0 0 25 0 5 0 517288841 293482496 24829 33554432000 4194304 4338100 140736875573392 18446744073709551615 239476800031 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=286604 CPUtime=4.8 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 10654 0 0 0 478 2 0 0 25 0 5 0 517288841 293482496 24829 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=286604 CPUtime=4.8 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 2575 0 0 0 479 1 0 0 25 0 5 0 517288841 293482496 24829 33554432000 4194304 4338100 140736875573392 18446744073709551615 239476800031 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 20.69
Current children cumulated vsize (KiB) 286604

[startup+12.701 s]
/proc/loadavg: 6.63 7.70 7.83 9/186 22229
/proc/meminfo: memFree=21815536/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=363760 CPUtime=46.25 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 45291 0 0 0 4609 16 0 0 25 0 5 0 517288692 372490240 45225 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 90940 45225 198 36 0 88845 0
[pid=22219/tid=22226] ppid=22217 vsize=363760 CPUtime=11.14 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 17512 0 0 0 1111 3 0 0 25 0 5 0 517288841 372490240 45225 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=363760 CPUtime=11.2 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 5228 0 0 0 1116 4 0 0 25 0 5 0 517288841 372490240 45225 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=363760 CPUtime=11.2 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 16248 0 0 0 1117 3 0 0 25 0 5 0 517288841 372490240 45225 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=363760 CPUtime=11.2 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 5537 0 0 0 1116 4 0 0 25 0 5 0 517288841 372490240 45225 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488520885 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 46.25
Current children cumulated vsize (KiB) 363760

[startup+25.5005 s]
/proc/loadavg: 6.93 7.72 7.83 9/186 22230
/proc/meminfo: memFree=21233652/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=436416 CPUtime=97.36 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 62105 0 0 0 9708 28 0 0 25 0 5 0 517288692 446889984 62031 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 109104 62031 198 36 0 107009 0
[pid=22219/tid=22226] ppid=22217 vsize=436416 CPUtime=23.87 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 26232 0 0 0 2381 6 0 0 25 0 5 0 517288841 446889984 62031 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=436416 CPUtime=23.99 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 7026 0 0 0 2391 8 0 0 25 0 5 0 517288841 446889984 62031 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488520885 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=436416 CPUtime=23.98 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 20773 0 0 0 2394 4 0 0 25 0 5 0 517288841 446889984 62031 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=436416 CPUtime=23.99 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 7308 0 0 0 2391 8 0 0 25 0 5 0 517288841 446889984 62031 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 97.36
Current children cumulated vsize (KiB) 436416

[startup+51.1006 s]
/proc/loadavg: 7.30 7.74 7.83 9/186 22230
/proc/meminfo: memFree=20202328/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=558664 CPUtime=199.61 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 93309 0 0 0 19918 43 0 0 25 0 5 0 517288692 572071936 93235 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 139666 93235 198 36 0 137571 0
[pid=22219/tid=22226] ppid=22217 vsize=558664 CPUtime=49.32 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 35595 0 0 0 4923 9 0 0 25 0 5 0 517288841 572071936 93235 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=558664 CPUtime=49.59 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 9657 0 0 0 4948 11 0 0 25 0 5 0 517288841 572071936 93235 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=558664 CPUtime=49.59 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 38243 0 0 0 4949 10 0 0 25 0 5 0 517288841 572071936 93235 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=558664 CPUtime=49.59 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 9048 0 0 0 4947 12 0 0 25 0 5 0 517288841 572071936 93235 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 199.61
Current children cumulated vsize (KiB) 558664

[startup+102.301 s]
/proc/loadavg: 7.77 7.79 7.84 11/186 22232
/proc/meminfo: memFree=19029532/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=695420 CPUtime=404.1 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 126494 0 0 0 40340 70 0 0 25 0 5 0 517288692 712110080 126420 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 173855 126420 198 36 0 171760 0
[pid=22219/tid=22226] ppid=22217 vsize=695420 CPUtime=100.26 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 47413 0 0 0 10011 15 0 0 25 0 5 0 517288841 712110080 126420 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=695420 CPUtime=100.78 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 11586 0 0 0 10061 17 0 0 25 0 5 0 517288841 712110080 126420 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488520885 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=695420 CPUtime=100.76 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 54804 0 0 0 10059 17 0 0 25 0 5 0 517288841 712110080 126420 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=695420 CPUtime=100.78 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 11925 0 0 0 10058 20 0 0 25 0 5 0 517288841 712110080 126420 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 404.1
Current children cumulated vsize (KiB) 695420

[startup+162.3 s]
/proc/loadavg: 7.93 7.83 7.85 9/186 22267
/proc/meminfo: memFree=21769084/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=815812 CPUtime=643.53 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 153701 0 0 0 64251 102 0 0 25 0 5 0 517288692 835391488 153627 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 203953 153627 198 36 0 201858 0
[pid=22219/tid=22226] ppid=22217 vsize=815812 CPUtime=160.05 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 67646 0 0 0 15980 25 0 0 25 0 5 0 517288841 835391488 153627 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=815812 CPUtime=160.74 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 11817 0 0 0 16048 26 0 0 25 0 5 0 517288841 835391488 153627 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=815812 CPUtime=160.71 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 59808 0 0 0 16052 19 0 0 25 0 5 0 517288841 835391488 153627 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=815812 CPUtime=160.52 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 13664 0 0 0 16021 31 0 0 25 0 5 0 517288841 835391488 153627 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 643.53
Current children cumulated vsize (KiB) 815812

[startup+222.307 s]
/proc/loadavg: 8.10 7.89 7.86 9/186 22269
/proc/meminfo: memFree=21299016/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=900524 CPUtime=883.2 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 174953 0 0 0 88195 125 0 0 25 0 5 0 517288692 922136576 174879 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 225131 174879 198 36 0 223036 0
[pid=22219/tid=22226] ppid=22217 vsize=900524 CPUtime=220.04 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 77458 0 0 0 21975 29 0 0 25 0 5 0 517288841 922136576 174879 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0

################
# More data... #
################

[pid=22219/tid=22227] ppid=22217 vsize=1598160 CPUtime=700.17 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 20451 0 0 0 69934 83 0 0 25 0 5 0 517288841 1636515840 341142 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1598160 CPUtime=700.36 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 162975 0 0 0 69975 61 0 0 25 0 5 0 517288841 1636515840 341142 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1598160 CPUtime=698.17 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 21224 0 0 0 69725 92 0 0 25 0 5 0 517288841 1636515840 341142 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2799.75
Current children cumulated vsize (KiB) 1598160

[startup+762.3 s]
/proc/loadavg: 8.11 8.07 7.95 9/186 22283
/proc/meminfo: memFree=19290332/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=1717280 CPUtime=3039.32 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 371082 0 0 0 303605 327 0 0 25 0 5 0 517288692 1758494720 371008 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 429320 371008 198 36 0 427225 0
[pid=22219/tid=22226] ppid=22217 vsize=1717280 CPUtime=759.49 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 149284 0 0 0 75885 64 0 0 25 0 5 0 517288841 1758494720 371008 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=1717280 CPUtime=759.81 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 20482 0 0 0 75888 93 0 0 25 0 5 0 517288841 1758494720 371008 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1717280 CPUtime=760.35 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 179288 0 0 0 75968 67 0 0 25 0 5 0 517288841 1758494720 371008 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1717280 CPUtime=758.13 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 21262 0 0 0 75713 100 0 0 25 0 5 0 517288841 1758494720 371008 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3039.32
Current children cumulated vsize (KiB) 1717280

[startup+822.301 s]
/proc/loadavg: 8.18 8.10 7.97 9/186 22285
/proc/meminfo: memFree=19192732/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=1761200 CPUtime=3278.79 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 383378 0 0 0 327535 344 0 0 25 0 5 0 517288692 1803468800 383304 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 440300 383304 198 36 0 438205 0
[pid=22219/tid=22226] ppid=22217 vsize=1761200 CPUtime=819.46 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 151335 0 0 0 81880 66 0 0 25 0 5 0 517288841 1803468800 383304 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=1761200 CPUtime=819.41 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 21386 0 0 0 81842 99 0 0 25 0 5 0 517288841 1803468800 383304 33554432000 4194304 4338100 140736875573392 18446744073709551615 4288188 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1761200 CPUtime=820.34 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 188218 0 0 0 81963 71 0 0 25 0 5 0 517288841 1803468800 383304 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1761200 CPUtime=818.06 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 21673 0 0 0 81700 106 0 0 25 0 5 0 517288841 1803468800 383304 33554432000 4194304 4338100 140736875573392 18446744073709551615 4240519 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3278.79
Current children cumulated vsize (KiB) 1761200

[startup+882.307 s]
/proc/loadavg: 8.18 8.11 7.98 9/186 22286
/proc/meminfo: memFree=19111868/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=1783224 CPUtime=3518.39 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 389019 0 0 0 351479 360 0 0 25 0 5 0 517288692 1826021376 388945 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 445806 388945 198 36 0 443711 0
[pid=22219/tid=22226] ppid=22217 vsize=1783224 CPUtime=879.44 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 154205 0 0 0 87877 67 0 0 25 0 5 0 517288841 1826021376 388945 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=1783224 CPUtime=879.06 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 21469 0 0 0 87800 106 0 0 25 0 5 0 517288841 1826021376 388945 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1783224 CPUtime=880.34 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 190572 0 0 0 87962 72 0 0 25 0 5 0 517288841 1826021376 388945 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1783224 CPUtime=878.04 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 22007 0 0 0 87691 113 0 0 25 0 5 0 517288841 1826021376 388945 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3518.39
Current children cumulated vsize (KiB) 1783224

[startup+942.301 s]
/proc/loadavg: 8.06 8.08 7.98 9/186 22288
/proc/meminfo: memFree=18971488/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=1843936 CPUtime=3757.88 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 397959 0 0 0 375412 376 0 0 25 0 5 0 517288692 1888190464 397885 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 460984 397885 198 36 0 458889 0
[pid=22219/tid=22226] ppid=22217 vsize=1843936 CPUtime=939.29 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 157445 0 0 0 93860 69 0 0 25 0 5 0 517288841 1888190464 397885 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=1843936 CPUtime=938.81 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 24691 0 0 0 93768 113 0 0 25 0 5 0 517288841 1888190464 397885 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488520885 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1843936 CPUtime=940.27 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 192486 0 0 0 93953 74 0 0 25 0 5 0 517288841 1888190464 397885 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1843936 CPUtime=938 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 22571 0 0 0 93681 119 0 0 25 0 5 0 517288841 1888190464 397885 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3757.88
Current children cumulated vsize (KiB) 1843936

[startup+1002.3 s]
/proc/loadavg: 8.17 8.10 7.99 9/186 22290
/proc/meminfo: memFree=18812880/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=1872216 CPUtime=3997.51 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 404900 0 0 0 399357 394 0 0 25 0 5 0 517288692 1917149184 404826 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 468054 404826 198 36 0 465959 0
[pid=22219/tid=22226] ppid=22217 vsize=1872216 CPUtime=998.93 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 160723 0 0 0 99822 71 0 0 25 0 5 0 517288841 1917149184 404826 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=1872216 CPUtime=998.8 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 24718 0 0 0 99761 119 0 0 25 0 5 0 517288841 1917149184 404826 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1872216 CPUtime=1000.27 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 196096 0 0 0 99951 76 0 0 25 0 5 0 517288841 1917149184 404826 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488509072 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1872216 CPUtime=997.99 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 22597 0 0 0 99673 126 0 0 25 0 5 0 517288841 1917149184 404826 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3997.51
Current children cumulated vsize (KiB) 1872216

[startup+1062.3 s]
/proc/loadavg: 8.13 8.09 7.99 9/186 22291
/proc/meminfo: memFree=18634304/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=1969312 CPUtime=4237.08 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 429308 0 0 0 423296 412 0 0 25 0 5 0 517288692 2016575488 429234 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 492328 429234 198 36 0 490233 0
[pid=22219/tid=22226] ppid=22217 vsize=1969312 CPUtime=1058.57 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 171598 0 0 0 105781 76 0 0 25 0 5 0 517288841 2016575488 429234 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=1969312 CPUtime=1058.79 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 24735 0 0 0 105755 124 0 0 25 0 5 0 517288841 2016575488 429234 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=1969312 CPUtime=1060.23 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 209594 0 0 0 105943 80 0 0 25 0 5 0 517288841 2016575488 429234 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=1969312 CPUtime=1057.97 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 22615 0 0 0 105666 131 0 0 25 0 5 0 517288841 2016575488 429234 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4237.08
Current children cumulated vsize (KiB) 1969312

[startup+1122.3 s]
/proc/loadavg: 8.11 8.09 8.00 9/186 22293
/proc/meminfo: memFree=18515984/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=2022744 CPUtime=4476.73 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 442887 0 0 0 447244 429 0 0 25 0 5 0 517288692 2071289856 442813 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 505686 442813 198 36 0 503591 0
[pid=22219/tid=22226] ppid=22217 vsize=2022744 CPUtime=1118.23 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 181628 0 0 0 111742 81 0 0 25 0 5 0 517288841 2071289856 442813 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=2022744 CPUtime=1118.78 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 24759 0 0 0 111749 129 0 0 25 0 5 0 517288841 2071289856 442813 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=2022744 CPUtime=1120.23 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 213093 0 0 0 111941 82 0 0 25 0 5 0 517288841 2071289856 442813 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=2022744 CPUtime=1117.97 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 22641 0 0 0 111662 135 0 0 25 0 5 0 517288841 2071289856 442813 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4476.73
Current children cumulated vsize (KiB) 2022744

[startup+1182.3 s]
/proc/loadavg: 8.16 8.10 8.01 9/186 22294
/proc/meminfo: memFree=18338408/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=2107328 CPUtime=4716.26 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 464033 0 0 0 471175 451 0 0 25 0 5 0 517288692 2157903872 463959 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 526832 463959 198 36 0 524737 0
[pid=22219/tid=22226] ppid=22217 vsize=2107328 CPUtime=1177.84 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 200884 0 0 0 117695 89 0 0 25 0 5 0 517288841 2157903872 463959 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=2107328 CPUtime=1178.74 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 24783 0 0 0 117740 134 0 0 25 0 5 0 517288841 2157903872 463959 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=2107328 CPUtime=1180.21 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 214932 0 0 0 117938 83 0 0 25 0 5 0 517288841 2157903872 463959 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=2107328 CPUtime=1177.95 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 22668 0 0 0 117652 143 0 0 25 0 5 0 517288841 2157903872 463959 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4716.26
Current children cumulated vsize (KiB) 2107328



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.3 s]
/proc/loadavg: 8.11 8.10 8.01 9/186 22295
/proc/meminfo: memFree=18317572/32950928 swapFree=67087032/67111528
[pid=22219] ppid=22217 vsize=2113584 CPUtime=4800.1 cores=1,3,5,7
/proc/22219/stat : 22219 (ttsth-5-4) S 22217 22219 22182 0 -1 4202496 468883 0 0 0 479554 456 0 0 25 0 5 0 517288692 2164310016 468809 33554432000 4194304 4338100 140736875573392 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/22219/statm: 528396 468809 198 36 0 526301 0
[pid=22219/tid=22226] ppid=22217 vsize=2113584 CPUtime=1198.71 cores=1,3,5,7
/proc/22219/task/22226/stat : 22226 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 202087 0 0 0 119781 90 0 0 25 0 5 0 517288841 2164310016 468809 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=22219/tid=22227] ppid=22217 vsize=2113584 CPUtime=1199.74 cores=1,3,5,7
/proc/22219/task/22227/stat : 22227 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 25627 0 0 0 119838 136 0 0 25 0 5 0 517288841 2164310016 468809 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=22219/tid=22228] ppid=22217 vsize=2113584 CPUtime=1201.21 cores=1,3,5,7
/proc/22219/task/22228/stat : 22228 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 215826 0 0 0 120038 83 0 0 25 0 5 0 517288841 2164310016 468809 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=22219/tid=22229] ppid=22217 vsize=2113584 CPUtime=1198.92 cores=1,3,5,7
/proc/22219/task/22229/stat : 22229 (ttsth-5-4) R 22217 22219 22182 0 -1 4202560 24577 0 0 0 119747 145 0 0 25 0 5 0 517288841 2164310016 468809 33554432000 4194304 4338100 140736875573392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 2113584

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.4
CPU time (s): 4800.22
CPU user time (s): 4795.57
CPU system time (s): 4.64629
CPU usage (%): 398.889
Max. virtual memory (cumulated for all children) (KiB): 2113584

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.57
system time used= 4.64629
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 468883
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2329
involuntary context switches= 20836

runsolver used 2.03769 second user time and 4.51631 second system time

The end

Launcher Data 

Begin job on node148 at 2011-04-29 08:01:56
IDJOB=3273404
IDBENCH=83138
IDSOLVER=1654
FILE ID=node148/3273404-1304056916
RUNJOBID= node148-1304056916-22201
PBS_JOBID= 13170435
Free space on /tmp= 71436 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n160-s40283905-sat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3273404-1304056916/watcher-3273404-1304056916 -o /tmp/evaluation-result-3273404-1304056916/solver-3273404-1304056916 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273404-1304056916.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8e3c134eb643f2d9a45c884774c054a8
RANDOM SEED=861986582

node148.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      22616460 kB
Buffers:       2017660 kB
Cached:        7710040 kB
SwapCached:       3548 kB
Active:        1738340 kB
Inactive:      8032484 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      22616460 kB
SwapTotal:    67111528 kB
SwapFree:     67087032 kB
Dirty:             544 kB
Writeback:           0 kB
AnonPages:       42136 kB
Mapped:          13416 kB
Slab:           498436 kB
PageTables:       4512 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   197900 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71436 MiB
End job on node148 at 2011-04-29 08:22:02