Trace number 3259186

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.29 1205.39

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v760c43649g2.cnf
MD5SUM791cdb75c1a209efd7f7547f06e0fed9
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark1235.8
Satisfiable
(Un)Satisfiability was proved
Number of variables760
Number of clauses43649
Sum of the clauses size87978
Maximum clause length19
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243609
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259186-1304054236.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 760
0.00/0.02	c Clauses = 43649
0.00/0.02	c Literals = 87978
0.00/0.02	c a2
0.00/0.05	c a1
0.09/0.10	c Independent sub-problems = 1
0.09/0.10	c Sub-problem with 760 variables and 43649 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 1
0.09/0.10	c Ordering finished 0.090 seconds
0.09/0.10	c mapped 0.090 seconds
0.09/0.10	c permuted 0.090 seconds
0.09/0.10	c about to sort
0.09/0.10	c sorted
0.09/0.19	c clauses now 43649
0.09/0.19	c init finished 0.110 seconds
0.09/0.19	c tree built 0.180 seconds
0.09/0.19	c 
0.09/0.19	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.19	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.19	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.19	c ................................ | ..........                  1 |                                  0.2         5.6             
0.09/0.19	c ................................ | ..........                  2 |                                  0.2        11.1             
0.09/0.19	c ................................ | ..........                  3 |                                  0.2        16.7             
0.09/0.19	c ................................ | ..........                  4 |                                  0.2        22.2             
0.09/0.19	c ................................ | ..........                  5 |                                  0.2        27.8             
0.09/0.19	c ................................ | ..........               1000 | 7            182                 0.2      5263.2         9351
95.98/24.31	c ................................ | ..........            3592000 | 70553        763806             96.0     37408.9     24592747
191.75/48.46	c ................................ | ..........            6329000 | 123316       1324259           192.0     32961.8     36132685
287.89/72.63	c ................................ | ..........            8779000 | 168512       1809525           288.0     30482.6     43608514
383.76/96.77	c ................................ | ..........           10869000 | 207838       2248138           384.1     28301.0     50305809
479.77/120.86	c ................................ | *.........           12778000 | 245506       2658533           480.0     26620.3     56768706
575.83/144.94	c ................................ | *.........           14833000 | 285613       3049517           576.0     25751.7     61509837
671.66/169.09	c ................................ | *.........           16742000 | 332078       3417344           672.0     24911.8     65627919
767.71/193.18	c ................................ | *.........           18469000 | 374326       3767474           768.0     24046.9     73852044
863.90/217.24	c ................................ | *.........           20114000 | 413322       4094367           864.1     23278.5     80405288
959.67/241.28	c ................................ | **........           21781000 | 450786       4447311           960.0     22687.8     88776129
1055.90/265.32	c ................................ | **........           23357000 | 487363       4779699          1056.0     22118.2     92507308
1151.62/289.40	c ................................ | **........           24946000 | 523051       5084104          1152.0     21653.9     97779622
1247.88/313.53	c ................................ | **........           26627000 | 562796       5391415          1248.0     21335.7    102917146
1343.75/337.67	c ................................ | **........           28217000 | 597029       5692897          1344.0     20994.3    110113800
1439.65/361.79	c ................................ | ***.......           29553000 | 625090       5976384          1440.0     20522.6    110439502
1535.94/385.93	c ................................ | ***.......           30793000 | 653477       6255200          1536.1     20046.6    112220779
1631.95/410.01	c ................................ | ***.......           32022000 | 682213       6523565          1632.0     19621.2    114862300
1727.93/434.12	c ................................ | ***.......           33311000 | 713133       6776260          1728.0     19276.8    117503342
1823.88/458.26	c ................................ | ***.......           34590000 | 743657       7029289          1824.1     18962.6    119834606
1919.85/482.34	c ................................ | ****......           35850000 | 767024       7273140          1920.0     18671.7    121638483
2015.80/506.45	c ................................ | ****......           37333000 | 797777       7538144          2016.0     18518.1    126337945
2111.75/530.56	c ................................ | ****......           38709000 | 829747       7806872          2112.0     18328.1    129480468
2207.95/554.62	c ................................ | ****......           40012000 | 856100       8055803          2208.0     18121.0    131474343
2304.00/578.71	c ................................ | ****......           41233000 | 883221       8314809          2304.0     17896.0    132082618
2399.95/602.81	c ................................ | *****.....           42356000 | 909653       8562756          2400.0     17648.3    133037130
2495.99/626.91	c ................................ | *****.....           43440000 | 935470       8805746          2496.0     17403.6    136935122
2591.73/650.98	c ................................ | *****.....           44561000 | 959496       9058847          2592.1     17191.4    142077255
2687.84/675.04	c ................................ | *****.....           45702000 | 985325       9297277          2688.0     17002.2    143214178
2783.99/699.12	c ................................ | *****.....           46880000 | 1015167      9528414          2784.1     16838.7    147361231
2879.77/723.17	c ................................ | ******....           47942000 | 1043963      9761507          2880.1     16646.2    147788757
2975.87/747.24	c ................................ | ******....           49009000 | 1070518      10000981         2976.1     16467.7    150865858
3071.93/771.33	c ................................ | ******....           50124000 | 1098520      10238568         3072.1     16316.0    153526518
3167.87/795.46	c ................................ | ******....           51243000 | 1123141      10484271         3168.1     16174.7    157487153
3263.89/819.54	c ................................ | ******....           52275000 | 1144365      10705962         3264.1     16015.4    159618225
3359.80/843.65	c ................................ | *******...           53339000 | 1169478      10934483         3360.0     15874.6    166112032
3455.75/867.77	c ................................ | *******...           54511000 | 1196930      11181763         3456.0     15772.7    170840356
3551.76/891.87	c ................................ | *******...           55610000 | 1219331      11416618         3552.1     15655.7    172882220
3647.86/915.94	c ................................ | *******...           56684000 | 1245516      11648300         3648.0     15538.3    178087645
3743.89/940.03	c ................................ | *******...           57803000 | 1271507      11885829         3744.0     15438.7    185036404
3840.02/964.10	c ................................ | ********..           58893000 | 1298121      12129729         3840.0     15336.6    192156410
3935.72/988.17	c ................................ | ********..           59940000 | 1321168      12348980         3936.0     15228.6    197654760
4031.65/1012.30	c ................................ | ********..           61064000 | 1347908      12583169         4032.1     15144.7    198313143
4127.99/1036.40	c ................................ | ********..           62248000 | 1378920      12806921         4128.0     15079.5    200414193
4224.02/1060.51	c ................................ | ********..           63274000 | 1405747      13015488         4224.1     14979.5    200962697
4319.65/1084.60	c ................................ | *********.           64459000 | 1438933      13218982         4320.1     14920.9    202243206
4416.01/1108.70	c ................................ | *********.           65510000 | 1464495      13427067         4416.1     14834.5    203059611
4512.02/1132.80	c ................................ | *********.           66728000 | 1490015      13643046         4512.0     14788.9    204092876
4608.03/1156.91	c ................................ | *********.           67850000 | 1514037      13851319         4608.1     14724.1    208112389
4704.03/1181.00	c ................................ | *********.           68863000 | 1539487      14049279         4704.1     14639.1    212017232
4799.93/1205.11	c CPU limit (4800) reached
4799.93/1205.11	c Time = 4800.00 seconds
4799.93/1205.11	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259186-1304054236/watcher-3259186-1304054236 -o /tmp/evaluation-result-3259186-1304054236/solver-3259186-1304054236 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259186-1304054236.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.39 7.92 7.96 3/177 583
/proc/meminfo: memFree=27127680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=8980 CPUtime=0 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) R 581 583 330 0 -1 4202496 300 0 0 0 0 0 0 0 25 0 1 0 517022066 9195520 243 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204308165 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/583/statm: 2245 244 146 36 0 150 0

[startup+0.0796291 s]
/proc/loadavg: 7.39 7.92 7.96 3/177 583
/proc/meminfo: memFree=27127680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=18400 CPUtime=0.07 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) R 581 583 330 0 -1 4202496 3476 0 0 0 7 0 0 0 25 0 1 0 517022066 18841600 2626 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204308165 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/583/statm: 4600 2630 155 36 0 2505 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 18400

[startup+0.100618 s]
/proc/loadavg: 7.39 7.92 7.96 3/177 583
/proc/meminfo: memFree=27127680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=18532 CPUtime=0.09 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) R 581 583 330 0 -1 4202496 3527 0 0 0 9 0 0 0 25 0 1 0 517022066 18976768 2677 33554432000 4194304 4338100 140733219334016 18446744073709551615 4228256 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/583/statm: 4633 2677 161 36 0 2538 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18532

[startup+0.350479 s]
/proc/loadavg: 7.39 7.92 7.96 3/177 583
/proc/meminfo: memFree=27127680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=163260 CPUtime=0.81 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 11704 0 0 0 80 1 0 0 25 0 5 0 517022066 167178240 10681 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 41597 10944 176 36 0 39502 0
Current children cumulated CPU time (s) 0.81
Current children cumulated vsize (KiB) 163260

[startup+0.891391 s]
/proc/loadavg: 7.39 7.92 7.96 3/177 583
/proc/meminfo: memFree=27127680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=188680 CPUtime=3.08 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 18702 0 0 0 305 3 0 0 25 0 5 0 517022066 193208320 17679 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 48734 18258 176 36 0 46639 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 188680

[startup+2.00701 s]
/proc/loadavg: 7.39 7.92 7.96 3/177 583
/proc/meminfo: memFree=27127680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=304796 CPUtime=7.49 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 31249 0 0 0 743 6 0 0 25 0 5 0 517022066 312111104 30226 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 76199 30226 176 36 0 74104 0
Current children cumulated CPU time (s) 7.49
Current children cumulated vsize (KiB) 304796

[startup+3.10102 s]
/proc/loadavg: 7.39 7.92 7.96 9/186 592
/proc/meminfo: memFree=26848680/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=350308 CPUtime=11.75 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 41697 0 0 0 1166 9 0 0 25 0 5 0 517022066 358715392 40674 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 87577 40675 176 36 0 85482 0
[pid=583/tid=585] ppid=581 vsize=350308 CPUtime=2.9 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 10398 0 0 0 288 2 0 0 24 0 5 0 517022085 358715392 40675 33554432000 4194304 4338100 140733219334016 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=350308 CPUtime=2.84 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 6822 0 0 0 282 2 0 0 18 0 5 0 517022085 358715392 40675 33554432000 4194304 4338100 140733219334016 18446744073709551615 4255940 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=587] ppid=581 vsize=350308 CPUtime=2.9 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 12031 0 0 0 289 1 0 0 24 0 5 0 517022085 358715392 40675 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=588] ppid=581 vsize=350308 CPUtime=2.89 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 3395 0 0 0 287 2 0 0 18 0 5 0 517022085 358715392 40676 33554432000 4194304 4338100 140733219334016 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.75
Current children cumulated vsize (KiB) 350308

[startup+6.30049 s]
/proc/loadavg: 7.52 7.94 7.97 9/186 592
/proc/meminfo: memFree=26647348/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=469436 CPUtime=24.47 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 72765 0 0 0 2427 20 0 0 25 0 5 0 517022066 480702464 71742 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 117359 71742 176 36 0 115264 0
[pid=583/tid=585] ppid=581 vsize=469436 CPUtime=6.09 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 20521 0 0 0 604 5 0 0 25 0 5 0 517022085 480702464 71742 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=469436 CPUtime=5.97 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 12158 0 0 0 593 4 0 0 18 0 5 0 517022085 480702464 71742 33554432000 4194304 4338100 140733219334016 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=587] ppid=581 vsize=469436 CPUtime=6.1 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 22880 0 0 0 607 3 0 0 25 0 5 0 517022085 480702464 71742 33554432000 4194304 4338100 140733219334016 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=588] ppid=581 vsize=469436 CPUtime=6.09 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 8153 0 0 0 603 6 0 0 18 0 5 0 517022085 480702464 71742 33554432000 4194304 4338100 140733219334016 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.47
Current children cumulated vsize (KiB) 469436

[startup+12.7004 s]
/proc/loadavg: 7.56 7.94 7.97 9/186 592
/proc/meminfo: memFree=26311836/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=670324 CPUtime=49.93 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 122885 0 0 0 4950 43 0 0 25 0 5 0 517022066 686411776 121862 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 167581 121862 176 36 0 165486 0
[pid=583/tid=585] ppid=581 vsize=670324 CPUtime=12.48 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 37483 0 0 0 1239 9 0 0 25 0 5 0 517022085 686411776 121862 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=670324 CPUtime=12.24 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 19874 0 0 0 1217 7 0 0 18 0 5 0 517022085 686411776 121862 33554432000 4194304 4338100 140733219334016 18446744073709551615 4253920 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=587] ppid=581 vsize=670324 CPUtime=12.48 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 41683 0 0 0 1241 7 0 0 25 0 5 0 517022085 686411776 121862 33554432000 4194304 4338100 140733219334016 18446744073709551615 4263586 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=588] ppid=581 vsize=670324 CPUtime=12.5 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 14792 0 0 0 1233 17 0 0 25 0 5 0 517022085 686411776 121862 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.93
Current children cumulated vsize (KiB) 670324

[startup+25.5003 s]
/proc/loadavg: 7.66 7.94 7.97 9/186 593
/proc/meminfo: memFree=25719000/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=1033372 CPUtime=100.74 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 212712 0 0 0 9987 87 0 0 25 0 5 0 517022066 1058172928 211689 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 258343 211690 176 36 0 256248 0
[pid=583/tid=585] ppid=581 vsize=1033372 CPUtime=25.2 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 67701 0 0 0 2504 16 0 0 25 0 5 0 517022085 1058172928 211690 33554432000 4194304 4338100 140733219334016 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=586] ppid=581 vsize=1033372 CPUtime=24.78 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 34668 0 0 0 2462 16 0 0 18 0 5 0 517022085 1058172928 211690 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=587] ppid=581 vsize=1033372 CPUtime=25.26 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 73342 0 0 0 2512 14 0 0 25 0 5 0 517022085 1058172928 211690 33554432000 4194304 4338100 140733219334016 18446744073709551615 4241440 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=588] ppid=581 vsize=1033372 CPUtime=25.28 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 27949 0 0 0 2490 38 0 0 25 0 5 0 517022085 1058172928 211690 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 100.74
Current children cumulated vsize (KiB) 1033372

[startup+51.101 s]
/proc/loadavg: 7.83 7.96 7.97 9/186 593
/proc/meminfo: memFree=25008504/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=1420964 CPUtime=202.48 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 307605 0 0 0 20094 154 0 0 25 0 5 0 517022066 1455067136 306582 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 355241 306582 176 36 0 353146 0
[pid=583/tid=585] ppid=581 vsize=1420964 CPUtime=50.63 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 106923 0 0 0 5034 29 0 0 25 0 5 0 517022085 1455067136 306582 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204343808 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=586] ppid=581 vsize=1420964 CPUtime=49.95 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 45573 0 0 0 4970 25 0 0 18 0 5 0 517022085 1455067136 306582 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=587] ppid=581 vsize=1420964 CPUtime=50.83 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 110788 0 0 0 5062 21 0 0 25 0 5 0 517022085 1455067136 306582 33554432000 4194304 4338100 140733219334016 18446744073709551615 4256007 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=588] ppid=581 vsize=1420964 CPUtime=50.84 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 35268 0 0 0 5007 77 0 0 25 0 5 0 517022085 1455067136 306582 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.48
Current children cumulated vsize (KiB) 1420964

[startup+102.3 s]
/proc/loadavg: 8.04 8.00 7.98 9/186 605
/proc/meminfo: memFree=24145588/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=1936432 CPUtime=406.04 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 433726 0 0 0 40335 269 0 0 25 0 5 0 517022066 1982906368 432703 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 484108 432703 176 36 0 482013 0
[pid=583/tid=585] ppid=581 vsize=1936432 CPUtime=101.44 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 169084 0 0 0 10099 45 0 0 25 0 5 0 517022085 1982906368 432707 33554432000 4194304 4338100 140733219334016 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=586] ppid=581 vsize=1936432 CPUtime=100.42 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 51965 0 0 0 10001 41 0 0 18 0 5 0 517022085 1982906368 432707 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=587] ppid=581 vsize=1936432 CPUtime=101.96 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 162305 0 0 0 10164 32 0 0 25 0 5 0 517022085 1982906368 432707 33554432000 4194304 4338100 140733219334016 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=588] ppid=581 vsize=1936432 CPUtime=102.01 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 41323 0 0 0 10052 149 0 0 25 0 5 0 517022085 1982906368 432707 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.04
Current children cumulated vsize (KiB) 1936432

[startup+162.3 s]
/proc/loadavg: 8.12 8.03 7.99 9/186 606
/proc/meminfo: memFree=23491852/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=2371856 CPUtime=645.01 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 542979 0 0 0 63956 545 0 0 25 0 5 0 517022066 2428780544 541956 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 592964 541956 176 36 0 590869 0
[pid=583/tid=585] ppid=581 vsize=2371856 CPUtime=161.27 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 223703 0 0 0 16062 65 0 0 25 0 5 0 517022085 2428780544 541956 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=586] ppid=581 vsize=2371856 CPUtime=159.98 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 58077 0 0 0 15852 146 0 0 18 0 5 0 517022085 2428780544 541956 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=587] ppid=581 vsize=2371856 CPUtime=161.78 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 203611 0 0 0 16138 40 0 0 25 0 5 0 517022085 2428780544 541956 33554432000 4194304 4338100 140733219334016 18446744073709551615 4229577 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=2371856 CPUtime=161.77 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 48535 0 0 0 15884 293 0 0 25 0 5 0 517022085 2428780544 541956 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 645.01
Current children cumulated vsize (KiB) 2371856

[startup+222.306 s]
/proc/loadavg: 8.04 8.02 7.99 9/186 608
/proc/meminfo: memFree=22607460/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=2902864 CPUtime=884.29 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 675968 0 0 0 87588 841 0 0 25 0 5 0 517022066 2972532736 674945 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 725716 674945 176 36 0 723621 0
[pid=583/tid=585] ppid=581 vsize=2902864 CPUtime=221.2 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 278546 0 0 0 22040 80 0 0 25 0 5 0 517022085 2972532736 674945 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 94478 0 0 0 68696 1277 0 0 25 0 5 0 517022085 5152079872 1202884 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2796.76
Current children cumulated vsize (KiB) 5031328

[startup+762.3 s]
/proc/loadavg: 8.15 8.10 8.02 9/186 622
/proc/meminfo: memFree=18624356/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=5214072 CPUtime=3036.12 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1250186 0 0 0 300829 2783 0 0 25 0 5 0 517022066 5339209728 1249163 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1303518 1249163 176 36 0 1301423 0
[pid=583/tid=585] ppid=581 vsize=5214072 CPUtime=760.11 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 553214 0 0 0 75833 178 0 0 25 0 5 0 517022085 5339209728 1249168 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=5214072 CPUtime=756.51 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 99951 0 0 0 74586 1065 0 0 18 0 5 0 517022085 5339209728 1249168 33554432000 4194304 4338100 140733219334016 18446744073709551615 4285989 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=5214072 CPUtime=759.7 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 493296 0 0 0 75864 106 0 0 25 0 5 0 517022085 5339209728 1249168 33554432000 4194304 4338100 140733219334016 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=5214072 CPUtime=759.58 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 94677 0 0 0 74526 1432 0 0 25 0 5 0 517022085 5339209728 1249168 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3036.12
Current children cumulated vsize (KiB) 5214072

[startup+822.3 s]
/proc/loadavg: 8.13 8.09 8.02 9/186 624
/proc/meminfo: memFree=18161704/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=5424968 CPUtime=3275.03 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1303795 0 0 0 324642 2861 0 0 25 0 5 0 517022066 5555167232 1302772 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1356242 1302772 176 36 0 1354147 0
[pid=583/tid=585] ppid=581 vsize=5424968 CPUtime=820.05 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 582890 0 0 0 81818 187 0 0 25 0 5 0 517022085 5555167232 1302772 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204042483 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=5424968 CPUtime=815.92 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 100140 0 0 0 80520 1072 0 0 18 0 5 0 517022085 5555167232 1302772 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=5424968 CPUtime=819.33 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 512006 0 0 0 81823 110 0 0 25 0 5 0 517022085 5555167232 1302772 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=5424968 CPUtime=819.51 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 99706 0 0 0 80461 1490 0 0 25 0 5 0 517022085 5555167232 1302772 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3275.03
Current children cumulated vsize (KiB) 5424968

[startup+882.306 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 625
/proc/meminfo: memFree=17506820/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=5823544 CPUtime=3513.97 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1403547 0 0 0 348445 2952 0 0 25 0 5 0 517022066 5963309056 1402524 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1455886 1402524 176 36 0 1453791 0
[pid=583/tid=585] ppid=581 vsize=5823544 CPUtime=879.97 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 618653 0 0 0 87800 197 0 0 25 0 5 0 517022085 5963309056 1402524 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204645435 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=5823544 CPUtime=875.38 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 112676 0 0 0 86455 1083 0 0 18 0 5 0 517022085 5963309056 1402524 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=5823544 CPUtime=878.96 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 548706 0 0 0 87778 118 0 0 25 0 5 0 517022085 5963309056 1402524 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=5823544 CPUtime=879.45 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 114459 0 0 0 86393 1552 0 0 25 0 5 0 517022085 5963309056 1402524 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3513.97
Current children cumulated vsize (KiB) 5823544

[startup+942.3 s]
/proc/loadavg: 8.02 8.06 8.01 9/186 627
/proc/meminfo: memFree=16893120/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=6240888 CPUtime=3753.08 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1508211 0 0 0 372103 3205 0 0 25 0 5 0 517022066 6390669312 1507188 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1560222 1507188 176 36 0 1558127 0
[pid=583/tid=585] ppid=581 vsize=6240888 CPUtime=939.81 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 657146 0 0 0 93773 208 0 0 25 0 5 0 517022085 6390669312 1507188 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=6240888 CPUtime=935.1 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 127701 0 0 0 92323 1187 0 0 25 0 5 0 517022085 6390669312 1507189 33554432000 4194304 4338100 140733219334016 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=6240888 CPUtime=938.81 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 592874 0 0 0 93752 129 0 0 25 0 5 0 517022085 6390669312 1507189 33554432000 4194304 4338100 140733219334016 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=583/tid=588] ppid=581 vsize=6240888 CPUtime=939.14 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 121438 0 0 0 92235 1679 0 0 25 0 5 0 517022085 6390669312 1507189 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204042686 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3753.08
Current children cumulated vsize (KiB) 6240888

[startup+1002.3 s]
/proc/loadavg: 8.07 8.06 8.01 9/186 629
/proc/meminfo: memFree=16194360/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=6633208 CPUtime=3992.19 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1606925 0 0 0 395771 3448 0 0 25 0 5 0 517022066 6792404992 1605902 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1658302 1605902 176 36 0 1656207 0
[pid=583/tid=585] ppid=581 vsize=6633208 CPUtime=999.67 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 688114 0 0 0 99749 218 0 0 25 0 5 0 517022085 6792404992 1605902 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=6633208 CPUtime=994.82 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 141830 0 0 0 98194 1288 0 0 18 0 5 0 517022085 6792404992 1605902 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=6633208 CPUtime=998.61 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 632587 0 0 0 99723 138 0 0 25 0 5 0 517022085 6792404992 1605902 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=6633208 CPUtime=998.85 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 135341 0 0 0 98084 1801 0 0 25 0 5 0 517022085 6792404992 1605902 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3992.19
Current children cumulated vsize (KiB) 6633208

[startup+1062.3 s]
/proc/loadavg: 8.12 8.08 8.01 9/186 630
/proc/meminfo: memFree=15986760/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=6725240 CPUtime=4231.14 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1633911 0 0 0 419599 3515 0 0 25 0 5 0 517022066 6886645760 1632888 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1681310 1632888 176 36 0 1679215 0
[pid=583/tid=585] ppid=581 vsize=6725240 CPUtime=1059.59 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 701104 0 0 0 105734 225 0 0 25 0 5 0 517022085 6886645760 1632888 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=6725240 CPUtime=1054.3 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 145552 0 0 0 104135 1295 0 0 18 0 5 0 517022085 6886645760 1632888 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=6725240 CPUtime=1058.26 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 642689 0 0 0 105685 141 0 0 25 0 5 0 517022085 6886645760 1632888 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=6725240 CPUtime=1058.77 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 135513 0 0 0 104024 1853 0 0 25 0 5 0 517022085 6886645760 1632888 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4231.14
Current children cumulated vsize (KiB) 6725240

[startup+1122.3 s]
/proc/loadavg: 7.95 8.04 8.00 9/186 632
/proc/meminfo: memFree=15845872/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=6826952 CPUtime=4470.2 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1655037 0 0 0 443441 3579 0 0 25 0 5 0 517022066 6990798848 1654014 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1706738 1654014 176 36 0 1704643 0
[pid=583/tid=585] ppid=581 vsize=6826952 CPUtime=1119.53 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 714239 0 0 0 111722 231 0 0 25 0 5 0 517022085 6990798848 1654014 33554432000 4194304 4338100 140733219334016 18446744073709551615 210204645435 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=586] ppid=581 vsize=6826952 CPUtime=1113.84 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 149821 0 0 0 110083 1301 0 0 18 0 5 0 517022085 6990798848 1654014 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=6826952 CPUtime=1117.87 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 644872 0 0 0 111646 141 0 0 25 0 5 0 517022085 6990798848 1654014 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=588] ppid=581 vsize=6826952 CPUtime=1118.74 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 137052 0 0 0 109971 1903 0 0 25 0 5 0 517022085 6990798848 1654014 33554432000 4194304 4338100 140733219334016 18446744073709551615 4285683 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4470.2
Current children cumulated vsize (KiB) 6826952

[startup+1182.3 s]
/proc/loadavg: 8.09 8.06 8.01 9/186 633
/proc/meminfo: memFree=15484380/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=7115916 CPUtime=4709.21 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) S 581 583 330 0 -1 4202496 1728027 0 0 0 467262 3659 0 0 25 0 5 0 517022066 7286697984 1727004 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 1778979 1727004 176 36 0 1776884 0
[pid=583/tid=585] ppid=581 vsize=7115916 CPUtime=1179.23 cores=0,2,4,6
/proc/583/task/585/stat : 585 (ttsth-5-4) R 581 583 330 0 -1 4202560 748344 0 0 0 117682 241 0 0 25 0 5 0 517022085 7286697984 1727004 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=583/tid=586] ppid=581 vsize=7115916 CPUtime=1173.36 cores=0,2,4,6
/proc/583/task/586/stat : 586 (ttsth-5-4) R 581 583 330 0 -1 4202560 149979 0 0 0 116028 1308 0 0 18 0 5 0 517022085 7286697984 1727004 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=583/tid=587] ppid=581 vsize=7115916 CPUtime=1177.72 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4202560 683439 0 0 0 117622 150 0 0 25 0 5 0 517022085 7286697984 1727004 33554432000 4194304 4338100 140733219334016 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=583/tid=588] ppid=581 vsize=7115916 CPUtime=1178.68 cores=0,2,4,6
/proc/583/task/588/stat : 588 (ttsth-5-4) R 581 583 330 0 -1 4202560 137212 0 0 0 115910 1958 0 0 25 0 5 0 517022085 7286697984 1727004 33554432000 4194304 4338100 140733219334016 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4709.21
Current children cumulated vsize (KiB) 7115916



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.2 s]
/proc/loadavg: 7.74 7.99 7.98 2/173 652
/proc/meminfo: memFree=21994868/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=0 CPUtime=4800.09 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) Z 581 583 330 0 -1 4203548 1747857 0 0 0 476311 3698 0 0 18 0 2 0 517022066 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 0 0 0 0 0 0 0
[pid=583/tid=587] ppid=581 vsize=0 CPUtime=1200.59 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4203588 692717 0 0 0 119899 160 0 0 25 0 2 0 517022085 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.31 s]
/proc/loadavg: 7.74 7.99 7.98 2/173 652
/proc/meminfo: memFree=21994868/32951124 swapFree=67088964/67111528
[pid=583] ppid=581 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/583/stat : 583 (ttsth-5-4) Z 581 583 330 0 -1 4203548 1747857 0 0 0 476311 3709 0 0 18 0 2 0 517022066 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/583/statm: 0 0 0 0 0 0 0
[pid=583/tid=587] ppid=581 vsize=0 CPUtime=1200.7 cores=0,2,4,6
/proc/583/task/587/stat : 587 (ttsth-5-4) R 581 583 330 0 -1 4203588 692717 0 0 0 119899 171 0 0 25 0 2 0 517022085 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.39
CPU time (s): 4800.29
CPU user time (s): 4763.12
CPU system time (s): 37.1703
CPU usage (%): 398.235
Max. virtual memory (cumulated for all children) (KiB): 7197640

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4763.12
system time used= 37.1703
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1747857
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 19584
involuntary context switches= 20703

runsolver used 1.9837 second user time and 4.53631 second system time

The end

Launcher Data 

Begin job on node130 at 2011-04-29 07:17:16
IDJOB=3259186
IDBENCH=71514
IDSOLVER=1654
FILE ID=node130/3259186-1304054236
RUNJOBID= node130-1304051825-348
PBS_JOBID= 13170415
Free space on /tmp= 71992 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v760c43649g2.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259186-1304054236/watcher-3259186-1304054236 -o /tmp/evaluation-result-3259186-1304054236/solver-3259186-1304054236 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259186-1304054236.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 791cdb75c1a209efd7f7547f06e0fed9
RANDOM SEED=1928952380

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27127960 kB
Buffers:       1156364 kB
Cached:        4096488 kB
SwapCached:      22432 kB
Active:         410820 kB
Inactive:      4980884 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27127960 kB
SwapTotal:    67111528 kB
SwapFree:     67088964 kB
Dirty:            4380 kB
Writeback:          36 kB
AnonPages:      116060 kB
Mapped:          15228 kB
Slab:           368620 kB
PageTables:       4136 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   259580 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71992 MiB
End job on node130 at 2011-04-29 07:37:23