Trace number 3250868

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.37 1209.76

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1327.5-n60-p30-s52708253.cnf
MD5SUM472b10ce0a2683d9355981037eea091e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark7.17391
Satisfiable
(Un)Satisfiability was proved
Number of variables3672
Number of clauses1163952
Sum of the clauses size2331456
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21163892
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data 

0.79/0.80	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250868-1304105581.cnf 
0.79/0.80	c Using SATTIMEOUT of 4800
0.79/0.80	c Using SATRAM of 15500
0.79/0.80	c Size of subset node = 32
0.79/0.80	c Variables = 3672
0.79/0.80	c Clauses = 1163952
0.79/0.80	c Literals = 2331456
0.79/0.80	c Reduced size:
0.79/0.80	c Variables = 3672
0.79/0.80	c Clauses = 1163952
0.79/0.80	c Literals = 2331456
0.79/0.80	c a2
1.39/1.43	c a1
4.19/4.21	c Independent sub-problems = 1
4.19/4.21	c Sub-problem with 3672 variables and 1163952 clauses
4.19/4.21	c using permutation method 4
4.19/4.21	c count = 1
4.19/4.21	c Ordering finished 4.070 seconds
4.19/4.21	c mapped 4.200 seconds
4.19/4.21	c permuted 4.210 seconds
4.19/4.21	c about to sort
4.29/4.31	c sorted
10.10/10.12	c clauses now 1163952
10.10/10.12	c init finished 4.690 seconds
10.10/10.12	c tree built 10.120 seconds
10.10/10.12	c 
10.10/10.12	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
10.10/10.12	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
10.10/10.12	c ................................ | ..........                    |                                 10.1         0.0             
10.10/10.12	c ................................ | ..........                  1 |                                 10.1         0.1             
10.10/10.12	c ................................ | ..........                  2 |                                 10.1         0.2             
10.10/10.12	c ................................ | ..........                  3 |                                 10.1         0.3             
10.10/10.12	c ................................ | ..........                  4 |                                 10.1         0.4             
10.10/10.12	c ................................ | ..........                  5 |                                 10.1         0.5             
10.10/10.13	c ................................ | ..........               1000 |              63                 10.1        98.6         1920
95.89/31.62	c ................................ | ..........            3707000 | 554          250680             96.0     38614.6     14959760
191.76/55.65	c ................................ | ..........            7644000 | 1306         524824            192.0     39812.5     25508235
287.62/79.69	c ................................ | ..........           11588000 | 2261         810944            288.0     40234.7     40725844
383.89/103.72	c ................................ | ..........           15425000 | 3091         1067563           384.0     40169.3     52084453
479.76/127.76	c ................................ | *.........           19244000 | 4131         1334632           480.0     40090.8     62640577
575.64/151.79	c ................................ | *.........           23096000 | 4652         1593980           576.0     40097.2     75826403
671.90/175.82	c ................................ | *.........           26736000 | 5359         1848640           672.0     39785.1     86130930
767.74/199.86	c ................................ | *.........           30393000 | 5977         2104396           768.0     39573.2     97351693
863.62/223.89	c ................................ | *.........           34072000 | 6948         2359141           864.0     39435.2    106105942
959.88/247.93	c ................................ | **........           37687000 | 7785         2614581           960.0     39256.5    118472235
1055.73/271.96	c ................................ | **........           41151000 | 8375         2849982          1056.0     38968.8    119941627
1151.99/296.00	c ................................ | **........           44553000 | 8916         3085174          1152.0     38673.8    122740487
1247.86/320.03	c ................................ | **........           48139000 | 9399         3339982          1248.0     38572.9    127386712
1343.73/344.07	c ................................ | **........           51578000 | 10098        3586168          1344.0     38375.9    131586984
1439.98/368.10	c ................................ | ***.......           54988000 | 11053        3824844          1440.0     38185.8    137638950
1535.82/392.14	c ................................ | ***.......           58563000 | 12114        4080002          1536.0     38127.0    147764644
1631.69/416.18	c ................................ | ***.......           61916000 | 13234        4322369          1632.0     37938.3    158786248
1727.93/440.22	c ................................ | ***.......           65314000 | 13871        4563329          1728.0     37797.0    170915903
1823.82/464.25	c ................................ | ***.......           68624000 | 14388        4796657          1824.0     37622.6    176073717
1919.67/488.28	c ................................ | ****......           71986000 | 14814        5031190          1920.0     37492.7    185964078
2015.93/512.32	c ................................ | ****......           75388000 | 15385        5280354          2016.0     37394.7    189996028
2111.77/536.35	c ................................ | ****......           79026000 | 16042        5529944          2112.0     37417.6    200905895
2207.62/560.39	c ................................ | ****......           82482000 | 16427        5772268          2208.0     37355.6    203685147
2303.89/584.43	c ................................ | ****......           85961000 | 16971        6010248          2304.0     37309.3    213670486
2399.71/608.47	c ................................ | *****.....           89177000 | 17669        6233610          2400.0     37156.8    213803785
2495.99/632.51	c ................................ | *****.....           92550000 | 18159        6459700          2496.0     37079.2    213803785
2591.82/656.54	c ................................ | *****.....           95773000 | 18565        6669645          2592.0     36949.5    213803785
2687.67/680.58	c ................................ | *****.....           99025000 | 19035        6888395          2688.0     36839.7    217665585
2783.93/704.62	c ................................ | *****.....          102444000 | 19645        7112399          2784.0     36797.3    226032590
2879.79/728.65	c ................................ | ******....          105672000 | 20143        7331783          2880.0     36691.4    232614630
2975.66/752.69	c ................................ | ******....          108871000 | 20705        7574290          2976.0     36582.8    241018844
3071.93/776.72	c ................................ | ******....          112026000 | 21258        7801965          3072.0     36466.6    247514070
3167.81/800.75	c ................................ | ******....          115327000 | 21802        8027233          3168.0     36403.7    254018417
3263.66/824.78	c ................................ | ******....          118607000 | 22086        8266145          3264.0     36337.9    263583433
3359.89/848.83	c ................................ | *******...          121802000 | 23791        8492675          3360.0     36250.4    271517951
3455.70/872.87	c ................................ | *******...          125120000 | 25434        8724803          3456.0     36203.7    273348015
3551.93/896.91	c ................................ | *******...          128317000 | 26628        8948834          3552.0     36125.3    274440640
3647.76/920.95	c ................................ | *******...          131476000 | 28830        9177810          3648.0     36040.6    275516644
3743.98/945.00	c ................................ | *******...          134490000 | 30382        9397500          3744.0     35921.5    275516644
3839.86/969.04	c ................................ | ********..          137692000 | 30812        9628202          3840.0     35857.3    275516644
3935.72/993.08	c ................................ | ********..          140596000 | 31910        9831869          3936.0     35720.3    275516644
4031.98/1017.11	c ................................ | ********..          143672000 | 33705        10037981         4032.0     35632.7    276042511
4127.85/1041.14	c ................................ | ********..          146710000 | 34703        10241946         4128.0     35540.0    277713529
4223.72/1065.17	c ................................ | ********..          149771000 | 36074        10453618         4224.0     35457.0    280079209
4319.98/1089.21	c ................................ | *********.          152896000 | 37499        10681472         4320.0     35392.5    285152593
4415.84/1113.25	c ................................ | *********.          156208000 | 38529        10910058         4416.0     35373.1    294095440
4512.01/1137.30	c ................................ | *********.          159184000 | 39431        11122960         4512.0     35280.1    296452419
4607.87/1161.33	c ................................ | *********.          162321000 | 40500        11346324         4608.0     35225.8    296452419
4703.74/1185.37	c ................................ | *********.          165450000 | 41319        11554463         4704.0     35172.0    299954200
4799.98/1209.40	c CPU limit (4800) reached
4799.98/1209.40	c Time = 4800.01 seconds
4799.98/1209.40	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250868-1304105581/watcher-3250868-1304105581 -o /tmp/evaluation-result-3250868-1304105581/solver-3250868-1304105581 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250868-1304105581.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.93 7.86 7.96 2/177 3096
/proc/meminfo: memFree=30826708/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=9368 CPUtime=0 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 409 0 0 0 0 0 0 0 25 0 1 0 502974070 9592832 352 33554432000 4194304 4338100 140733873790560 18446744073709551615 220816236288 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 2342 352 147 36 0 247 0

[startup+0.0884099 s]
/proc/loadavg: 6.93 7.86 7.96 2/177 3096
/proc/meminfo: memFree=30826708/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=26452 CPUtime=0.08 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 4582 0 0 0 8 0 0 0 25 0 1 0 502974070 27086848 4525 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 6613 4527 148 36 0 4518 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26452

[startup+0.100406 s]
/proc/loadavg: 6.93 7.86 7.96 2/177 3096
/proc/meminfo: memFree=30826708/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=28696 CPUtime=0.09 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 5174 0 0 0 9 0 0 0 25 0 1 0 502974070 29384704 5117 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 7174 5118 148 36 0 5079 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28696

[startup+0.300371 s]
/proc/loadavg: 6.93 7.86 7.96 2/177 3096
/proc/meminfo: memFree=30826708/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=69504 CPUtime=0.29 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 15058 0 0 0 28 1 0 0 25 0 1 0 502974070 71172096 15001 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 17376 15002 148 36 0 15281 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 69504

[startup+0.700311 s]
/proc/loadavg: 6.93 7.86 7.96 2/177 3096
/proc/meminfo: memFree=30826708/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=148408 CPUtime=0.69 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 34363 0 0 0 65 4 0 0 25 0 1 0 502974070 151969792 34306 33554432000 4194304 4338100 140733873790560 18446744073709551615 4220820 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 37102 34306 150 36 0 35007 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 148408

[startup+1.50015 s]
/proc/loadavg: 6.93 7.86 7.96 6/182 3103
/proc/meminfo: memFree=30562320/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=250036 CPUtime=1.49 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 54290 0 0 0 143 6 0 0 25 0 1 0 502974070 256036864 54038 33554432000 4194304 4338100 140733873790560 18446744073709551615 4228108 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 62509 54039 723 36 0 60414 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 250036

[startup+3.10085 s]
/proc/loadavg: 6.85 7.83 7.95 6/182 3103
/proc/meminfo: memFree=30531568/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=250200 CPUtime=3.09 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 81276 0 0 0 300 9 0 0 25 0 1 0 502974070 256204800 58635 33554432000 4194304 4338100 140733873790560 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 62550 58635 155 36 0 60455 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 250200

[startup+6.30026 s]
/proc/loadavg: 6.85 7.83 7.95 6/182 3103
/proc/meminfo: memFree=30273220/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=348320 CPUtime=6.29 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) R 3094 3096 2468 0 -1 4202496 109762 0 0 0 616 13 0 0 25 0 1 0 502974070 356679680 84847 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815894298 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/3096/statm: 87080 84847 164 36 0 84985 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 348320

[startup+12.7011 s]
/proc/loadavg: 6.81 7.79 7.93 9/186 3109
/proc/meminfo: memFree=29627432/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=774424 CPUtime=20.39 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 189682 0 0 0 1983 56 0 0 25 0 5 0 502974070 793010176 164766 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 193606 164767 176 36 0 191511 0
[pid=3096/tid=3104] ppid=3094 vsize=774424 CPUtime=2.57 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 3885 0 0 0 256 1 0 0 22 0 5 0 502975083 793010176 164767 33554432000 4194304 4338100 140733873790560 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=774424 CPUtime=2.57 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1927 0 0 0 240 17 0 0 21 0 5 0 502975083 793010176 164767 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815897633 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=774424 CPUtime=2.56 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 5235 0 0 0 256 0 0 0 25 0 5 0 502975083 793010176 164767 33554432000 4194304 4338100 140733873790560 18446744073709551615 4288399 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=774424 CPUtime=2.55 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 2929 0 0 0 238 17 0 0 22 0 5 0 502975083 793010176 164767 33554432000 4194304 4338100 140733873790560 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 20.39
Current children cumulated vsize (KiB) 774424

[startup+25.5007 s]
/proc/loadavg: 6.92 7.78 7.93 9/186 3112
/proc/meminfo: memFree=28798776/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=1032544 CPUtime=71.53 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 262825 0 0 0 6912 241 0 0 25 0 5 0 502974070 1057325056 237909 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 258136 237909 176 36 0 256041 0
[pid=3096/tid=3104] ppid=3094 vsize=1032544 CPUtime=15.37 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 28229 0 0 0 1528 9 0 0 25 0 5 0 502975083 1057325056 237909 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631483 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=1032544 CPUtime=15.31 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 14235 0 0 0 1427 104 0 0 25 0 5 0 502975083 1057325056 237909 33554432000 4194304 4338100 140733873790560 18446744073709551615 4285040 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=1032544 CPUtime=15.36 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 28597 0 0 0 1532 4 0 0 25 0 5 0 502975083 1057325056 237909 33554432000 4194304 4338100 140733873790560 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=1032544 CPUtime=15.32 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 16058 0 0 0 1430 102 0 0 25 0 5 0 502975083 1057325056 237911 33554432000 4194304 4338100 140733873790560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 71.53
Current children cumulated vsize (KiB) 1032544

[startup+51.101 s]
/proc/loadavg: 7.47 7.84 7.94 9/186 3112
/proc/meminfo: memFree=27622664/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=1501300 CPUtime=173.79 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 374194 0 0 0 16759 620 0 0 25 0 5 0 502974070 1537331200 349278 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 375325 349278 176 36 0 373230 0
[pid=3096/tid=3104] ppid=3094 vsize=1501300 CPUtime=40.96 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 68842 0 0 0 4076 20 0 0 25 0 5 0 502975083 1537331200 349278 33554432000 4194304 4338100 140733873790560 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=1501300 CPUtime=40.78 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 29342 0 0 0 3798 280 0 0 25 0 5 0 502975083 1537331200 349278 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828049102 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=1501300 CPUtime=40.96 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 66271 0 0 0 4083 13 0 0 25 0 5 0 502975083 1537331200 349278 33554432000 4194304 4338100 140733873790560 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=1501300 CPUtime=40.92 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) S 3094 3096 2468 0 -1 4202560 34032 0 0 0 3807 285 0 0 25 0 5 0 502975083 1537331200 349279 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062068 0 0 4096 0 18446604468606239104 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 173.79
Current children cumulated vsize (KiB) 1501300

[startup+102.307 s]
/proc/loadavg: 7.88 7.89 7.95 9/186 3114
/proc/meminfo: memFree=25737616/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=2475572 CPUtime=378.32 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 600911 0 0 0 36393 1439 0 0 25 0 5 0 502974070 2534985728 575995 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 618893 575995 176 36 0 616798 0
[pid=3096/tid=3104] ppid=3094 vsize=2475572 CPUtime=92.17 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 153428 0 0 0 9169 48 0 0 25 0 5 0 502975083 2534985728 575995 33554432000 4194304 4338100 140733873790560 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=2475572 CPUtime=91.71 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 59852 0 0 0 8505 666 0 0 25 0 5 0 502975083 2534985728 575995 33554432000 4194304 4338100 140733873790560 18446744073709551615 4285491 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=2475572 CPUtime=92.17 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 145646 0 0 0 9188 29 0 0 25 0 5 0 502975083 2534985728 575995 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815932958 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=2475572 CPUtime=92.1 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 66278 0 0 0 8536 674 0 0 25 0 5 0 502975083 2534985728 575995 33554432000 4194304 4338100 140733873790560 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 378.32
Current children cumulated vsize (KiB) 2475572

[startup+162.3 s]
/proc/loadavg: 7.91 7.89 7.95 9/186 3147
/proc/meminfo: memFree=27013284/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=3432724 CPUtime=617.97 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 821970 0 0 0 59462 2335 0 0 25 0 5 0 502974070 3515109376 797054 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 858181 797054 176 36 0 856086 0
[pid=3096/tid=3104] ppid=3094 vsize=3432724 CPUtime=152.17 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 252358 0 0 0 15137 80 0 0 25 0 5 0 502975083 3515109376 797054 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631841 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=3432724 CPUtime=151.39 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 79430 0 0 0 14049 1090 0 0 25 0 5 0 502975083 3515109376 797054 33554432000 4194304 4338100 140733873790560 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=3432724 CPUtime=152.17 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 218138 0 0 0 15172 45 0 0 25 0 5 0 502975083 3515109376 797054 33554432000 4194304 4338100 140733873790560 18446744073709551615 4245205 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=3432724 CPUtime=152.1 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 96337 0 0 0 14110 1100 0 0 25 0 5 0 502975083 3515109376 797054 33554432000 4194304 4338100 140733873790560 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 617.97
Current children cumulated vsize (KiB) 3432724

[startup+222.3 s]
/proc/loadavg: 7.97 7.91 7.95 9/186 3149
/proc/meminfo: memFree=25128424/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=4296396 CPUtime=857.62 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 1022718 0 0 0 82463 3299 0 0 25 0 5 0 502974070 4399509504 997802 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 1074099 997802 176 36 0 1072004 0
[pid=3096/tid=3104] ppid=3094 vsize=4296396 CPUtime=212.15 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 345740 0 0 0 21106 109 0 0 25 0 5 0 502975083 4399509504 997802 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815932940 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=4296396 CPUtime=211.29 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 93134 0 0 0 19578 1551 0 0 25 0 5 0 502975083 4399509504 997802 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631431 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=4296396 CPUtime=212.16 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 294973 0 0 0 21155 61 0 0 25 0 5 0 502975083 4399509504 997802 33554432000 4194304 4338100 140733873790560 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=4296396 CPUtime=211.85 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 113164 0 0 0 19629 1556 0 0 25 0 5 0 502975083 4399509504 997802 33554432000 4194304 4338100 140733873790560 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 857.62
Current children cumulated vsize (KiB) 4296396

[startup+282.3 s]
/proc/loadavg: 7.98 7.92 7.95 9/186 3150
/proc/meminfo: memFree=23904248/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=4822500 CPUtime=1097.27 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 1157863 0 0 0 105478 4249 0 0 25 0 5 0 502974070 4938240000 1132947 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 1205625 1132947 176 36 0 1203530 0
[pid=3096/tid=3104] ppid=3094 vsize=4822500 CPUtime=272.15 cores=1,3,5,7

################
# More data... #
################

/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 971002 0 0 0 74857 348 0 0 25 0 5 0 502975083 9336877056 2134739 33554432000 4194304 4338100 140733873790560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=9118044 CPUtime=750.31 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 143214 0 0 0 69108 5923 0 0 25 0 5 0 502975083 9336877056 2134739 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631746 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=9118044 CPUtime=750.95 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 690195 0 0 0 74944 151 0 0 25 0 5 0 502975083 9336877056 2134739 33554432000 4194304 4338100 140733873790560 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=9118044 CPUtime=750.98 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 179537 0 0 0 69238 5860 0 0 25 0 5 0 502975083 9336877056 2134739 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631399 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3014.44
Current children cumulated vsize (KiB) 9118044

[startup+822.301 s]
/proc/loadavg: 8.05 8.04 8.00 9/186 3166
/proc/meminfo: memFree=14178584/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=9662212 CPUtime=3254.08 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2295488 0 0 0 312074 13334 0 0 25 0 5 0 502974070 9894105088 2270572 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2415553 2270573 176 36 0 2413458 0
[pid=3096/tid=3104] ppid=3094 vsize=9662212 CPUtime=812.04 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1052955 0 0 0 80823 381 0 0 25 0 5 0 502975083 9894105088 2270573 33554432000 4194304 4338100 140733873790560 18446744073709551615 4288277 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=9662212 CPUtime=810.29 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 147282 0 0 0 74612 6417 0 0 18 0 5 0 502975083 9894105088 2270573 33554432000 4194304 4338100 140733873790560 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=9662212 CPUtime=810.63 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 736207 0 0 0 80900 163 0 0 25 0 5 0 502975083 9894105088 2270573 33554432000 4194304 4338100 140733873790560 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=9662212 CPUtime=810.97 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 183338 0 0 0 74746 6351 0 0 25 0 5 0 502975083 9894105088 2270573 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631376 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3254.08
Current children cumulated vsize (KiB) 9662212

[startup+882.301 s]
/proc/loadavg: 8.16 8.07 8.01 9/186 3167
/proc/meminfo: memFree=13355668/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=10063532 CPUtime=3493.63 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2385481 0 0 0 335019 14344 0 0 25 0 5 0 502974070 10305056768 2360565 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2515883 2360565 176 36 0 2513788 0
[pid=3096/tid=3104] ppid=3094 vsize=10063532 CPUtime=871.99 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1101781 0 0 0 86795 404 0 0 25 0 5 0 502975083 10305056768 2360565 33554432000 4194304 4338100 140733873790560 18446744073709551615 4240522 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=10063532 CPUtime=870.25 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 158283 0 0 0 80112 6913 0 0 25 0 5 0 502975083 10305056768 2360565 33554432000 4194304 4338100 140733873790560 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=10063532 CPUtime=870.6 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 757014 0 0 0 86893 167 0 0 25 0 5 0 502975083 10305056768 2360565 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631585 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=10063532 CPUtime=870.64 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 192696 0 0 0 80225 6839 0 0 25 0 5 0 502975083 10305056768 2360565 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815632259 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3493.63
Current children cumulated vsize (KiB) 10063532

[startup+942.301 s]
/proc/loadavg: 8.17 8.09 8.01 9/186 3169
/proc/meminfo: memFree=12879848/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=10282724 CPUtime=3733.2 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2423650 0 0 0 357960 15360 0 0 25 0 5 0 502974070 10529509376 2398734 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2570681 2398734 176 36 0 2568586 0
[pid=3096/tid=3104] ppid=3094 vsize=10282724 CPUtime=931.97 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1114946 0 0 0 92779 418 0 0 25 0 5 0 502975083 10529509376 2398734 33554432000 4194304 4338100 140733873790560 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=10282724 CPUtime=930.15 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 172060 0 0 0 85598 7417 0 0 25 0 5 0 502975083 10529509376 2398734 33554432000 4194304 4338100 140733873790560 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=10282724 CPUtime=930.58 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 757016 0 0 0 92891 167 0 0 25 0 5 0 502975083 10529509376 2398734 33554432000 4194304 4338100 140733873790560 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=10282724 CPUtime=930.33 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 203921 0 0 0 85696 7337 0 0 25 0 5 0 502975083 10529509376 2398734 33554432000 4194304 4338100 140733873790560 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3733.2
Current children cumulated vsize (KiB) 10282724

[startup+1002.3 s]
/proc/loadavg: 8.20 8.10 8.02 9/186 3171
/proc/meminfo: memFree=12568180/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=10282724 CPUtime=3972.86 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2440126 0 0 0 380963 16323 0 0 25 0 5 0 502974070 10529509376 2415210 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2570681 2415210 176 36 0 2568586 0
[pid=3096/tid=3104] ppid=3094 vsize=10282724 CPUtime=991.8 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1114960 0 0 0 98752 428 0 0 25 0 5 0 502975083 10529509376 2415210 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631797 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=10282724 CPUtime=990.05 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 180297 0 0 0 91110 7895 0 0 25 0 5 0 502975083 10529509376 2415210 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815932896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=10282724 CPUtime=990.58 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 757024 0 0 0 98891 167 0 0 25 0 5 0 502975083 10529509376 2415210 33554432000 4194304 4338100 140733873790560 18446744073709551615 4288258 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=10282724 CPUtime=990.27 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 212138 0 0 0 91215 7812 0 0 25 0 5 0 502975083 10529509376 2415210 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815631785 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3972.86
Current children cumulated vsize (KiB) 10282724

[startup+1062.3 s]
/proc/loadavg: 8.22 8.11 8.02 9/186 3172
/proc/meminfo: memFree=11840628/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=10347844 CPUtime=4212.54 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2473195 0 0 0 404010 17244 0 0 25 0 5 0 502974070 10596192256 2448279 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2586961 2448279 176 36 0 2584866 0
[pid=3096/tid=3104] ppid=3094 vsize=10347844 CPUtime=1051.47 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1138405 0 0 0 104704 443 0 0 25 0 5 0 502975083 10596192256 2448279 33554432000 4194304 4338100 140733873790560 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=10347844 CPUtime=1050.05 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 182757 0 0 0 96654 8351 0 0 25 0 5 0 502975083 10596192256 2448279 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=10347844 CPUtime=1050.59 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 761469 0 0 0 104890 169 0 0 25 0 5 0 502975083 10596192256 2448279 33554432000 4194304 4338100 140733873790560 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=10347844 CPUtime=1050.27 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 214857 0 0 0 96768 8259 0 0 25 0 5 0 502975083 10596192256 2448279 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4212.54
Current children cumulated vsize (KiB) 10347844

[startup+1122.3 s]
/proc/loadavg: 8.14 8.11 8.02 9/186 3177
/proc/meminfo: memFree=10873152/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=10876148 CPUtime=4452.16 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2601773 0 0 0 426944 18272 0 0 25 0 5 0 502974070 11137175552 2576857 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2719037 2576857 176 36 0 2716942 0
[pid=3096/tid=3104] ppid=3094 vsize=10876148 CPUtime=1111.11 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1210844 0 0 0 110638 473 0 0 25 0 5 0 502975083 11137175552 2576857 33554432000 4194304 4338100 140733873790560 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=10876148 CPUtime=1110.05 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 183613 0 0 0 102161 8844 0 0 25 0 5 0 502975083 11137175552 2576857 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=10876148 CPUtime=1110.57 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 816103 0 0 0 110877 180 0 0 25 0 5 0 502975083 11137175552 2576857 33554432000 4194304 4338100 140733873790560 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=10876148 CPUtime=1110.26 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 215506 0 0 0 102273 8753 0 0 25 0 5 0 502975083 11137175552 2576857 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4452.16
Current children cumulated vsize (KiB) 10876148

[startup+1182.3 s]
/proc/loadavg: 8.05 8.08 8.02 9/186 3179
/proc/meminfo: memFree=10320208/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=11360532 CPUtime=4691.76 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) S 3094 3096 2468 0 -1 4202496 2645511 0 0 0 449912 19264 0 0 25 0 5 0 502974070 11633184768 2620595 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828039141 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 2840133 2620595 176 36 0 2838038 0
[pid=3096/tid=3104] ppid=3094 vsize=11360532 CPUtime=1170.79 cores=1,3,5,7
/proc/3096/task/3104/stat : 3104 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 1234509 0 0 0 116588 491 0 0 25 0 5 0 502975083 11633184768 2620595 33554432000 4194304 4338100 140733873790560 18446744073709551615 220815632110 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=3096/tid=3106] ppid=3094 vsize=11360532 CPUtime=1170.01 cores=1,3,5,7
/proc/3096/task/3106/stat : 3106 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 189825 0 0 0 107674 9327 0 0 25 0 5 0 502975083 11633184768 2620595 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=3096/tid=3107] ppid=3094 vsize=11360532 CPUtime=1170.57 cores=1,3,5,7
/proc/3096/task/3107/stat : 3107 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 824698 0 0 0 116875 182 0 0 25 0 5 0 502975083 11633184768 2620595 33554432000 4194304 4338100 140733873790560 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=11360532 CPUtime=1170.23 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4202560 220772 0 0 0 107781 9242 0 0 25 0 5 0 502975083 11633184768 2620595 33554432000 4194304 4338100 140733873790560 18446744073709551615 220828062389 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4691.76
Current children cumulated vsize (KiB) 11360532



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1209.5 s]
/proc/loadavg: 8.03 8.08 8.01 6/183 3180
/proc/meminfo: memFree=12786556/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) Z 3094 3096 2468 0 -1 4203548 2682859 0 0 0 460265 19746 0 0 18 0 2 0 502974070 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 0 0 0 0 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=0 CPUtime=1197.43 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4203588 220773 0 0 0 110263 9480 0 0 25 0 2 0 502975083 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1209.61 s]
/proc/loadavg: 8.03 8.08 8.01 6/183 3180
/proc/meminfo: memFree=12786556/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=0 CPUtime=4800.22 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) Z 3094 3096 2468 0 -1 4203548 2682859 0 0 0 460265 19757 0 0 18 0 2 0 502974070 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 0 0 0 0 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=0 CPUtime=1197.53 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4203588 220773 0 0 0 110263 9490 0 0 25 0 2 0 502975083 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

[startup+1209.7 s]
/proc/loadavg: 8.03 8.08 8.01 6/183 3180
/proc/meminfo: memFree=12786556/32951124 swapFree=67105204/67111528
[pid=3096] ppid=3094 vsize=0 CPUtime=4800.3 cores=1,3,5,7
/proc/3096/stat : 3096 (ttsth-5-4) Z 3094 3096 2468 0 -1 4203548 2682859 0 0 0 460265 19765 0 0 18 0 2 0 502974070 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/3096/statm: 0 0 0 0 0 0 0
[pid=3096/tid=3109] ppid=3094 vsize=0 CPUtime=1197.62 cores=1,3,5,7
/proc/3096/task/3109/stat : 3109 (ttsth-5-4) R 3094 3096 2468 0 -1 4203588 220773 0 0 0 110263 9499 0 0 25 0 2 0 502975083 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1209.76
CPU time (s): 4800.37
CPU user time (s): 4602.66
CPU system time (s): 197.715
CPU usage (%): 396.805
Max. virtual memory (cumulated for all children) (KiB): 11481420

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4602.66
system time used= 197.715
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2682859
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 11150
involuntary context switches= 32894

runsolver used 2.0027 second user time and 4.5883 second system time

The end

Launcher Data 

Begin job on node147 at 2011-04-29 21:33:02
IDJOB=3250868
IDBENCH=88203
IDSOLVER=1654
FILE ID=node147/3250868-1304105581
RUNJOBID= node147-1304105580-3012
PBS_JOBID= 13170622
Free space on /tmp= 72644 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1327.5-n60-p30-s52708253.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250868-1304105581/watcher-3250868-1304105581 -o /tmp/evaluation-result-3250868-1304105581/solver-3250868-1304105581 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250868-1304105581.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 472b10ce0a2683d9355981037eea091e
RANDOM SEED=1768799284

node147.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30827244 kB
Buffers:        282884 kB
Cached:        1384160 kB
SwapCached:       3064 kB
Active:         224764 kB
Inactive:      1574672 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30827244 kB
SwapTotal:    67111528 kB
SwapFree:     67105204 kB
Dirty:           17040 kB
Writeback:           0 kB
AnonPages:      130332 kB
Mapped:          14592 kB
Slab:           261152 kB
PageTables:       4032 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   276460 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72636 MiB
End job on node147 at 2011-04-29 21:53:13