Trace number 3280637

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Black Hole SAT 2011-03-08? (TO) 1300.06 1300.02

General information on the benchmark

Nameindustrial/palacios/hard/
sortnet-8-ipc5-h19-sat.cnf
MD5SUM409f4401e9e8b60ee665ac5567a13e41
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1110.09
Satisfiable
(Un)Satisfiability was proved
Number of variables359341
Number of clauses1254773
Sum of the clauses size3046189
Maximum clause length57
Minimum clause length1
Number of clauses of size 1256
Number of clauses of size 2896240
Number of clauses of size 3346435
Number of clauses of size 4830
Number of clauses of size 5571
Number of clauses of size over 510441

Solver Data 

18.60/18.67	c RESTART
27.50/27.51	c RESTART
33.79/33.85	c RESTART
40.60/40.61	c RESTART
45.00/45.04	c RESTART
59.40/59.41	c RESTART
66.19/66.20	c RESTART
73.19/73.20	c RESTART
81.30/81.35	c RESTART
92.09/92.15	c RESTART
99.39/99.41	c RESTART
106.60/106.60	c RESTART
116.69/116.74	c RESTART
122.49/122.55	c RESTART
128.39/128.43	c RESTART
133.00/133.02	c RESTART
141.09/141.12	c RESTART
146.29/146.37	c RESTART
154.89/154.99	c RESTART
162.49/162.59	c RESTART
169.59/169.64	c RESTART
180.49/180.51	c RESTART
188.29/188.32	c RESTART
195.09/195.17	c RESTART
201.59/201.66	c RESTART
206.99/207.06	c RESTART
213.40/213.46	c RESTART
219.89/219.95	c RESTART
225.69/225.71	c RESTART
234.59/234.62	c RESTART
240.90/240.93	c RESTART
247.79/247.81	c RESTART
253.09/253.13	c RESTART
259.79/259.88	c RESTART
264.69/264.73	c RESTART
270.19/270.24	c RESTART
277.49/277.55	c RESTART
288.49/288.56	c RESTART
293.70/293.77	c RESTART
307.60/307.64	c RESTART
313.51/313.58	c RESTART
320.01/320.00	c RESTART
324.20/324.20	c RESTART
330.80/330.88	c RESTART
334.70/334.74	c RESTART
342.40/342.45	c RESTART
350.00/350.04	c RESTART
354.11/354.17	c RESTART
360.50/360.58	c RESTART
366.60/366.60	c RESTART
373.60/373.64	c RESTART
381.80/381.81	c RESTART
389.41/389.43	c RESTART
397.01/397.08	c RESTART
403.90/403.99	c RESTART
407.20/407.29	c RESTART
410.90/410.94	c RESTART
416.30/416.30	c RESTART
421.60/421.60	c RESTART
429.40/429.44	c RESTART
434.60/434.67	c RESTART
452.71/452.74	c RESTART
464.00/464.07	c RESTART
468.40/468.41	c RESTART
472.51/472.59	c RESTART
478.11/478.19	c RESTART
483.21/483.24	c RESTART
489.31/489.30	c RESTART
493.81/493.86	c RESTART
497.80/497.87	c RESTART
501.61/501.62	c RESTART
505.80/505.89	c RESTART
511.51/511.57	c RESTART
517.20/517.21	c RESTART
521.41/521.46	c RESTART
526.20/526.20	c RESTART
529.70/529.73	c RESTART
535.20/535.22	c RESTART
567.30/567.35	c RESTART
572.80/572.82	c RESTART
580.40/580.46	c RESTART
586.81/586.84	c RESTART
591.31/591.35	c RESTART
598.81/598.86	c RESTART
610.91/610.90	c RESTART
616.21/616.24	c RESTART
620.61/620.61	c RESTART
626.11/626.13	c RESTART
631.51/631.53	c RESTART
636.11/636.10	c RESTART
641.71/641.77	c RESTART
651.21/651.24	c RESTART
654.41/654.45	c RESTART
660.21/660.29	c RESTART
667.41/667.46	c RESTART
676.41/676.49	c RESTART
683.71/683.75	c RESTART
691.21/691.23	c RESTART
713.11/713.14	c RESTART
720.71/720.75	c RESTART
745.72/745.80	c RESTART
749.21/749.26	c RESTART
776.91/776.92	c RESTART
789.11/789.15	c RESTART
805.41/805.46	c RESTART
818.91/818.90	c RESTART
829.32/829.40	c RESTART
842.62/842.61	c RESTART
870.62/870.64	c RESTART
876.02/876.02	c RESTART
896.92/896.91	c RESTART
910.82/910.81	c RESTART
917.42/917.41	c RESTART
934.72/934.77	c RESTART
971.82/971.89	c RESTART
993.82/993.82	c RESTART
1016.33/1016.35	c RESTART
1021.52/1021.58	c RESTART
1026.92/1026.96	c RESTART
1040.22/1040.22	c RESTART
1049.62/1049.64	c RESTART
1061.32/1061.32	c RESTART
1074.63/1074.67	c RESTART
1093.02/1093.07	c RESTART
1112.22/1112.24	c RESTART
1140.22/1140.22	c RESTART
1157.62/1157.66	c RESTART
1163.12/1163.15	c RESTART
1167.42/1167.44	c RESTART
1176.12/1176.13	c RESTART
1182.02/1182.04	c RESTART
1186.02/1186.03	c RESTART
1264.22/1264.24	c RESTART
1267.42/1267.43	c RESTART
1276.22/1276.25	c RESTART
1282.53/1282.52	c RESTART
1288.43/1288.49	c RESTART
1293.33/1293.32	c RESTART

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3280637-1303749649/watcher-3280637-1303749649 -o /tmp/evaluation-result-3280637-1303749649/solver-3280637-1303749649 -C 2400 -W 1300 -M 7680 saturn HOME/instance-3280637-1303749649.cnf 

running on 2 cores: 0,2

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 7864320 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 7915520 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.94 2.06 2.68 4/183 29686
/proc/meminfo: memFree=16982408/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=43080 CPUtime=0 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 2015 0 0 0 0 0 0 0 21 0 1 0 565195573 44113920 1957 33554432000 4194304 4210924 140736950384064 18446744073709551615 260020089949 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 10770 1969 81 5 0 3556 0

[startup+0.0676439 s]
/proc/loadavg: 0.94 2.06 2.68 4/183 29686
/proc/meminfo: memFree=16982408/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=159356 CPUtime=0.05 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 14097 0 0 0 4 1 0 0 21 0 1 0 565195573 163180544 14037 33554432000 4194304 4210924 140736950384064 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 39839 14040 384 5 0 32625 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 159356

[startup+0.100664 s]
/proc/loadavg: 0.94 2.06 2.68 4/183 29686
/proc/meminfo: memFree=16982408/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=165032 CPUtime=0.08 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 16376 0 0 0 7 1 0 0 21 0 1 0 565195573 168992768 16316 33554432000 4194304 4210924 140736950384064 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 41291 16319 640 5 0 34077 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 165032

[startup+0.300626 s]
/proc/loadavg: 0.94 2.06 2.68 4/183 29686
/proc/meminfo: memFree=16982408/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=198956 CPUtime=0.28 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 29854 0 0 0 26 2 0 0 22 0 1 0 565195573 203730944 29794 33554432000 4194304 4210924 140736950384064 18446744073709551615 260020057121 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 49739 29798 2199 5 0 42525 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 198956

[startup+0.700558 s]
/proc/loadavg: 0.94 2.06 2.68 4/183 29686
/proc/meminfo: memFree=16982408/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=264560 CPUtime=0.69 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 56255 0 0 0 64 5 0 0 23 0 1 0 565195573 270909440 56195 33554432000 4194304 4210924 140736950384064 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66140 56198 5425 5 0 58926 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 264560

[startup+1.50041 s]
/proc/loadavg: 0.94 2.06 2.68 5/190 29699
/proc/meminfo: memFree=16360236/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=249212 CPUtime=1.48 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61210 0 0 0 143 5 0 0 25 0 1 0 565195573 255193088 54762 33554432000 4194304 4210924 140736950384064 18446744073709551615 4198572 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62303 54762 97 5 0 60774 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 249212

[startup+3.10112 s]
/proc/loadavg: 0.94 2.06 2.68 5/190 29699
/proc/meminfo: memFree=16343120/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=249344 CPUtime=3.09 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61254 0 0 0 303 6 0 0 25 0 1 0 565195573 255328256 54806 33554432000 4194304 4210924 140736950384064 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62336 54806 97 5 0 60807 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 249344

[startup+6.30053 s]
/proc/loadavg: 1.26 2.11 2.69 5/190 29699
/proc/meminfo: memFree=16338944/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=249344 CPUtime=6.29 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61273 0 0 0 623 6 0 0 25 0 1 0 565195573 255328256 54825 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62336 54825 97 5 0 60807 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 249344

[startup+12.7004 s]
/proc/loadavg: 1.48 2.14 2.70 5/190 29699
/proc/meminfo: memFree=16337852/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=249344 CPUtime=12.69 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61279 0 0 0 1263 6 0 0 25 0 1 0 565195573 255328256 54831 33554432000 4194304 4210924 140736950384064 18446744073709551615 4201865 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62336 54831 97 5 0 60807 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 249344

[startup+25.501 s]
/proc/loadavg: 2.04 2.23 2.72 5/190 29700
/proc/meminfo: memFree=16337988/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=249744 CPUtime=25.49 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61393 0 0 0 2543 6 0 0 25 0 1 0 565195573 255737856 54945 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206192 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62436 54945 103 5 0 60907 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 249744

[startup+51.1003 s]
/proc/loadavg: 2.71 2.37 2.75 5/190 29700
/proc/meminfo: memFree=16337124/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=250148 CPUtime=51.09 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61517 0 0 0 5103 6 0 0 25 0 1 0 565195573 256151552 55069 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204590 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62537 55069 104 5 0 61008 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 250148

[startup+102.307 s]
/proc/loadavg: 3.44 2.62 2.82 5/190 29702
/proc/meminfo: memFree=16332220/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=251348 CPUtime=102.3 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 61881 0 0 0 10224 6 0 0 25 0 1 0 565195573 257380352 55433 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204590 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 62837 55433 104 5 0 61308 0
Current children cumulated CPU time (s) 102.3
Current children cumulated vsize (KiB) 251348

[startup+162.301 s]
/proc/loadavg: 3.79 2.87 2.89 5/190 29703
/proc/meminfo: memFree=16329324/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=252544 CPUtime=162.29 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 62169 0 0 0 16223 6 0 0 25 0 1 0 565195573 258605056 55721 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204590 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 63136 55721 104 5 0 61607 0
Current children cumulated CPU time (s) 162.29
Current children cumulated vsize (KiB) 252544

[startup+222.301 s]
/proc/loadavg: 3.92 3.07 2.95 5/190 29705
/proc/meminfo: memFree=16326904/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=254028 CPUtime=222.29 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 62531 0 0 0 22223 6 0 0 25 0 1 0 565195573 260124672 56083 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204590 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 63507 56083 104 5 0 61978 0
Current children cumulated CPU time (s) 222.29
Current children cumulated vsize (KiB) 254028

[startup+282.301 s]
/proc/loadavg: 3.97 3.24 3.02 5/190 29706
/proc/meminfo: memFree=16325260/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=255368 CPUtime=282.29 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 62889 0 0 0 28223 6 0 0 25 0 1 0 565195573 261496832 56441 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204590 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 63842 56441 104 5 0 62313 0
Current children cumulated CPU time (s) 282.29
Current children cumulated vsize (KiB) 255368

[startup+342.301 s]
/proc/loadavg: 4.04 3.39 3.08 5/190 29708
/proc/meminfo: memFree=16320132/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=257684 CPUtime=342.3 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 63460 0 0 0 34223 7 0 0 25 0 1 0 565195573 263868416 57012 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 64421 57012 104 5 0 62892 0
Current children cumulated CPU time (s) 342.3
Current children cumulated vsize (KiB) 257684

[startup+402.301 s]
/proc/loadavg: 4.01 3.50 3.13 5/190 29710
/proc/meminfo: memFree=16318384/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=258644 CPUtime=402.3 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 63712 0 0 0 40223 7 0 0 25 0 1 0 565195573 264851456 57264 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206991 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 64661 57264 104 5 0 63132 0
Current children cumulated CPU time (s) 402.3
Current children cumulated vsize (KiB) 258644

[startup+462.301 s]
/proc/loadavg: 4.00 3.59 3.19 5/190 29711
/proc/meminfo: memFree=16316752/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=259440 CPUtime=462.3 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 63933 0 0 0 46223 7 0 0 25 0 1 0 565195573 265666560 57485 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206991 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 64860 57485 104 5 0 63331 0
Current children cumulated CPU time (s) 462.3
Current children cumulated vsize (KiB) 259440

[startup+522.301 s]
/proc/loadavg: 4.05 3.67 3.24 5/190 29713
/proc/meminfo: memFree=16314748/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=261068 CPUtime=522.3 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 64327 0 0 0 52223 7 0 0 25 0 1 0 565195573 267333632 57879 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 65267 57879 104 5 0 63738 0
Current children cumulated CPU time (s) 522.3
Current children cumulated vsize (KiB) 261068

[startup+582.301 s]
/proc/loadavg: 4.02 3.73 3.28 5/190 29714
/proc/meminfo: memFree=16313732/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=261476 CPUtime=582.31 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 64442 0 0 0 58224 7 0 0 25 0 1 0 565195573 267751424 57994 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206988 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 65369 57994 104 5 0 63840 0
Current children cumulated CPU time (s) 582.31
Current children cumulated vsize (KiB) 261476

[startup+642.301 s]
/proc/loadavg: 4.04 3.79 3.33 5/190 29716
/proc/meminfo: memFree=16311856/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=261744 CPUtime=642.31 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 64519 0 0 0 64224 7 0 0 25 0 1 0 565195573 268025856 58071 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206192 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 65436 58071 104 5 0 63907 0
Current children cumulated CPU time (s) 642.31
Current children cumulated vsize (KiB) 261744

[startup+702.301 s]
/proc/loadavg: 4.01 3.83 3.37 5/190 29718
/proc/meminfo: memFree=16309100/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=263496 CPUtime=702.31 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 64968 0 0 0 70224 7 0 0 25 0 1 0 565195573 269819904 58520 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206991 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 65874 58520 104 5 0 64345 0
Current children cumulated CPU time (s) 702.31
Current children cumulated vsize (KiB) 263496

[startup+762.307 s]
/proc/loadavg: 4.00 3.86 3.41 5/190 29719
/proc/meminfo: memFree=16307224/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=264448 CPUtime=762.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65199 0 0 0 76224 8 0 0 25 0 1 0 565195573 270794752 58751 33554432000 4194304 4210924 140736950384064 18446744073709551615 4205177 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66112 58751 104 5 0 64583 0
Current children cumulated CPU time (s) 762.32
Current children cumulated vsize (KiB) 264448

[startup+822.301 s]
/proc/loadavg: 4.00 3.88 3.44 5/190 29721
/proc/meminfo: memFree=16304600/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=264584 CPUtime=822.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65228 0 0 0 82224 8 0 0 25 0 1 0 565195573 270934016 58780 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206192 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66146 58780 104 5 0 64617 0
Current children cumulated CPU time (s) 822.32
Current children cumulated vsize (KiB) 264584

[startup+882.301 s]
/proc/loadavg: 4.00 3.90 3.47 5/190 29722
/proc/meminfo: memFree=16303452/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=264848 CPUtime=882.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65298 0 0 0 88224 8 0 0 25 0 1 0 565195573 271204352 58850 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204590 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66212 58850 104 5 0 64683 0
Current children cumulated CPU time (s) 882.32
Current children cumulated vsize (KiB) 264848

[startup+942.301 s]
/proc/loadavg: 4.04 3.93 3.51 5/190 29724
/proc/meminfo: memFree=16302320/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=265260 CPUtime=942.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65387 0 0 0 94224 8 0 0 25 0 1 0 565195573 271626240 58939 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206988 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66315 58939 104 5 0 64786 0
Current children cumulated CPU time (s) 942.32
Current children cumulated vsize (KiB) 265260

[startup+1002.3 s]
/proc/loadavg: 4.01 3.94 3.54 5/190 29726
/proc/meminfo: memFree=16301304/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=265796 CPUtime=1002.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65537 0 0 0 100224 8 0 0 25 0 1 0 565195573 272175104 59089 33554432000 4194304 4210924 140736950384064 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66449 59089 104 5 0 64920 0
Current children cumulated CPU time (s) 1002.32
Current children cumulated vsize (KiB) 265796

[startup+1062.3 s]
/proc/loadavg: 4.07 3.96 3.57 5/190 29727
/proc/meminfo: memFree=16299284/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=267008 CPUtime=1062.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65827 0 0 0 106224 8 0 0 25 0 1 0 565195573 273416192 59379 33554432000 4194304 4210924 140736950384064 18446744073709551615 4198621 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66752 59379 104 5 0 65223 0
Current children cumulated CPU time (s) 1062.32
Current children cumulated vsize (KiB) 267008

[startup+1122.3 s]
/proc/loadavg: 4.02 3.97 3.59 5/190 29729
/proc/meminfo: memFree=16297764/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=267004 CPUtime=1122.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65827 0 0 0 112224 8 0 0 25 0 1 0 565195573 273412096 59379 33554432000 4194304 4210924 140736950384064 18446744073709551615 4204561 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66751 59379 104 5 0 65222 0
Current children cumulated CPU time (s) 1122.32
Current children cumulated vsize (KiB) 267004

[startup+1182.3 s]
/proc/loadavg: 4.06 3.99 3.62 5/190 29730
/proc/meminfo: memFree=16296992/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=267004 CPUtime=1182.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65828 0 0 0 118224 8 0 0 25 0 1 0 565195573 273412096 59380 33554432000 4194304 4210924 140736950384064 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66751 59380 104 5 0 65222 0
Current children cumulated CPU time (s) 1182.32
Current children cumulated vsize (KiB) 267004

[startup+1242.3 s]
/proc/loadavg: 4.02 3.99 3.64 5/190 29741
/proc/meminfo: memFree=16294912/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=267400 CPUtime=1242.32 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65940 0 0 0 124224 8 0 0 25 0 1 0 565195573 273817600 59492 33554432000 4194304 4210924 140736950384064 18446744073709551615 4206489 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66850 59492 104 5 0 65321 0
Current children cumulated CPU time (s) 1242.32
Current children cumulated vsize (KiB) 267400



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 4.01 3.99 3.66 4/188 29744
/proc/meminfo: memFree=16368156/32951124 swapFree=67111528/67111528
[pid=29686] ppid=29684 vsize=267400 CPUtime=1300.03 cores=0,2
/proc/29686/stat : 29686 (saturn) R 29684 29686 29242 0 -1 4202496 65940 0 0 0 129995 8 0 0 25 0 1 0 565195573 273817600 59492 33554432000 4194304 4210924 140736950384064 18446744073709551615 4198674 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29686/statm: 66850 59492 104 5 0 65321 0
Current children cumulated CPU time (s) 1300.03
Current children cumulated vsize (KiB) 267400

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1300.02
CPU time (s): 1300.06
CPU user time (s): 1299.96
CPU system time (s): 0.101984
CPU usage (%): 100.003
Max. virtual memory (cumulated for all children) (KiB): 267400

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1299.96
system time used= 0.101984
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 65940
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3
involuntary context switches= 2434

runsolver used 2.28065 second user time and 4.82827 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-25 18:40:50
IDJOB=3280637
IDBENCH=20410
IDSOLVER=1583
FILE ID=node110/3280637-1303749649
RUNJOBID= node110-1303749648-29656
PBS_JOBID= 13154052
Free space on /tmp= 72220 MiB

SOLVER NAME= Black Hole SAT 2011-03-08
BENCH NAME= SAT07/industrial/palacios/hard/sortnet-8-ipc5-h19-sat.cnf
COMMAND LINE= saturn BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3280637-1303749649/watcher-3280637-1303749649 -o /tmp/evaluation-result-3280637-1303749649/solver-3280637-1303749649 -C 2400 -W 1300 -M 7680  saturn HOME/instance-3280637-1303749649.cnf

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 7680 MiB
NBCORE= 2

MD5SUM BENCH= 409f4401e9e8b60ee665ac5567a13e41
RANDOM SEED=1666623873

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16982812 kB
Buffers:       2323204 kB
Cached:       12939336 kB
SwapCached:          0 kB
Active:        8091520 kB
Inactive:      7308140 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16982812 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           69428 kB
Writeback:           0 kB
AnonPages:      135980 kB
Mapped:          14880 kB
Slab:           505636 kB
PageTables:       5044 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   291388 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72224 MiB
End job on node110 at 2011-04-25 19:02:32