Trace number 3282937

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Black Hole SAT 2011-03-08? (TO) 1300.04 1300.01

General information on the benchmark

NameAPPLICATIONS/bioinfo/
q_query_3_l46_lambda.cnf
MD5SUMeb4d0d02a25a8a84182ddd3b6d1fa8b5
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark84.3072
Satisfiable
(Un)Satisfiability was proved
Number of variables33090
Number of clauses165828
Sum of the clauses size478271
Maximum clause length9
Minimum clause length1
Number of clauses of size 14268
Number of clauses of size 241089
Number of clauses of size 397665
Number of clauses of size 415752
Number of clauses of size 56916
Number of clauses of size over 5138

Solver Data 

0.18/0.21	c RESTART
0.28/0.31	c RESTART
0.28/0.38	c RESTART
0.48/0.54	c RESTART
0.58/0.67	c RESTART
0.68/0.74	c RESTART
0.78/0.82	c RESTART
0.88/0.91	c RESTART
1.28/1.37	c RESTART
1.38/1.45	c RESTART
1.48/1.56	c RESTART
1.68/1.70	c RESTART
1.98/2.02	c RESTART
2.09/2.14	c RESTART
2.19/2.26	c RESTART
2.28/2.38	c RESTART
2.48/2.50	c RESTART
2.58/2.61	c RESTART
15.18/15.26	c RESTART
18.18/18.20	c RESTART
18.28/18.33	c RESTART
18.38/18.46	c RESTART
21.28/21.33	c RESTART
21.48/21.55	c RESTART
21.68/21.72	c RESTART
21.89/21.92	c RESTART
21.99/22.07	c RESTART
22.28/22.32	c RESTART
22.38/22.49	c RESTART
22.58/22.65	c RESTART
22.78/22.81	c RESTART
22.88/22.96	c RESTART
22.99/23.07	c RESTART
23.18/23.22	c RESTART
78.69/78.71	c RESTART
78.79/78.87	c RESTART
78.98/79.03	c RESTART
81.49/81.51	c RESTART
81.59/81.69	c RESTART
81.79/81.89	c RESTART
100.69/100.74	c RESTART
100.89/100.93	c RESTART
100.99/101.09	c RESTART
126.19/126.20	c RESTART
128.49/128.56	c RESTART
128.69/128.73	c RESTART
128.79/128.87	c RESTART
132.99/133.08	c RESTART
143.69/143.76	c RESTART
144.39/144.42	c RESTART
144.49/144.52	c RESTART
144.59/144.66	c RESTART
144.69/144.77	c RESTART
144.89/144.92	c RESTART
145.59/145.64	c RESTART
146.49/146.55	c RESTART
146.69/146.72	c RESTART
149.19/149.25	c RESTART
279.99/280.06	c RESTART
310.89/310.94	c RESTART
310.99/311.07	c RESTART
311.79/311.85	c RESTART
323.79/323.89	c RESTART
327.89/327.96	c RESTART
413.90/413.91	c RESTART
623.31/623.34	c RESTART
625.91/625.95	c RESTART
626.10/626.14	c RESTART
626.20/626.26	c RESTART
626.40/626.46	c RESTART
626.50/626.59	c RESTART
627.10/627.10	c RESTART
638.10/638.18	c RESTART
638.30/638.34	c RESTART
638.80/638.84	c RESTART
639.01/639.09	c RESTART
665.71/665.77	c RESTART
667.31/667.32	c RESTART
667.51/667.56	c RESTART
720.01/720.07	c RESTART
720.21/720.25	c RESTART
720.41/720.44	c RESTART
720.61/720.60	c RESTART
720.71/720.78	c RESTART
721.21/721.20	c RESTART
721.31/721.34	c RESTART
721.61/721.62	c RESTART
760.41/760.44	c RESTART
760.51/760.58	c RESTART
760.71/760.74	c RESTART
760.81/760.86	c RESTART
824.11/824.16	c RESTART
824.31/824.39	c RESTART
824.51/824.55	c RESTART
824.81/824.85	c RESTART
824.92/824.98	c RESTART
825.11/825.12	c RESTART
825.91/825.92	c RESTART
826.02/826.10	c RESTART
826.21/826.23	c RESTART
826.41/826.41	c RESTART
989.62/989.63	c RESTART
993.02/993.02	c RESTART
993.22/993.20	c RESTART
993.42/993.40	c RESTART
993.52/993.54	c RESTART
1037.42/1037.43	c RESTART
1039.02/1039.03	c RESTART
1039.22/1039.23	c RESTART
1039.32/1039.34	c RESTART
1039.52/1039.53	c RESTART
1039.72/1039.73	c RESTART
1039.92/1039.96	c RESTART
1040.12/1040.14	c RESTART
1041.12/1041.19	c RESTART
1041.32/1041.34	c RESTART
1044.52/1044.53	c RESTART
1044.72/1044.72	c RESTART
1044.82/1044.87	c RESTART
1045.12/1045.10	c RESTART
1045.32/1045.31	c RESTART
1045.82/1045.88	c RESTART
1046.02/1046.06	c RESTART
1046.22/1046.22	c RESTART
1125.82/1125.88	c RESTART
1136.42/1136.41	c RESTART
1155.22/1155.23	c RESTART
1155.42/1155.40	c RESTART
1155.62/1155.62	c RESTART
1159.52/1159.52	c RESTART
1160.92/1160.99	c RESTART
1161.22/1161.22	c RESTART

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3282937-1303762402/watcher-3282937-1303762402 -o /tmp/evaluation-result-3282937-1303762402/solver-3282937-1303762402 -C 2400 -W 1300 -M 7680 saturn HOME/instance-3282937-1303762402.cnf 

running on 2 cores: 0,2

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 7864320 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 7915520 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.68 3.91 3.66 5/186 8711
/proc/meminfo: memFree=27361180/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=11800 CPUtime=0 cores=0,2
/proc/8711/stat : 8711 (runsolver) R 8709 8711 8502 0 -1 4202560 41 0 0 0 0 0 0 0 25 0 1 0 468656128 12083200 52 33554432000 4194304 5434984 140734584333344 18446744073709551615 4881607 0 2147483391 4096 24578 0 0 0 17 2 0 0 0
/proc/8711/statm: 2950 52 31 303 0 2644 0

[startup+0.0680059 s]
/proc/loadavg: 3.68 3.91 3.66 5/186 8711
/proc/meminfo: memFree=27361180/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=32780 CPUtime=0.05 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 4994 0 0 0 5 0 0 0 25 0 1 0 468656128 33566720 4898 33554432000 4194304 4210924 140734920246080 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 8195 4901 521 5 0 5904 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 32780

[startup+0.101017 s]
/proc/loadavg: 3.68 3.91 3.66 5/186 8711
/proc/meminfo: memFree=27361180/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37532 CPUtime=0.08 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 6965 0 0 0 8 0 0 0 25 0 1 0 468656128 38432768 6859 33554432000 4194304 4210924 140734920246080 18446744073709551615 220815897285 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9383 6862 775 5 0 7092 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 37532

[startup+0.300982 s]
/proc/loadavg: 3.68 3.91 3.66 5/186 8711
/proc/meminfo: memFree=27361180/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=36596 CPUtime=0.28 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 7807 0 0 0 28 0 0 0 25 0 1 0 468656128 37474304 6924 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9149 6924 101 5 0 7620 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 36596

[startup+0.70089 s]
/proc/loadavg: 3.68 3.91 3.66 5/186 8711
/proc/meminfo: memFree=27361180/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=36596 CPUtime=0.68 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 7813 0 0 0 68 0 0 0 25 0 1 0 468656128 37474304 6930 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198629 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9149 6930 102 5 0 7620 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 36596

[startup+1.50076 s]
/proc/loadavg: 3.68 3.91 3.66 5/190 8718
/proc/meminfo: memFree=27258980/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=36728 CPUtime=1.48 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 7838 0 0 0 148 0 0 0 25 0 1 0 468656128 37609472 6955 33554432000 4194304 4210924 140734920246080 18446744073709551615 4207055 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9182 6955 102 5 0 7653 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 36728

[startup+3.10046 s]
/proc/loadavg: 3.68 3.91 3.66 5/190 8718
/proc/meminfo: memFree=27257484/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37124 CPUtime=3.08 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 7948 0 0 0 308 0 0 0 25 0 1 0 468656128 38014976 7065 33554432000 4194304 4210924 140734920246080 18446744073709551615 4201682 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9281 7065 102 5 0 7752 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 37124

[startup+6.30089 s]
/proc/loadavg: 3.79 3.93 3.66 5/190 8719
/proc/meminfo: memFree=27256728/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37124 CPUtime=6.28 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 7971 0 0 0 628 0 0 0 25 0 1 0 468656128 38014976 7088 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9281 7088 102 5 0 7752 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 37124

[startup+12.7007 s]
/proc/loadavg: 3.80 3.93 3.66 5/190 8719
/proc/meminfo: memFree=27255660/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37124 CPUtime=12.68 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 7971 0 0 0 1268 0 0 0 25 0 1 0 468656128 38014976 7088 33554432000 4194304 4210924 140734920246080 18446744073709551615 4200055 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9281 7088 102 5 0 7752 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 37124

[startup+25.5003 s]
/proc/loadavg: 3.85 3.93 3.67 5/190 8719
/proc/meminfo: memFree=27253292/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37256 CPUtime=25.48 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8004 0 0 0 2548 0 0 0 25 0 1 0 468656128 38150144 7121 33554432000 4194304 4210924 140734920246080 18446744073709551615 4205022 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9314 7121 102 5 0 7785 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 37256

[startup+51.1006 s]
/proc/loadavg: 3.96 3.95 3.68 5/190 8720
/proc/meminfo: memFree=27253064/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37256 CPUtime=51.08 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8004 0 0 0 5108 0 0 0 25 0 1 0 468656128 38150144 7121 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198612 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9314 7121 102 5 0 7785 0
Current children cumulated CPU time (s) 51.08
Current children cumulated vsize (KiB) 37256

[startup+102.307 s]
/proc/loadavg: 3.98 3.95 3.70 5/190 8721
/proc/meminfo: memFree=27251168/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37388 CPUtime=102.29 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8010 0 0 0 10229 0 0 0 25 0 1 0 468656128 38285312 7127 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198763 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9347 7127 102 5 0 7818 0
Current children cumulated CPU time (s) 102.29
Current children cumulated vsize (KiB) 37388

[startup+162.301 s]
/proc/loadavg: 3.99 3.96 3.71 5/190 8723
/proc/meminfo: memFree=27250244/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=162.29 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 16229 0 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4201755 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 162.29
Current children cumulated vsize (KiB) 37520

[startup+222.301 s]
/proc/loadavg: 3.99 3.97 3.73 5/190 8724
/proc/meminfo: memFree=27249592/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=222.29 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 22229 0 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204576 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 222.29
Current children cumulated vsize (KiB) 37520

[startup+282.301 s]
/proc/loadavg: 3.99 3.97 3.74 5/190 8726
/proc/meminfo: memFree=27249548/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=282.29 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 28229 0 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 282.29
Current children cumulated vsize (KiB) 37520

[startup+342.301 s]
/proc/loadavg: 3.99 3.97 3.75 5/190 8728
/proc/meminfo: memFree=27249528/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=342.29 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 34229 0 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198761 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 342.29
Current children cumulated vsize (KiB) 37520

[startup+402.301 s]
/proc/loadavg: 4.09 4.00 3.78 5/190 8729
/proc/meminfo: memFree=27249352/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=402.29 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 40229 0 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 402.29
Current children cumulated vsize (KiB) 37520

[startup+462.301 s]
/proc/loadavg: 4.11 4.02 3.79 5/190 8731
/proc/meminfo: memFree=27249424/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=462.3 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 46229 1 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4206973 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 462.3
Current children cumulated vsize (KiB) 37520

[startup+522.301 s]
/proc/loadavg: 4.04 4.01 3.80 5/190 8732
/proc/meminfo: memFree=27249132/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=522.3 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 52229 1 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 522.3
Current children cumulated vsize (KiB) 37520

[startup+582.301 s]
/proc/loadavg: 4.01 4.00 3.82 5/190 8734
/proc/meminfo: memFree=27249220/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37520 CPUtime=582.3 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8072 0 0 0 58229 1 0 0 25 0 1 0 468656128 38420480 7189 33554432000 4194304 4210924 140734920246080 18446744073709551615 4205022 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9380 7189 103 5 0 7851 0
Current children cumulated CPU time (s) 582.3
Current children cumulated vsize (KiB) 37520

[startup+642.301 s]
/proc/loadavg: 4.00 4.00 3.82 5/190 8736
/proc/meminfo: memFree=27248812/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37652 CPUtime=642.31 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8105 0 0 0 64230 1 0 0 25 0 1 0 468656128 38555648 7222 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9413 7222 103 5 0 7884 0
Current children cumulated CPU time (s) 642.31
Current children cumulated vsize (KiB) 37652

[startup+702.301 s]
/proc/loadavg: 4.00 4.00 3.83 5/190 8740
/proc/meminfo: memFree=27248028/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37784 CPUtime=702.31 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8139 0 0 0 70230 1 0 0 25 0 1 0 468656128 38690816 7256 33554432000 4194304 4210924 140734920246080 18446744073709551615 4206988 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9446 7256 103 5 0 7917 0
Current children cumulated CPU time (s) 702.31
Current children cumulated vsize (KiB) 37784

[startup+762.307 s]
/proc/loadavg: 4.00 4.00 3.83 5/190 8742
/proc/meminfo: memFree=27247948/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37784 CPUtime=762.31 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8139 0 0 0 76230 1 0 0 25 0 1 0 468656128 38690816 7256 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9446 7256 103 5 0 7917 0
Current children cumulated CPU time (s) 762.31
Current children cumulated vsize (KiB) 37784

[startup+822.3 s]
/proc/loadavg: 4.00 4.00 3.84 5/190 8743
/proc/meminfo: memFree=27248044/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=37916 CPUtime=822.31 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8172 0 0 0 82230 1 0 0 25 0 1 0 468656128 38825984 7289 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204583 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9479 7289 103 5 0 7950 0
Current children cumulated CPU time (s) 822.31
Current children cumulated vsize (KiB) 37916

[startup+882.3 s]
/proc/loadavg: 4.00 4.00 3.85 5/190 8745
/proc/meminfo: memFree=27248116/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38048 CPUtime=882.31 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8205 0 0 0 88230 1 0 0 25 0 1 0 468656128 38961152 7322 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198616 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9512 7322 103 5 0 7983 0
Current children cumulated CPU time (s) 882.31
Current children cumulated vsize (KiB) 38048

[startup+942.3 s]
/proc/loadavg: 4.00 4.00 3.85 5/190 8747
/proc/meminfo: memFree=27248464/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38048 CPUtime=942.31 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8205 0 0 0 94230 1 0 0 25 0 1 0 468656128 38961152 7322 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198726 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9512 7322 103 5 0 7983 0
Current children cumulated CPU time (s) 942.31
Current children cumulated vsize (KiB) 38048

[startup+1002.3 s]
/proc/loadavg: 4.00 4.00 3.86 5/190 8748
/proc/meminfo: memFree=27248792/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38048 CPUtime=1002.32 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8205 0 0 0 100231 1 0 0 25 0 1 0 468656128 38961152 7322 33554432000 4194304 4210924 140734920246080 18446744073709551615 4206988 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9512 7322 103 5 0 7983 0
Current children cumulated CPU time (s) 1002.32
Current children cumulated vsize (KiB) 38048

[startup+1062.3 s]
/proc/loadavg: 4.04 4.01 3.87 5/190 8750
/proc/meminfo: memFree=27248388/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38180 CPUtime=1062.32 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8239 0 0 0 106230 2 0 0 25 0 1 0 468656128 39096320 7356 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204753 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9545 7356 103 5 0 8016 0
Current children cumulated CPU time (s) 1062.32
Current children cumulated vsize (KiB) 38180

[startup+1122.3 s]
/proc/loadavg: 4.07 4.02 3.88 5/190 8751
/proc/meminfo: memFree=27248356/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38180 CPUtime=1122.32 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8239 0 0 0 112230 2 0 0 25 0 1 0 468656128 39096320 7356 33554432000 4194304 4210924 140734920246080 18446744073709551615 4198608 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9545 7356 103 5 0 8016 0
Current children cumulated CPU time (s) 1122.32
Current children cumulated vsize (KiB) 38180

[startup+1182.3 s]
/proc/loadavg: 4.02 4.02 3.89 5/190 8753
/proc/meminfo: memFree=27248288/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38312 CPUtime=1182.33 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8272 0 0 0 118231 2 0 0 25 0 1 0 468656128 39231488 7389 33554432000 4194304 4210924 140734920246080 18446744073709551615 4207045 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9578 7389 103 5 0 8049 0
Current children cumulated CPU time (s) 1182.33
Current children cumulated vsize (KiB) 38312

[startup+1242.3 s]
/proc/loadavg: 4.01 4.01 3.89 5/190 8755
/proc/meminfo: memFree=27247264/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38312 CPUtime=1242.33 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8272 0 0 0 124231 2 0 0 25 0 1 0 468656128 39231488 7389 33554432000 4194304 4210924 140734920246080 18446744073709551615 4202057 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9578 7389 103 5 0 8049 0
Current children cumulated CPU time (s) 1242.33
Current children cumulated vsize (KiB) 38312



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 4.00 4.01 3.90 3/188 8758
/proc/meminfo: memFree=27283820/32951124 swapFree=67111524/67111528
[pid=8711] ppid=8709 vsize=38312 CPUtime=1300.03 cores=0,2
/proc/8711/stat : 8711 (saturn) R 8709 8711 8502 0 -1 4202496 8272 0 0 0 130001 2 0 0 25 0 1 0 468656128 39231488 7389 33554432000 4194304 4210924 140734920246080 18446744073709551615 4204579 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/8711/statm: 9578 7389 103 5 0 8049 0
Current children cumulated CPU time (s) 1300.03
Current children cumulated vsize (KiB) 38312

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1300.01
CPU time (s): 1300.04
CPU user time (s): 1300.02
CPU system time (s): 0.022996
CPU usage (%): 100.003
Max. virtual memory (cumulated for all children) (KiB): 38312

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1300.02
system time used= 0.022996
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 8272
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3
involuntary context switches= 4802

runsolver used 2.26066 second user time and 4.85326 second system time

The end

Launcher Data 

Begin job on node147 at 2011-04-25 22:13:22
IDJOB=3282937
IDBENCH=70941
IDSOLVER=1583
FILE ID=node147/3282937-1303762402
RUNJOBID= node147-1303761099-8520
PBS_JOBID= 13157150
Free space on /tmp= 72164 MiB

SOLVER NAME= Black Hole SAT 2011-03-08
BENCH NAME= SAT09/APPLICATIONS/bioinfo/q_query_3_l46_lambda.cnf
COMMAND LINE= saturn BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3282937-1303762402/watcher-3282937-1303762402 -o /tmp/evaluation-result-3282937-1303762402/solver-3282937-1303762402 -C 2400 -W 1300 -M 7680  saturn HOME/instance-3282937-1303762402.cnf

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 7680 MiB
NBCORE= 2

MD5SUM BENCH= eb4d0d02a25a8a84182ddd3b6d1fa8b5
RANDOM SEED=1263225798

node147.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27361584 kB
Buffers:        956836 kB
Cached:        4084508 kB
SwapCached:          4 kB
Active:         607236 kB
Inactive:      4537744 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27361584 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            7864 kB
Writeback:           0 kB
AnonPages:      103912 kB
Mapped:          15464 kB
Slab:           379936 kB
PageTables:       5172 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   271108 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72160 MiB
End job on node147 at 2011-04-25 22:35:04