Trace number 302637

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
CMUSAT 2007-02-08? (TO) 1200.11 1200.46

General information on the benchmark

Namerandom/LargeSize/7SAT/v240/
unif-k7-r85-v240-c20400-S788797920-13.cnf
MD5SUM1a3e4a32f1ced60c9a6caa04c9b79230
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables240
Number of clauses20400
Sum of the clauses size142800
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 520400

Solver Data (download as text)

0.00/0.00	c Pre-processing provided 0 assignments
0.00/0.00	c |  Number of variables:  240                                                  |
0.00/0.03	c |  Number of clauses:    20400                                                |

Verifier Data (download as text)

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer:

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node11/watcher-302637-1172016227 -o ROOT/results/node11/solver-302637-1172016227 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/302637-1172016227/cmusat /tmp/evaluation/302637-1172016227/instance-302637-1172016227.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 1267
/proc/meminfo: memFree=1623240/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=2316 CPUtime=0
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 324 0 0 0 0 0 0 0 18 0 1 0 221577245 2371584 307 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135092575 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 579 308 96 243 0 333 0

[startup+0.107499 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 1267
/proc/meminfo: memFree=1623240/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=4196 CPUtime=0.1
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 805 0 0 0 10 0 0 0 18 0 1 0 221577245 4296704 788 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134530328 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 1049 788 107 243 0 803 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 4196

[startup+0.515546 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 1267
/proc/meminfo: memFree=1623240/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=4360 CPUtime=0.5
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 852 0 0 0 50 0 0 0 21 0 1 0 221577245 4464640 835 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134530504 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 1090 835 108 243 0 844 0
Current children cumulated CPU time (s) 0.5
Current children cumulated vsize (KiB) 4360

[startup+1.34166 s]
/proc/loadavg: 0.92 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1618624/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=6948 CPUtime=1.33
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 2079 0 0 0 133 0 0 0 25 0 1 0 221577245 7114752 1389 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135092575 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 1737 1389 118 243 0 1491 0
Current children cumulated CPU time (s) 1.33
Current children cumulated vsize (KiB) 6948

[startup+2.97784 s]
/proc/loadavg: 0.92 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1615872/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=9300 CPUtime=2.96
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 2869 0 0 0 295 1 0 0 25 0 1 0 221577245 9523200 2010 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134581522 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 2325 2010 118 243 0 2079 0
Current children cumulated CPU time (s) 2.96
Current children cumulated vsize (KiB) 9300

[startup+6.25422 s]
/proc/loadavg: 0.93 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1614272/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=10636 CPUtime=6.24
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 3343 0 0 0 622 2 0 0 25 0 1 0 221577245 10891264 2363 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135092575 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 2659 2363 118 243 0 2413 0
Current children cumulated CPU time (s) 6.24
Current children cumulated vsize (KiB) 10636

[startup+12.71 s]
/proc/loadavg: 0.93 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1611840/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=12524 CPUtime=12.7
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 4124 0 0 0 1268 2 0 0 25 0 1 0 221577245 12824576 2875 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 3131 2875 118 243 0 2885 0
Current children cumulated CPU time (s) 12.7
Current children cumulated vsize (KiB) 12524

[startup+25.5105 s]
/proc/loadavg: 0.95 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1603968/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=21244 CPUtime=25.49
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 6507 0 0 0 2546 3 0 0 25 0 1 0 221577245 21753856 4781 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589650 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 5311 4781 118 243 0 5065 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 21244

[startup+51.1235 s]
/proc/loadavg: 0.96 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1602240/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=22884 CPUtime=51.09
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 7210 0 0 0 5105 4 0 0 25 0 1 0 221577245 23433216 5283 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589715 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 5721 5283 118 243 0 5475 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 22884

[startup+102.341 s]
/proc/loadavg: 0.98 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1593600/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=31056 CPUtime=102.3
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 9851 0 0 0 10223 7 0 0 25 0 1 0 221577245 31801344 7427 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 7764 7427 118 243 0 7518 0
Current children cumulated CPU time (s) 102.3
Current children cumulated vsize (KiB) 31056

[startup+162.372 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1573056/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=53608 CPUtime=162.31
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 16083 0 0 0 16220 11 0 0 25 0 1 0 221577245 54894592 12548 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 13402 12548 118 243 0 13156 0
Current children cumulated CPU time (s) 162.31
Current children cumulated vsize (KiB) 53608

[startup+222.404 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1572736/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=53664 CPUtime=222.32
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 16335 0 0 0 22221 11 0 0 25 0 1 0 221577245 54951936 12619 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134590210 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 13416 12619 118 243 0 13170 0
Current children cumulated CPU time (s) 222.32
Current children cumulated vsize (KiB) 53664

[startup+282.431 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1572416/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=53724 CPUtime=282.33
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 16612 0 0 0 28221 12 0 0 25 0 1 0 221577245 55013376 12701 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589515 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 13431 12701 118 243 0 13185 0
Current children cumulated CPU time (s) 282.33
Current children cumulated vsize (KiB) 53724

[startup+342.461 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1572032/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=53792 CPUtime=342.35
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 16902 0 0 0 34222 13 0 0 25 0 1 0 221577245 55083008 12781 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 13448 12781 118 243 0 13202 0
Current children cumulated CPU time (s) 342.35
Current children cumulated vsize (KiB) 53792

[startup+402.497 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1564480/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=60972 CPUtime=402.37
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 19025 0 0 0 40221 16 0 0 25 0 1 0 221577245 62435328 14677 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 15243 14677 118 243 0 14997 0
Current children cumulated CPU time (s) 402.37
Current children cumulated vsize (KiB) 60972

[startup+462.53 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1563264/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=62212 CPUtime=462.39
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 19559 0 0 0 46222 17 0 0 25 0 1 0 221577245 63705088 14965 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134590277 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 15553 14965 118 243 0 15307 0
Current children cumulated CPU time (s) 462.39
Current children cumulated vsize (KiB) 62212

[startup+522.559 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1557312/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=67736 CPUtime=522.39
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 21309 0 0 0 52220 19 0 0 25 0 1 0 221577245 69361664 16448 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589771 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 16934 16448 118 243 0 16688 0
Current children cumulated CPU time (s) 522.39
Current children cumulated vsize (KiB) 67736

[startup+582.591 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 1268
/proc/meminfo: memFree=1554752/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=70292 CPUtime=582.41
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 21958 0 0 0 58221 20 0 0 25 0 1 0 221577245 71979008 17097 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135092575 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 17573 17097 118 243 0 17327 0
Current children cumulated CPU time (s) 582.41
Current children cumulated vsize (KiB) 70292

[startup+642.626 s]
/proc/loadavg: 1.05 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1551168/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=73688 CPUtime=642.43
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 23139 0 0 0 64222 21 0 0 25 0 1 0 221577245 75456512 17989 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589593 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 18422 17989 118 243 0 18176 0
Current children cumulated CPU time (s) 642.43
Current children cumulated vsize (KiB) 73688

[startup+702.657 s]
/proc/loadavg: 1.02 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1511488/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=117840 CPUtime=702.44
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 34622 0 0 0 70218 26 0 0 25 0 1 0 221577245 120668160 27881 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589593 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29460 27881 118 243 0 29214 0
Current children cumulated CPU time (s) 702.44
Current children cumulated vsize (KiB) 117840

[startup+762.692 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1511616/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=117840 CPUtime=762.46
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 34623 0 0 0 76219 27 0 0 25 0 1 0 221577245 120668160 27882 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589771 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29460 27882 118 243 0 29214 0
Current children cumulated CPU time (s) 762.46
Current children cumulated vsize (KiB) 117840

[startup+822.728 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1511040/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=117960 CPUtime=822.47
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35115 0 0 0 82219 28 0 0 25 0 1 0 221577245 120791040 28032 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589670 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29490 28032 118 243 0 29244 0
Current children cumulated CPU time (s) 822.47
Current children cumulated vsize (KiB) 117960

[startup+882.762 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1511040/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=117960 CPUtime=882.49
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35115 0 0 0 88220 29 0 0 25 0 1 0 221577245 120791040 28032 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589690 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29490 28032 118 243 0 29244 0
Current children cumulated CPU time (s) 882.49
Current children cumulated vsize (KiB) 117960

[startup+942.791 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1510976/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=117960 CPUtime=942.5
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35115 0 0 0 94221 29 0 0 25 0 1 0 221577245 120791040 28032 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589506 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29490 28032 118 243 0 29244 0
Current children cumulated CPU time (s) 942.5
Current children cumulated vsize (KiB) 117960

[startup+1002.82 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1511040/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=117960 CPUtime=1002.52
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35156 0 0 0 100222 30 0 0 25 0 1 0 221577245 120791040 28073 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29490 28073 118 243 0 29244 0
Current children cumulated CPU time (s) 1002.52
Current children cumulated vsize (KiB) 117960

[startup+1062.85 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1510464/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=118092 CPUtime=1062.52
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35653 0 0 0 106222 30 0 0 25 0 1 0 221577245 120926208 28198 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29523 28198 118 243 0 29277 0
Current children cumulated CPU time (s) 1062.52
Current children cumulated vsize (KiB) 118092

[startup+1122.88 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1510272/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=118352 CPUtime=1122.54
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35695 0 0 0 112223 31 0 0 25 0 1 0 221577245 121192448 28240 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134590210 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29588 28240 118 243 0 29342 0
Current children cumulated CPU time (s) 1122.54
Current children cumulated vsize (KiB) 118352

[startup+1182.92 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1510208/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=118352 CPUtime=1182.55
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35703 0 0 0 118224 31 0 0 25 0 1 0 221577245 121192448 28248 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29588 28248 118 243 0 29342 0
Current children cumulated CPU time (s) 1182.55
Current children cumulated vsize (KiB) 118352



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1200.44 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1510144/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=118536 CPUtime=1200.07
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35715 0 0 0 119976 31 0 0 25 0 1 0 221577245 121380864 28260 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589650 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29634 28260 118 243 0 29388 0
Current children cumulated CPU time (s) 1200.07
Current children cumulated vsize (KiB) 118536

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1200.44 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 1268
/proc/meminfo: memFree=1510144/2055920 swapFree=4192812/4192956
[pid=1267] ppid=1265 vsize=118536 CPUtime=1200.07
/proc/1267/stat : 1267 (cmusat) R 1265 1267 843 0 -1 4194304 35715 0 0 0 119976 31 0 0 25 0 1 0 221577245 121380864 28260 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589650 0 0 4096 35 0 0 0 17 1 0 0
/proc/1267/statm: 29634 28260 118 243 0 29388 0
Current children cumulated CPU time (s) 1200.07
Current children cumulated vsize (KiB) 118536

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1200.46
CPU time (s): 1200.11
CPU user time (s): 1199.77
CPU system time (s): 0.338948
CPU usage (%): 99.9704
Max. virtual memory (cumulated for all children) (KiB): 119004

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1199.77
system time used= 0.338948
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 35715
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6
involuntary context switches= 2392

runsolver used 1.40379 second user time and 2.77958 second system time

The end

Launcher Data (download as text)

Begin job on node11 on Wed Feb 21 00:03:47 UTC 2007

IDJOB= 302637
IDBENCH= 20088
IDSOLVER= 121
FILE ID= node11/302637-1172016227

PBS_JOBID= 3849156

Free space on /tmp= 66542 MiB

SOLVER NAME= CMUSAT 2007-02-08
BENCH NAME= HOME/pub/bench/SAT07/random/LargeSize/7SAT/v240/unif-k7-r85-v240-c20400-S788797920-13.cnf
COMMAND LINE= /tmp/evaluation/302637-1172016227/cmusat /tmp/evaluation/302637-1172016227/instance-302637-1172016227.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node11/watcher-302637-1172016227 -o ROOT/results/node11/solver-302637-1172016227 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/302637-1172016227/cmusat /tmp/evaluation/302637-1172016227/instance-302637-1172016227.cnf            

META MD5SUM SOLVER= af990fb12f8926fdd756b397adb3b5cf
MD5SUM BENCH=  1a3e4a32f1ced60c9a6caa04c9b79230

RANDOM SEED= 896805897

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node11.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.243
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.243
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1623648 kB
Buffers:         55512 kB
Cached:         261776 kB
SwapCached:          0 kB
Active:         169024 kB
Inactive:       191488 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1623648 kB
SwapTotal:     4192956 kB
SwapFree:      4192812 kB
Dirty:            3080 kB
Writeback:           0 kB
Mapped:          62132 kB
Slab:            57336 kB
Committed_AS:   715204 kB
PageTables:       1756 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66542 MiB

End job on node11 on Wed Feb 21 00:23:49 UTC 2007