Trace number 298373

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
CMUSAT 2007-02-08? (TO) 1200.04 1200.31

General information on the benchmark

Namerandom/OnTreshold/3SAT/v500/
unif-k3-r4.26-v500-c2130-S1211740352-20.UNSAT.shuffled.cnf
MD5SUM49141385aac03f506711f7c91a24b9ac
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data (download as text)

0.00/0.00	c Pre-processing provided 0 assignments
0.00/0.00	c |  Number of variables:  500                                                  |
0.00/0.00	c |  Number of clauses:    2130                                                 |

Verifier Data (download as text)

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer:

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node22/watcher-298373-1171854915 -o ROOT/results/node22/solver-298373-1171854915 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/298373-1171854915/cmusat /tmp/evaluation/298373-1171854915/instance-298373-1171854915.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.00 1.00 1.00 3/77 28252
/proc/meminfo: memFree=628968/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=2312 CPUtime=0
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 199 0 0 0 0 0 0 0 18 0 1 0 205445446 2367488 182 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134534648 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 578 182 109 243 0 332 0

[startup+0.106655 s]
/proc/loadavg: 1.00 1.00 1.00 3/77 28252
/proc/meminfo: memFree=628968/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=2768 CPUtime=0.1
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 307 0 0 0 10 0 0 0 18 0 1 0 205445446 2834432 290 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589663 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 692 290 118 243 0 446 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 2768

[startup+0.514685 s]
/proc/loadavg: 1.00 1.00 1.00 3/77 28252
/proc/meminfo: memFree=628968/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=3344 CPUtime=0.51
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 434 0 0 0 51 0 0 0 21 0 1 0 205445446 3424256 417 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589578 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 836 417 118 243 0 590 0
Current children cumulated CPU time (s) 0.51
Current children cumulated vsize (KiB) 3344

[startup+1.33475 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=627616/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=3624 CPUtime=1.33
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 506 0 0 0 133 0 0 0 25 0 1 0 205445446 3710976 489 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 906 489 118 243 0 660 0
Current children cumulated CPU time (s) 1.33
Current children cumulated vsize (KiB) 3624

[startup+2.97087 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=626848/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=5304 CPUtime=2.96
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 878 0 0 0 296 0 0 0 25 0 1 0 205445446 5431296 861 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589578 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 1326 861 118 243 0 1080 0
Current children cumulated CPU time (s) 2.96
Current children cumulated vsize (KiB) 5304

[startup+6.25311 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=625376/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=5700 CPUtime=6.24
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 1012 0 0 0 624 0 0 0 25 0 1 0 205445446 5836800 995 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589663 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 1425 995 118 243 0 1179 0
Current children cumulated CPU time (s) 6.24
Current children cumulated vsize (KiB) 5700

[startup+12.7036 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=622816/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=8264 CPUtime=12.69
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 1804 0 0 0 1269 0 0 0 25 0 1 0 205445446 8462336 1561 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589690 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 2066 1561 118 243 0 1820 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 8264

[startup+25.5095 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=620832/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=10056 CPUtime=25.49
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 2504 0 0 0 2548 1 0 0 25 0 1 0 205445446 10297344 2069 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589690 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 2514 2069 118 243 0 2268 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 10056

[startup+51.1224 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=616160/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=15192 CPUtime=51.1
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 4204 0 0 0 5108 2 0 0 25 0 1 0 205445446 15556608 3228 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 3798 3228 118 243 0 3552 0
Current children cumulated CPU time (s) 51.1
Current children cumulated vsize (KiB) 15192

[startup+102.34 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=612640/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=18492 CPUtime=102.31
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 5489 0 0 0 10227 4 0 0 25 0 1 0 205445446 18935808 4122 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 4623 4122 118 243 0 4377 0
Current children cumulated CPU time (s) 102.31
Current children cumulated vsize (KiB) 18492

[startup+162.375 s]
/proc/loadavg: 1.03 1.01 1.00 2/78 28253
/proc/meminfo: memFree=603680/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=28896 CPUtime=162.32
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 8640 0 0 0 16227 5 0 0 25 0 1 0 205445446 29589504 6338 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 7224 6338 118 243 0 6978 0
Current children cumulated CPU time (s) 162.32
Current children cumulated vsize (KiB) 28896

[startup+222.404 s]
/proc/loadavg: 1.01 1.01 1.00 2/78 28253
/proc/meminfo: memFree=603168/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=29380 CPUtime=222.34
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 8940 0 0 0 22228 6 0 0 25 0 1 0 205445446 30085120 6477 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134580032 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 7345 6477 118 243 0 7099 0
Current children cumulated CPU time (s) 222.34
Current children cumulated vsize (KiB) 29380

[startup+282.431 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=599776/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=32068 CPUtime=282.36
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 10142 0 0 0 28228 8 0 0 25 0 1 0 205445446 32837632 7309 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 8017 7309 118 243 0 7771 0
Current children cumulated CPU time (s) 282.36
Current children cumulated vsize (KiB) 32068

[startup+342.456 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=596640/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=34988 CPUtime=342.37
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 11131 0 0 0 34228 9 0 0 25 0 1 0 205445446 35827712 8085 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134590140 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 8747 8085 118 243 0 8501 0
Current children cumulated CPU time (s) 342.37
Current children cumulated vsize (KiB) 34988

[startup+402.488 s]
/proc/loadavg: 1.07 1.03 1.00 2/78 28253
/proc/meminfo: memFree=596000/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=35712 CPUtime=402.38
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 11297 0 0 0 40229 9 0 0 25 0 1 0 205445446 36569088 8251 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134590282 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 8928 8251 118 243 0 8682 0
Current children cumulated CPU time (s) 402.38
Current children cumulated vsize (KiB) 35712

[startup+462.519 s]
/proc/loadavg: 1.02 1.02 1.00 2/78 28253
/proc/meminfo: memFree=593376/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=37924 CPUtime=462.41
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 12178 0 0 0 46230 11 0 0 25 0 1 0 205445446 38834176 8899 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 9481 8899 118 243 0 9235 0
Current children cumulated CPU time (s) 462.41
Current children cumulated vsize (KiB) 37924

[startup+522.546 s]
/proc/loadavg: 1.04 1.03 1.00 2/78 28253
/proc/meminfo: memFree=577888/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=57264 CPUtime=522.41
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 17350 0 0 0 52228 13 0 0 25 0 1 0 205445446 58638336 12754 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14316 12754 118 243 0 14070 0
Current children cumulated CPU time (s) 522.41
Current children cumulated vsize (KiB) 57264

[startup+582.579 s]
/proc/loadavg: 1.01 1.02 1.00 2/78 28253
/proc/meminfo: memFree=577632/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=57616 CPUtime=582.44
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 17413 0 0 0 58230 14 0 0 25 0 1 0 205445446 58998784 12817 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134590140 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14404 12817 118 243 0 14158 0
Current children cumulated CPU time (s) 582.44
Current children cumulated vsize (KiB) 57616

[startup+642.611 s]
/proc/loadavg: 1.00 1.02 1.00 2/78 28253
/proc/meminfo: memFree=576928/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=57860 CPUtime=642.45
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 17872 0 0 0 64231 14 0 0 25 0 1 0 205445446 59248640 12995 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14465 12995 118 243 0 14219 0
Current children cumulated CPU time (s) 642.45
Current children cumulated vsize (KiB) 57860

[startup+702.638 s]
/proc/loadavg: 1.00 1.01 1.00 2/78 28253
/proc/meminfo: memFree=576480/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=58244 CPUtime=702.47
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 17978 0 0 0 70232 15 0 0 25 0 1 0 205445446 59641856 13101 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589684 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14561 13101 118 243 0 14315 0
Current children cumulated CPU time (s) 702.47
Current children cumulated vsize (KiB) 58244

[startup+762.669 s]
/proc/loadavg: 1.06 1.02 1.00 2/78 28253
/proc/meminfo: memFree=575520/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=59080 CPUtime=762.49
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 18553 0 0 0 76233 16 0 0 25 0 1 0 205445446 60497920 13334 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14770 13334 118 243 0 14524 0
Current children cumulated CPU time (s) 762.49
Current children cumulated vsize (KiB) 59080

[startup+822.704 s]
/proc/loadavg: 1.02 1.01 1.00 2/78 28253
/proc/meminfo: memFree=575584/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=59240 CPUtime=822.5
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 18593 0 0 0 82234 16 0 0 25 0 1 0 205445446 60661760 13374 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589771 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14810 13374 118 243 0 14564 0
Current children cumulated CPU time (s) 822.5
Current children cumulated vsize (KiB) 59240

[startup+882.734 s]
/proc/loadavg: 1.00 1.01 1.00 2/78 28253
/proc/meminfo: memFree=574560/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=59800 CPUtime=882.52
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19184 0 0 0 88235 17 0 0 25 0 1 0 205445446 61235200 13626 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14950 13626 118 243 0 14704 0
Current children cumulated CPU time (s) 882.52
Current children cumulated vsize (KiB) 59800

[startup+942.762 s]
/proc/loadavg: 1.07 1.02 1.00 2/78 28253
/proc/meminfo: memFree=574432/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=59932 CPUtime=942.54
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19211 0 0 0 94236 18 0 0 25 0 1 0 205445446 61370368 13653 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 14983 13653 118 243 0 14737 0
Current children cumulated CPU time (s) 942.54
Current children cumulated vsize (KiB) 59932

[startup+1002.79 s]
/proc/loadavg: 1.02 1.01 1.00 3/78 28253
/proc/meminfo: memFree=574304/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=60256 CPUtime=1002.55
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19244 0 0 0 100237 18 0 0 25 0 1 0 205445446 61702144 13686 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 15064 13686 118 243 0 14818 0
Current children cumulated CPU time (s) 1002.55
Current children cumulated vsize (KiB) 60256

[startup+1062.82 s]
/proc/loadavg: 1.01 1.01 1.00 2/78 28253
/proc/meminfo: memFree=573600/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=60708 CPUtime=1062.58
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19446 0 0 0 106239 19 0 0 25 0 1 0 205445446 62164992 13855 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 15177 13855 118 243 0 14931 0
Current children cumulated CPU time (s) 1062.58
Current children cumulated vsize (KiB) 60708

[startup+1122.85 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=573216/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=60856 CPUtime=1122.59
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19924 0 0 0 112239 20 0 0 25 0 1 0 205445446 62316544 13960 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134580862 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 15214 13960 118 243 0 14968 0
Current children cumulated CPU time (s) 1122.59
Current children cumulated vsize (KiB) 60856

[startup+1182.88 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=573152/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=60856 CPUtime=1182.6
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19939 0 0 0 118240 20 0 0 25 0 1 0 205445446 62316544 13975 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589601 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 15214 13975 118 243 0 14968 0
Current children cumulated CPU time (s) 1182.6
Current children cumulated vsize (KiB) 60856



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1200.29 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 28253
/proc/meminfo: memFree=573152/2055920 swapFree=4121028/4192956
[pid=28252] ppid=28250 vsize=60856 CPUtime=1200.02
/proc/28252/stat : 28252 (cmusat) R 28250 28252 21342 0 -1 4194304 19944 0 0 0 119981 21 0 0 25 0 1 0 205445446 62316544 13980 18446744073709551615 134512640 135508737 4294956640 18446744073709551615 134589771 0 0 4096 35 0 0 0 17 1 0 0
/proc/28252/statm: 15214 13980 118 243 0 14968 0
Current children cumulated CPU time (s) 1200.02
Current children cumulated vsize (KiB) 60856

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1200.31
CPU time (s): 1200.04
CPU user time (s): 1199.82
CPU system time (s): 0.219966
CPU usage (%): 99.9773
Max. virtual memory (cumulated for all children) (KiB): 60856

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1199.82
system time used= 0.219966
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19944
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6
involuntary context switches= 1125

runsolver used 1.35679 second user time and 2.89556 second system time

The end

Launcher Data (download as text)

Begin job on node22 on Mon Feb 19 03:15:15 UTC 2007

IDJOB= 298373
IDBENCH= 19984
IDSOLVER= 121
FILE ID= node22/298373-1171854915

PBS_JOBID= 3839650

Free space on /tmp= 66561 MiB

SOLVER NAME= CMUSAT 2007-02-08
BENCH NAME= HOME/pub/bench/SAT07/random/OnTreshold/3SAT/v500/unif-k3-r4.26-v500-c2130-S1211740352-20.UNSAT.shuffled.cnf
COMMAND LINE= /tmp/evaluation/298373-1171854915/cmusat /tmp/evaluation/298373-1171854915/instance-298373-1171854915.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node22/watcher-298373-1171854915 -o ROOT/results/node22/solver-298373-1171854915 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/298373-1171854915/cmusat /tmp/evaluation/298373-1171854915/instance-298373-1171854915.cnf            

META MD5SUM SOLVER= af990fb12f8926fdd756b397adb3b5cf
MD5SUM BENCH=  49141385aac03f506711f7c91a24b9ac

RANDOM SEED= 207533895

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node22.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.235
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.235
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:        629312 kB
Buffers:         52828 kB
Cached:        1209664 kB
SwapCached:      39232 kB
Active:         834544 kB
Inactive:       478648 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:        629312 kB
SwapTotal:     4192956 kB
SwapFree:      4121028 kB
Dirty:            2496 kB
Writeback:           0 kB
Mapped:          30472 kB
Slab:            99012 kB
Committed_AS:   642600 kB
PageTables:       1672 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66561 MiB

End job on node22 on Mon Feb 19 03:35:18 UTC 2007