Trace number 300974

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
TTS 4.0? (TO) 1200.07 1200.39

General information on the benchmark

Namerandom/LargeSize/5SAT/v600/
unif-k5-r20-v600-c12000-S811866391-08.cnf
MD5SUMe2bcd99348dd1a340f87926981c0a2e4
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark21.7367
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses12000
Sum of the clauses size60000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 512000
Number of clauses of size over 50

Solver Data (download as text)

Verifier Data (download as text)

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: 

Watcher Data (download as text)

runsolver version 3.2.0 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node5/watcher-300974-1171945850 -o ROOT/results/node5/solver-300974-1171945850 -C 1200 -M 1800 --output-limit 1,15 /tmp/evaluation/300974-1171945850/tts-4-0 /tmp/evaluation/300974-1171945850/instance-300974-1171945850.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 32224
/proc/meminfo: memFree=1053408/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=796 CPUtime=0
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 111 0 0 0 0 0 0 0 18 0 1 0 214539857 815104 96 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134647773 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 199 96 57 113 0 83 0

[startup+0.106347 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 32224
/proc/meminfo: memFree=1053408/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=2536 CPUtime=0.1
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 530 0 0 0 10 0 0 0 18 0 1 0 214539857 2596864 515 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134538267 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 634 515 59 113 0 518 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 2536

[startup+0.514394 s]
/proc/loadavg: 0.92 0.98 0.99 3/77 32224
/proc/meminfo: memFree=1053408/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=18140 CPUtime=0.5
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 4433 0 0 0 49 1 0 0 21 0 1 0 214539857 18575360 4416 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134627035 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 4535 4416 66 113 0 4419 0
Current children cumulated CPU time (s) 0.5
Current children cumulated vsize (KiB) 18140

[startup+1.33449 s]
/proc/loadavg: 0.92 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1029656/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=24600 CPUtime=1.32
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 6032 0 0 0 131 1 0 0 25 0 1 0 214539857 25190400 6015 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134641336 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 6150 6015 66 113 0 6034 0
Current children cumulated CPU time (s) 1.32
Current children cumulated vsize (KiB) 24600

[startup+2.97067 s]
/proc/loadavg: 0.93 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1028504/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=25496 CPUtime=2.96
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 6244 0 0 0 295 1 0 0 25 0 1 0 214539857 26107904 6227 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134552285 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 6374 6227 66 113 0 6258 0
Current children cumulated CPU time (s) 2.96
Current children cumulated vsize (KiB) 25496

[startup+6.24704 s]
/proc/loadavg: 0.93 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1028248/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=25632 CPUtime=6.24
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 6319 0 0 0 622 2 0 0 25 0 1 0 214539857 26247168 6302 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134591882 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 6408 6302 66 113 0 6292 0
Current children cumulated CPU time (s) 6.24
Current children cumulated vsize (KiB) 25632

[startup+12.7038 s]
/proc/loadavg: 0.94 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1027736/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=26080 CPUtime=12.69
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 6417 0 0 0 1267 2 0 0 25 0 1 0 214539857 26705920 6400 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134591879 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 6520 6400 66 113 0 6404 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 26080

[startup+25.5082 s]
/proc/loadavg: 0.95 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1026776/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=27048 CPUtime=25.49
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 6657 0 0 0 2546 3 0 0 25 0 1 0 214539857 27697152 6640 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134513482 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 6762 6640 66 113 0 6646 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 27048

[startup+51.1181 s]
/proc/loadavg: 0.96 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1025240/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=28452 CPUtime=51.09
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 7041 0 0 0 5106 3 0 0 25 0 1 0 214539857 29134848 7024 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134591729 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 7113 7024 66 113 0 6997 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 28452

[startup+102.335 s]
/proc/loadavg: 0.98 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1020184/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=33800 CPUtime=102.3
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 8327 0 0 0 10226 4 0 0 25 0 1 0 214539857 34611200 8310 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134513469 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 8450 8310 66 113 0 8334 0
Current children cumulated CPU time (s) 102.3
Current children cumulated vsize (KiB) 33800

[startup+162.368 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1012824/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=40732 CPUtime=162.32
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 10115 0 0 0 16226 6 0 0 25 0 1 0 214539857 41709568 10098 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534114 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 10183 10098 66 113 0 10067 0
Current children cumulated CPU time (s) 162.32
Current children cumulated vsize (KiB) 40732

[startup+222.395 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1006296/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=47332 CPUtime=222.32
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 11753 0 0 0 22225 7 0 0 25 0 1 0 214539857 48467968 11736 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134591721 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 11833 11736 66 113 0 11717 0
Current children cumulated CPU time (s) 222.32
Current children cumulated vsize (KiB) 47332

[startup+282.418 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=1001112/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=52728 CPUtime=282.34
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 13065 0 0 0 28225 9 0 0 25 0 1 0 214539857 53993472 13048 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134641336 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 13182 13048 66 113 0 13066 0
Current children cumulated CPU time (s) 282.34
Current children cumulated vsize (KiB) 52728

[startup+342.444 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=994840/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=58820 CPUtime=342.34
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 14623 0 0 0 34224 10 0 0 25 0 1 0 214539857 60231680 14606 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534071 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 14705 14606 66 113 0 14589 0
Current children cumulated CPU time (s) 342.34
Current children cumulated vsize (KiB) 58820

[startup+402.475 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=988248/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=65384 CPUtime=402.36
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 16237 0 0 0 40224 12 0 0 25 0 1 0 214539857 66953216 16220 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534130 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 16346 16220 66 113 0 16230 0
Current children cumulated CPU time (s) 402.36
Current children cumulated vsize (KiB) 65384

[startup+462.507 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=982936/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=70544 CPUtime=462.37
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 17558 0 0 0 46223 14 0 0 25 0 1 0 214539857 72237056 17541 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134513469 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 17636 17541 66 113 0 17520 0
Current children cumulated CPU time (s) 462.37
Current children cumulated vsize (KiB) 70544

[startup+522.535 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32225
/proc/meminfo: memFree=978840/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=74828 CPUtime=522.39
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 18599 0 0 0 52224 15 0 0 25 0 1 0 214539857 76623872 18582 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134535843 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 18707 18582 66 113 0 18591 0
Current children cumulated CPU time (s) 522.39
Current children cumulated vsize (KiB) 74828

[startup+582.562 s]
/proc/loadavg: 0.99 0.98 0.99 2/78 32227
/proc/meminfo: memFree=974360/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=79184 CPUtime=582.4
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 19686 0 0 0 58224 16 0 0 25 0 1 0 214539857 81084416 19669 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534117 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 19796 19669 66 113 0 19680 0
Current children cumulated CPU time (s) 582.4
Current children cumulated vsize (KiB) 79184

[startup+642.594 s]
/proc/loadavg: 1.06 1.00 1.00 2/78 32227
/proc/meminfo: memFree=969112/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=84340 CPUtime=642.41
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 20983 0 0 0 64224 17 0 0 25 0 1 0 214539857 86364160 20966 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534130 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 21085 20966 66 113 0 20969 0
Current children cumulated CPU time (s) 642.41
Current children cumulated vsize (KiB) 84340

[startup+702.62 s]
/proc/loadavg: 1.02 1.00 1.00 2/78 32227
/proc/meminfo: memFree=964376/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=89036 CPUtime=702.43
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 22192 0 0 0 70224 19 0 0 25 0 1 0 214539857 91172864 22175 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534120 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 22259 22175 66 113 0 22143 0
Current children cumulated CPU time (s) 702.43
Current children cumulated vsize (KiB) 89036

[startup+762.646 s]
/proc/loadavg: 1.01 1.00 1.00 2/78 32227
/proc/meminfo: memFree=960600/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=92792 CPUtime=762.44
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 23118 0 0 0 76224 20 0 0 25 0 1 0 214539857 95019008 23101 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534122 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 23198 23101 66 113 0 23082 0
Current children cumulated CPU time (s) 762.44
Current children cumulated vsize (KiB) 92792

[startup+822.679 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=956504/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=97012 CPUtime=822.45
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 24136 0 0 0 82224 21 0 0 25 0 1 0 214539857 99340288 24119 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534117 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 24253 24119 66 113 0 24137 0
Current children cumulated CPU time (s) 822.45
Current children cumulated vsize (KiB) 97012

[startup+882.709 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=953496/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=99824 CPUtime=882.46
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 24881 0 0 0 88224 22 0 0 25 0 1 0 214539857 102219776 24864 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534114 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 24956 24864 66 113 0 24840 0
Current children cumulated CPU time (s) 882.46
Current children cumulated vsize (KiB) 99824

[startup+942.741 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=950168/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=103340 CPUtime=942.48
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 25738 0 0 0 94225 23 0 0 25 0 1 0 214539857 105820160 25721 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534114 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 25835 25721 66 113 0 25719 0
Current children cumulated CPU time (s) 942.48
Current children cumulated vsize (KiB) 103340

[startup+1002.77 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=946968/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=106620 CPUtime=1002.49
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 26542 0 0 0 100225 24 0 0 25 0 1 0 214539857 109178880 26525 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134513469 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 26655 26525 66 113 0 26539 0
Current children cumulated CPU time (s) 1002.49
Current children cumulated vsize (KiB) 106620

[startup+1062.8 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=942296/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=111200 CPUtime=1062.51
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 27687 0 0 0 106226 25 0 0 25 0 1 0 214539857 113868800 27670 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534114 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 27800 27670 66 113 0 27684 0
Current children cumulated CPU time (s) 1062.51
Current children cumulated vsize (KiB) 111200

[startup+1122.82 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=937304/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=116128 CPUtime=1122.52
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 28939 0 0 0 112225 27 0 0 25 0 1 0 214539857 118915072 28922 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534114 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 29032 28922 66 113 0 28916 0
Current children cumulated CPU time (s) 1122.52
Current children cumulated vsize (KiB) 116128

[startup+1182.85 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=933336/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=120116 CPUtime=1182.53
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 29943 0 0 0 118225 28 0 0 25 0 1 0 214539857 122998784 29926 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534114 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 30029 29926 66 113 0 29913 0
Current children cumulated CPU time (s) 1182.53
Current children cumulated vsize (KiB) 120116



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1200.37 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=932120/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=121288 CPUtime=1200.04
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 30241 0 0 0 119976 28 0 0 25 0 1 0 214539857 124198912 30224 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534095 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 30322 30224 66 113 0 30206 0
Current children cumulated CPU time (s) 1200.04
Current children cumulated vsize (KiB) 121288

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1200.37 s]
/proc/loadavg: 1.00 1.00 1.00 2/78 32227
/proc/meminfo: memFree=932120/2055920 swapFree=4192812/4192956
[pid=32224] ppid=32222 vsize=121288 CPUtime=1200.04
/proc/32224/stat : 32224 (tts-4-0) R 32222 32224 31883 0 -1 4194304 30241 0 0 0 119976 28 0 0 25 0 1 0 214539857 124198912 30224 18446744073709551615 134512640 134976056 4294956624 18446744073709551615 134534095 0 0 4096 0 0 0 0 17 1 0 0
/proc/32224/statm: 30322 30224 66 113 0 30206 0
Current children cumulated CPU time (s) 1200.04
Current children cumulated vsize (KiB) 121288

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1200.39
CPU time (s): 1200.07
CPU user time (s): 1199.77
CPU system time (s): 0.308953
CPU usage (%): 99.9735
Max. virtual memory (cumulated for all children) (KiB): 121288

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1199.77
system time used= 0.308953
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 30241
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6
involuntary context switches= 1488

runsolver used 1.18682 second user time and 2.74758 second system time

The end

Launcher Data (download as text)

Begin job on node5 on Tue Feb 20 04:30:50 UTC 2007

IDJOB= 300974
IDBENCH= 20048
IDSOLVER= 98
FILE ID= node5/300974-1171945850

PBS_JOBID= 3848945

Free space on /tmp= 66551 MiB

SOLVER NAME= TTS 4.0
BENCH NAME= HOME/pub/bench/SAT07/random/LargeSize/5SAT/v600/unif-k5-r20-v600-c12000-S811866391-08.cnf
COMMAND LINE= /tmp/evaluation/300974-1171945850/tts-4-0 /tmp/evaluation/300974-1171945850/instance-300974-1171945850.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node5/watcher-300974-1171945850 -o ROOT/results/node5/solver-300974-1171945850 -C 1200 -M 1800 --output-limit 1,15  /tmp/evaluation/300974-1171945850/tts-4-0 /tmp/evaluation/300974-1171945850/instance-300974-1171945850.cnf            

META MD5SUM SOLVER= 73a7a27d0c2e86f9b1191433fde766ff
MD5SUM BENCH=  e2bcd99348dd1a340f87926981c0a2e4

RANDOM SEED= 15361344

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node5.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.224
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.224
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1053816 kB
Buffers:         61084 kB
Cached:         819584 kB
SwapCached:          0 kB
Active:         535444 kB
Inactive:       389128 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1053816 kB
SwapTotal:     4192956 kB
SwapFree:      4192812 kB
Dirty:             828 kB
Writeback:           0 kB
Mapped:          62796 kB
Slab:            63172 kB
Committed_AS:   779428 kB
PageTables:       1692 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66551 MiB

End job on node5 on Tue Feb 20 04:50:53 UTC 2007