Lookup slowness after ~5 minutes

[sf techguy]

My server is runing BIND 9.2.2 on Mac OS X Server 10.4.6 (Darwin
8.6.0). BIND is configured with a few simple ACL's and views, and is
set up as a caching forwarder for my rfc 1918 LAN (192.168.x.x). I
have one master zone, several slave zones, and lookups on any other
zones are forwarded out to my ISP nameservers (hence the ACL's/views).

When the server first boots, lookups from a LAN IP against my BIND
server are quick: usually the client has an answer in less than .5
seconds (when looking up a non-local domain name that i know isn't
cached).

After about 5 minutes of the server running, LAN client DNS lookups
(again, also of known un-cached, non-local domain names) slow to over
5 seconds. If I stop and start BIND, lookups are fast again, and then
after running for 5 minutes or so, they return to being slow.

If I do lookups from a LAN client against my ISP's public nameservers
(instead of against my LAN BIND server), answers are always under a
second or so, and never slow down noticeably.

Anyone know what could cause this?