open-source tool for filter out stats from dns logs

[Mike Hoskins (michoski)]

-----Original Message-----

From: Jeff Wright <jwright at isc.org>
Date: Thursday, January 3, 2013 8:41 AM
To: "bind-users at lists.isc.org" <bind-users at lists.isc.org>
Subject: Re: open-source tool for filter out stats from dns logs

>There might be some tools already out there (like Splunk) that do this
>for you.  I think you can get a free Splunk license if you parse
>relatively small amounts of daily data.  If you're particularly
>concerned about open-source, this thread might also help:
>http://stackoverflow.com/questions/183977/what-commercial-and-open-source-
>competitors-are-there-to-splunk.

Just wanted to add a few things based on some research I've been doing...
By all means, start with the SO thread above and [your favorite search
engine] as I did.  This may just save folks some time.  :-)

Splunk is an amazing tool, but gets expensive fast when indexing much
data...  With the maturity of many OSS solutions, I'm not sure it even
makes sense on a small scale these days (unless you plan to stick with it).

After reading through several SO threads and spending many late nights
searching, I've mostly concluded that there are two OSS "solutions" (a mix
of technologies/tools) that can fill this gap.  You can go the "neato"
(newer, being discussed more) way of [ logstash + graylog + elastic search
] or the "oldschool" (relatively at this point) of [ syslog-ng + mysql +
sphinx ] (ELSA).

For the prior, my initial research let to buzzword/acronym overload.  This
post helped immensely:

http://jpmens.net/2012/08/06/my-logstash-and-graylog2-notes/

And also led me to find this useful ES utility:

http://jpmens.net/2012/08/09/must-have-ui-for-elasticsearch/

These are also obvious places to start playing (the first is worth
visiting just to watch the, hilarious IMCO, video on the front page):

http://logstash.net/

http://graylog2.org/

http://www.elasticsearch.org/

Of course after setting all that up, some conclude it's too slow for
real-time analytics.  There's discussion about this on SO and other
places.  Based on your use cases, you might not care.  If you do, consider
ELSA:

https://code.google.com/p/enterprise-log-search-and-archive/

Somewhat dated, but great overview by the author (refer to the docs for
latest features):

http://ossectools.blogspot.com/2011/03/fighting-apt-with-open-source-softwa
re.html

We are in the process of building prototype environments for both of these
atm, so wanted to share.

hth