This is a great blog entry on exactly the kind of system I am trying to build. When we went thru the evaluation for this stack - Elasticsearch came out as the choice for the datastore and querying part. Where we are still not decided is using Flume vs logstash. Have you compared the two? We will be building our own UI ...