---

Is there an open source thing out there I could use??

So if I only want to use mostly free and open source it there's a bunch of tools one need to glue together:

These days I'd like to for primary ingestion have a BGP ECMP/anycast for rsyslog receivers. These also run logstash (or beat?). Or maybe one can have a load balancer up front which redirects traffic based on incoming port (and maybe a syslog tag for some 'authentication' ? ) to a set of logparsing/rsyslog servers.

These would write to a Kafka cluster.

Then we would need more readers to stream events on to elastic, siems or Hadoop or for example longer term storage engines.

For the as a Service bit I'd like to play with Rundeck and have users configure most of the bits themselves. Logstash grokking/parsing though needs outsourcing too. Fewer rules means more throughput so would be good with different logstash processes for different logs. Could like loggly direct users to ship logs with a tag to get them into the correct lane.

For reading just grafana and kibana should be a good start.