Skip Navigation Linksfun-with-the-twitter-stream-api-elk-stack-rabbitmq-serilog-and-high-performance-sql

​Fun with the Twitter Stream API - ELK Stack, RabbitMQ, Serilog and High Performance SQL


Dave Mateer


Dave is a senior developer for qnrl.com working on enterprise systems. He is a speaker, dad, rock climber, guitar player, ex oil rig worker, ex outdoor instructor, bad joke teller and lover of technology.

Brief Overview


Taking an existing system which consumes 4.3million tweets per day then taking the next step to it making it more Scalabe and Testable.

Description


Sign up to Twitter
  • API Simplest possible C# app working consuming the Twitter stream:
  • RabbitMQ
  • ELK (Elasticsearch, Logstash, Kibana) stack
  • Great data exploration dashboard (Tweets)
  • Great devops (software) dashboard - where are the errors..404, 500 etc..
  • SQL Tips for Sorting, Filter, Page
  • Dapper as the ORM
This application has been in development for 6 months. It is born out of 7 years of writing similar enterprise software - can my system handle 2x or 10x the load if I need it? Can I prove my system is working ie Auditable/Testable and Dashboard overviews.