Last week I had a chance to attend and speak at my first QCon conference: QCon London 2018. QCon has been an example of an extraordinary tech conference for me - great organization, amazing lineup, great tracks covering bleeding-edge tech, as well as various best practices around distributed systems. Also, they managed to get a decent vegetarian food right 😉
Keep reading if you’re curious about the best talks I attended and my speaking experience.
Best Talks I Attended
Day 1: Stream Processing in the Modern Age
I spent my first day at the Stream Processing in the Modern Age track hosted by Tyler Akidau.
Drivetribe: A Social Network on Streams
Drivetribe is a popular motoring social network. Instead of relying on a classic 3-tier architecture or embracing microservices they decided to do something different. Drivetribe uses stream processing for generating all relevant content in write time. So, when you open a page everything was already precalculated before, including the number of likes and comments. How? They heavily rely on write time aggregation. Here’s the high-level diagram of the system:
Drivetribe also employs various algebras to deal with duplicate and out of order events:
- Semigroup/Monoid for adding events
- [Band](https://en.wikipedia.org/wiki/Band_(mathematics)) for duplicated events
- Semilattice for out of order and duplicated events
Next Steps in Stateful Streaming with Apache Flink
Stephan Ewen talked about the stateful operations in Apache Flink and whether it’s possible to combine Batch and Streaming APIs in one processor. Recommended for anyone who uses Apache Flink in production.
Streaming SQL Foundations: Why I ❤ Streams+Tables
Tyler Akidau did really great job explaining the Stream/Table duality, something that Kafka Streams implements as a first-class citizen.
He also showed us how (and why) you can extend SQL to support streaming.
Day 2: Architectures You’ve Always Wondered About
I spent most of my second day at the Architectures You've Always Wondered About track hosted by Wesley Reisz.
It seems like everybody wanted to attend this talk 😉 And it was definitely worth it! We learned a lot about the navigation systems, all kinds of sensors and various use-cases for Machine Learning: perception, prediction and planning.
Testing self-driving cars is a tricky topic as well. We discovered pros and cons of offline testing via simulation as well as online testing with real vehicles.
Logistics as a Service: Building the Ocado Smart Platform
This presentation consisted from two parts. In the first part Paul Sweetman described how and why Ocado uses multiple clouds (AWS and GCP). The second part by Alex Harvey was really mind-blowing! Check this short video, looks impressive, doesn’t it? Alex actually talks about the infrastructure they use to power all that automation and robots. Great talk.
Lambda Architectures: a Snapshot, a Stream, & a Bunch of Deltas
I was really surprised to see another application built on top of stream processing. Lots of really interesting details about Hudson’s Bay microservices setup with Lambda Architecture, caching pros & cons, Apache Kafka usage and more. Very practical and quite entertaining.
Day 3: a bit of everything
Third day was the most diverse in terms of tracks I decided to attend.
Slides | Video (not yet available)
Probably one of the best introductions to actor systems and Akka (Java API) I’ve seen! Focused on real-world performance and reasoning behind some of the Akka design decisions.
Cloud-Native and Scalable Kafka Architecture
This is a true gem of the conference for me, because, well, I deal with Apache Kafka clusters a lot. Netflix has the same problems with scaling Kafka clusters quickly up and down. Adding an extra broker to the existing cluster is not fast and a proper rebalancing can take days.
They solved the problem by scaling the number of clusters, not the number of brokers. With proper automation and custom producers they’re able to quickly add extra capacity by creating additional clusters when needed. Producers know how to deal with it and as a bonus they also know how to failover quickly when necessary.
Netflix also uses Kafka ACL and quotas for proper multi-tenancy.
Great talk by Michael Hussey (for some reason there were just a few attendees, but I loved it!). He talked about various general use-cases for Pub/Sub systems like using different messaging topics for migrating data formats, monitoring, IoT and even geo-location!
Cluster Consensus: When Aeron Met Raft
Slides | Video (not yet available)
Another excellent talk by @mjpt777 at #QConLondon pic.twitter.com/ThfbkD3gVP
— Yaroslav Tkachenko (@sap1ens) March 7, 2018
Martin Thompson is a well-known high-performance expert and, apparently, he’s aware of every computer science paper related to performance, concurrency or messaging for the last few decades 😉
He talked about Aeron, which is a high-performance message transport. He covered the way Aaron uses Raft for clustering and shared some of the Java NIO pain.
My Talks
Presentation
I had a chance to talk about the two of my most favourite topics (aside from the data pipelines) at the same time: actors and event-driven systems. You can find an article about my Actors or Not: Async Event Architectures presentation here, video here and the slides below:
In this presentation I talk about two different event-driven [micro]service environments: “classic” Pub/Sub in Python that we’ve been building at Demonware and actor-based messaging on top of message queue with some routing that we’ve designed and built at Bench Accounting a few years ago.
My talk hasn’t received very high ratings (¯\(ツ)/¯), but I’ve got A LOT of questions after, which always feels good - people seem to be interested in learning best practices and adopting event-driven architecture more widely nowadays.
Here’s the interesting part though: apparently, my talk is #3 Best Attended talk during the conference!
AMA: Actor-Based Architectures
I also had an honour to host an AMA session (Ask-Me-Anything) together with Jonas Boner and Kiki Carter.
I counted only ~10 people in total, but it was amazing to see people like Martin Thompson and Dave Farley among our guests.
We discussed a lot of concerns and challenges related to the Actor systems. And I didn’t notice how 50 minutes just flew by.
Summary
QCon has 12 years of experience organizing top notch tech conferences and you can clearly see it in every detail. That + one of the best experts in the industry is a simple recipe for success. I’m looking forward to my next QCon!