Event-Driven Architectures Done Right, Apache Kafka • Tim Berglund • Devoxx Poland 2021

190,418
222
Published 2022-05-26
Subscribe to our channel: youtube.pl/c/DevoxxPoland?sub...

Far from a controversial choice, Kafka is now a technology developers and architects are adopting with enthusiasm. And it’s often not just a good choice, but a technology enabling meaningful improvements in complex, evolvable systems that need to respond to the world in real time. But surely it's possible to do wrong! In this talk, we'll look at common mistakes in event-driven systems built on top of Kafka:

-Deploying Kafka when an event-driven architecture is not the best choice.
-Ignoring schema management. Events are the APIs of event-driven systems!
-Writing bespoke consumers when stream processing is a better fit.
-Using stream processing when you really need a database.
-Trivializing the task of elastic scaling in all parts of the system.

It's highly likely for medium- and large-scale systems that an event-first perspective is the most helpful one to take, but it's early days, and it's still possible to get this wrong. Come to this talk for a survey of mistakes not to make.

Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the Senior Director of Developer Advocacy. He can frequently be found at speaking at conferences in the United States and all over the world. He is the co-presenter of various training videos on topics ranging from Git to Distributed Systems to Apache Kafka. He tweets as @tlberglund, blogs very occasionally at timberglund.com/, and lives in Littleton, CO, USA with his wife, their three children having grown up.

Topics covered:
-What is Event-Driven Architecture
-Data Mesh Principles
-Scaling
-State management

Recorded at Devoxx Poland 2021

Twitter: twitter.com/DevoxxPL
Instagram: www.instagram.com/DevoxxPL

Join us also here:
Devflix: devflix.pl/

#Devoxx #DevoxxPoland #IT #Development #SoftwareDevelopment

All Comments (21)
  • "If it is small program, you don't call it a monolith. Architecturally it IS one. I want to give you permission just embrace that" - BRILLIANT! Noted for further discussions with middle developers, thanks a lot Tim!
  • this guy is brilliant, I love his sense of humor and what a great talk !!!
  • @atkinpaul
    Thanks! I hope you do engage more with Data Mesh. Zhamak seems to de-prioritize operational data flows but there is a massive opportunity to converge operational and analytic pipelines when you use an event driven architecture combined with polyglot persistence.
  • @vanivari359
    Don't want to complain too much, but from an expert like him i kinda expected a bit more than: "only use EDA if you need it", "Use DBs if they are a good idea", "the event schema is kinda important because stuff changes" and "stuff is hard, don't implement your own scaling or state management". I was already suspicious after the claim that there are 5 ways to fail with EDA because i've seen at least 10. And the 5 in the talk are for sure valid points, but not the stuff (except scaling) projects typically struggle with. The real challenge is stuff like eventual consistency and the lack of transactions - issues which could have been discussed pretty well with that reference architecture. But hey, now i know that he wanted to buy tailored shirts once... but didn't. ;)
  • @KirBirger
    Thanks for this. There's a shortage of honest content out there that doesn't simply evangelize concepts and approaches.
  • @3enny3oy
    The last 15 minutes of 2001 a Space Odyssey 😂 I’d say that’s where all the meaning of the movie is. And yeah, don’t be afraid of a monolith if you’re building something small and isolated. If you can get something up and running to do the job with little up front effort or admin overhead, it’s probably fine to spend the extra effort to migrate to an event driven architecture in the future once you’re actually generating value. If you over engineer up front it will be far longer before you start to generate business value and it will probably cost more in the long run. Build fast, fail fast and iterate. Just avoid decisions which will lock you down and make iteration difficult. Be pragmatic and agile.
  • @tanertasim3637
    This presentation is just gold!! Very clean and precise explanations! Thanks!
  • @marekchodak6705
    I usually watch on 1.5x speed but he is so good to listen i kinda dont want to
  • @joshzwicker
    Asked event-driven-architecture into your heart 😄👍nice metaphor
  • @cxmais
    do things right, don’t do things wrong
  • @mehrdadk.6816
    The reason for developers think when using event-driven is a database replacement can be , there is no example that uses the combination in Confluent materials. Maybe this can be getting done
  • @user-sg5qk6wy3p
    Given the event log is the system of record, and lets say we want the customer's address updated in the user service. Does that mean we have to fetch the current version of the user from the event log? Seems challenging and requires specific technologies. Alternatively you'll use the DB in the user service. By doing that, the DB has effectively become the system of record, right?
  • @smaug9833
    Another pitfall of event driven architecture is that one service derives the whole context from the message it consumes. But sometimes, there is a business need for a service to maintain its own context in addition to what it reads from the topics. I know this is not the way to go for purely event driven services, but sometimes you are forced into a situation where you have to do this.