Ask HN: Examples of Moving to Event Streaming?

by ultra-jeremyx on 1/17/2023, 3:04:52 PM with 1 comments
My company is struggling with scaling up our applications and APIs because a good chunk of the core data lives within a single database. This DB/Server has been fined-tuned over the years to handle the ever-increasing load, but it's not sustainable. I am investigating moving some of this data into an event-stream as a path forward to reduce pressure on the DB and facilitate scaling APIs.

The high-level architecture:

  - DB change detection publishes events to an immutable event stream.
  - Event-stream processing service projects the data from the event-stream into a database for particular API.
  - Data served from the API is sourced from the projected data in the API database.
The trade offs with this architecture are:

  - Eventual consistency
  - Copies of data
  - Managing complexity
  - Schema changes
I'm looking for some real-world examples of companies that have made the transition from a single DB to an event-driven architecture. Do you know of any case studies, interviews, etc. ?

  • by bob1029 on 1/17/2023, 3:21:58 PM

    I don't think other real-world examples are going to tell you if this is actually a good idea for your business or not.

    Understanding your business & workload in more detail would be critical in developing even the remotest amount of realistic advice on this matter.

    > Managing complexity

    Event streaming is almost always going to be more complicated than boring old SQL database access. This is what would probably kill the project more than any specific performance consideration.