It’s Time to Rethink Event Sourcing

I've always been fascinated by Event Sourcing (ES) and other Domain-Driven Design (DDD) concepts. At some point, I even built a prototype of an event-sourced system called LMAX, which handles 6M orders per second as a high-frequency trading platform.

Unfortunately, the traditional approach to implementing Event Sourcing comes with its own set of challenges. In this blog post, I’ll share new ideas on how to achieve 80% of the Event Sourcing benefits with 20% effort.

Event Sourcing is a unicorn idea that captivates many developers, but it is rarely adopted and implemented successfully.

Why Use Event Sourcing

At its core, Event Sourcing is a simple architectural design pattern. All data changes are recorded as an immutable sequence of events in an append-only store, which becomes the main source of truth for application data. That’s it.

Event Sourcing is a simple, yet powerful concept.

This design pattern provides many advantages:

Traditional Event Sourcing system

Event Sourcing Examples

Most of us use existing event-sourced systems every day and can’t imagine living without them.

Git and bank ledger are frequently used Event Sourcing systems.

Bank ledger account

When you load information about your bank account, most online banks will show you recent ledger transactions, which represent event-sourced records of every money movement in your account.

The idea of recording ledgers as an Event Sourcing system was used way before computer systems were invented. Around 7000 years ago, ledgers were used to record lists of expenditures and goods traded on clay tablets, while temples were considered the banks of the time.

Clay tablets as bank ledgers

Version control system

Version control systems, such as Git, are examples of Event Sourcing systems. Commits represent code changes that are recorded sequentially and become the main source of truth.

Additionally, commits record information about ‘who’ made the change, ‘when’ the change happened, and ‘why’ it was made via a commit message.

Git as an Event Sourcing concept

This means that you can view a history of all changes, time travel by checking to a previous commit, rollback changes, troubleshoot issues by using a binary search, analyze code changes, and so on. You’ve got the idea.

Issues with Traditional Event Sourcing

While Event Sourcing has many benefits, it also comes with many disadvantages that prevent it from being adopted more widely.

Event Sourcing is a simple idea that is very hard to implement.
Developer productivity over time with a CRUD system vs Event Sourcing

Is there a way to get most of the Event Souring benefits while avoiding its disadvantages?

The New Approach to Event Sourcing

The disadvantages of Event Sourcing listed above make it a complete nonstarter for most companies. Let's reconsider the traditional Event Sourcing approach by taking a closer look at how we use a version control system like Git.

Rethinking traditional Event Sourcing through the lens of a version control system

As you can see, with Git:

We can’t, however, blindly copy the Git model and apply it to build “Git for data”. The main reason is that Git commits are usually committed manually by developers, while data in applications is frequently changed automatically. Instead, we need to use a slightly different approach.

Change Data Capture, and its limitations

Change Data Capture (CDC) is a design pattern used to identify and capture changes made to data in a database in real-time. For example, when moving data from an online transaction processing (OLTP) database like PostgreSQL to an online analytical processing (OLAP) system like Snowflake, people typically use CDC to ingest changes and record them in a data warehouse.

{
   "table": "shopping_cart_items",
   "primary_key": 1,
   "operation": "UPDATE",
   "committed_at": "2024-09-01 17:09:15+00",
   "before":{
      "id": 1,
      "quantity": 1,
      ...
   },
   "after":{
      "id": 1,
      "quantity": 2,
      ...
   }
}

Captured change

We could continue performing CRUD operations in a regular database (behaves like the latest snapshot) without rearchitecting our application, use CDC to capture all data changes in the background and store them as an immutable audit log (behaves like an event store).

There is, however, one big fundamental difference between Event Sourcing and Change Data Capture:

Similarities between CDC and version control systems

To bridge the gap and make database changes captured with CDC meaningful and consistent, we can use a couple of different approaches.

Approach 1: Outbox pattern with Change Data Capture

The Outbox pattern allows to atomically update data in a database and record messages that need to be sent in order to guarantee data consistency.

When performing regular database record changes, we can also insert event records in an “ephemeral” outbox table within the same transactions:

BEGIN;
  UPDATE shopping_cart_items SET quantity = 2 WHERE id = 1;
  UPDATE products SET in_stock_count = in_stock_count - 1 WHERE id = 123;
  INSERT INTO outbox_events (event_type, entity_type, entity_id, payload) VALUES (...);
COMMIT;

Inserting events using the Outbox pattern

After the transaction completes, the domain-specific events can be reliably captured by CDC and permanently stored in an event store similarly to a traditional Event Sourcing approach.

Event Sourcing using the Outbox pattern and Change Data Capture

With this approach, we get the simplicity of a typical CRUD system and the benefits of an immutable and consistent append-only event store derived from data changes with CDC.

Approach 2: Contextualized Change Data Capture

Another slightly simplified and more practical approach is to contextualize data changes in CDC pipelines without making any modifications to the underlying data structure and database queries.

With a database like PostgreSQL, it’s possible to pass additional context with queries that can only be visible by a CDC system. Here is a simple code example written in JavaScript using Prisma ORM:

setContext({
  // Event-related data
  eventType: 'SHOPPING_CART_ITEM_QUANTITY_UPDATED',
  entityType: 'SHOPPING_CART_ITEM',
  entityId: 1,
  quantity: 2,
  // Additional context
  userId: currentUser.id,
  apiEndpoint: req.url,
});

await prisma.shoppingCartItem.update({
  where: { id: 1 },
  data: { quantity: 2 },
});
await prisma.products.update({
  where: { id: 123 },
  data: { inStockCount: product.inStockCount - 1 },
});

After the changes are committed to the database, we can reliably capture them, stitch with the context, and store as audit trail records.

Event Sourcing using Change Data Capture and data change contextualization

With this approach, we can continue using CRUD operations and store all event data as context in an immutable and reliable audit trail. This allows us, for example, to query all events by a particular “Shopping Cart Item” and see all underlying data changes made as part of these events.

Conclusion

It’s time to rethink Event Sourcing and stop trying to reinvent the wheel every time we want to implement it in our applications.

In some regulated industries like accounting there are already well-established industry standards for using Event Sourcing in a form of a double-entry bookkeeping system, such as a General Ledger.

In 95% of other cases, you can get most of the Event Sourcing benefits by using Change Data Capture enriched with your domain-specific information. The Change Data Capture data design pattern allows to reliably track and record all data changes made in a database. This, in combination with the Outbox pattern or data change contextualization implemented in the application, allows you to achieve the Event Sourcing advantages mentioned at the beginning of this blog post.

It is possible to event-source any system by implementing Change Data Capture and enriching it with domain-specific information.

This essentially flips the paradigm and allows deriving an immutable log of domain-specific events from regular database changes. Note that the described approaches are not meant to replace the business layer in your application. You still need to think about your domain design and implement it in your code.

About us

If you need help with event-sourcing your system, check out Bemi. Our solution can help you enable automatic data change tracking for your database in a few minutes, integrate it with your ORM for data change contextualization, and have a full audit trail automatically stored in a serverless PostgreSQL database.

Event Sourcing via CDC vs traditional Event Sourcing

For scaling a centralized Postgres data store, check out the BemiDB Github repo.

Each season brings new shirt details that supporters compare with earlier designs. Anyone reviewing club collections can use Paris Saint-Germain jersey（camiseta del Paris Saint-Germain） to focus on the matching design. Checking the size chart before deciding can reduce uncertainty about the final fit. bonus at 1Win Moldova