Backward compatibility when applying a schema change

I was reading Designing Data-Intensive Applications by Martin Kleppmann. Chapter 4, “Encoding and evolution,” brought back a mistake I made while I was a developer at TNG Digital Sdn Bhd.

Why this mistake matters

The chapter discusses schema changes, such as adding a field or changing the type of an existing field. When a schema changes, the application must keep working while different versions of the code and data coexist. That means supporting both directions of compatibility:

• Backward compatibility: Newer code can read data written by older code.
• Forward compatibility: Older code can read data written by newer code.

Why it matters in production

We typically use rolling deployments. We deploy a new version to a few pods, monitor it, then roll it out to the rest. For part of that window, old and new versions of the application run in production at the same time.

We usually apply database schema changes before deployment, although some changes need to happen afterwards. Either way, the application has to tolerate data created by both versions.

What I got wrong

My code did not handle backward compatibility. I added a field to a registration form. After a user submitted the form, the request was converted to JSON and stored in a VARCHAR column in the table that held registration requests.

When an admin approved a request, the application read that JSON into a Java POJO. Later in the approval flow, the code read the new field from the POJO and used it in the business logic.

New submissions worked after the release. The problem was the older records already in the database: their JSON did not contain the new field. When an admin approved one of those records, the application threw a NullPointerException while trying to read the missing value.

Embarrassing? Yes. But it was a useful lesson.

The fix

The new field was a String, so I treated a missing value as an empty string before using it:

String newField = Optional.ofNullable(request.getNewField()).orElse("");

Use the getter that matches your POJO. The important part is ofNullable: it lets newer code safely process older JSON that does not include the field.

Takeaway

The book puts it well:

We must assume that different nodes are running different versions of our application’s code. Thus, it is important that all data flowing around the system is encoded in a way that provides backward compatibility (new code can read old data) and forward compatibility (old code can read new data).