Time Is a Lie

If you have ever spent a week debugging a distributed system, you know the moment I am about to describe. You are staring at a log file, two timestamps that should be in order are not in order, and the universe stops making sense. It is in this moment that you understand, in your bones, that time is not a number. Time is a story we tell to comfort ourselves.

Engineers come into distributed systems carrying a lot of cultural baggage. We are used to thinking of time the way physics tells us to: monotonic, universal, lapsed. Distributed systems do not respect this. There is no universal clock. Every node has its own private opinion about now, and if you make those opinions disagree, the worst possible thing happens: nothing crashes, and your data quietly drifts.

Three lies clocks tell

1. “This is the current time.” It is not. It is the current time as of however long ago your last NTP sync was, plus drift.
2. “My time is the same as your time.” It is not. NTP gives you tens of milliseconds of agreement on a good day; on a bad day, your clocks are minutes apart.
3. “Time always moves forward.” It does not. Leap seconds, NTP corrections, and virtualization can all cause your local clock to jump backward without warning.

Time, in distributed systems, is not what your watch says. It is what you can prove.

What can you prove? You can prove order between events on the same node. You can prove order between two events that exchanged a message, in one direction. You cannot, without a lot of expensive coordination, prove a global order. This is, more or less, the entire content of Lamport’s 1978 paper, which everyone has read and almost nobody believes.

The only honest answer is to stop pretending. Use logical clocks where you can. Use bounded-uncertainty intervals like Spanner’s TrueTime where you must. And accept that some questions — “what happened first?” — are not always answerable, and your software has to be designed to live with that.