Every distributed system you depend on relies on clocks agreeing with each other. Database replication, TLS handshakes, Kerberos authentication, log correlation during incidents. All of it falls apart if your servers can't agree on what time it is and clocks constantly lie.
Why Clocks Drift
Computers keep time using crystal oscillators, usually quartz. The system counts vibrations at a nominal frequency to track the passage of time, but quartz isn't perfect, so temperature changes, voltage fluctuations, crystal age, even the specific batch it was cut from can introduce error. A typical commodity server clock drifts somewhere between 10 and 200 parts per million. That sounds negligible, but 100 ppm actually works out to about 8.6 seconds per day. If you leave a server alone for a month you could be four minutes out.
On a single machine this barely matters, if the clock is wrong, but everything on that box is wrong in the same direction. But the moment you add a second machine, a database replica, a microservice, a log aggregator, you now have two clocks that are wrong in different ways.
- Distributed databases like Spanner and PostgreSQL logical replication use timestamps to determine event ordering. If Node A thinks it's 14:00:00.000 and Node B thinks it's 14:00:00.350, a write on B can be ordered before a write on A even though it happened after. In serialisable isolation this causes phantom reads or lost updates. Google built custom atomic clocks and GPS receivers into their data centres over a decade ago specifically to keep Spanner's TrueTime uncertainty window below 7ms. That should tell you how much this matters.
- Kerberos authentication, the backbone of Active Directory, rejects tickets when the clock skew between client and server exceeds five minutes. If your domain controller drifts over the weekend and nobody notices, you'll get an eventful Monday morning.
- TLS certificate validation depends on the client's local clock to check the
notBeforeandnotAfterfields. A clock that has drifted forward makes a valid certificate look expired. A clock that has drifted backward makes a revoked certificate look valid, so neither is great. - Log correlation during incident response becomes basically impossible when your servers disagree on time. If your web server, app server, and database are each a few seconds apart, reconstructing the sequence of events in a post-mortem is just guesswork.
Enter NTP
The Network Time Protocol has been solving this problem since 1985, making it one of the oldest internet protocols still in active use. Its job is to synchronise your system clock with a reliable reference source and keep it synchronised.
NTP uses a hierarchical model based on strata. Stratum 0 sources are the reference clocks themselves, atomic clocks, GPS receivers, radio clocks synced to national time standards. Stratum 1 servers connect directly to stratum 0 hardware. Stratum 2 servers sync to stratum 1, and so on down the chain. Each hop adds a small amount of uncertainty, but stratum 2 or 3 is more than good enough for nearly everything.
The genius part is how NTP handles network latency. The naive approach of just asking a remote server for the time and setting your clock to the answer doesn't work because the response takes time to arrive. NTP sends multiple packets, measures the round-trip delay, and estimates the offset by assuming the network path is roughly symmetric. It filters outliers, runs statistical algorithms to pick the best source from a pool of configured servers, then adjusts the local clock gradually through a discipline loop rather than jumping it.
When the offset is small, under 128ms by default, ntpd turns the clock by speeding it up or slowing it down slightly until it converges. When the offset is larger but still under the panic threshold of 1000 seconds, it steps the clock in one go. Beyond the panic threshold it refuses to act and exits, on the assumption that something is so broken that automated correction would be dangerous.
NTP vs Chrony vs PTP
The original ntpd reference implementation has largely been replaced by Chrony in Linux environments. Chrony handles intermittent connectivity better, converges faster after boot, and behaves well in virtualised environments where the clock can be particularly unpredictable thanks to CPU steal and live migration.
For workloads where NTP still isn't accurate enough, like high-frequency trading, scientific instrumentation, telecom, there's Precision Time Protocol (PTP, IEEE 1588). PTP operates at the hardware level using timestamping built into the NIC itself. It can hit sub-microsecond accuracy on a local network but needs hardware support and is considerably more complex to deploy.
For most infrastructure, Chrony pointed at a pool of stratum 2 servers gets you within a few milliseconds of UTC. That covers log correlation, certificate validation, database consistency, and auth protocols just fine.
What can you do about it?
You should actually monitor your NTP offset. Every monitoring stack supports it. Prometheus has node_timex_offset_seconds from the node exporter. Set an alert at something like 50ms and investigate anything that trips it. A drifting clock can be an early indicator of other problems like high CPU load starving the NTP daemon, a misconfigured VM host, or a network path change disrupting your time source (DNS?).
Use at least three time sources. NTP's source selection algorithm needs multiple inputs to detect a falseticker, a source returning bad time. With one source you can't tell if it's wrong. With two you can tell something is off but not which one. Three is the minimum for reliable detection.
Consider running a local stratum 1 server if you have enough kit to justify it. A Raspberry Pi with a GPS/RTC HAT and a PPS (pulse-per-second) output makes a very capable time source for a homelab or small office and it also removes your dependency on external NTP pools and gives you accuracy within a few microseconds.
If you run VMs, pay attention to the hypervisor's timekeeping configuration. Virtualisation adds a layer of clock abstraction that can amplify drift. Most modern hypervisors like KVM, Hyper-V, and VMware provide a paravirtual clock source that is more stable than the emulated hardware clock, but it needs to be set up correctly.
Clock synchronisation is unglamorous work but future you staring at logs at 2am will be grateful :)