How a 32-Day Clock Jump Erased My VictoriaMetrics History

I was testing a new ComposeFlux feature on my homeserver. As part of some maintenance work, I restarted the servers and opened Grafana a few minutes later to check whether everything had come back normally.

That was when I noticed every metric before 19:18 was gone. New metrics were arriving normally, but the entire history ended at the same minute.

Because this happened immediately after testing ComposeFlux and restarting the servers, my first suspicion was obvious: did ComposeFlux recreate the VictoriaMetrics container or prune its Docker volume?

It did not. The real cause was much more interesting: my Raspberry Pi rebooted with its clock 32 days behind, and VictoriaMetrics treated the existing data as being far in the future.

With the help of Codex, I was able to troubleshoot this quickly. I checked the Docker volume and container metadata, ComposeFlux logs, the host journal, VictoriaMetrics logs, and finally the stored metrics through the VictoriaMetrics API.

Checking the Docker Volume

The VictoriaMetrics volume still existed and had not been recreated:

Name:      victoriametrics_data
Created:   2026-06-16T14:17:25+02:00
Mounted:   /victoria-metrics-data

The VictoriaMetrics container was also created on August 24, several days before the incident. ComposeFlux had no VictoriaMetrics deployment, stack removal, or Docker volume-prune entries around 19:18.

That ruled out a new empty volume. The same container was still using the same volume.

The Timeline

The host logs explained what happened:

The journal recorded the correction clearly:

ntpd: CLOCK: time stepped by 2772598.848847
ntpd: CLOCK: time changed from 2026-07-28 to 2026-08-29

The Pi has no real-time clock:

RTC time: n/a

It restored an old date during boot and started Docker before NTP had synchronized. VictoriaMetrics therefore reopened its storage while the machine believed it was July 28.

Why VictoriaMetrics Removed the Data

My VictoriaMetrics configuration uses a 30-day retention period:

command:
  - "-retentionPeriod=30d"

VictoriaMetrics also accepts future samples only up to now+2d by default. This is controlled by -futureRetention.

When VictoriaMetrics started with the July 28 clock, the data already stored for August appeared to be between 27 and 32 days in the future. That was far outside the default two-day future-retention window.

After NTP moved the clock forward, VictoriaMetrics logged the opposite problem for samples collected during the bad clock window:

cannot insert row with too small timestamp;
minimum allowed timestamp is 1785431897000;
probably you need updating -retentionPeriod or -maxBackfillAge

The storage numbers confirmed that this was real data loss, not a Grafana query problem:

Storage opened before clock correction:  76.8 MB
Storage after clock correction:           39.6 MB

A direct query to the VictoriaMetrics API returned no up samples before 19:18, while samples after that time were present.

The Docker volume survived. Its historical data parts did not.

Preventing This

I still need to check how to prevent this from happening again.