Working across changing log formats

This guide covers a common operational problem: log formats change over time. A service may move from plain text to JSON, a library upgrade may rename fields, or a migration may leave old and new formats mixed in the same retention window.

The how-to section comes after the main product chapters because it assumes you already know the ingest and query model. Use it when you need practical patterns for real-world operating problems rather than a conceptual introduction.

logging24 keeps this manageable because the raw events remain searchable. You do not need to rebuild an index or redesign a schema before you can query across the old and new formats.

Example: plain text to JSON

Older log format:

2025-06-25 10:01:23 INFO User login: id=1234

Newer log format:

{"timestamp":"2025-06-25T10:01:23Z","level":"INFO","event":"user_login","user_id":1234}

Query each format separately

For the older line structure:

.*User login: id=.*

For the newer JSON structure:

.*"event":"user_login".*

Query both formats together

If your retention window contains both variants, combine them with alternation:

(.*User login: id=.*)|(.*"event":"user_login".*)

This lets you search across migration periods without splitting the investigation by format version.

Why this matters

Format changes do not force an immediate ingest redesign.
Old and new log shapes remain searchable in the same system.
You can update queries incrementally instead of rebuilding pipelines first.
Teams can keep investigating while services and logging libraries evolve.

Practical guidance

When formats are in transition, start with broad regex matches that identify the old and new variants, then refine them as needed. If you also need numeric or textual analysis, capture values only after you are confident the surrounding pattern is stable.