How to Parse Batch Epoch Time Strings with Mixed Millisecond and Second Formats

1699999999 vs 1699999999000: When Your Log Pipeline Meets Both

You inherited the mess. Three application servers, two cloud providers, and a legacy database all funnel their event logs into a single analytics dashboard. Server A writes timestamps as ten-digit Unix epoch seconds. Server B upgraded last year and now spits out thirteen-digit millisecond epochs. Server C? Nobody knows what Server C is doing, but its timestamps look like they were generated by a random number generator that once read a Wikipedia article about time.

Here is the exact moment everything breaks: your batch parser assumes one format, applies it uniformly, and suddenly your dashboard shows events from January 1970 sitting next to events from the year 5138. The dashboard does not care. It just renders the data. Your stakeholders, however, care a lot.

This is a pitfalls guide for engineers who need to parse batch epoch time strings with mixed millisecond and second formats — without accidentally rewriting history or projecting events into a future none of us will live to see.

The Wrong Way: Blind Division vs Smart Detection

What Fails: Dividing Everything by 1000

Let us look at the most common mistake first. You discover mixed formats in your batch, so you write a quick normalization step: divide every timestamp by 1000 to convert milliseconds to seconds. Clean, simple, wrong.

Take the timestamp 1700000000. This is a legitimate ten-digit epoch representing November 14, 2023, at 22:13:20 UTC. Divide it by 1000 and you get 1700000 — which is January 20, 1970. You have just demoted a 2023 event to the early Nixon administration.

Now flip it. Take 1700000000000, a valid thirteen-digit millisecond timestamp for the same moment. If your parser treats it as seconds, you get a date in the year 55874. Your monitoring tool will either crash, silently drop the record, or display it with a straight face.

What Works: Length-Based Heuristics First

Before you touch any arithmetic, count the digits. Unix epoch seconds from 2001 to 2286 fall between 10 and 11 digits. Millisecond epochs from 2001 to 2286 fall between 13 and 14 digits. This gives you a fast, cheap first-pass filter:

• 10 digits → almost certainly seconds
• 13 digits → almost certainly milliseconds
• 11 digits → seconds, post-2001 (still safe)
• 14 digits → milliseconds, post-2001 (still safe)
• Anything else → flag for manual review, do not guess

This heuristic is not perfect, but it eliminates the vast majority of misclassifications in a batch parse job. The edge cases — where it breaks — are what the rest of this guide addresses.

Assuming Clean Input vs Handling Dirty Reality

The Trap: Trusting Your Data Source

Your batch ingestion script reads a CSV, pulls the timestamp column, and runs every value through a Unix timestamp converter. The documentation says the column is in milliseconds. The documentation was written by someone who left the company in 2019. The format changed twice since then without anyone updating the docs.

Here is a real pattern we see in production log files from a mixed infrastructure stack:

event_id,timestamp,user_id
1001,1700000000,user_a
1002,1700000000000,user_b
1003,1700000000,user_c
1004,1.700000000E9,user_d

Row 1 is seconds. Row 2 is milliseconds. Row 3 is seconds again. Row 4 is scientific notation, because someone's Python script used float() instead of int() before writing to the log. If your batch parser does not account for this, you will produce a dataset where the same real-world second appears as three different dates.

The Fix: Normalize Before You Classify

Strip everything down to raw digits before applying your length heuristic. Remove decimal points. Remove scientific notation by converting to a full integer string. Strip whitespace and trailing newlines that snuck in during file transfer. Only then do you count digits and classify.

A robust batch parser pipeline looks like this:

1. Read raw string — do not cast to int or float yet
2. Strip whitespace and remove any non-numeric characters except a leading minus sign (for pre-1970 timestamps, rare but possible)
3. Convert scientific notation to a full digit string if detected
4. Count digits and apply the length-based heuristic
5. Convert to seconds — divide by 1000 only if classified as milliseconds
6. Validate the result against a sane date range (more on this below)

One Global Rule vs Per-Source Rules

What Fails: A Single Parsing Strategy for All Sources

You have five data sources. Each one has its own timestamp format. You write one parser, apply it to all five, and hope for the best. The hope is misplaced.

Source 1 outputs seconds. Source 2 outputs milliseconds. Source 3 outputs seconds but occasionally pads with trailing zeros, making a ten-digit timestamp look like thirteen digits. Source 4 outputs milliseconds but truncates the last digit during export, giving you twelve digits instead of thirteen. Source 5 is fine, but only because nobody has touched it in six years, and the moment someone does, it will stop being fine.

A single global rule cannot handle this. You need per-source metadata.

What Works: Tag Every Source, Parse Per Tag

When you ingest a batch, record where it came from. Attach a source identifier to every record before parsing. Then apply source-specific rules:

• Source A (known seconds) → parse as seconds, no division
• Source B (known milliseconds) → parse as milliseconds, divide by 1000
• Source C (mixed, unknown) → apply the length heuristic, then validate
• Source D (known broken) → quarantine for manual review, do not auto-parse

This approach turns a chaotic batch parse job into a structured pipeline. When a new source appears, you classify it once, tag it, and every future batch from that source gets handled correctly without guesswork.

Skipping Validation vs Catching the Unthinkable

The Trap: Parse and Ship

You classified the timestamp. You converted it. You wrote it to your output store. You did not check whether the result makes sense. This is how a thirteen-digit timestamp that was actually a ten-digit timestamp with three zeros of padding becomes a date in the year 53914 — and nobody notices until a customer asks why their event log shows activity from the 54th century.

The Fix: Range-Check Every Converted Timestamp

After conversion, validate the result against a reasonable bounds window. For most applications, any timestamp before January 1, 2000, or after December 31, 2100, is suspicious. Define your own window based on your data, but define one.

Here is a concrete example. You parse the string 1700000000000. Your heuristic says thirteen digits, so it must be milliseconds. You divide by 1000 and get 1700000000 seconds. Convert that to a date: November 14, 2023. That falls within your window. Good.

Now you parse 1700000000 but your heuristic misfires because of a trailing zero bug in the source data, producing 1700000000000 internally. You divide by 1000, get 1700000000, and the date is still November 14, 2023. The range check passes. But the original value was seconds, and you treated it as milliseconds. The date happens to be correct only because dividing by 1000 and then the bug adding three zeros canceled out. This is a false positive, and it is why range-checking is necessary but not sufficient.

For critical pipelines, add a second layer: spot-check a random sample of converted timestamps against the raw source. If five percent of your sample looks wrong, stop the batch and investigate before shipping.

Building a Resilient Batch Epoch Parser

The goal is not to write a parser that handles every possible timestamp format. The goal is to write one that fails loudly when it encounters something it cannot handle, instead of failing silently by producing wrong dates.

Here is the summary of what a resilient batch epoch parser for mixed millisecond and second formats should do:

1. Never assume a single format across all records in a batch, even if the documentation says otherwise
2. Normalize raw strings — strip whitespace, handle scientific notation, remove non-numeric characters before classification
3. Use digit count as the primary heuristic — 10-11 digits means seconds, 13-14 digits means milliseconds, anything else gets flagged
4. Tag sources and apply per-source rules when the origin is known, falling back to the heuristic only for unknown or mixed sources
5. Range-check every result against a sane date window and quarantine anything outside it
6. Log every classification decision so you can audit why a timestamp was treated as seconds or milliseconds after the fact

Mixed-format epoch timestamps are not going away. As long as different systems, different SDKs, and different generations of infrastructure coexist in the same pipeline, you will encounter batches where 1699999999 and 1699999999000 sit in the same column. The difference between a parser that handles this gracefully and one that does not is not complexity — it is discipline. Count the digits. Check the range. Tag the source. Fail loudly. Your dashboard, your stakeholders, and your future self will all benefit.