The formats that outlive us

The comma-separated value file is older than most careers in this industry and was only formally described in 2005, when RFC 4180 arrived to document, with visible resignation, what everyone had already been doing for thirty years. The RFC standardized nothing so much as the disagreements: the escaping edge cases, the header ambiguity, the encoding silence. CSV is, by any engineering standard, a poor format. It is also, by observed behavior, the most successful data interchange technology ever shipped, and the gap between those two sentences is this essay's subject.

Formats are contracts, and they are the longest-lived contracts computing produces. Applications are rewritten, platforms migrate, vendors rise and are acquired; the files remain, because the files are where the past is stored, and the past does not refactor itself. A format choice is therefore a commitment made on behalf of one's successors, with a duration measured in decades, which would suggest the industry chooses formats with corresponding gravity. The industry chooses them in a sprint, from whatever the framework defaults to.

Why did CSV win? Not despite its poverty; because of it. The format is readable by a human squinting at a text editor, writable by any language's print statement, and parseable, badly, by code a student writes in an afternoon. Its floor is so low that everything can trip over it, which means everything did, which means every system that has ever existed grew a CSV path, and ubiquity compounds the way stable interfaces always compound: each adopter makes the next adoption more inevitable. The lesson is uncomfortable for engineers who equate quality with merit. In formats, accessibility is fitness, and the theoretically superior format that requires a library loses, decade after decade, to the mediocre one that requires only split(',') and courage.

Parquet, the columnar workhorse of the modern data estate, looks like the counterexample and is closer to a confirmation. It is a genuinely good format, columnar, typed, compressed, and it won its niche not on those virtues alone but by being boringly, promiscuously implementable: an open specification, readers in every ecosystem, and a governance home at a neutral foundation rather than a vendor. The table-format settlement of the past few years, Iceberg and its peers converging as open standards, is the same selection pressure operating one layer up: formats win by being nobody's property and everybody's floor.

The failure cases prove the rule by dying. Proprietary formats with single readers, the binary export of a vendor now three acquisitions deep, the "optimized" internal format whose parser lives in one team's repository: these are where data goes to become archaeology. Every enterprise of any age carries tapes and files it can no longer confidently read, and each one traces to the same decision: a format chosen for the writer's convenience rather than the reader's future. The write happens once, this quarter, under deadline. The reads go on forever, under whoever inherits.

Hence the essay's practical residue, three questions to ask of any format decision that will outlive its meeting. Can a motivated stranger implement a reader from public documentation alone? If not, the data's lifespan is the vendor's lifespan, discounted by their acquisition risk. Is there more than one independent implementation in the wild today? One implementation is a product; two is a format. And what is the degradation path: when the tooling is gone, does the file yield to a text editor and patience, or to nothing? CSV's answer to that last question, embarrassingly, is the best in the industry, which is why the format that should have died in 1995 will attend all of our funerals.

Choose formats the way one chooses foundations, on the assumption that everything above them will change. It will. They won't. That is what they are for.