go-richdoc logo go-richdoc
LICENSE=BSD-3-ClauseCGO_ENABLED=0
Fork me on GitHub

One document model,
and the formats that meet in it.

A document is worth one representation, not one per format. richdoc is that representation, and the rest of this organisation reads and writes it — so Markdown, reStructuredText, LaTeX, HTML, an EPUB, a FictionBook, an email and a CSV all meet in the middle instead of pairwise. Pure Go, CGO_ENABLED=0, GOOS=js included. BSD-3-Clause.

The model → Read the source
10 modules10 build targets72 pairwise converters not written
main.go
// Read one format, write another. The middle is the point.
doc, _ := markdown.Parse(src)
out, _ := rst.Write(doc)

// An ebook, a letter, a page, a paper, some rows.
doc, _ = epub.Parse(book, epub.Limits{})
doc, _ = email.Parse(eml, email.Limits{})
doc, _ = html.Parse(page, html.Limits{})
doc, _ = latex.Parse(paper)
doc, _ = csv.Parse(rows, csv.Options{})

// And what a document IS, all of it:
Heading Paragraph List CodeBlock BlockQuote
Table ThematicBreak MathBlock RawBlock
Text Emph Strong Strikethrough Code Link
Image Math LineBreak RawInline
One organisation, 10 modules, every one of them pure Go
richdocmarkdownrstlatexdatafb2htmlepubmobiemail
building for 10 targets, js/wasm among them
linux/amd64linux/arm64linux/riscv64linux/loong64linux/ppc64lelinux/s390xdarwin/amd64darwin/arm64windows/amd64js/wasm
Why a model at all

9 formats pairwise would be 72 converters.

Each would have to understand two formats at once, and every format added would need another for each one already there. Through a model it is two pieces of work per format — read it, write it — and a reader written once is a reader every writer can use. That is the whole architecture, and the arithmetic is why.

🔒

Closed interface sets

Every block and inline type carries an unexported marker method, so a converter can type-switch over all of them and the compiler knows when a case is missing. A model open to new node types would make every converter silently incomplete.

📦

Nothing but the tree

richdoc does no rendering, no parsing and no I/O. It is the one dependency every converter here shares, and the reason it can be shared is that it has nothing in it to disagree with.

🪆

A cell holds blocks

Not a line of text. reST's grid tables and HTML's <td> both allow a paragraph, a list and a table inside one cell, so a model whose cell is a string cannot carry what either format wrote.

🧾

RawBlock and RawInline

Format-specific passthrough, kept with the format's name beside it. It is what lets a LaTeX round trip return the macro nobody has modelled yet, instead of dropping it and calling the result lossless.

The two directions

Read it once, write it anywhere.

Markdown, reStructuredText and LaTeX read and write; HTML, EPUB, FictionBook, email and the data formats read. Which directions a module has is in its description below, because "converter" said of a one-way reader is the kind of claim that costs somebody an afternoon — and a module that only reads is not unfinished: an EPUB and a mailbox are sources, and writing one back is a job nobody has asked for.

zsh
$ richdata sales.csv
| Region | Units | Revenue |
|:-------|------:|--------:|   a table is the obvious shape for a CSV;
| North  | 1 204 | 48 160  |   what the package decides is everything else
$ richdata -format json -title Orders orders.json
$ richdata -o notes.md -verbatim server.log
a log file left verbatim, rather than parsed
into something it is not
The organisation

One module a format, and the model in the middle.

Each is its own module with its own release line, so a program takes the model and the one format it reads — never a stack of parsers it will not call.

richdoc ready

The model itself — an ordered list of blocks and a map[string]string of metadata, with no rendering, no parsing and no I/O at all. ⛔ Blocks and inlines are closed interface sets: every concrete type carries an unexported marker method, so a converter can type-switch over them and the compiler knows when a case is missing. RawBlock and RawInline carry format-specific passthrough, which is what lets a round trip keep what the model has no word for. A cell holds blocks, not a line of text, because reST’s grid tables and HTML’s <td> both allow a paragraph, a list and a table inside one cell.

repository · pkg.go.dev

markdown ready

CommonMark and GFM, both directions. Parsing delegates to goldmark, the reference Go implementation — tables, strikethrough, PHP-Markdown-Extra footnotes — rather than hand-rolling a CommonMark parser. ⛔ WithAutoHeadingID is deliberately off: auto ids would fabricate an anchor on every heading, and the writer would then stamp a {#id} the source never had. Only an anchor an author wrote is kept, which is what makes the round trip faithful rather than merely reversible.

repository · pkg.go.dev

rst ready

reStructuredText, both directions, on go-docutils’s engine. ⛔ It ships no reST parser of its own, because there is no gap to fill: docutils/rst is already a full, tested, pure-Go doctree producer, and a second one here would duplicate real work for no fidelity gain. That dependency doubles as this package’s own correctness proof.

repository · pkg.go.dev

latex ready

A practical subset of LaTeX in, a compilable article out. ⛔ This one does ship its own parser, and the reason is worth knowing: go-tex/engine is a typesetting engine whose tokenizer is internal — it exposes a compile API, not a reusable parse tree. The preamble is mined for metadata, the document environment is read for content, and anything the model has no node for survives verbatim as RawInline/RawBlock with Format: "latex".

repository · pkg.go.dev

data ready

CSV, JSON, XML and plain text into a document, plus the richdata command that writes the result as Markdown. A table is the obvious shape for a CSV; what the package decides is everything else — when a JSON object is a table and when it is a list, and when a log file is better left verbatim than parsed into something it is not.

repository · pkg.go.dev

fb2 ready

FictionBook 2 (.fb2) in — the one ebook format a reader can do with encoding/xml and nothing else: a single XML file, no ZIP, no package document, no spine, images base64 inside that same file. ⛔ Section nesting is where a heading’s level comes from, since FB2 has no <h2>; a title broken over several <p> is one heading; an image becomes its bytes as a data URL, because #cover.jpg is a name nothing can resolve once the file is closed. One token representation rather than two typed fields, after encoding/xml silently filled only one of two ,any fields and every test came back empty with no error.

repository · pkg.go.dev

html ready

HTML and XHTML in, with the parse delegated to golang.org/x/net/html — the piece the ebook formats were missing, since EPUB’s chapters are XHTML and an email’s body usually is too. ⛔ What it drops is the interesting part: script, style, noscript and the head’s contents go completely, because a stylesheet is text in the tree like any other text and is the reason naive converters emit a page of CSS as the first paragraph. The test for it reads a real published page and fails if prefers-color-scheme reaches the document. ⛔ Parsing XHTML with the HTML parser is also the safe choice: the HTML5 algorithm has no DTD, so an ebook cannot ask it to fetch a file or expand a billion laughs.

repository · pkg.go.dev

epub ready

EPUB 2 and 3 in. An EPUB is three things in a trench coat — a ZIP, a manifest and an order — and none of them is the content. ⛔ Nothing is found by its path: the container names the package document, which names its content, and the spine is the only thing that says what order to read it in; an alphabetical walk of the archive puts chapter 10 before chapter 2. An image becomes its bytes, and the same picture referred to fifty times is inflated once. Measured on eleven real books (7 480 to 377 683 words each) which are deliberately not in the repository: they are somebody’s library, and a fixture a test ships has to be one the repository may carry.

repository · pkg.go.dev

mobi ready

Kindle MOBI in — three formats stacked on each other, and reading one is reading all three: a Palm database of records, a record 0 holding the PalmDOC and MOBI headers, and the book itself as ONE HTML document cut into 4096-byte pieces and each piece compressed on its own. ⛔ The bytes at the end of a text record are not text and are not compressed — indexing data, and the overlap of a multibyte character split across two records — and feeding them to the decompressor does not fail: it produces plausible rubbish at the end of every record. ⛔ Pictures are named by record number, two different ways, and kindle:embed:003C is base 32 and one-based: read as hexadecimal it would be wrong for every picture past the thirty-first and still look right for the first nine. Measured against Project Gutenberg’s own Pride and Prejudice with Gutenberg’s plain text of the same book as the judge: 125 816 words against 126 630, and 3 638 of 4 215 long sentences found verbatim. DRM, HUFF/CDIC and KF8 are refused by name, because none of those files is defective.

repository · pkg.go.dev

email ready

RFC 5322 messages and MIME in — a .eml, or whatever an IMAP server hands back. ⛔ Which part is the body is the whole job: an alternative is the same message twice and must be read once, a related is one message plus the pictures it names by Content-Id, a mixed is a sequence, and a signed is the message plus a signature that is not content. ⛔ Nothing is fetched, ever: an HTML mail’s remote <img> is a tracking pixel as often as a picture, so only cid: parts that travelled inside the message are inlined — there is no network code in the package and no way to ask for any.

repository · pkg.go.dev

What a round trip is for

Reversible is easy. Faithful is the work.

A converter that loses nothing it can represent is still wrong if it adds what the author never wrote — and that is the harder failure, because nothing is missing to notice.

🏷️

An anchor nobody wrote

goldmark can generate an id for every heading. markdown leaves that off, because the writer would then stamp a {#id} into source that never had one — a diff on every heading of every file, from a converter that was asked to preserve the document.

🔁

Delegate where a parser exists

CommonMark goes through goldmark and reST through go-docutils, because both are full, tested, pure-Go parsers. latex ships its own only because go-tex's tokenizer is internal — a compile API, not a parse tree.

📐

A heading's level is where it sat

FictionBook has no <h2>, only a section inside a section, and a title broken over several <p> for the line breaks. Three headings where a reader sees one is a document that navigates wrongly.

⛔

Said, when it is not kept

<sub> and <sup> have no node here, so FictionBook's text is kept and its position is not: losing the 2 of H2O to say so would be the worse trade. Every such decision is in the module's README, with the reason.

Where it runs

10 build targets, and the suites run on amd64.

⛔ Said precisely, because the precise claim is cheap and the vague one is not true: every module here cross-builds for linux/amd64, linux/arm64, linux/riscv64, linux/loong64, linux/ppc64le, linux/s390x, darwin/amd64, darwin/arm64, windows/amd64 and js/wasm, and runs its tests on amd64 with the race detector. Where byte order is the risk rather than a worry — a ZIP's little-endian integers in go-ooxml — the suite runs under qemu on five more architectures, big-endian included. These libraries are a tree and a great deal of string handling, which is a different risk.

🌐

js/wasm is a target, not an afterthought

Nothing here opens a file: a converter takes bytes and returns bytes. An editor that converts a document in a browser tab needs no server, and nothing it reads leaves the machine.

🧮

CGO is off everywhere

Which is what makes js/wasm and the cross-builds possible at all, and why a binary that embeds any of these carries no C toolchain's worth of assumptions.

📚

Documentation of its own

The docs site is built by MkDocs and versioned with mike, so a reader can look at the release they are using rather than at main.

🖨️

And out the other side, a PDF

go-pdfkit composes a richdoc document, which is what makes .fb2 → PDF or .md → PDF a two-step with no format-specific code in between.

The model, 9 formats, and one place where they meet.

A document read here can be written as Markdown, reStructuredText or LaTeX, laid out as a PDF by go-pdfkit, or edited in a browser tab — because all of them speak the same tree.

Read the source on GitHub →