# The fetcher, fetch.mjs

fetch.mjs reads the run context and your downloaded files, prints one record a line, and stops loudly when a file is not what it knows.

Kanboombo starts `node fetch.mjs` in your folder, writes the run context to its
standard input, and reads records from its standard output. That is the whole
protocol: files in, records out. The
[quickstart](/developers/plugins/quickstart/#4-teach-it-your-file) writes one in
twenty lines.

## The run context {#run-context}

```json norun="Kanboombo writes this; you only read it"
{
  "contract": 1,
  "plugin": "harbour-almanac",
  "now": "2026-09-30T19:30:00.000Z",
  "today": "2026-10-01",
  "backfill": true,
  "backfillFrom": "2025-10-01",
  "sources": { "tides": "/sources/tides", "catch": "/sources/catch", "council": "/sources/council", "notices": "/sources/notices" },
  "series": { "tide": { "last": null, "count": 0 }, "catch": { "last": null, "count": 0 }, "council": { "last": null, "count": 0 }, "notices": { "last": null, "count": 0 } }
}
```

Use `today`, the date in your zone, not `now`, the moment in UTC. Read each
source at the path `sources` gives (under `/sources/` on the Server, your
fixture under `check`); never build one. Under `check`, `now` and `today` come
from `fixtures/context.json`, and each `fixtures/runs/<case>` runs with
`backfill` false.

### PluginRunContext {#pluginruncontext}

What the fetcher reads on stdin. sources maps each declared source's name to a file it may read.

| Field | Type | Required | Description |
|---|---|---|---|
| `contract` | `1` | yes | The contract the run follows: 1. |
| `plugin` | [PluginId](/developers/plugins/reference/manifest/#pluginid) | yes | Your plug-in's id. |
| `now` | string; at most 40 characters | yes | The moment the run started, in UTC (ISO 8601). |
| `today` | [PluginDate](/developers/plugins/concepts/records/#plugindate) | yes | Today's date in your manifest's time zone. Nothing you print may be dated after it, except a scheduled event. |
| `backfill` | boolean | yes | True on the first run, when nothing is stored yet: print the whole history your backfill allows. |
| `backfillFrom` | [PluginDate](/developers/plugins/concepts/records/#plugindate) or null |  | The first date a backfill fills, from your manifest's backfill span; null for all. |
| `sources` | object of string | yes | Each declared source's name and the file you may read it from. Never build these paths yourself. |
| `series` | object of [PluginSeriesContext](/developers/plugins/build/fetcher/#pluginseriescontext) | yes | What is already stored for each series. |

### PluginSeriesContext {#pluginseriescontext}

What is stored for one series before this run.

| Field | Type | Required | Description |
|---|---|---|---|
| `last` | [PluginDate](/developers/plugins/concepts/records/#plugindate) or null | yes | The date of its newest stored record on or before today, or null. |
| `count` | integer; at least 0 | yes | How many of its records are stored. |

## The rules it lives by {#rules}

- **Date by `today`**, never `new Date()`; nothing after it but a scheduled
  event.
- **Records on stdout, everything else on stderr**, which readers never see;
  a failed run's last stderr line (at most 300 characters) is its reason.
- **Exit 0 when the file reads as expected**, even with nothing new; **exit 2
  when it is not what you know** (a renamed sheet, a missing column, a new
  status), with one line saying why. Nothing is written, and readers see it
  could not refresh: nothing beats something wrong.
- **Print the whole truth for your snapshot windows**: a record you stop
  printing inside one is withdrawn after three runs. More costs only time;
  Kanboombo writes only what changed.
- **The same files give the same bytes**: sort, and print no timestamps or
  random numbers. The Server runs your fixtures twice.
- **Stay in your folder.** Import only your own files and `node:` modules. No
  network, database, keys or environment beyond the locale; only the files the
  digest covers are there. A run stops at 120 seconds and 256 MB, or your lower
  `limits`.

```js norun="the pattern every example uses"
function fail(reason) {
  process.stderr.write(`my-plugin: ${reason}\n`);
  process.exit(2);
}
```

The publishing portal takes no dependencies: copy what you need into your
folder as readable `.mjs`. (A plug-in in Kanboombo's own release may declare
them in `package.json` with a `package-lock.json`.)

### How far back to print {#how-far-back}

On the first run, print what `backfillFrom` allows; the first date is yours to
keep. After that, print at least each series' snapshot window, and further
back when its newest stored record is older, so days missed while runs failed
are filled:

```js norun="from plugins/harbour-almanac/fetch.mjs, which check runs"
function since(series, days) {
  if (context.backfill) return context.backfillFrom ?? "0000-01-01";
  const window = addDays(today, -days);
  const last = context.series?.[series]?.last ?? null;
  return last !== null && last < window ? last : window;
}
```

## Read CSV, JSON and XLSX {#reading-files}

Both examples carry their readers, MIT-0, to copy. Whatever the format, find
columns by name, never position; stop on what you do not know; and convert
nothing you need not: a date written `2026-10-01` stays that string, never
passed through `new Date()`.

- **CSV**: [Harbour Almanac's `csv.mjs`](/developers/plugins/examples/harbour-almanac/#csv-mjs)
  reads quotes, line breaks inside them, CRLF or LF and a byte-order mark, keys
  each row by its header, and refuses a file without the columns you need.
  Convert a decimal comma (`1.234,5`) or another encoding
  (`new TextDecoder("windows-1252")`) on purpose, and fail on what will not
  convert.
- **JSON**: `JSON.parse` reads; check the shape before you trust it, and map
  the source's words to the contract's in one table (`held` to `happened`),
  failing on any word you have never seen.
- **XLSX**: a workbook is a ZIP of XML. [DOP Economy's](/developers/plugins/examples/dop-economy/)
  `xlsx.mjs` reads the ZIP's directory, inflates the parts it needs with
  `node:zlib`, and returns one sheet as rows.

```js title="plugins/dop-economy/xlsx.mjs" source=plugins/dop-economy/xlsx.mjs head=24
// Reads the cells of one sheet of an .xlsx workbook with nothing but Node.
//
// A workbook is a zip of XML files. This reads the zip's central directory,
// inflates the four parts it needs (the workbook, its relationships, the
// shared strings and the one sheet) and returns the sheet as rows of cells.
// It imports nothing outside Node, because a plug-in imports nothing outside
// its own folder (docs/kanboombo/PLUGIN-CONTRACT.md).

import { inflateRawSync } from "node:zlib";

const EOCD = 0x06054b50;
const CENTRAL = 0x02014b50;
const LOCAL = 0x04034b50;

export class WorkbookError extends Error {}

/** The zip's entries, by name: { method, compressedSize, offset }. */
function entries(buf) {
  let end = -1;
  for (let i = buf.length - 22; i >= Math.max(0, buf.length - 65557); i -= 1) {
    if (buf.readUInt32LE(i) === EOCD) { end = i; break; }
  }
  if (end < 0) throw new WorkbookError("not a zip file (no end of central directory)");
  const count = buf.readUInt16LE(end + 10);
```

For a large file (up to 20 MB): read it once, keep only what you print, and
print as you go.