- tags
- #help #implementations #architecture #wip
- description
- How to organise the code of a Node app so it stays workable as it grows, for people and for AI agents: folders by feature, a kernel below them, one order between them, and a check that holds it.
- brief
- The guidelines every Node or TypeScript build follows: folders by feature with the same layers inside, a kernel that imports no feature, one order between features, features that register themselves, and a check that refuses an import against the rules.
Node apps: how to organise the code
For any Node or TypeScript build, a server above all. A new build starts this way; an existing one gets there by the steps at the end. d2's own server follows it, and is the worked example.
The rules in one screen:
- Folders by feature, the same few layers inside each.
- A kernel below the features, which imports none of them.
- One order between the features: each imports only the ones before it.
- A feature registers itself; one file starts the app and only collects.
- What a lower module needs from a higher one is passed in, declared below, or sent as a message.
- A check holds all of it, with a baseline for what exists today, so it only gets better.
- Small files, no import cycles.
- Tests are picked by what changed.
Folders by feature
The top folders of src/ are the app's features, named as its spec names them (accounts, payments, mailer), never by kind of file (controllers, services, models). A change to a feature then touches one folder.
Inside each feature, the same layers:
| Layer | Holds | May import |
|---|---|---|
core/ |
the rules and the records; runs with no request | other core/, the kernel |
web/ |
routes, request handling, the feature's own wiring | core/, pages/, the kernel |
pages/ |
what a person sees | core/, the kernel |
- A
core/file never imports aweb/orpages/file. If it seems to need one, see what a lower module needs. - Keep a feature under about 5,000 lines. Past that, it is two features: cut it by the spec's names.
A kernel below the features
src/kernel/ holds what every feature stands on: the log, ids, the store, the message bus, the request helpers. One rule defines it: the kernel imports no feature.
- A file joins the kernel when many features import it and it needs nothing outside the kernel to run.
- A file that is widely used but imports a feature is not kernel yet: first cut what it needs from above, then move it.
- Do not make a
utils/orshared/folder. Whatever is not a feature is kernel, and the kernel has the rule.
One order between the features
Write the features down in one line, lowest first: each imports only the ones before it (and the kernel).
quotas < accounts < vault < mailer < payments < notices
- Choose the order by what each feature is, not by which imports are cheapest to fix today. An identity does not need a mailbox to exist; a mailbox needs to know whose it is.
- Two features that import each other are one feature, or one of them holds something that belongs lower.
- A page that shows parts of several features sits above them all, or asks each for its part (registration).
A feature registers itself
One file starts the app (server.ts, the composition root). It may import everything, and nothing imports it. It holds only what must run in a fixed order at start; everything else a feature declares in its own folder:
- its routes, as a list the root collects;
- its listeners, caches and timed jobs, in one
register()the root calls; - what it adds to shared lists (a menu, a home page's cards), as a contribution the list's owner asks for.
A new feature then adds its own files and edits no shared one. That is what lets several people, or several agents, work at once without meeting in the same file.
What a lower module needs from a higher one
Sooner or later the store needs a tag parser, or accounts needs to send a mail. Never import upward. In order of preference:
- Pass it in. The caller that knows both hands the function over:
backfillTags(readTags). - Declare it below, register it above. The lower module declares the small interface it calls; the higher one registers its implementation in its
register(). - Send a message. The lower module raises an event or a request on the bus; the higher one listens. Use this when nothing comes back, or when the answer can wait.
- Move the code. If the lower module needs most of the higher one, the code is in the wrong place.
Avoid one shared hooks object that everyone fills: it hides who depends on whom. Avoid a dependency-injection container until the first three stop being enough; most apps never get there.
A check holds it
A rule that is only written down is broken by the next change. Put the rules in the project's check, the one command that runs before every merge:
| Rule | Refuses |
|---|---|
| kernel | a kernel file importing a feature |
| layer | a core/ file importing a web/ or pages/ file; anything importing the root |
| order | a feature importing one that comes after it |
| cycle | a file joining a cycle of imports that run at load |
- Start with a baseline. On an existing app, write down every import that breaks a rule today. The check passes now; a new one is refused; and an entry the code no longer needs fails too, so the list only shrinks.
- A refusal says what to do instead, naming both files. An agent reads the message and fixes it in one try.
- The tool: dependency-cruiser or eslint-plugin-boundaries do this. A script of your own is about 150 lines, runs in under a second and has no dependency; d2 uses one, because the tools that read TypeScript's own API lag behind a new TypeScript.
- Count only
import typeas a type import. Animport()at call time is how a cycle is cut on purpose, so it is not a cycle edge.
Small files, no cycles
- Most files under 300 lines; none over 800 without a reason written at its top. A long file is read whole by an agent every time it is touched.
- One thing per file, named for it. A file named
helpersis the start of a second kernel with no rule. - No import cycles at load. A cycle means neither file can be loaded, tested or moved alone.
- Say why at the top of a file, in two lines: what it is for and what it must not do.
Tests
What a large app's tests cost is mostly how they are organised, not what they check. These are the rules d2's own tests were rebuilt on (about 1,300 test files, 4,900 tests), with what each one saved or prevented.
Four kinds, by what a test needs
| Kind | What it needs | Where it runs | When |
|---|---|---|---|
| Unit | no server, no child process | all in one process, first | every change |
| Server | the app answering requests | one shared server per feature | every change that touches the feature |
| Page | a real browser | a shared server of their own per feature, apart from the server tests | a release, or a change to pages or styles |
| Rare | a server of its own: it restarts, stops or kills it, or writes its files by hand | alone | the nightly run and a major release |
- A test's kind is a fact the runner can see (does it start a server, a child process, a browser), not a folder or a mark someone keeps.
- One server per feature for its server tests. A second one only for its page tests, because they are several times slower and would slow the run a small fix waits on.
- Rare is the one exception to sharing, and it is named as such with its reason on the test.
A test says what it is
- Put what a test is on the test's own line: its feature, and whether it is core, rare, late or a page test. The runner picks by that.
- Do not keep lists of test files by name (this Suite's files, the slow ones, the ones allowed an exception). Every such list goes stale the first time a file is split, moved or renamed, and a stale list either drops a test from a run without anyone noticing or fails a check for no reason. Most of the breakage in d2's reorganisation came from these lists, not from tests.
- Where a list is unavoidable, a check must refuse an entry whose file is gone and a file the list should hold and does not.
Few files, not one per change
- A file per ticket is how a suite ends up with a third of its files holding one test. Each file on a shared server still pays to start a process and to build its own project and people: about 2.6 s a file in d2, several minutes a run, for nothing a test checks.
- Group tests by area within a feature (vault, tokens, plans), a few files per feature. Not one file per feature either: a file of hundreds of tests is read whole by an agent each time and collides on every merge.
- Load a feature's files into one process and let them share one world where they can.
Writing a test that can share a server
A test that passes alone and fails beside others almost always breaks one of these:
- Take every name from the test's world: project names, emails, handles. A shared server numbers them, so a name written by hand is someone else's or nobody's.
- Count only your own things. Never assert on "all projects", "the whole audit log" or "the admin's list": other tests' rows are in it.
- Never assume the server is empty or new. In particular, never rely on being the first account. If the first sign-up gets special rights, the harness takes that slot itself and a test that needs those rights asks for that account by name. In d2, 361 files had passed only because their first person happened to be the administrator.
- Put back anything server-wide you change (a setting, a shared page, a base record), in a
finally. The file that leaves it changed is the one to fix, not the file that trips on it. - Ask for the shared server's settings, not your own. A setting a test really needs different is set for that file by the runner and put back; that only works for settings the app reads on each use, so prefer those over settings read once at start.
- Do not read the server's files. A test that opens the data folder to see that something was recorded reads the same record through the API instead. A test about what is stored (a secret sealed at rest, a record's shape) runs against the store in process with a folder of its own, as a unit test. A test that writes a file by hand and restarts is rare.
Tests that are slow on purpose
- A test that trips a rate limit, a quota or a guard on purpose goes in a file of its own and runs only at a major release. On a shared server it throttles everyone else's test, and it usually waits a window out.
- A test that only waits (a real 15-second hold, a retry window) is marked as a timing test and runs after the release, not in a fix. Better: make the wait a setting a test can shorten.
- Tests of the test harness itself that run nested suites are slow by nature: late too.
Weigh every test against its cost
- Measure per test, not per run: the runner records each test's seconds. Add its share of its file's setup; a file with one small test is an expensive test.
- A test over a few seconds has to earn it: it is the test of a core rule, or the only test of its rule. Otherwise it is late, or it waits until it is cheaper. Never delete a test to save time: park it, with why.
- Measure before you optimise. Two guesses were wrong in d2. A browser launch was assumed to cost a second and cost 50 ms, so sharing one browser across tests saved 2%. Running page tests on a DOM library with no browser was assumed to remove most of their time and removed a quarter, for a 20 MB dependency and no layout: the time was in the pages' own timers and real round trips. The lever was the waits inside the tests.
What runs when
- A change runs the unit tests, then the tests of the files it touched and of what imports them. Everything runs at a release; the rare and late tests at a major release and nightly.
- When tests are missing from a gate, the gate says so in its pass line ("24 of 25 listed files not checked"). A gate that passes quietly on nothing is worse than no gate.
- When files move, they keep the tests they had: prove it by comparing the picked tests before and after.
Reorganising a large suite
What worked, in order:
- Park every test in a folder beside the tests, same sub-folders, moved and not copied. Keep the harness and the fixtures in place, and make every check and runner pass on nothing.
- Bring them back in order of cost and risk: unit tests; the server tests that share a server as written; then the rest by cause (names written by hand, settings, the first-account habit, the data folder), reworking each to share. Whatever cannot share honestly stays parked with its reason.
- Keep the branch releasable at every step: each batch is a move plus the smallest edits, the checks pass, the suites it touched pass twice in a row.
- Never weaken an assertion to make a test share. After a rework, break the code on purpose and see the test fail: twice in d2 a new security test still passed with its guard removed.
- A reviewer opens the test that is said to cover a dropped case. That claim was false in six of eight batches.
- With several agents at once: one cause or one folder each, each adding its files in a table of its own so merges do not collide, one heavy run at a time on a small box, and push straight after fetching.
Node running TypeScript directly
Node 22 and later run .ts files with no build, by erasing the types. It shapes the code:
- Only syntax that can be erased: no
enum, no parameter properties, no decorators (so no NestJS, no tsyringe). SeterasableSyntaxOnlyandverbatimModuleSyntax. - Imports carry the
.tsextension and are relative.tsconfigpaths do not exist at run time; if you want an alias, use"imports"inpackage.json(#kernel/log.ts). tsc --noEmitis the type check, a step of the check.
What not to reach for first
- Workspace packages (a monorepo of
packages/*): they give real walls, but packages cannot import each other in a circle, so the untangling above comes first anyway, and then they add little. Use them when a part is published or deployed on its own. - A framework for structure (NestJS and the like): it brings its own module system and decorators. Folders, a kernel and a check do the same with nothing to learn or upgrade.
- Layers by kind (
controllers/,services/,repositories/at the top): every change then crosses every folder.
Starting a new app
src/
server.ts the root: starts the app, collects the features
kernel/ log.ts ids.ts store/ bus.ts server/ (request helpers)
<feature>/
core/ rules and records
web/ routes, register()
pages/
test/
<feature>/ the feature's tests
scripts/
check.ts type check + the import rules, one command
importcheck.ts the four rules and the order
AGENTS.md ten lines: the check's command, the order of the features, this page's address
Write the order of the features and the check on day one, with an empty baseline. It costs an hour then and weeks later.
Reorganising an existing app
In this order; each step lands alone and changes no behaviour:
- Measure: which features import which, the cycles, the files everyone imports, the largest files.
- The check, with a baseline of what exists. From here nothing gets worse.
- The kernel: move the widely imported files that need no feature. Move files with a script that rewrites every import, never by hand.
- Registration: move each feature's wiring out of the root into the feature.
- Cut the big features by the spec's names, one commit each.
- The order: write it, baseline what runs against it, turn those imports around a few at a time.
- Split the long files last: it is easy once the walls exist.
The worked example: d2's own server
A TypeScript server on Node, no build, about 130,000 lines in 770 files. Measured before the work: 18 features where 82 pairs imported each other, 81 imports from core up to web, 56 files in import cycles, one 4,000-line root that most changes edited, and no check on imports. The check with its baseline went in first; then 25 files moved to a kernel in one scripted commit that rewrote about 1,300 import paths and kept every file's tests the same; then the largest feature was cut into six along the spec's component names, with the order above.
See also
- Implementations: one spec, several builds; New implementation links this page into a Node build.
- Connect GitHub: repos for your AI agents.
- Outside d2: VS Code's source organisation (layers and a kernel, checked), Node.js best practices (folders by component), Shopify's Packwerk retrospective (what boundary checks did and did not prove), Node's TypeScript support.