MUI Docs Infra

Warning

This is an internal project, and is not intended for public use. No support or stability guarantees are provided.

Architecture

This page explains the overall architecture of @mui/internal-docs-infra, including data flows, key design decisions, and how the various pieces fit together.

Overview

The package is built around one core principle: move locally-derived computation to build time. The cache only updates whenever those files change.

┌─────────────────────────────────────────────────────────────────┐
│                           BUILD TIME                            │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │  TypeScript  │    │    Source    │    │   Markdown   │       │
│  │   Sources    │    │    Files     │    │    Files     │       │
│  └──────┬───────┘    └──────┬───────┘    └──────┬───────┘       │
│         │                   │                   │               │
│         ▼                   ▼                   ▼               │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │ typescript-  │    │ Starry Night │    │    Remark    │       │
│  │api-extractor │    │ Highlighter  │    │   Plugins    │       │
│  └──────┬───────┘    └──────┬───────┘    └──────┬───────┘       │
│         │                   │                   │               │
│         └───────────────────┼───────────────────┘               │
│                             ▼                                   │
│                    ┌──────────────┐                             │
│                    │     HAST     │  (JSON-serializable AST)    │
│                    └──────┬───────┘                             │
│                           │                                     │
│                           ▼                                     │
│                    ┌──────────────┐                             │
│                    │  precompute  │  (injected into source)     │
│                    └──────┬───────┘                             │
└───────────────────────────┼─────────────────────────────────────┘
                            │
┌───────────────────────────┼─────────────────────────────────────┐
│                           ▼              RUNTIME                │
│                    ┌──────────────┐                             │
│                    │  hastToJsx   │  (HAST → React.ReactNode)   │
│                    └──────┬───────┘                             │
│                           │                                     │
│                           ▼                                     │
│                    ┌──────────────┐                             │
│                    │   Rendered   │                             │
│                    │  Components  │                             │
│                    └──────────────┘                             │
└─────────────────────────────────────────────────────────────────┘

Why this architecture?

  • Syntax highlighting libraries are large (~500KB+) - keep them out of client bundles
  • TypeScript parsing is slow - do it once, cache the results
  • HAST is JSON - webpack can cache it, and it serializes across server/client boundaries

Data Flows

Code Demos Pipeline

When you write a demo with createDemo():

demos/basic/index.ts (createDemo call)
         │
         ▼ [loadPrecomputedCodeHighlighter loader]
         │
    ┌────┴────┐
    │ Parse   │  Find createDemo() call, extract variants
    │ Factory │
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │  Load   │  Read each variant file + dependencies
    │  Files  │  Track all files for webpack watching
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │ Starry  │  Syntax highlight with grammar scopes
    │  Night  │  Add line gutters
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │Transform│  TypeScript → JavaScript (optional)
    │  TS→JS  │  via Sucrase
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │ Inject  │  Insert { precompute: { ... } } into source
    │  HAST   │
    └────┬────┘
         │
         ▼
    createDemo(import.meta.url, Component, { precompute: { ... } })

Key files:

Type Documentation Pipeline

Type documentation has two paths that converge on the same highlighting steps:

Build-time path (precomputed)

When you write types with createTypes():

types.ts (createTypes call)
         │
         ▼ [loadPrecomputedTypes loader]
         │
    ┌────┴────┐
    │ Parse   │  Find createTypes() call
    │ Factory │
    └────┬────┘
         │
         ▼ [loadServerTypes with sync: true]
         │
         ▼ [Worker Thread - preserves TS language service cache]
         │
    ┌──────────┐
    │typescript│  Parse TypeScript AST
    │   -api-  │  Extract exports, components, props
    │extractor │
    └────┬─────┘
         │
         ▼
    ┌─────────┐
    │ Format  │  Format props, find DataAttributes/CssVars enums
    │  Types  │  Parse JSDoc descriptions as markdown
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │  Write  │  Generate and write types.md
    │ types.md│
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │Highlight│  Syntax highlight type strings → HAST
    │  Types  │
    └────┬────┘
         │
         ▼
    createTypes(import.meta.url, Component, { precompute: { ... } })

Runtime path (from existing types.md)

When types are already generated, loadServerTypes can read an existing types.md instead of re-extracting from TypeScript source. This is useful in RSC pages, custom build tools, or when types are committed to the repository:

types.md (already on disk)
         │
         ▼ [loadServerTypes with sync: false (default)]
         │
    ┌──────────┐
    │  Parse   │  Read types.md
    │ Markdown │  Parse back into structured TypesSourceData
    └────┬─────┘
         │
         ▼
    ┌─────────┐
    │Highlight│  Same highlighting as the build-time path
    │  Types  │  Type strings → HAST
    └────┬────┘
         │
         ▼
    HighlightedTypesMeta with HAST fields

Both paths produce identical HighlightedTypesMeta with syntax-highlighted HAST — the only difference is where the type data comes from.

Key files:

Search Pipeline

 
                    [Out of band - transformMarkdownMetadata]
                                     │
    page.mdx files ──────────────────┼──▶ index page.mdx files
    (title, description, sections)   │    (aggregated metadata)
                                     │
─────────────────────────────────────┼─────────────────────────
                                     │
app/sitemap/index.ts (createSitemap call)
         │
         ▼ [loadPrecomputedSitemap loader]
         │
    ┌────┴────┐
    │ Parse   │  Find createSitemap() call
    │ Factory │  Extract page imports
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │  Load   │  Import each index page.mdx
    │ Indexes │  Read pre-extracted metadata
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │ Build   │  Create Orama-compatible schema
    │  Index  │  Index all searchable content
    └────┬────┘
         │
         ▼
    { schema: {...}, data: {...} }  // Injected as precompute
         │
         ▼ [Runtime - Client]
         │
    ┌──────────┐
    │ useSearch│  Create Orama instance from precomputed data
    │   hook   │  Provide search API to components
    └──────────┘

Key files:

Coordinated Loading & Streaming

The pipelines above decide what crosses the server→client boundary and how small it is. A separate runtime architecture decides when deferred content swaps in — and coordinates that swap across every block on the page so it settles as one update instead of a staggered cascade. It is layered, and each layer is usable on its own:

useStream · abstractCreateStream   load a list of pieces   (client stream · server/build)
      ▲
CoordinatedLazy     one deferred piece: fallback ⇄ content swap + coordinated settle
                    (CoordinatedLazy, createCoordinatedLazy, useChunk, LazyContent)
      ▲
useCoordinated      the settle gate + page-wide coordination

Each layer composes on the one below and is usable on its own.

The settle gate

createSettleGate() is the shared "all sources settled" primitive. A source register()s on mount and calls the returned settle() once it reaches its stable layout; the gate opens when every source has settled. It opens once — a source mounting after the page settled adopts the open state rather than re-closing it. Two opt-in completion signals extend it for content that arrives over time: expect(n) (known count) and markLast() (last chunk).

One module-global instance, pageSettleGate, is what the page-wide layoutShiftGate and every CoordinatedLazy swap register with — so useCoordinated can hold a page-wide transform/variant change until the initial swaps have all landed.

The swap — CoordinatedLazy

CoordinatedLazy generalizes the fallback↔content swap that was previously welded into CodeHighlighterClient:

Server: pre-render BOTH                Client: state-driven swap
  fallback = <ContentLoading/>           1. force-mount fallback once
  content  = <Content/>          ──▶     2. fallback hoists data up
                                         3. ready → swap to content
                                            (hoisted data flows down)
  • Force-mount-once — even when the content is precomputed, the loading component mounts for one commit so its hoist hook runs (the same trick CodeHighlighter uses to recover its decompression dictionary).
  • Hoist up / content-context down — the fallback hoists data via useCoordinatedFallback; the swap collects it and hands it to the content via useCoordinatedContent, so the full content reuses what the fallback already fetched (a DEFLATE dictionary, or a low-resolution chart line).
  • Nested suppression — an inner instance nested inside an outer's still-showing fallback stays in its own fallback, collapsing a "fallback → content → fallback → content" flicker into one transition.
  • Settle registration — each swap registers with the page gate (and, inside a controller, that controller's gate too) so the page can coordinate.

A codeblock frame is the simplest instance: its plain-text data.fallback → highlighted HAST is exactly a loading→full swap, and a multi-frame file is serial chunking of one source.

Self-loading pieces & streaming

createCoordinatedLazy() builds a self-loading component that loads one piece's data and swaps via CoordinatedLazy. Its source is a discriminated union (no overloads, no runtime return-sniffing):

source.modeLoads
dataone value directly (optionally a quick initial first)
urlsa list of chunk URLs, then each URL's data
streampushes chunks into an array and yields — progressive reveal

useChunk() loads a single chunk. Driving a list is the layer above, split by who loads: useStream() streams the list in on the client (accumulating into state) and useStreamController() scopes a group so the page can tell when they have all loaded — via known count or a last-chunk flag — while abstractCreateStream() is the server/build counterpart that injects precomputed data. A list controller hands its gate down as the ambient gate, so each chunk's CoordinatedLazy registers with it without a threaded gate prop. On the server, each chunk is its own async component under a per-chunk Suspense boundary, so React flushes them as they resolve.

Speculative preload

When the fallback hoists its data, CoordinatedLazy fires a preload(hoisted) callback. A consumer uses it to start dynamic import()s of the heavy helpers the data implies — a transform function, say — in parallel with loading the full content, instead of the content mounting and then requesting them in a serial roundtrip. PreloadProvider / usePreload dedup those imports across every instance on the page, so a shared helper is fetched once.

Why this architecture?

  • The swap, hoist, and coordination logic is identical whether content is precomputed, server-loaded, or client-loaded — so it lives in one tested primitive instead of being copied per feature.
  • Coordinating swaps avoids layout-shift cascades: blocks settle together, and a later page-wide change can wait for the initial settle.
  • Streaming plus per-chunk Suspense lets large content (a detailed chart, a long file) reveal progressively without blocking first paint.

Key files: useCoordinated/ (settle gate + page coordination), CoordinatedLazy/ (the swap plus createCoordinatedLazy / useChunk / LazyContent / the *Server render functions — one deferred piece), useStream/ (useStream / useStreamController / streamChunks — client list loading), ChunkProvider/ (lazy client-loader context + usePreload dedup), abstractCreateStream/ (the server/build createStream factory).

The Factory Pattern

All precomputable features use the same pattern:

// 1. Define a factory in your app
export const createDemo = createDemoFactory({ DemoContent });

// 2. Use the factory in index.ts files
export const DemoCheckbox = createDemo(import.meta.url, Checkbox);

// 3. Webpack loader detects the factory call
// 4. Loader processes and injects precompute
// 5. Factory receives precomputed data at runtime

Why import.meta.url?

  • Provides the file's absolute URL at both build and runtime
  • Loaders can derive names, paths, and relationships from the URL
  • Enables filesystem-based conventions (like demos/basic/index.ts)

See Built Factories for details.

Worker Thread Architecture

Type processing uses a singleton worker thread to preserve the TypeScript language service cache:

┌─────────────────────────────────────────────────────────────┐
│                    Main Thread (Webpack)                    │
│                                                             │
│  Loader 1 ──┐                                               │
│  Loader 2 ──┼──▶ WorkerManager ──▶ Worker Thread            │
│  Loader 3 ──┘         │                   │                 │
│                       │                   │                 │
│                       │          ┌────────────────┐         │
│                       │          │ TS Language    │         │
│                       │          │ Service Cache  │         │
│                       │          └────────────────┘         │
│                       │                                     │
│              Request/Response via postMessage               │
└─────────────────────────────────────────────────────────────┘

Why a worker?

  • TypeScript's language service maintains a cache of parsed files
  • Creating a new program for each loader call is expensive
  • A persistent worker preserves the cache across all loader invocations
  • Uses Symbol.for() on process to ensure singleton across Turbopack contexts

See workerManager.ts for implementation.

Module Organization
ModuleEnvironmentPurpose
CodeHighlighter/IsomorphicMain code display component
CodeProvider/ClientClient-side code loading context
CoordinatedLazy/IsomorphicSwap + self-loading piece + Suspense
useStream/ClientClient list loading + completion
ChunkProvider/ClientLazy client loaders + preload dedup
abstractCreate*/ServerFactory utilities ('server-only')
use*/ClientReact hooks (state/effects)
pipeline/loadPrecomputed*BuildWebpack loaders
pipeline/loadIsomorphicCodeVariantIsomorphicRuntime code loading/transforms
pipeline/loadServer*ServerRSC data loading
pipeline/syncTypesBuildTypeScript parsing, worker threads
withDocsInfra/BuildNext.js configuration plugin

Key Design Decisions

Why HAST?
  1. JSON-serializable - Can be cached by webpack, stored in bundles, passed across server/client boundary
  2. Framework-agnostic - Convert to React, Vue, or any framework
  3. Transformable - Rich ecosystem of rehype plugins for post-processing
  4. Safer - Avoids dangerouslySetInnerHTML and its XSS risks

Why webpack loaders over runtime processing?
  1. Caching - Webpack caches loader results, rebuilds only changed files
  2. Dependency tracking - Loaders can declare file dependencies for watch mode
  3. Zero runtime cost - Client never parses or highlights code
  4. Works with RSC - Precomputed data is just JSON, serializes cleanly

Why the factory pattern?
  1. Single source of truth - Factory controls all behavior, index files are copy-paste
  2. Future-proof - Change factory implementation without touching hundreds of files
  3. Type-safe - TypeScript infers types from factory configuration
  4. Loader-friendly - Predictable pattern for loaders to detect and process

Performance Characteristics
OperationWhenTypical Time
TypeScript parsing (cold)Build500-2000ms
TypeScript parsing (warm)Build50-200ms
Syntax highlightingBuild10-50ms per file
HAST → JSX conversionRuntimeunder 10ms
Search index creationRuntime (once)50-100ms
Search queryRuntimeunder 10ms

The worker thread architecture makes TypeScript parsing ~10x faster after the first file by preserving the language service cache.