Sightline docs

What it does

An assistant built into your app can answer "why can't I click Generate?" from what's actually rendered instead of from documentation that may be out of date. Sightline gives it that view. It reads the accessibility tree the way a screen reader does and writes one short line per heading, landmark and control:

Page: /api-keys
Title: "Anam Lab"
Viewport: 1440px wide
Scroll: top, more below
navigation "Main navigation"
  link "Personas" → / [ref=e1]
  link "API Keys" → /api-keys [current] [ref=e2]
  link "Sessions" → /sessions [ref=e3]
main
  h1 "API Keys"
  button "New Key" [ref=e4]
  searchbox "Search API keys..." [empty] [ref=e5]
  table: columns Key ID | Label | Created | Actions; 9 rows
    button "1675c7a…3cd2" [ref=e6]
    button "Open actions for production" [collapsed, opens menu] [ref=e7]
    … 6 more similar items

It's built for an assistant inside your own product, looking at a real user's screen, so it follows a few rules a testing snapshot doesn't need:

• It never reads field values, passwords or contenteditable text. Email addresses become [email].
• With a modal open, only the modal is listed.
• Repeats and tables are summarised, and the whole outline fits a character budget.
• It runs in the page with no CDP, extension, server or dependencies, in a few milliseconds.
• A skip selector keeps your agent's own panel out.
• Refs stay stable between captures, so the agent can point at a control after the page changes.

Install

Sightline is an ES module with TypeScript types and no runtime dependencies. It needs a DOM, so call it in the browser, in the same page as the UI it describes.

Quick start

Create one Sightline per page and capture an outline whenever the user asks your assistant something. Keep the instance around: it remembers which element has which ref.

import { createSightline } from '@smidge/sightline';

const sightline = createSightline({
  skip: '[data-agent-panel]', // never describe the agent's own UI
  viewport: true, // mark [offscreen] controls, add a Scroll: line
});

// Send with each message to your LLM.
const outline = sightline.capture();
await chat.send({ text: question, pageOutline: outline?.text });

// Let the agent point at things.
sightline.highlight('e5'); // by ref from the outline
sightline.highlight('New Key'); // or by visible name

capture() returns null when the page has nothing to list, so a failed capture never blocks the message.

Reading an outline

An outline is plain text, one item per line. Indentation shows nesting: controls sit under the landmark, dialog or table that holds them. Each line starts with a role and, when there is one, the accessible name in quotes. Everything after the name is optional.

textbox "Label" [required, empty, invalid] [ref=e1] — "Enter a label."

That line is a text field named "Label". It's required, has nothing in it and is marked invalid, its ref is e1, and the text after the dash is the error message it points to with aria-describedby or aria-errormessage.

Header lines

Turn all of these off with header: false.

Controls

States, in the order they appear: checked, unchecked, mixed, on and off for switches, selected, current, pressed, expanded or collapsed, opens menu (or opens dialog, opens listbox and so on), disabled, required, empty, invalid and offscreen.

Structure

Paragraph text isn't listed on its own. It reaches the outline when it names something: a button's text, a field's label, a dialog's description, an error message.

Sending it to a model

Send a fresh outline with every user message, and tell the model what it is. Refs give the model a way to name a control exactly, so ask it to use them, then point at what it named:

const outline = sightline.capture();

const system = [
  'You are the assistant inside Acme.',
  'Each message includes an outline of the screen the user is looking at.',
  'Answer from the outline. To point at a control, include its ref, like [ref=e12].',
].join('\n');

const reply = await model.respond({
  system,
  message: `${question}\n\n<screen>\n${outline?.text ?? 'No outline available.'}\n</screen>`,
});

for (const [, ref] of reply.matchAll(/\[ref=(e\d+)\]/g)) sightline.highlight(ref);

A few things that help:

• Keep the default budget. 6,000 characters is about 1,500 tokens, and most app pages fit. Raise maxChars for long documents rather than dropping it.
• Turn on viewport if the agent gives directions. It can then say "scroll down to Webhooks" for an [offscreen] control.
• For a live agent, send the watch diff instead of the whole outline each time. change.diff.added holds just the new lines.

createSightline

createSightline(options?: SightlineOptions): Sightline

Takes every outline option, plus document to read a document other than the current one (a same-origin iframe, for example). Refs are on by default. The returned object keeps element refs stable across captures.

capture(overrides?)

capture(overrides?: Partial<OutlineOptions>): Outline | null

Outlines the page now. overrides change options for this capture only, such as { infer: true }. Returns null when nothing useful was found.

type Outline = {
  text: string; // the outline
  lines: string[]; // the same, one entry per line
  refs: Map<string, Element>; // e1 → element, for this capture
  stats: {
    chars: number;
    lines: number;
    visited: number; // elements walked
    controls: number;
    truncated: boolean;
    ms: number;
  };
};

resolve(ref)

resolve(ref: string): Element | null

The element behind a ref from an earlier capture, or null once it has left the page. Accepts e12 or ref=e12.

find(target, role?)

find(target: string, role?: string | null): Element | null

A visible control by ref or by the name the outline shows. An exact name beats a partial one. Pass role to narrow it: find('Save', 'button').

highlight(target, options?)

highlight(target: string | Element, options?: HighlightOptions & { role?: string | null }): boolean

Scrolls a control into view and draws a pulsing ring over it, so the user can see what the agent means. The ring sits above the page, ignores the pointer and is hidden from assistive technology; the element itself isn't changed. Returns false when nothing matched.

watch(onChange, options?)

watch(onChange: (change: PageChange) => void, options?: WatchOptions): () => void

Reports the page each time it settles into a new state. Returns a function that stops watching. See Watching a live page.

track(onInteraction, options?)

track(onInteraction: (interaction: Interaction & { ref: string }) => void, options?: TrackOptions): () => void

Reports clicks, toggles, chosen options, focus moving into a field and fields being filled or cleared, each with the control's ref. Returns a function that stops tracking. See Tracking the user.

Options

createSightline and outline take the same options.

Without refs, navigation and tabs collapse onto one line each:

navigation "Settings": Profile · API keys [current] · Billing
main
  h1 "API keys"
  button "New key"

Watching a live page

A live agent, such as a voice or avatar assistant, needs to know when the screen changes without polling. watch calls back once with the initial page, then each time the page settles into a new state.

const stop = sightline.watch((change) => {
  if (change.kind === 'update') return; // only navigation, dialogs and alerts
  agent.addContext(`[screen] ${change.kind} ${change.path}\n${change.outline.text}`);
});

It captures at most once every 150 ms while the page is busy. It reports a change only after the page has held still for 250 ms, or 700 ms after the address changes so the new page can load. A poll every 1.5 s catches changes that no mutation reports, and mutations inside skip are ignored.

Tracking the user

track tells the agent what the user did, by the control's role and name, in the same words as the outline. It never reports what was typed: an edit is only filled or empty.

import { describeInteraction } from '@smidge/sightline';

const stop = sightline.track((interaction) => {
  agent.addContext(`[user] ${describeInteraction(interaction)} (${interaction.ref})`);
});

Each interaction also carries role, name, state (for toggles and edits), option (for selects), hint (the hintAttribute value), element and at, a timestamp. Repeats of the same interaction within 300 ms (dedupeMs) are dropped. With infer, clicks on the clickable elements it finds are reported too.

Pages without semantic HTML

Many sites build buttons from plain divs with a click handler, leave icon buttons unnamed, and fake headings and modals with styling. With infer: true, Sightline reads those as a sighted user would. Everything it infers is marked as a guess.

textbox near "× Summer sale Email Add to bag" [empty] [ref=e1]

clickable "shopping bag" (guessed from image file name) [ref=e1]
clickable "close" (guessed from ×) [ref=e2]
h2 "Summer sale"
textbox near "Email" [empty] [ref=e3]
clickable "Add to bag" [ref=e4]

• An element that reacts to clicks without a button or link role is listed as clickable. Sightline counts an onclick attribute or property, React (__reactProps$…) and Vue click props, jQuery handlers, and the element where a pointer cursor starts. It leaves out a wrapper around a real link or button, a container holding form fields, anything covering half the screen, and a clickable with nothing to call it.
• A control with no name gets a guess, marked with its source, such as button "close" (guessed from icon). The guess comes from a tooltip attribute, the icon inside (fa-trash, lucide-x, <use href="#icon-cart">, Material UI's data-testid="DeleteIcon"), an image file name, a test id, name, id or the class. A guess from a file name, id or class is kept only if it contains a word that says what the control does. A field takes the short text just before it, and a control showing only × or ☰ reads as close or menu.
• Short text set noticeably larger and bolder than the body text becomes a heading. Prices are left out.
• A fixed element over the centre of the viewport that covers a fifth of it, holds something to click and isn't the whole app counts as an open dialog. Only the dialog is shown, and watch reports it opening and closing.
• Open shadow roots, their slots and same-origin frames are read. A frame from another site is listed by its title.

find, highlight, track and describeElement use the same names, so an agent can ask to click "close" when the page shows ×. Inference costs a computed-style read per element, a few milliseconds on a busy page. Listeners added with addEventListener are invisible to page script, so a div with one and no pointer cursor stays unlisted.

Privacy

Sightline is meant to run on real users' screens, so it reads less than a testing tool would:

• Text fields show [empty] or nothing. Their values, passwords and contenteditable text never appear, in outlines or in track events.
• A select shows its chosen option, since that is a fixed choice the page offered, not typed text.
• Email addresses in names and text become [email]. Pass keepEmail to keep, say, your support address.
• redact runs on every name and text excerpt, for anything else you don't want to leave the page, such as account numbers.
• Chat logs (role="log") are listed by name only.

createSightline({
  keepEmail: /@acme\.com$/,
  redact: (text) => text.replace(/\b\d{8,}\b/g, '[number]'),
});

Sightline sends nothing anywhere. What leaves the page is whatever your code sends to your model.

Lower-level functions

createSightline is built from these, and each is exported for custom setups.

Performance

Measured in Chromium 143 on generated pages (nav, labelled forms, a large table, card grids and a dialog), with refs and viewport on. Each figure is the median of 20 runs after 3 warm-ups (pnpm bench):

Long runs of similar items are counted and skipped, not walked, so cost follows how varied a page is rather than how big it is. With no character cap at all, the 50,000-element page still took 1.7 ms. On real Anam Lab pages an outline took under 6 ms, including the first, unwarmed capture.

With infer on, measured on live sites:

On an accessible page, inference adds nothing but time.

Accessible names are the contract

Sightline sees what a screen reader sees. An icon button with no aria-label reaches the agent as button near "…", named from the text beside it, or button (unlabelled). Naming your controls properly makes the agent accurate, and it's good accessibility practice anyway. For pages you don't control, infer fills some of the gaps with marked guesses.

To see what your own app looks like to an agent, run the bookmarklet on it, or try a public page in the demo.

Development

pnpm install
pnpm test        # Vitest + happy-dom
pnpm bench       # Chromium benchmark via Playwright (needs a local Chromium)
pnpm build       # emits dist/
pnpm site        # builds this site into _site/ and serves it on :4173

Source, issues and releases are on GitHub. Sightline is MIT licensed.