Sightline runs inside your page and reads it the way a screen reader does. Your in-app assistant gets a short outline of the headings, controls, states, open dialog and errors in front of the user, so it answers from what is actually there. It never reads what anyone typed.
npm install @smidge/sightline
import { createSightline } from '@smidge/sightline';
const sightline = createSightline({ skip: '[data-agent-panel]' });
// Send with each message to your model.
const outline = sightline.capture();
await chat.send({ text: question, pageOutline: outline?.text });Read the docs MIT licence · no dependencies · 13 KB gzipped
Desktop, 1440 × 900
Drawn here without its scripts. Click anything to find it in the
outline, or scroll it to watch [offscreen] move.
How this works. Browsers can't fetch other sites, so yours asks Jina Reader, a free service, for the page's HTML. The demo removes the scripts, draws the page in a sandboxed frame here and runs Sightline on it in your browser, the same code you install. Pages that build everything with JavaScript come out empty this way; for those, use the bookmarklet.
Playwright's AI snapshot and its in-page forks produce a similar outline. Sightline has a different job: an assistant in your own app, looking at a real person's screen. That changes the rules.
Fields show their label and whether they're [empty],
never their value. Passwords and contenteditable text stay out of
the outline and out of interaction events. Email addresses
become [email] unless you allow them.
form "Invite a teammate" textbox "Work email" [ref=e1] textbox "Password" [ref=e2] textbox "Note" [empty] [ref=e3] button "Send invite to [email]" [ref=e4]
Content behind a modal (aria-hidden or
inert) is skipped, so the agent describes what the
user can act on now, not the page underneath.
dialog "Cancel your subscription?" (open) h2 "Cancel your subscription?" button "Keep my plan" [ref=e1] button "Cancel subscription" [ref=e2]
Runs of similar items shorten to the first three and a count. Tables become their columns and a row count. The outline stops at 6,000 characters by default, about 1,500 tokens.
main h1 "Products" link "Product 1" → /p/1 [ref=e1] link "Product 2" → /p/2 [ref=e2] link "Product 3" → /p/3 [ref=e3] … 45 more similar items table: columns Order | Customer | Total | Status; 120 rows
Every control gets its own line and a ref, so a ref always sits
next to its name. In live testing, a model reading several
Name [ref=eN] pairs on one line picked the wrong
one. Refs stay the same between captures, so an agent can say
"highlight e4" after the page has changed around it.
navigation "Settings" link "Profile" → /settings/profile [ref=e1] link "API keys" → /settings/api-keys [current] [ref=e2] link "Billing" → /settings/billing [ref=e3] main h1 "API keys" button "New key" [ref=e4] button "Open actions for production" [collapsed, opens menu] [ref=e5]
No CDP, browser extension or server, and no dependencies. Long runs of similar items are counted and skipped rather than walked, so cost follows how varied a page is, not how big. On real Anam Lab pages an outline took under 6 ms, including the first capture.
| Elements | Median | p95 | Output |
|---|---|---|---|
| 515 | 0.9 ms | 1.1 ms | 5,971 chars |
| 2,015 | 0.8 ms | 0.9 ms | 4,661 chars |
| 10,015 | 0.9 ms | 1.3 ms | 4,661 chars |
| 50,015 | 1.7 ms | 1.8 ms | 4,662 chars |
Pass a selector and the chat panel never describes itself. The
same selector keeps the panel's own changes from waking
watch.
createSightline({ skip: '[data-agent-panel]' })main h1 "Orders" button "Export CSV" [ref=e1]
watch reports navigation, dialogs and alerts once
the page has settled. track reports what the user
clicked or toggled, by name. Together they tell a live agent
what just happened without resending the whole page.
[user] clicked button "New key" (e3) [screen] dialog-opened /settings/api-keys dialog "Create API key" (open) h2 "Create API key" textbox "Label" [required, empty] [ref=e4] button "Create" [disabled] [ref=e5] button "Cancel" [ref=e6]
With infer: true, a div with a click handler is
listed as clickable, an unnamed icon gets a guess
that says where it came from, and big bold text counts as a
heading. Without it, this page is one confusing field.
textbox near "× Summer sale Email Add to bag" [empty] [ref=e1]
infer: trueclickable "shopping bag" (guessed from image file name) [ref=e1] clickable "close" (guessed from ×) [ref=e2] h2 "Summer sale" textbox near "Email" [empty] [ref=e3] clickable "Add to bag" [ref=e4]
Sightline was built for Sophie, the in-app assistant in Anam Lab. Anam asked Sophie questions about the open page, with and without the outline. It now also powers a live avatar version of Sophie that watches the page and guides people step by step.
| Setup | Questions answered correctly | Median time to answer |
|---|---|---|
| With the outline | 30 of 30 runs | 3.8 s |
| Without it | 3 of 6 runs | 10.0 s |
Sightline is an ES module with TypeScript types. It needs a DOM, so it runs in the browser, next to your chat UI.
Send outline.text with the user's question. Most pages fit in 1,500 tokens.
Tell the model it can answer with a ref, then call highlight so the user sees which control it means.
watch and track report what changed and what the user did, in the same vocabulary.
import { createSightline } from '@smidge/sightline';
const sightline = createSightline({
skip: '[data-agent-panel]', // never describe the agent's own UI
viewport: true, // mark [offscreen] controls, add a Scroll: line
});
// Send with each message to your LLM.
const outline = sightline.capture();
await chat.send({ text: question, pageOutline: outline?.text });
// Let the agent point at things.
sightline.highlight('e5'); // by ref from the outline
sightline.highlight('New Key'); // or by visible name
// Keep a live agent up to date.
const stopWatching = sightline.watch((change) => {
agent.addContext(`[screen] ${change.kind} ${change.path}\n${change.outline.text}`);
});
const stopTracking = sightline.track((interaction) => {
agent.addContext(`[user] ${interaction.type} ${interaction.role} "${interaction.name}"`);
});The demo above reads a page's HTML without its scripts. To see what Sightline reports on a live app, including pages you're signed in to, add this bookmarklet and press it on any page. The panel updates as the page changes, and nothing leaves the page.
Drag it to your bookmarks bar, or press it here to try it on this page. Press it again to close the panel. In Chrome it also runs on sites with strict security policies, such as GitHub and YouTube.