lumen

lumen is a single-binary session service. Agents drive real pages over a capability API or raw CDP, read and click a Qt application through its accessibility tree, and read terminals as text; the human sees the active surface live, can take control at any moment, and annotates the region that matters. It is one process, one config file, and one viewer.

Isolated sessions Live viewer Accessibility tree Human feedback Loopback only

Four surfaces, one contract

Browser sessions

CDP

Isolated Chromium, driven over the DevTools protocol

Each session is its own Chromium with a private profile. Agents attach playwright-cli over CDP to navigate, snapshot the accessibility tree with element refs, click and fill by ref, and screenshot. Tabs the browser opens on its own are adopted automatically, so the viewer always shows the tab the browser is actually displaying.

Qt applications

Accessibility

Read and click a real desktop app

An absolute program runs under its own headless Sway compositor and private D-Bus. The application publishes its controls as a structured tree of role, name, state, and bounds; an agent clicks a node's ref through the same virtual pointer the viewer uses. The endpoint refuses a tree that published nothing addressable instead of serving a skeleton the caller cannot distinguish from a real one.

Terminal sessions

PTY

Run any program unmodified

A real pseudoterminal is parsed into the same structured cell grid every other surface produces, so an unmodified ratatui app, htop, or a shell becomes a session an agent reads as text with no screenshot and no vision. Apps that link the lumen-ratatui crate skip the PTY and ship ratatui's own buffer diff instead.

Quickshell desktops

Wayland

A full shell per session

Each session runs one headless Sway compositor and one Quickshell process. The human watches the native Wayland output and drives mouse, wheel, and text input. Quickshell publishes no accessible objects, so these sessions stay screenshot-only.

Human feedback

Loop

A note the agent reads back

The human drags a rectangle and types a note; lumen captures those pixels as a PNG the moment the note is sent, stores it in SQLite, and hands it to the agent on its next lumen feedback call. Notes are keyed by session name and survive the session ending, so nothing is lost when a backend exits.

One viewer for every surface

The view plane is unified: browser sessions stream over CDP screencasting, Qt and Quickshell sessions use native Wayland screencopy, and terminal sessions stream the app’s own buffer diff. All of it renders to one canvas, and zoom is view-only—scroll zooms and drag pans without sending input until the human takes control.

Architecture

graph TD
    Agent[Agent CLI or CDP] --> API[HTTP/JSON control plane /v1]
    Human[Human viewer] --> View[View plane: CDP, Wayland, cell diff]
    API --> Supervisor[lumen supervisor]
    Supervisor --> Chromium[Chromium session]
    Supervisor --> Qt[Qt session + AT-SPI]
    Supervisor --> Quickshell[Quickshell + Sway]
    Supervisor --> Pty[Terminal PTY]
    Chromium --> View
    Qt --> View
    Quickshell --> View
    Pty --> View
    Human --> Feedback[Feedback in SQLite]
    Feedback --> Agent

A local, bounded service

Navigation host policy lives under [policy] in config/lumen.toml, and it is enforced per tab inside the browser: Lumen installs a navigation check on every tab it mediates, so a blocked navigation is refused no matter which session issues it—Lumen’s API, an agent’s own CDP connection, or a redirect. Coverage has an explicit boundary: a tab an agent creates entirely outside Lumen’s API is not checked until the API touches it. The policy bounds Lumen-mediated browsing; it is not a sandbox for an agent’s direct browser control. The audit trail at GET /v1/audit records navigations, tab operations, raw CDP calls, feedback, and session reaps, so a session can vanish from the list without a DELETE—but never without a trace.

Repository

Source and setup