A phone that thinks before it acts.

Jenni · architecture

Follow one request through Jenni, from the orb to the thing it does on the phone. Every tier is a chance to stop early, and every action passes a gate. Send your own requests below.

Guided tour

1. One request, eight parts: A request enters at the orb, may be answered early by a cheap tier, passes a gate, then acts. Memory, jobs and proof sit around it. Send your own below.
2. The orb: listening on the phone: Speech is recognised offline. One arbiter owns the microphone, and a spoken “stop” can interrupt at any time.
3. Fast lanes: reflexes and a tiny router: Eight rules answer the easy things with no model at all. A 270M router handles plain commands, and code double-checks it. Unsure requests drop down.
4. The brain: one model, a short tool list: The on-phone model sees only a few tools and reuses its cached prompt prefix. Each step is one model call.
5. The gate: nothing risky runs unasked: Every tool runs through the ToolExecutor, then a gate. Messages need two cards, and some things are refused outright.
6. The hands: native tools first, the screen second: Most things use Android’s own APIs. Only when none fits does the agent read the screen, one checked action at a time.
7. Memory: written through a gate: Facts must be backed by what the user said. Secrets are never stored, and forgetting sweeps every layer.
8. Night shift: jobs that survive a crash: Jobs hold a lease and an effect journal. Kill the app mid-job and nothing is sent twice.
9. The proof: graded from the phone: Tasks are checked against real device state. The numbers carry their labels: emulator, laptop model, subset, rented phones.

Parts you can inspect

The orb: Where a request starts

Tap the orb, type, or use the opt-in wake word. Speech is recognised offline on the phone and the reply is spoken by the phone’s own offline voice.

• Orb states: idle, listening, thinking, working, speaking, asking
• Scheduled and notification jobs enter the same pipeline

Mic arbiter: One owner for the microphone

Four priority rings: wake word, stop word, speech, recording. A higher priority preempts a lower one, and a lease is released only after the hardware handle is closed.

• Priorities: WAKE < STOPWORD < ASR < RECORD
• Stop-word barge-in works, free-form interruption does not

Offline speech: Vosk, fully offline

The system recogniser is never used because it might go to the network. Small Vosk models run on the phone’s own audio loop, with an energy and text-stability end-of-speech check.

• English-India small model is 36 MB
• Hindi and Hinglish recognition quality is untested
• Devanagari is transliterated to Roman in code

Wake word: Opt-in, off by default

A tiny Vosk grammar listens in a foreground service. Android limits when that service can start, so it is switched on from the app.

• Partly done: not yet confirmed on a physical phone
• It is a setting, not a shipped “Hey Jenni” phrase

Tier 0 · reflexes: Eight rules, zero model calls

Plain regex rules answer the easy things: greetings, the time, battery or Wi-Fi status, flashlight, volume, “open an app”. Anything with extra words falls through.

• Still goes through the ToolExecutor, so permissions and tracing apply
• Hinglish verbs are covered

Tier 1 · router: A 270M model picks one of 15 tools

A tiny FunctionGemma model makes one call and reports how sure it is. Code decides whether to trust it.

• Median 106 ms per call on the dev laptop; speed on a phone is not measured
• Quote it as “routes most plain commands, code double-checks”, not a headline accuracy

Confidence ≥ 0.9: The lowest token probability

The router’s confidence is the lowest probability among its tokens. Below 0.9, the request passes down to a bigger model instead of guessing.

• Router timeout is 4 seconds, then it escalates

Six trust checks: Code verifies what the model said

Every argument must be words the user actually said, no “him” or “her” as a contact, and times and numbers are re-derived by code. A recurrence word vetoes a one-time reminder.

• A wrong action would be worse than a slower one
• That is why the checks exist: the harder test showed 92% tool and 84% exact arguments

Tier 3 · the brain: One on-phone model, one step at a time

The supervisor is a Qwen3-1.7B model run by llama.cpp. Each step is one model call that returns tool calls or an answer, up to 10 steps.

• Replies are validated against a schema after generation
• If no model file exists it fails locally, it never silently uses the laptop

Narrowed tool list: 3 core tools plus one group

Of 56 tools offered to the model, it sees three core tools and one fixed group picked by the request. The ring shows the few that are lit.

• 58 tools registered, 2 hidden, 15 callable by the router
• Narrowing is a speed-up, never a limit: the runner widens if the model names another tool

Prefix cache: Reuse what was already read

The system prompt and tool list stay identical for the same kind of request, so llama.cpp reuses the cached prefix and reads only the new part.

• Rented Galaxy S24, proxy: 0.37 s with a cached prefix, 13.7 s cold for a 1.1k-token prompt
• Decode about 20 tokens a second on that phone

On-phone models: What runs where

Brain: Qwen3-1.7B. Router: FunctionGemma 270M. Speech: Vosk. Memory search: a small multilingual embedder. OCR: ML Kit. Voice: the phone’s offline TTS.

• Brain, voice and embedder downloads are hash-pinned; the router’s pin is not set yet
• No fine-tuned model is deployed. The CPU is the shipped path

ToolExecutor: The one place every tool runs

Reflex, router and brain all come through it: a permission pre-check, a resource lock so tools do not fight over the screen, a trace, and a time limit.

• A missing permission becomes a spoken “I need the X permission”
• Default timeout is 10 minutes, and a cancel kills only that tool

Allow: The safe lane

Plain actions go straight through: opening an app, setting an alarm, reading a file.

Confirm: A human card first

Money, deletes, uploads, submits and publishing need a confirmation. Messages use two cards: “Is this who you mean?” and a send / edit / cancel review.

• The message is typed, not sent, until you approve
• After sending, the input box is read back to prove it left

Refuse: Some things are never done

It never types passwords, PINs or card numbers, refuses to message “everyone”, and obeys “do not hit save”. These are code rules, not prompt sentences.

• Refusals group in the bench: 11 pass and 1 skipped of 12, no side effects
• Prompt injection is defended in depth, not claimed immune

Stop barrier: Stop means stop

Stop cancels every await, discards late model replies and silences speech. A run-id counter makes the native side refuse any action that was queued behind the one running.

• Sources: orb, overlay pill, notification, or the spoken word “stop”

phone_task: The screen loop

When no native tool fits, the agent reads the screen, makes one model call, takes one to three guarded actions, and repeats. Code judges whether each action changed the screen as expected.

• Capped at 30 steps or 5 minutes
• A completion checker reads a fresh screen before it says “done”; if it cannot check, it says “unknown”

Accessibility tree: What it “sees”

Android’s accessibility service gives the visible screen as a list of nodes. Protected screens read as empty, and it does not fall back to screenshots there.

• OCR is used only when the tree is nearly empty

Native tools first: Intents and system APIs

Alarms, reminders, calendar, files, settings and notifications go through Android’s own APIs. Driving the screen is the fallback.

• Files: copy, move, rename and append in shared folders only. Never deletes or overwrites
• Edits to a document are saved as a copy

Memory: Six layers, one local database

Working memory, a core profile, memories, a people graph, episodes and documents, stored in one on-phone SQLite file with full-text search.

• Nothing leaves the phone by default
• A recall mixes keyword, vector, entity and recency scores

Claim gate: The store refuses what is not backed

A fact said by the user must quote the user. Sensitive facts, such as relationships or health, can only come from the user, or become a question for them. Lower-trust claims never overwrite higher-trust ones.

• Born from a real bug: a model copied a family-relationship fact from an example prompt

SQLite + FTS5: One file

The knowledge base is one SQLite file with full-text search. Jobs live in a second database.

Dream pass: A slow night-time tidy

About once a day, when there are enough memories, a stronger model merges duplicates, deletes stale items and rewrites the compact core profile.

Forget: Forgetting reaches everywhere

Forgetting removes the fact from memories, claims and their raw episodes, turns, events, the profile and indexed chunks, then a tombstone stops it being re-learned.

• Press “Forget a fact” to watch it sweep every layer

Secrets never stored: The shield

Passwords, PINs, one-time codes, CVVs, card numbers and keys are refused or replaced with [removed] before anything reaches disk.

• Bank and one-time-code notifications are classified by rules and never saved

Durable jobs: Work that survives the app dying

A job is a database row with a lease. If the app dies, the lease expires and the job goes back in the queue, with back-offs of 30 seconds, 2 minutes and 8 minutes.

• A job runs through the normal pipeline, so it gets the same gates
• Unattended runs only do safe things like reminders and digests

Leases + back-off: Recover without guessing

A runner holds a lease. When it expires, the job is requeued or closed as interrupted. A wake that runs out of time does not burn an attempt.

Effect journal: Never twice

Every state-changing call writes an intent first and a receipt after. A retry never repeats an effect that already finished, and an unknown outcome is left for you to settle.

• This is what stops “remind me and message a contact” from messaging twice

Notification trigger: Jobs that wait for a match

A job can wait for a matching notification. The text it reads is marked as untrusted, and the job returns to waiting afterwards.

Learned routines: Replay without a model

After a verified success of two or more actions, a routine is saved by selectors and becomes active after two good replays. A routine that does the opposite of the request is refused.

• Never learned from read-only, destructive or multi-task goals
• Teach mode exists but is not confirmed on a phone

Graded bench: Graded from the phone, not the answer

Each task is checked by reading the device’s real state, never the assistant’s own words. The latest readable full run was 48 records: 42 pass, 3 fail, 2 skipped, 1 inconclusive, with 1 false claim.

• Emulator with a laptop-hosted model, not phone-only accuracy
• A log of about 120 problems, each with its fix, shows the method

AndroidWorld subset: 15 of 22, partial

The best result per task, retried after fixes, on a 22-task subset. It uses native tools, runs on an emulator with a laptop model, and is not the 116-task suite.

• Not comparable with published scores
• Open: timer keypad, some note edits, deletes that need a confirm

Hardware lab: Rented phones, honest numbers

On a rented Galaxy S24 the CPU decodes about 20 tokens a second. An OpenCL build reads a long prompt 3.4 times faster but decodes no faster, Vulkan crashed, and the NPU only handles int8 graphs.

• Single samples on lab phones, not the owner’s phone
• The shipped path is the CPU

What it is not: Said plainly

Not released. It is sideloaded for personal use, and Google Play’s accessibility rules make distribution an open decision. Speed and wake-word reliability are not measured on the owner’s phone.

• No claim of airplane-mode use
• No fine-tuned model is shipped

Things you can try

• Open an app: A reflex: zero model calls
• Set an alarm: The tiny router handles it
• Message a contact: Two cards, then a read-back
• Edit a document: Escalates to the on-phone brain
• Message everyone: Refused in code
• Press Stop: Cuts the run mid-flight
• Forget a fact: Sweeps every memory layer
• Kill the app mid-job: Lease, retry, no repeat

A schematic. Speeds on lab phones are proxies, not the owner’s own phone, and the benchmark numbers carry their labels. Everything else comes from the project’s own code and notes. Developer tooling is shown for context, not as a user feature.