Contactless HCI / Built for Mac

Your intention,
in motion.

Point with your hand. Speak what you see. Control your Mac with AI that stays on your Mac.

Download source ↗See how it works ↓

Apple Silicon · macOS 14+ · Offline after setup

Jarvis / Preview● On-device
THE MERIDIAN / NEWS

A wider view.
A closer connection.

WorldTechnologyCulture

Voice + hand / local

“Technology”

Target locked · pinch to click

Say a visible label. Local AI finds the target.

Interactive illustrationNo camera access
Voice finds the targetGestures make the moveInference stays local
01 / A more direct connection

From “over there”
to right there.

Voice and gestures share the work. Say the label you’re looking at, let the cursor settle, then make a deliberate pinch to act.

01 — FIND IT
“Technology”

Say what you see.

Local speech recognition turns your words into text. Screen understanding finds readable labels near your cursor. Multiple matches? Say the number of your choice.

02 — REACH IT
↗

Let your hand lead.

Your index finger moves the pointer. Adjustable movement gain and smoothing help you cross the screen; a target lock steadies the cursor for your next action.

03 — DO IT
● ↔ ●

Pinch with purpose.

Thumb + middle finger clicks. Thumb + ring finger right-clicks. Hold thumb + pinky and move vertically to scroll. Two quick left pinches give you a double-click.

02 / Beyond a single demo

Your everyday apps.
A different way in.

Jarvis reads accessible controls and visible text across macOS. The demo website is a place to practice; the interaction system lives at the desktop level.

↗

Find the right thing

Natural-language target matching, numbered choices for duplicate labels, and a visible sensing radius around the pointer.

⇄

Navigate the workspace

Say “tab left,” “tab right,” “close tab,” or “last page.” “All windows” toggles the overview; “switch windows” invokes the Mac app switcher.

✎

Catch a thought

“Hey Jarvis” opens a notepad for your next speech block. Say “Bye Jarvis” or stop speaking for five seconds to save a text file in your chosen folder.

◎

See the interaction

A floating pane shows speech feedback and a live camera preview with the hand skeleton overlaid. Target highlights and local sound cues make actions easier to follow.

03 / Local by design

Turn off Wi-Fi.
Keep the connection.

The connection between you and your computer, that is. After downloading the models and preparing the runtime, speech, hand tracking, and target interpretation run locally.

No cloud inference is required for the control loop. The marketing site and GitHub download still need internet access.

01 / HEARArgmax WhisperKit

A cached Whisper model transcribes speech on the Mac.

02 / SEEApple Vision + Accessibility

Hand landmarks guide gestures. OCR and accessibility labels describe visible controls.

03 / UNDERSTAND & ACTBonsai + native macOS

Bonsai interprets requests through a local MLX service. A native bridge moves the pointer and sends supported input actions.

04 / Make it yours

Explore the code.
Try the experience.

Get the complete project from GitHub, prepare the local models, and launch on your Mac.

Download source ZIP ↗View on GitHub ↗

Source code, not a prebuilt installer. Model weights are prepared separately.

  1. 01 Apple Silicon Mac running macOS 14 or newer
  2. 02 Follow the repository’s setup and model instructions
  3. 03 Allow Camera, Microphone, Accessibility, and Screen Recording
  4. 04 Launch Jarvis and start with the offline news demo
  5. Read the installation guide ↗
A few useful details
Does this run entirely offline?

The desktop app’s control loop can run offline once its models and dependencies are installed. Initial setup and downloading source code require connectivity. Your browser still needs internet for online websites.

Will it work with every app and every target?

It works across apps using screen text and accessibility labels. Hidden controls, unreadable text, and unlabeled custom canvases can be difficult to target. Screen discovery currently focuses on the primary display. Hand tracking depends on lighting, visibility, and camera angle.

Does the website use my camera or microphone?

No. The interactive illustration above simulates the workflow. Camera and microphone use happens in the native desktop app after you grant macOS permissions.

Where can I find the full feature list?

Read the feature guide on GitHub for supported gestures, voice commands, settings, architecture, and current limits.