Exploring multimodal interaction for complex operational software

Most AI interfaces add a conversation layer to an existing product: a chat box in the corner, a copilot beside the workflow, or a separate conversational destination.

I wanted to explore something different.

Could AI become part of the interface itself, allowing people to operate a complex product through natural language rather than navigating every control manually?

A mission-control environment made the question concrete. Operators may have their hands occupied, their attention split across screens, or no convenient access to a mouse. Voice and gesture could potentially become useful interaction modes rather than novelty features.

But there was a harder problem underneath:

Multimodal interaction only works if different inputs mean the same thing.

Fast is easy. Coherent is hard.

SkyWatch supports three ways to operate the interface.

Voice

Spoken intent is converted into structured commands such as sorting, filtering, opening panels or switching to a triage view.

Gesture

Hand tracking through the webcam lets the user grab, move, resize and scroll interface panels without touching a mouse.

Intent

Natural-language commands can change the structure of the interface itself. Asking for a problems dashboard, for example, can reconfigure the layout, filter anomalies and surface the relevant alerts.

The more interesting interaction happens when they work together.

A user can pinch a fleet list and say:

“Sort this by strongest signal.”

The gesture identifies the object. Voice provides the action. Both operate on the same underlying command model.

The central design decision was to make the interface input-agnostic.

Mouse, gesture and voice all feed into one command dispatcher. Behaviours are defined once, then inherited by every input mode.

That meant the product did not have three separate implementations of the same interaction.

It had one interaction model with multiple ways to express intent.

The architecture was the design thesis.

This also changed how I evaluated the experiment. I wasn't asking whether voice worked, or whether gesture worked.

I was asking whether the product still behaved coherently when the input changed.

Some of the most useful findings only appeared once the system was working.

The microphone initially picked up the application's own spoken responses. The system would hear itself say “Done” and try to interpret that as a new command.

The fix was simple: suspend microphone input while the application is speaking.

That kind of failure would have been easy to miss in a static prototype. Building the interaction exposed it immediately.

The same happened with voice output.

I initially switched from Claude Haiku to ElevenLabs because the latter produced a more sophisticated voice. Technically, it was better.

It was also wrong for the product.

A mission-control interface needed short, neutral acknowledgements, not a voice that sounded like it was reading an audiobook. I switched back.

The lesson was simple:

The best implementation is not necessarily the best product decision. Context defines quality.

Gesture interaction worked well under controlled conditions: one operator, a clear background and reasonable lighting.

It became much less reliable outside those conditions.

A second hand in the camera frame could confuse tracking. Low light could make the pinch gesture unreliable precisely when precision mattered most.

That changed how I thought about the interaction.

The answer wasn't to pretend gesture could replace the mouse everywhere.

A real deployment would need constrained camera positioning, confidence thresholds or another input mode that could take over when tracking became unreliable.

Voice could become that fallback.

Knowing where an input modality fails is as important as making it work.

The prototype uses real satellite positions for the operational context.

ISS, Hubble and Starlink positions are calculated from live orbital data and updated every two seconds.

The fictional 24-satellite Meridian constellation and its telemetry are deliberately simulated.

That distinction matters.

Real operators keep operational telemetry private, so inventing the mission data allowed me to explore the interaction model without implying that fictional information represented a real satellite operation.

Knowing what to make real and what to fake is part of the product design.

The experiment started as a question about replacing the mouse.

It became a question about interaction architecture.

The difficult part of multimodal design isn't adding voice, gesture or another input method. It is creating an underlying model where different inputs can express the same intent without creating different versions of the product.

Building the prototype also changed how I think about AI interfaces.

AI doesn't have to become another destination in the product. It can become part of the interaction layer itself, interpreting intent while the product retains its existing structure and behaviours.

The prototype made those ideas tangible in a way that a specification could not.

I built SkyWatch solo with Claude Code as the implementation partner, directing the product concept, interaction model, visual design and technical architecture while using AI to move through unfamiliar engineering territory.

The result was a working prototype rather than a static concept: voice recognition, intent parsing, gesture tracking, a shared command dispatcher, live satellite data and the interface itself.

The point of the experiment was not to prove that I could build a satellite dashboard.

It was to use building as a way to investigate a difficult interaction problem.