Speak normally. The model chooses the JSON route.
Your image and question go to the VLM together. In voice mode, the waveform goes directly to the model without transcription. A fixed system instruction asks the model to infer your intent and return exactly one of four machine-readable response types:
| What you mean | Try saying | JSON route | What the demo does |
|---|---|---|---|
| Describe the whole view | What am I looking at? |
caption |
Shows and speaks a scene description |
| Answer a focused question | How many cars are visible? |
answer |
Shows and speaks the factual answer |
| Choose somewhere to land or deliver | Where could I lower this parcel? |
location |
Draws an amber candidate or broad directional cue |
| Find one visible object | Where is the white car? |
pointing + locate |
Draws a precise green point |
| Mark one visible object | Point out the white car. |
pointing + point |
Draws a precise green point |
| Follow one visible object | Track the white car. |
pointing + track |
Seeds the live video tracker when the point is valid |
The returned type, plus intent or action where needed, tells downstream
software how to use the answer. You never need to say "JSON," name a route, or
paste a schema.
A precise pointing result may start the green tracker. A precise location
result gets an amber landing or delivery candidate, but never starts an object
tracker. A broad coarse_grid_direction is only a general cue. Null coordinates,
invalid JSON, or a non-target status remove the marker.
Coordinates use the model's 0..1000 image-relative frame: left to right for
x, top to bottom for y. These are research outputs for inspection and
downstream experiments, not certified flight-control commands.
Pick a clip, pause the drone, and ask out loud. The video resumes with the answer. Ask it to track one visible object and a precise pointing / track response starts an in-browser optical-flow tracker. It is an early research demo, not flight-control authority.
Select an example video
Start simple or pick a busier aerial scene. Click a thumbnail, play to any frame, then speak to the drone.
Choose Center square crop to shade everything the model will not see. The yellow outline is the exact source crop; coordinates are mapped back into the original video before tracking.
The model's reply is also spoken in plain English. Coordinates are translated into frame directions.
Recording stops and submits automatically after six seconds. A progress overlay remains visible during inference; the same player then resumes with the response. Freeze it again to ask another question.
This is the early version, and that is the interesting bit: a small aerial model can already turn an ordinary question into structured perception that software can inspect, draw, speak, or track. Precise points may seed the visual tracker; coarse directions are review cues only. Nothing here is a certified flight-control command.