The Inspiration
It started with curiosity
It all started out of curiosity: what can we build if we can track hand and finger poses accurately? The American Sign Language manual alphabet, where every letter is a precise handshape, became our inspiration.
We wanted to make that precision visible. Each recognized letter turns into light, and the light becomes something visitors can interact with.
ASL is a complete language with its own grammar, and it uses far more than the hands. Synapse works with only one small part of it, the manual alphabet, and we hope it sparks curiosity about the language and the Deaf community.
The Experience
One camera, no controllers, no touch
A standard webcam and real-time hand tracking. Visitors choose between four modes with held gestures. Nothing to install, nothing to touch.
Fingerspelling Interactive Demo
Short interactive challenges built around the ASL manual alphabet. A machine learning model running in the browser recognizes static fingerspelled letters, and the particles form each letter as it's recognized.
Paint in the air
Computer vision tracks your hand and fingers in real time. Pinch your thumb and index finger to paint in the air, with one hand or both.
Play with particles
Particles follow your open hand. Close your fist to charge an energy ball, then open it to launch it. Clap, or make a heart with both hands, for more effects.
Watch the alphabet
A short show in which particles form handshapes from the manual alphabet, letter by letter. A first look at fingerspelling, no hands needed.
Inside the Fingerspelling Interactive Demo
Three fingerspelling challenges
- Open-endedFree SpellFingerspell a word or your name, send it, and get a short reply in particles, text and speech.
- 10 rounds · 7 s eachMatch the LetterA letter appears in particles. Match its handshape within 7 seconds. Ten rounds.
- 60 secondsBeat the ClockHow many letters can you match in 60 seconds? Build a streak.
The Build
From a hand-pose experiment to a full experience
Synapse is led by Alejandro Mendoza. While building the flow for tracking hand and finger poses, he found his inspiration in the ASL manual alphabet, where every letter is a precise handshape that a camera has to read correctly.
That spark grew into a complete experience. A rule-based detector that covered about ten letters became a custom-trained model that reads 21 3D landmarks per hand. Around it, the team built three fingerspelling challenges, a paint-in-the-air mode, a particle playground, a 12,000-particle engine, and an on-site AI device that works alongside Amazon Bedrock in the cloud. All of it runs from a standard webcam.
- 12,000particles rendered in real time
- 213D landmarks tracked per hand
- 24letters in the challenges
- 3fingerspelling challenges
- 0controllers or touchscreens
Synapse is
- An interactive demo that combines hand tracking, in-browser ML, and generative AI on AWS
- An interactive first encounter with the shapes of the ASL manual alphabet
- One-way: it recognizes static fingerspelled letters, shows them as particles and text, and replies with speech
- Built for real booth conditions: the language model knows which letters are commonly confused, so it makes a smarter guess at the word
Synapse isn't
- A course or tool for learning ASL. It doesn't teach handshape, movement or palm orientation
- A translator. It doesn't cover ASL vocabulary, grammar, facial expressions or conversation
- A communication aid for Deaf and hard-of-hearing people
- A wearable device. It uses only a standard camera
Want to learn ASL? Learn from Deaf teachers and Deaf-led organizations. Ask our team at the booth for more information and insights.
Take it Home
Leave with your card
Your fingerspelled word ends on screen with a QR code. Scan it to save your card to your phone.
- Create. Fingerspell a word or your name.
- Scan. Point your phone camera at the QR code on screen.
- Keep. Save the card and share it.
Come Closer
From logo to light
When the booth is idle, the particles hold the shape of a rotating AWS logo. The moment a hand appears, the logo dissolves into the live experience.
- AWS logo
- Hands detected
- Dissolve
- Interact
- Hold a pinchPaint in the air
- Point and holdChoose a mode
- Close your fist, then open itConfirm, or launch an energy ball in Play
- Thumbs up or downRate the result
Under the Hood
Real time at the edge, generative in the cloud
Hand tracking and letter recognition run in the browser, so camera video never leaves the booth. AWS receives only the spelled text, take-home cards, anonymous votes and usage counts.
- MediaPipe hand tracking in the browser, from a standard webcam.
- Custom fingerspelling model served with ONNX Runtime Web and trained on hand landmarks.
- Gesture-driven menus: point and hold, or close and open your fist, to choose a mode.
- On-site AI device: an NVIDIA Jetson at the booth runs a compact language model and text-to-speech locally, with Amazon Nova Lite on Amazon Bedrock as the cloud alternative.
- Amazon Bedrock Guardrails filter cloud replies for harmful content and profanity before they're shown or spoken.
- Amazon API Gateway, AWS Lambda, Amazon S3, Amazon CloudFront and Amazon DynamoDB provide a serverless backend for take-home cards and visitor votes.
- AWS CDK for infrastructure. Vite + TypeScript particle engine with 12,000 particles rendered in real time.
Built with Feedback
Listening first
Because Synapse draws on a language that belongs to the Deaf community, we asked for feedback and used it to shape how we present the project.
Deaf community feedback
A member of Fundación Mexicana de Sordos Gastronomía EnSeñas A.C., a Deaf foundation in Toluca, Mexico, gave feedback on the experience.
We'd love your feedback
If you find our project interesting, we'd love to hear what you think. Share your ideas and feedback with our team at the booth.
Visit Us
Raise your hands.
Find Synapse at the AWS re:Invent 2026 Builders' Fair. Step up to the camera, and the particles will respond to you.
Nov 30 – Dec 4, 2026 · Las Vegas, NV