CapSL

An Indonesian Sign Language learning app with real-time camera gesture feedback

React
Vite
MediaPipe
Python
scikit-learn

Screenshot of CapSL practicing sign language with the camera

model accuracy: 89.9%
SIBI letters: 24
servers, fully in-browser: 0

Background

Learning Indonesian Sign Language (SIBI) from static images is hard: learners never know whether their hand movements are correct. Most apps only offer a dictionary, with no feedback at all.

Together with one teammate (who handled content research and QA), I took this on as our capstone project: a web app that corrects gestures in real time, straight from the browser camera.

What I Did

As the only developer on the team, I built the entire technical side from scratch, logging every major decision as an ADR.

Set up the data pipeline: a fingerspelling dataset, then MediaPipe hand-landmark extraction into feature vectors
Trained an MLP model and exported it to pure JavaScript without a TensorFlow runtime, so all inference runs in the user's browser
Built the learning experience: guided practice with progress, geometry-based corrective feedback, memory quizzes, and XP & streak gamification
Kept quality in check with 25 unit tests and five ADRs

Results

The model reached 89.9% accuracy on held-out data, beating the PRD target of 85%. The app covers 24 SIBI letters and runs fully in the browser without a server, published through Vercel.

Lessons

Constraints actually sped us up. Dynamic letters J and Z were deliberately excluded, and BISINDO support was postponed, so the first version could truly ship. Calmly evaluating trade-offs through ADRs turned out to be more valuable than adding features.

Try the App

Next project

Luma