Building face check-in for a martial arts school

The instructor takes one photo of the class. DojoEasy, the school-management app I built on my own, finds the faces in it and suggests which students they belong to. The instructor confirms the names and attendance is done.

The model came off npm. Most of my time went into what happens around it, and a surprising amount went into the moments when it can’t tell two people apart.

Picking the library

MediaPipe Tasks Vision was the obvious choice. It has the better detector, and it can’t do the second half of the job at all. Its ImageEmbedder places whole images in a space where pictures that look alike land close together, and two kids in matching white gis, photographed against the same wall, look very alike to it. It would have looked great in a demo and then put the wrong child’s name on a face.

@vladmandic/human handles detection and recognition, and it ships age, gender and emotion classifiers alongside them. I wasn’t going to point an emotion classifier at kids. That ruled it out on scope.

I went with @vladmandic/face-api and copied three of its seven networks into the app: a detector, the 68-point landmark model and the recogniser.

Same origin, on purpose

The weights live in public/models/face/ and load from DojoEasy’s own domain, which keeps the Content Security Policy closed. Pulling them from a CDN would mean adding that CDN to connect-src, and every check-in would send a request off the device announcing that face recognition was about to run.

The library itself loads through a dynamic import(), so its 1.3 MB only lands on the check-in route:

const faceapi = await import("@vladmandic/face-api");

Weights come to about twelve megabytes. Over gym wifi, that’s the download most likely to die halfway through.

loadEngine keeps the in-flight promise in a module variable, so two components asking at the same moment share one download:

let loading: Promise<FaceApi> | null = null;

export function loadEngine(): Promise<FaceApi> {
  loading ??= attempt();
  return loading;
}

Failure is where this bites. If the promise rejects and stays in loading, every later caller gets handed that same old rejection, and recognition stays broken for the rest of the page’s life even after the wifi recovers. The only fix would be a reload nobody thinks to try, so a failed attempt clears itself and the next caller starts a fresh download.

Where the face data lives

Each face comes out as a 128-number descriptor. That’s a biometric template, which Brazil’s LGPD treats as dado pessoal sensível, sensitive personal data. Storing them in a server table would create a new kind of risk. People already post photos of themselves everywhere. A leaked template can be matched against other systems, and nobody can issue you a new face the way they issue a new password.

Turning thirty profile photos into descriptors costs thirty fetches and several seconds of GPU work, so the results need caching. They’re cached in IndexedDB on the instructor’s own device. A descriptor never leaves a device that was already allowed to see the photo it came from.

Entries die four ways, and the server takes part in none of them.

Uploading a new photo is the first. The cache key includes the avatar’s storage path, <user_id>/<uuid>.<ext>, and each upload gets a fresh UUID, so the old entry is never looked up again. There’s no invalidation call to forget.

Withdrawn consent is the second, and I’m happiest with this one. Every roster load returns the full list of students who opted in, and a prune step deletes every cached descriptor for anyone missing from it. When a student withdraws, their descriptor disappears from each instructor’s device at that instructor’s next session. I didn’t need a purge endpoint or a push channel, and both of those can fail silently.

The other two are simple. Entries expire after 30 days. Each one also records the model version that produced it, and a model change throws the old ones out, because their numbers describe positions in a space the new model doesn’t use.

Students already upload a profile picture so the front desk recognises them. Using that picture to match faces is a new purpose, and Article 6(I) of the LGPD doesn’t let a purpose change quietly. The consent checkbox is where it stops being quiet.

It defaults to false, and silence means no, whereas leaving the monthly leaderboard is opt-out because that one is only a preference. A face falls under Articles 5 and 11 as sensitive data, and a good share of the students are children, which brings in Article 14.

Consent sits on the membership row. Someone who trains at two franchises might want this at one and not the other, and the agreement is with that dojo.

The rules for minors live in the database. A rule that only exists in the interface lasts until someone opens DevTools. Minors can’t consent to face recognition, can’t upload a profile photo and can’t post on the academy wall. Age is self-declared, so a minor’s attempt to consent gets recorded and blocks every later attempt until an instructor clears it. Otherwise a kid could fix the birth year and click yes again.

Postgres bit the first version of that function. It recorded the attempt and then raised an exception, and raise exception rolls back the whole transaction, including the block the line before had just written. It returns a status now.

One rule I wanted, the server can’t enforce. The Cloudflare Worker has no image library and a tight memory ceiling, so it can’t check whether an avatar contains a human face. Only the browser sees the face, and anything the browser reports is a claim from the client. So the database enforces the only version it can: minors upload no avatar at all.

Deciding who is who

The matching itself is plain arithmetic, kept out of the browser on purpose, because the model runs there and can’t be unit-tested. The deciding code lives in a shared package with its own tests.

export const MATCH_THRESHOLD   = 0.6;
export const HIGH_CONFIDENCE   = 0.45;
export const MEDIUM_CONFIDENCE = 0.55;
export const AMBIGUITY_MARGIN  = 0.06;
export const MIN_FACE_PX       = 36;

Distance is Euclidean over the 128 dimensions. Under 0.45 the box can be ticked for the instructor. Between that and 0.55 the name shows up unticked, and at 0.6 or beyond the faces belong to different people. That last number comes with the network. If results disappoint, the bug is somewhere else.

The naive approach gives each face its nearest student, one face at a time. Picture a student standing next to a cousin who looks like them. Both faces get the same name, and the instructor confirming the list has no idea which one is right.

So matching runs greedily over every face and student pair, sorted by distance. The most certain pair in the photo gets settled first and both sides come off the board, then the next most certain, and so on down the list. The Hungarian algorithm would find the optimal assignment, but at a dojo’s scale the difference never shows up, and greedy still guarantees that a student’s name can’t land on two faces in one photo.

Siblings train together all the time. Two brothers in the same gi can land within a hair of each other, and picking the closer one would be a coin flip presented as a confident answer. When the runner-up is within 0.06 of the winner, the face comes back as unrecognised. The instructor sees that the app doesn’t know, which is true.

Faces under 36 pixels on their long side get refused before they’re measured, because below that size the crop is noise shaped like a person and the model will still return a confident answer for it. Ties break alphabetically by name, so the same photo always produces the same list in the same order.