How the AI Works (A Technical Look at Face Matching)

Give this site a single face photo, and it looks for actresses with similar facial features from among about 16,422 people. This page explains how that works, and the limits we found by actually measuring it. It is written from the builder’s point of view, so it leans a little technical.

The overall flow

  1. Find the face in the photo (face detection)
  2. Turn the face into 128 numbers (a feature vector)
  3. Compare them with the numbers for about 16,000 actresses (matching)
  4. Sort by closeness and show the results

All four steps run inside the browser you are using. The models are downloaded into the browser and run with an inference engine built on WebAssembly (ONNX Runtime Web), so the photo you upload is never sent to a server and never stored.

1. Face detection and feature vectors

For face detection we use YuNet, a lightweight model published in OpenCV Zoo, and for turning a face into numbers we likewise use SFace. A detected face is aligned in orientation and size based on the position of the eyes, cropped to 112×112 pixels, and passed to SFace. What comes out is a sequence of 128 numbers (the feature vector).

The model is trained so that faces of the same person produce similar sequences of numbers. So whether two faces look alike can be measured by how much the two sequences of numbers point in the same direction (cosine similarity).

2. Matching against about 16,000 actresses

The actresses’ feature vectors are computed in advance from their profile images. Your browser downloads them, compares them exhaustively against the feature vector of your photo, and sorts the results by similarity. There is no processing that queries a server.

We also show a per-part "closeness" for the eyes, nose, mouth and outline, but this is a simple method that splits the 128 numbers into ranges and compares those. It does not strictly cut out and measure each facial part, so please treat it as a rough guide.

Baseline values for similarity

A similarity number cannot be read without knowing the baseline. So we measured the similarity of 4,000 randomly selected pairs of different people from the profile images. The mean was 0.105, the standard deviation 0.117, 95% of pairs came out at 0.308 or below, and the maximum was 0.503. The commonly used rule of thumb for treating two faces as the same person with SFace is around 0.36.

In other words, values around 0.3 occur often even between completely different people. Ranking "lookalikes" on that basis would produce a meaningless list. The percentage shown on screen is a display conversion based on this distribution, not a probability. How to read the diagnosis results is summarized in How to use the AV actress diagnosis.

How the facial-feature rankings are measured

Features such as eye size and face outline are obtained by detecting facial landmarks (478 points) and expressing them as ratios divided by the width of the face. Using absolute lengths would make them depend on the image size, so everything is normalized by face width. On top of that, we added two steps to keep the results reliable.

  • Exclude photos facing sideways, looking down, or with eyes closed.Photos whose ratios cannot be measured correctly return wrong values instead of an error. About half of all photos passed this condition
  • Remove the effect of smiling.When you smile, your eyes look narrower and your lips thinner. We use the residuals of a regression with smile intensity as an explanatory variable

The results can be seen in the facial feature rankings and search by facial features.

Profile photos and faces inside the works

A profile photo can give a different impression from the actual works, because of retouching and shooting conditions. So we also extracted faces from the sample images of the works she appears in and compared them.

There was a pitfall here. Even when only one face appeared in a sample image, our measurements showed that 62.7% of the time it was not the actress herself — because her co-stars are in the frame too. So we do not treat an extracted face as the person as-is; we only use faces that were confirmed to be her by matching against the profile image.

The 1,124 actresses for whom we could confirm 15 or more in-work faces, ranked by how consistent they are with the impression of their profile photo. Here are the top examples of consistency.

Ranking of actresses whose profile image and in-work face give the same impression / Ranking of actresses with the biggest impression gap

Limits

  • What is being measured is a photograph. It is affected by retouching, makeup trends, and the era in which it was taken. There were cases where the average value of the same feature moved by more than 20 points depending on the debut era
  • Faces looking alike is a separate thing from the overall vibe or body type looking alike
  • Even a high similarity is no guarantee that it is the same person. This is meant to be used as entertainment

This is an unofficial site run by an individual, and it has no affiliation with FANZA or DMM.com. Our data sources and how we think about running the site are written up in About this site and Frequently asked questions.

Read next

WEB SERVICE BY FANZA