Skip to main content

ICP face recognition

This is an ICP smart contract runs face detection and face recognition of user's photo that can be uploaded either from a camera or a local file.

The smart contract consists of two canisters:

  • The backend canister embeds the the Tract ONNX inference engine with two ONNX models. One model is used to detect a face in the photo and return its bounding box. Another model is used for computing face embeddings.
  • The frontend canister contains the Web assets such as HTML, JS, CSS that are served to the browser.

Models

The smart contract uses two models: one for detecting the face and another for recognizing the face.

Face detection​

A face detection model finds the bounding box of a face in the image. You can download Ultraface - ultra-lightweight face detection model - [here].

Alternatively, you can run

./download-face-detection-model.sh

Face recognition​

A face recognition model computes a vector embedding of an image with a face. You can obtain a pretrained model from facenet-pytorch as follows.

pip install facenet-pytorch
pip install torch
pip install onnx

  • Step 3: Export ONNX model. Start a python shell and run the following commands or create a python file and run it:​

import torch
import facenet_pytorch
resnet = facenet_pytorch.InceptionResnetV1(pretrained='vggface2').eval()
input = torch.randn(1, 3, 160, 160)
torch.onnx.export(resnet, input, "face-recognition.onnx", verbose=False, opset_version=11)

  • Step 4: This should produce face-recognition.onnx. Copy the file to the root of this repository.​

Prerequisites​

  • Install the IC SDK. For local testing, dfx >= 0.22.0 is required.
  • Clone the example dapp project: git clone https://github.com/dfinity/examples
  • Install wasi2ic: Follow the steps in https://github.com/wasm-forge/wasi2ic and make sure that wasi2ic binary is in your $PATH.
  • Install wasm-opt: cargo install wasm-opt

Build the application​

dfx start --background
dfx deploy

If the deployment is successful, the it will show the frontend URL. Open that URL in browser to interact with the smart contract.

Chunk uploading of models​

Since the models are large, they cannot be embedded into the Wasm binary of the smart contract. Instead they should be uploaded separately.

DecideAI implemented a tool for incremental uploading of models: https://github.com/modclub-app/ic-file-uploader/tree/main.

You can install the tool with

cargo install ic-file-uploader

Afterwards, execute the upload-models-to-canister.sh script, which runs the following commands:

dfx canister call backend clear_face_detection_model_bytes
dfx canister call backend clear_face_recognition_model_bytes
ic-file-uploader backend append_face_detection_model_bytes version-RFB-320.onnx
ic-file-uploader backend append_face_recognition_model_bytes face-recognition.onnx
dfx canister call backend setup_models

Credits​

Thanks to DecideAI for discussions and providing ic-file-uploader.