Synthesis AI for ID Verification
Overview
Smartphones, PCs, and even financial, legal, government, healthcare and other institutions are increasingly using facial recognition for identity verification. But recent events have highlighted the strong bias in existing datasets — so much so, that according to Gartner, by 2022, more than 95% of RFPs for “will contain clear requirements regarding minimizing demographic bias” (2020 Gartner Market Guide for Identity Proofing and Affirmation).
Synthesis AI features over 40,000 unique identities across a broad spectrum of skin tones and ages. We also feature glasses, face masks, hats, headphones, facial hair, and infinite lighting and camera controls -- we can help your facial recognition models meet today's requirements.
Our API provides the ability to mimic indoor and outdoor environments so you can quickly test and iterate machine learning models across a broader set of settings and situations without spinning up a massive data collection effort. This includes granular control of mouth opening, gaze angle, head pose, accessories, gestures, lighting environments, camera systems (e.g. multi-camera modules), and more.
This document aims to provide copy-and-paste input examples to our API that serve many of the facial re-identification scenarios, including:
- Skin Tone and Age
- Camera Placement
- Occlusion
- Hair and Facial Hair
- Illumination
Workflow
The standard workflow for the Synthesis AI Face API is:
- Register Account, Download & Setup CLI
- Select Identities for the Face API with the Identities API
- Create Face API job with input JSON
- Download job output with CLI
- Parse outputs into your ML pipeline
Hello Re-ID World
Re-identification is one of the most challenging tasks in computer vision, and the most important to remove all forms of bias. Bias shows up in many forms from human demographics, to camera angle and human pose, and environmental factors. Confounds also create issues for re-identification models, like changing hair, facial hair growth throughout the day, hats, facial masks, and headphones.
To make it easy to get these replicate all of these potential video conference settings, the JSON provided contains the following features:
- 4 identities with 1 renders per identitiy (selected via Identities API)
- Camera placement similar to both desktop and mobile phones
- Dark & light indoor and outdoor environments
- Sunglasses (20% of images)
- Headwear (20% of images)
- Head Turn: head turn upwards (25), 20% of images
- Facial Hair (20% of images)
- Variety of clothing
To process the outputs more easily, we provide a synthesisai library in Python, and output documentation.
Quickstart JSON for ID Verification
{
"humans": [{
"identities": {
"ids": [
192
],
"renders_per_identity": 1
},
"skin": {
"highest_resolution": true
},
"environment": {
"hdri": {
"name": [
"home_breakfast_nook"
],
"intensity": {
"type": "list",
"values": [
0.1
]
}
}
},
"camera_and_light_rigs": [{
"cameras": [{
"name": "default",
"specifications": {
"focal_length": {
"type": "list",
"values": [
150.0
]
},
"sensor_width": {
"type": "list",
"values": [
150.0
]
},
"wavelength": "visible",
"resolution_w": 1280,
"resolution_h": 960
}
}],
"lights": [{
"type": "rect",
"size_meters": {
"type": "range",
"values": {
"min": 0.03,
"max": 0.03
}
},
"intensity": {
"type": "range",
"values": {
"min": 0.05,
"max": 0.05
}
},
"color": {
"red": {
"type": "range",
"values": {
"min": 142,
"max": 142
}
},
"green": {
"type": "range",
"values": {
"min": 188,
"max": 188
}
},
"blue": {
"type": "range",
"values": {
"min": 234,
"max": 234
}
}
},
"wavelength": "visible",
"relative_location": {
"z": {
"type": "list",
"values": [0.3]
}
}
}],
"location": {
"z": {
"type": "list",
"values": [
0.85
]
}
}
}]
},
{
"identities": {
"ids": [
119
],
"renders_per_identity": 1
},
"skin": {
"highest_resolution": true
},
"accessories": {
"glasses": [{
"style": [
"female_oval_plastic"
],
"lens_color": ["gray"],
"metalness": {
"type": "list",
"values": [
0.8
]
},
"transparency": {
"type": "list",
"values": [
0.95
]
},
"sex_matched_only": true,
"percent": 100
}]
},
"environment": {
"hdri": {
"name": [
"photo_studio_01"
]
}
},
"camera_and_light_rigs": [{
"cameras": [{
"name": "default",
"specifications": {
"focal_length": {
"type": "list",
"values": [
150.0
]
},
"sensor_width": {
"type": "list",
"values": [
150.0
]
},
"wavelength": "visible",
"resolution_w": 1280,
"resolution_h": 960
}
}],
"lights": [{
"wavelength": "visible"
}],
"location": {
"z": {
"type": "list",
"values": [
0.55
]
}
},
"type": "head_orbit_with_tracking"
}]
},
{
"identities": {
"ids": [
26
],
"renders_per_identity": 1
},
"skin": {
"highest_resolution": true
},
"facial_attributes": {
"facial_hair": [{
"style": [
"full_generic_01"
],
"color": [
"chocolate_brown"
],
"match_hair_color": true,
"relative_length": {
"type": "list",
"values": [
0.5
]
},
"relative_density": {
"type": "list",
"values": [
1
]
},
"percent": 100
}]
},
"environment": {
"hdri": {
"name": [
"photo_studio_01"
]
}
},
"camera_and_light_rigs": [{
"cameras": [{
"name": "default",
"specifications": {
"focal_length": {
"type": "list",
"values": [
150.0
]
},
"sensor_width": {
"type": "list",
"values": [
150.0
]
},
"wavelength": "visible",
"resolution_w": 1280,
"resolution_h": 960
}
}],
"lights": [{
"wavelength": "visible"
}],
"location": {
"z": {
"type": "list",
"values": [
0.55
]
}
},
"type": "head_orbit_with_tracking"
}]
},
{
"identities": {
"ids": [
46
],
"renders_per_identity": 1
},
"skin": {
"highest_resolution": true
},
"facial_attributes": {
"head_turn": [{
"pitch": {
"type": "list",
"values": [
-25
]
},
"percent": 100
}]
},
"environment": {
"hdri": {
"name": [
"photo_studio_01"
]
}
},
"camera_and_light_rigs": [{
"cameras": [{
"name": "default",
"specifications": {
"focal_length": {
"type": "list",
"values": [
150.0
]
},
"sensor_width": {
"type": "list",
"values": [
150.0
]
},
"wavelength": "visible",
"resolution_w": 1280,
"resolution_h": 960
}
}],
"lights": [{
"wavelength": "visible"
}],
"location": {
"z": {
"type": "list",
"values": [
0.55
]
}
}
}]
},
{
"identities": {
"ids": [
192
],
"renders_per_identity": 1
},
"skin": {
"highest_resolution": true
},
"environment": {
"hdri": {
"name": [
"greenwich_park_03"
],
"intensity": {
"type": "list",
"values": [
0.75
]
}
}
},
"camera_and_light_rigs": [{
"cameras": [{
"name": "default",
"specifications": {
"focal_length": {
"type": "list",
"values": [
150.0
]
},
"sensor_width": {
"type": "list",
"values": [
150.0
]
},
"wavelength": "visible",
"resolution_w": 1280,
"resolution_h": 960
}
}],
"lights": [{
"wavelength": "visible"
}],
"location": {
"z": {
"type": "list",
"values": [
0.85
]
}
}
}]
},
{
"identities": {
"ids": [
119
],
"renders_per_identity": 1
},
"skin": {
"highest_resolution": true
},
"accessories": {
"headwear": [{
"style": [
"female_hijab_1"
],
"sex_matched_only": true,
"percent": 100
}]
},
"environment": {
"hdri": {
"name": [
"photo_studio_01"
]
}
},
"camera_and_light_rigs": [{
"cameras": [{
"name": "default",
"specifications": {
"focal_length": {
"type": "list",
"values": [
150.0
]
},
"sensor_width": {
"type": "list",
"values": [
150.0
]
},
"wavelength": "visible",
"resolution_w": 1280,
"resolution_h": 960
}
}],
"lights": [{
"wavelength": "visible"
}],
"location": {
"z": {
"type": "list",
"values": [
0.55
]
}
},
"type": "head_orbit_with_tracking"
}]
}
]
}Visual examples of quickstart JSON:
Reducing Demographic Bias: Sex, Age, Ethnicity
Our API features over 40,000 unique identities across a broad spectrum of skin tones and ages. Our Identities API allows you to programmatically select a distribution of identities that match your requirements across sex, skin tone, and age.
We provide a sample JSON with IDs that span the age and skin tone spectrum:
JSON example of multiple IDs
"humans": [{
"identities": {
"ids": [137,80,185,93,406,468,286,125,180,242],
"renders_per_identity": 1
},
...
}]
Example IDs
|
Skin Tone |
 |
 |
 |
 |
 |
|
Age |
 |
 |
 |
 |
 |
Reducing Head Pose & Camera Angle Bias
Few datasets have a wide distribution of head turn angles and extreme camera angles. Our API provides programmatic control of head turn which allow for simulation of many scenarios; and comes with both face and body 3d landmarks.
These head turns are compatible with hats, glasses, and face masks, which serve as effective occlusions to train against.
In addition to head turn, we provide samples of camera angles. To reduce bias, consider turning the camera's "follow_head_orientation" to true, and then adding vertical and horizontal camera angle ranges (see camera location documentation). With "follow_head_orientation", the camera first follows the front of the face while head turn is applied, and subsequently, the additional camera vertical and horizontal range is applied.
Example JSON of head pose
"facial_attributes": {
"hair": [{
"style": ["quiff_01"],
"color": ["chocolate_brown"],
"sex_matched_only": true,
"ethnicity_matched_only": true,
"percent": 100
}],
"eyebrows": [{
"style": ["high_03"],
"sex_matched_only": true,
"match_hair_color": true,
"percent": 100
}],
"head_turn": [{
"pitch": {
"type": "list",
"values": [0]
},
"yaw": {
"type": "list",
"values": [25]
},
"roll": {
"type": "list",
"values": [0]
},
"percent": 100
}]
},
"camera_and_light_rigs": [{
"type": "head_orbit_with_tracking ",
"location": {
"pitch": {
"type": "range",
"values": {
"min": 0,
"max": 0
}
},
"yaw": {
"type": "range",
"values": {
"min": 0,
"max": 0
}
},
"roll": {
"type": "list",
"values": [0]
},
"x": {
"type": "range",
"values": {
"min": 0,
"max": 0
}
},
"y": {
"type": "range",
"values": {
"min": 0,
"max": 0
}
},
"z": {
"type": "range",
"values": {
"min": 0.55,
"max": 0.55
}
}
},
"cameras": [{
"name": "default",
"specifications": {
"resolution_h": 960,
"resolution_w": 1280,
"focal_length": {
"type": "list",
"values": [150.0]
},
"sensor_width": {
"type": "list",
"values": [150.0]
},
"wavelength": "visible"
},
"relative_location": {
"pitch": {
"type": "list",
"values": [0]
},
"yaw": {
"type": "list",
"values": [0]
},
"roll": {
"type": "list",
"values": [0]
},
"x": {
"type": "list",
"values": [0]
},
"y": {
"type": "list",
"values": [0]
},
"z": {
"type": "list",
"values": [0]
}
},
"advanced": {
"denoise": true,
"noise_threshold": 0.017,
"lens_shift_vertical": {
"type": "list",
"values": [0]
},
"lens_shift_horizontal": {
"type": "list",
"values": [0]
},
"window_offset_vertical": {
"type": "list",
"values": [0]
},
"window_offset_horizontal": {
"type": "list",
"values": [0]
}
}
}]
}],
"environment": {
"hdri": {
"name": ["photo_studio_01"]
}
}
Visual examples of head poses:
Accessory Occlusions
ID verification systems also need to tolerate occlusions by accessories. Our API offers programatic additions of many varieties of hats, glasses, and face masks.
On the output side all generated images come with segmentation (Jupyter Notebook) and face and body 3d landmarks.
Example JSON of glasses accessories
"accessories": {
"glasses": [{
"style": ["all"],
"lens_color": ["default"],
"transparency": {
"type": "range",
"values": {
"min": 1,
"max": 1
}
},
"metalness": {
"type": "range",
"values": {
"min": 0.03,
"max": 0.03
}
},
"sex_matched_only": true,
"percent": 100
}]
}
...
Visual examples of accessories:
Varying Hair & Facial Hair per Identity
Changing hairstyles, eyebrow styles, and especially facial hair are major confounds to some ID verification systems. Our API provides the ability to output a wide variety of hair styles and facial hair grooms, and also vary the color and length for any given identity.
By setting multiple renders_per_identity, each identity will have varying styles and color and increase robustness of your models. See the following JSON example and images.
Example JSON of IDs associated with hair
{
"humans": [{
"identities": {
"ids": [25, 119, 121, 128, 132, 80, 46, 77, 106, 129, 341, 93, 1037, 2415, 31, 76, 97, 94, 35, 36, 90, 26, 186],
"renders_per_identity": 100
},
"facial_attributes": {
"hair": [{
"style": ["all"],
"color": ["all"],
"sex_matched_only": true,
"ethnicity_matched_only": true,
"percent": 100
}],
"eyebrows": [{
"style": ["all"],
"sex_matched_only": true,
"match_hair_color": true,
"percent": 100
}],
"facial_hair": [{
"style": ["all"],
"color": ["all"],
"match_hair_color": true,
"percent": 100
}],
}
}]
}
...
Visual examples of hair styles and colors:
Lighting Variety
Being robust to "harsh" lighting environments is also a challenge for ID verification systems. Our APIs not only provide a large collection of indoor environments, outdoor environments, but also allows for placement of custom directional lights to simulate additional variety of light colors and intensities. Additionally, we allow for NIR-spectrum lighting and cameras.
The following JSON example uses directional lights to simulate glow from a computer monitor.
JSON environment lights simulating a "monitor" glow
"lights": [{
"type": "rect",
"color": {
"red": {
"type": "list",
"values": [142]
},
"blue": {
"type": "list",
"values": [234]
},
"green": {
"type": "list",
"values": [188]
}
},
"intensity": {
"type": "list",
"values": [0.05]
},
"wavelength": "visible",
"size_meters": {
"type": "list",
"values": [0.3]
},
"relative_location": {
"x": {
"type": "list",
"values": [0]
},
"y": {
"type": "list",
"values": [0]
},
"z": {
"type": "list",
"values": [0.3]
},
"yaw": {
"type": "list",
"values": [0]
},
"roll": {
"type": "list",
"values": [0]
},
"pitch": {
"type": "list",
"values": [0]
}
}
}]
Visual examples of lighting:
