Face masks have made facial recognition software less accurate, says federal report

It’s now common to see people donning face coverings, especially in public places, in an effort to mitigate the spread of COVID-19. But with face coverings becoming standard attire, experts from the National Institute of Standards and Technology (NIST) had concerns about how these can impact facial recognition software. In their report, the researchers revealed that the coverings made facial recognition software less effective. In particular, even the best facial recognition algorithms developed prior to the pandemic had error rates between five to 50 percent when they attempted to identify people wearing masks. The report is the first in a series of NIST studies looking at how coverings affect facial recognition software.

“With the arrival of the pandemic, we need to understand how face recognition technology deals with masked faces,” explained Mei Ngan, a NIST computer scientist and a co-author of the study. “We have begun by focusing on how an algorithm developed before the pandemic might be affected by subjects wearing face masks.”

The team plans to test the accuracy of algorithms designed with face coverings in mind later this year.

Face coverings impact software accuracy

For the federal study, the team investigated whether facial recognition software can perform “one-to-one” matching, which compares two different photos of the same person. This function is widely used in verification, such as unlocking a smartphone or checking a passport.

To do this, the team tested the algorithms on two datasets of around six million photos used by the government. One dataset contained images from immigrant benefit applications, while the other dataset contained photos taken at U.S. border crossings. Then, they digitally applied mask shapes to the original photos and tested the algorithms again. Given that people wear different masks in real-world settings, the researchers used nine mask types with different shapes, colors and nose coverage. (Related: FACT CHECK: Natural News remains the most outspoken pro-mask independent news publisher, even as InfoWars and others have become strongly anti-mask.)

The researchers used this setup to mimic real-life applications of facial recognition technology. For one, officials should be able to identify mask-wearing individuals as they present unmasked photos for identification.

The findings revealed a significant increase in error rates, even among the best facial recognition algorithms. These algorithms, in particular, tend to fail in matching 0.3 percent of the time, but these jumped to five percent after the digital masks were applied. Lower quality algorithms fared worse, having error rates of up to 50 percent.

“We can draw a few broad conclusions from the results, but there are caveats,” Ngan added. “None of these algorithms were designed to handle face masks, and the masks we used are digital creations, not the real thing.”

In addition, making a second attempt at identification won’t help much, according to the researchers. Without a mask, a user can just adjust his posture or expression for the algorithm to get a match. This won’t work for people with masks on, especially if the failure is rooted in a “systemic property of the algorithm.”

The shape and color of the masks can also affect the accuracy of the software. In particular, mask shapes that covered the nose are where algorithms struggled the most. The researchers added that exploring this is necessary, as most studies recommend covering the nose to prevent the spread of COVID-19.

“Unsurprisingly masks that occlude more of the face give larger false nonmatch rates,” the team wrote in the report.

The researchers also noted some limitations in their report. For one, while the digital masks covered various shapes, styles and colors, the shapes presented in the study are not very similar or representative of the masks that people wear in real-world settings. The study also didn’t include those who wear glasses with their masks, as well as how facial recognition software affects various demographic groups.

Sources include:

NextGov.com

NIST.gov