I: Introduction

Advances in sensor technology, remote bio-signal processing, and facial recognition are enabling governments and individuals to covertly gather private health information at unprecedented scales.

Countries around the world have taken drastic public health monitoring measures during the COVID-19 pandemic ranging from augmented reality thermal vision goggles goggles (Fig. 1) to integrated mobile phone applications for dynamic individualized risk assessments. While these systems have enabled rapidly adaptive public health policy, they have also opened the doors for dystopic levels of “automated tyranny” [1].

At the individual level, research prototypes such as ones presented in Sensing of the Self, Society, and Environment [2] have suggested the feasibility of similar health monitoring. Facial recognition, user biometrics, and environmental noise data are gathered continuously during use at a relatively low cost. As a member of the research team, I became familiar with modern bio-signal processing methodologies discussed in Section II.1.

Our prototypes built for government-funded research grant applications also incorporated systems for remotely estimating biometrics of those in the user’s viscinity. Naturally, the prototypes raised concerns about non-consensual breach of private health information, especially in light of the continuous facial recognition and indexing technology presented in [2].

---

In this paper, I seek to characterize this technology and understand the ethical and moral implications of its application. To this end, I survey the existing peer-reviewed literature on each key remote bio-signal processing technologies and describe the current use of these and similar technologies by governments around the world to contextualize its use at the state level. Finally, I conclude with an ethical analysis of these systems at the government and individual levels through the lenses of utiliarianism, human rights ethics, and current legislation.

II: Characterizing the Technology

The technology in question is the combination of facial recognition and remote bio-signal processing technologies. When these two are combined, it enables the collection of individually identifiable health information at a large scale. This presents a novel ethical challenge for existing health information privacy practices and legislation. The technology in question is fundamentally reliant on AI methods in image classification and segmentation for facial recognition and tracking. Furthermore, cutting-edge bio-signal processing methods make substantial use of learning methods for improved estimation (discussed in Section II.1). It is therefore effective to classify the technologies in question together as an AI system.

II.1: Existing Technology Overview

Here, I provide a technical overview of select facial recognition and remote bio-signal processing methodologies for estimating common biometrics including heart rate/variability, respiratory rate/variability, and body temperature.

II.1 A: Facial Recognition

Facial recognition refers to the automated detection and recognition of individuals in photos and videos. Early efforts had revolved around hand measurements of features [3]. Sirvoich and Kirby [4] were the first to apply principal component analysis (PCA, Fig. 3) to efficiently encode face images in 1988. In 1991, this “eigenfaces” technique was applied by Turk and Pentland [5] in an attempt to solve facial recognition. By calculating the Euclidian distance between projection coefficients of a new face and the projection coefficients of each known face, they were able to create a new state-of-the-art for automated facial recognition.

While there are a multitude of modern methods for facial recognition, this general method of PCA → low-dimensionality matching is the most well-known and widely accepted method. Since PCA reduces image dimensionality so much, relatively few training samples are required to learn the distribution of a new person’s face.

---

Accuracy and robustness are the most important metrics for these systems. According to a 2019 report by the National Institute of Standards and Technology [8], there have been massive gains in accuracy between 2013 and 2018. For high quality head-on photos, the 127 algorithms from industrial and academic labs tested on a gallery of 12 million photos regularly achieved error rates below 0.2%. While there are still challenges associated with ageing, injury, and low-quality photos, this AI technology is clearly able to achieve high performance on these relevant metrics.

II.1 B: Remote Heart Rate Estimation

There is a large body of literature on the estimation of heart rate from a distance (known as rPPG) indicating substantial feasibility. Even with a simple RGB camera, one can detect frequency spectrum peaks in pixel luminosity at the same frequency as the heart rate [9] (see Fig. 4).

More recently, a 2020 paper from Neurodata Lab proposed a machine learning approach for accurate heart rate estimation even during motion and lighting changes [10]. By segmenting footage using computer vision methods to narrow in on regions of interest, interpolating luminance data, and applying band-pass filtering, an accurate estimate of central pulse rate can be obtained. The proposed system was particularly effective relative to status quo methods during movement (p < 0.001). A fine-tuning step for estimating heart rate variation also enabled strong moment-to-moment estimation.

Recent progress in this field indicates high potential for accurate and robust remote heart rate and heart rate variability estimation using relatively low-cost consumer RGB cameras.

II.1 C: Remote Respiratory Rate

Systems for remote respiratory rate estimation bare substantial resemblance to those for heart rate estimation. Since respiration is known to affect heart rate and heart rate variability, there is experimental evidence for achieving fairly accurate respiratory rate estimation solely from rPPG data from face videos described above [11].

A 2020 paper by researchers at Universidad Panamericana compares a variety of techniques such as Eulerian motion video magnification and Hermite transform to a convolutional neural network (CNN)-based technique for respiration estimation based on simple RGB video [12]. The CNN was trained to classify a short sequence of images extracted from the video as either inhalations or exhalations. When combined with motion amplification, this technique was able to achieve over 98% test classification accuracy on all subjects in a variety of body positions.

Thermal imaging approaches show substantial promise for respiration rate estimation on moving subjects as well. Using facial recognition algorithms to localize regions of interest, respiration rate was accurately tracked in real time based on infrared radiation emission fluctuations [13]. This research used a FLIR A40 thermal imaging camera, which would have likely cost several thousand dollars [14].

Since thermal imaging technologies are becoming more and more affordable [14] and computer vision techniques for this problem continue to be investigated, remote respiration rate estimation clearly has a bright (and increasingly more affordable) future.

II.1 D: Body Temperature Estimation

According to the US Food and Drug Administration, thermal imaging systems can accurately measure people’s skin temperature from a distance for COVID-19 screening [15]. While there exist limitations in terms of temperature detection en masse and in environments with widely varying humidity and temperatures, this estimation problem can be considered solved for the purposes of fever detection based on the FDA’s documentation.

II.2: Current Use Cases

The Chinese government’s response to COVID-19 offers the most comprehensive example of a government integrating a wide range of surveillance, facial recognition, and remote biometric technologies to inform individualized public health policy.

According to an April 2020 Human Rights Watch article [1], the Chinese government-sponsored “Health Code” app indirectly and directly integrates many of the aforementioned technologies and more. After filling out their personal information (including symptoms and personal connections), a color code is issued ranging from green to red. The color code can dictate freedom of movement as certain areas use facial recognition technology to only allow those with a green code to enter. In addition to sharing location data with police, it is unclear what data streams are used to adapt an individual’s color code over time.

Concern is intensified as the Chinese government has substantially increased surveillance and facial recognition measures during the pandemic [16] with added focus on thermal imaging and ethnicity classification. Security robots and police outfitted with thermal imaging systems were also deployed [17].

---

The work in Sensing of the Self, Society, and Environment [2] and the associated publications is one relevant example of similar technology being used at the individual level. In [2], facial recognition is actively performed by the wearable system. While the this aspect is a new addition to the underlying Eyetap design, the Continuous lifelong capture of personal experience with EyeTap [18] is far from new (Fig. 5). University of Toronto Professor Steve Mann has long championed the virtues of this “sousveillance”: the inverse of surveillance wherein a participant in an activity records the activity [19] in contrast to being recorded (“surveilled”) by the administrators of said activity.

Regardless of the ethics of “sousveillance” for remote biometric systems (discussed in III: Ethical Critique), the continuous recording of people and events by individuals is not without contention. Even without remote biosensing or facial recognition, Prof. Mann allegedly suffered assault by employees at a Paris McDonalds because of his refusal to remove his Eyetap headset [20].

II.3: Technological Conclusion

There is clear novelty in the combination of facial recognition technology with recent advances in remote bio-signal processing. For the first time, one can covertly collect certain private health information from a distance. While the aforementioned biometrics may not seem like an extreme privacy violation, this information forms the basis for many medical diagnostics [21]. Given the trajectory of the technologies described, questions remain on the moral and ethical implications of its use now and in the future.

III: Ethical Critique

III.1: Ethical Framework

In order to understand the ethics of remote bio-signal processing in conjunction with facial recognition, I will make use of concepts from utiliarianism, human rights ethics, and existing legislative precedents.

Utilitarianism is a moral philosophy that seeks the maximum satisfaction and prosperity for the greatest number of people [22]. While some may argue that the freedom or privacy of a society is necessary for citizens to experience maximum satisfaction [23], I do not include this view in my definition of utilitarianism as it is more clear to contextualize privacy and freedom in human rights ethics. In this discussion, utiliarianism therefore refers to maximizing health, safety, and convenience the maximum number of people.

Human rights ethics offers a valuable counterpoint to utilitarianism by asserting certain inalienable rights that must not be violated if one is to create a righteous society. A central document in human rights ethics is the Universal Declaration on Human Rights (UDHR). Of the 30 articles, the following are the most relevant points to the present subject are summarized here [24]:

• Right to life, liberty, and security of person.
• No arbitrary attacks on privacy, family, home, or correspondence.
• No discrimination.

Finally, existing legislation in privacy (especially health information privacy) will be taken into account. The Health Insurance Portability and Accountability (HIPAA) is a piece of US legislation that outlines requirements for how personally identifiable information in electronic health records should be dealt with by healthcare and health insurance organizations [25]. It stipulates that “covered entities” (medical service providers, insurers, etc.) must ensure confidentiality and security of the identifyable healthcare data. The Confidential Information Protection and Statistical Efficiency Act (CIPSEA) is another US law establishing standards for protecting citizens from being personally exposed by census, labor statistics, and economic analysis data [26]. Finally, the General Data Protection Regulation (GDPR) is a sweeping piece of data protection legislation enacted by the European Union. Among other things, it stipulates that “data subjects” must give explicit, unambiguous consent before collection of personal data. Subjects have the right to be informed, to access their data, to correct their data, erase, restrict processing, and the right to object to the use of their data [27].

III.2: Critique

For both government and individual level applications of the technology in question, a motivation for the value of health information privacy is beneficial. According to the book, “Beyond the HIPAA Privacy Rule: Enhancing Privacy, Improving Health Through Research” [28]:

The bioethics principle nonmaleficence requires safeguarding personal privacy. Breaches of privacy and confidentiality not only may affect a person’s dignity, but can cause harm. When personally identifiable health information, for example, is disclosed to an employer, insurer, or family member, it can result in stigma, embarrassment, and discrimination.

At its core, the maintenance of health information privacy is a measure to reduce harm. Even for seemingly minor biometrics discussed in Section II, privacy considerations are warranted, particularly as researchers continue to work on sensing and bio-signal processing technologies.

III.2 A: Government Use

At the government level, the technology in question has massive potential for good. From a utiliarian perspective, this is a straight forward evaluation. Particularly during pandemics, public health monitoring through these systems has the ability to transform the way that governments adapt their public health guidelines. Many of the unknowns about disease transmission dynamics [29] could be answered using the data gathered by facial recognition and remote biosensing systems, especially if used in conjunction with a symptom logging application.

The ethical value of these systems at the government level is less clear from a human rights perspective, however. While the technology may increase one’s access to “life, liberty, and security of person” [24] during a pandemic, some may regard the large-scale and covert gathering of identifyable private health information as an attack on privacy. Furthermore, if movement rights are restricted at an individual level based on black-box systems as it has been in China, these systems could form the basis for a hyper-efficient method for discrimination. Even now, this is not an irrational concern in China given that the government heavily invested in facial recognition technology specifically designed to perform ethnicity classification [16].

Existing legislation and practices pertaining to the gathering and use of citizens’ data likely holds much of the answer to resolving these issues at the government level. De-identification, transparency, and statistical efficiency are central to proper management in this case. HIPAA’s requirements on de-identification offer excellent guidelines on how to prepare datasets for use in health policy research such that individuals cannot be identified. CIPSEA’s standards for statistical efficiency can be readily applied to data gathered from the public with such systems. Finally, key points from the GDPR such as the right to be informed, to acces, and to correct one’s data can help to assuage concerns of black-box systems enabling covert discrimination. Although GDPR and HIPAA are targeted toward corporations and healthcare providers respectively, their core principles are highly applicable to this dilemma. Thus, these technologies have a strong potential for good at the government level.

---

III.2 B: Individual Use

At the individual level, ethical valuations are substantially more speculative given the lack of accessible/direct precedents. Generally, I am of the mind that these advanced bio-signal processing and facial recognition systems should not be used arbitrarily by members of the public.

One of the main arguments for “sousveillance” is that it helps to keep the user safer [19] as recording dissuades others from doing harm. In the case of the research grants I helped write as part of the research team for wearable systems that incorporate remote bio-signal processing systems and facial recognition, the primary benefit we communicated was that users could screen those around them for symptoms of COVID-19 to adjust their actions and estimate their own risk of contracting the disease.

Unfortunately, this line of reasoning is tenuous at best when applied to non-medical professional citizens. The technologies in Section II.1 are merely a method for screening. At the individual level, the value of this type of “sousveillance” of people’s biometrics therefore has very little value in terms of reducing one’s risk of becoming diseased. This leaves the potentially non-consensual measurements of people’s health information as the main function of this technology at the individual level. The discomfort and potential damage caused by this breach of privacy outlined in [28] make a utiliarian argument for this application infeasible. The spirit of current legislation [25, 27] (Section III.1) further damns the ethical merit of applying the technology on an individual level for the same reasons.

From a human rights perspective, this would be a violation of privacy and would increase the risk of discrimination on the basis of medical conditions. While one may argue that the ability to use such technology at the individual level is necessary to maintain the right to liberty, this argument fails in light of the myriad laws and practices around other potentially invasive or disruptive technologies and practice like radio jammers [30].

III.3: Conclusion

Advances in facial recognition and bio-signal processing have and will continue to augment the ability of governments and individuals to covertly gather private health information. It is likely possible for governments to make use of these technologies to improve public health during pandemics. Assuming adherence to privacy, transparency, and statistical effiency standards, they may provide valuable information about the dynamics of disease spread without substantially negatively affecting citizens.

Meanwhile, it is unlikely that the use of these systems by individuals will be positive for society (or for the individual). The current technology offers only a limited number of remote biometrics and is nowhere near providing reliable diagnostics at a distance. At a the scale of an individual, this type of screening is of little use during a pandemic and, when combined with facial recognition technology, is generally negative because of its invasiveness.

Bibliography

[1] M. Wang, “China: Fighting COVID-19 With Automated Tyranny,” Human Rights Watch, 28-Oct-2020. [Online]. Available: https://www.hrw.org/news/2020/04/01/china-fighting-covid-19-automated-tyranny. [Accessed: 17-Mar-2021].

[2] S. Mann, C. Pierce, A. Bhargava, C. Tong, K. Desai, and K. O’Shaughnessy, “Sensing of the Self, Society, and the Environment,” 2020 IEEE SENSORS, 2020.

[3] P. Kaur, K. Krishan, S. K. Sharma, and T. Kanchan, “Facial-recognition algorithms: A literature review,” Medicine, Science and the Law, vol. 60, no. 2, pp. 131–139, 2020.

[4] L. Sirovich and M. Kirby, “Low-dimensional procedure for the characterization of human faces,” Journal of the Optical Society of America A, vol. 4, no. 3, p. 519, 1987.

[5] M. A. Turk and A. P. Pentland, “Face recognition using eigenfaces,” Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1991.

[6] A. S. Georghiades, P. N. Belhumeur, and D. J. Kriegman, “From few to many: illumination cone models for face recognition under variable lighting and pose,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp. 643–660, 2001.

[7] S. Draper, “ECE367: Matrix Algebra and Optimization — Assignment 3,” in University of Toronto, 2020.

[8] P. Grother, M. Ngan, and K. Hanaoka, “Ongoing face recognition vendor test (FRVT) part 2:” National Institute of Standards an Technology, 2018.

[9] S. Benedetto, C. Caldato, D. C. Greenwood, N. Bartoli, V. Pensabene, and P. Actis, “Remote heart rate monitoring – Assessment of the Facereader rPPg by Noldus,” PLOS ONE, vol. 14, no. 11, 2019.

[10] M. Artemyev, M. Churikova, M. Grinenko, and O. Perepelkina, “Robust algorithm for remote photoplethysmography in realistic conditions,” Digital Signal Processing, vol. 104, p. 102737, 2020.

[11] M. Chen, Q. Zhu, H. Zhang, M. Wu, and Q. Wang, “Respiratory Rate Estimation from Face Videos,” 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), 2019.

[12] J. Brieva, H. Ponce, and E. Moya-Albor, “A Contactless Respiratory Rate Estimation Method Using a Hermite Magnification Technique and Convolutional Neural Networks,” Applied Sciences, vol. 10, no. 2, p. 607, 2020.

[13] H. Elphick, A. Alkali, R. Kingshott, D. Burke, and R. Saatchi, “Thermal imaging method for measurement of respiratory rate,” 7.1 Paediatric Respiratory Physiology and Sleep, 2015.

[14] “FLIR One Pro LT,” FLIR ONE Pro LT Thermal Camera for Smartphones | FLIR Systems. [Online]. Available: https://www.flir.ca/products/flir-one-pro-lt/. [Accessed: 17-Mar-2021].

[15] Center for Devices and Radiological Health, “Thermal Imaging (Infrared Thermographic Systems / Thermal Cameras),” U.S. Food and Drug Administration. [Online]. Available: https://www.fda.gov/medical-devices/general-hospital-devices-and-supplies/thermal-imaging-systems-infrared-thermographic-systems-thermal-imaging-cameras#:~:text=Thermal imaging systems generally detect,be contagious without a fever. [Accessed: 17-Mar-2021].

[16] D. Byler, “State of Surveillance,” ChinaFile, 30-Dec-2020. [Online]. Available: https://www.chinafile.com/state-surveillance-china. [Accessed: 17-Mar-2021].

[17] T. W. Martin and L. Lin, “Fever-Detecting Goggles and Disinfectant Drones: Countries Turn to Tech to Fight Coronavirus,” The Wall Street Journal, 10-Mar-2020. [Online]. Available: https://www.wsj.com/articles/fever-detecting-goggles-and-disinfectant-drones-countries-turn-to-tech-to-fight-coronavirus-11583832616. [Accessed: 17-Mar-2021].

[18] S. Mann, “Continuous lifelong capture of personal experience with EyeTap,” Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences – CARPE’04, 2004.

[19] S. Mann, J. Nolan, and B. Wellman, “Sousveillance: Inventing and Using Wearable Computing Devices for Data Collection in Surveillance Environments.,” Surveillance & Society, vol. 1, no. 3, pp. 331–355, 2002.

[20] B. Popper, “New evidence emerges in alleged assault on cyborg at Paris McDonald’s,” The Verge, 19-Jul-2012. [Online]. Available: https://www.theverge.com/2012/7/19/3169889/steve-mann-cyborg-assault-mcdonalds-eyetap-paris. [Accessed: 17-Mar-2021].

[21] “Vital Signs (Body Temperature, Pulse Rate, Respiration Rate, Blood Pressure),” Johns Hopkins Medicine. [Online]. Available: https://www.hopkinsmedicine.org/health/conditions-and-diseases/vital-signs-body-temperature-pulse-rate-respiration-rate-blood-pressure. [Accessed: 17-Mar-2021].

[22] J. Driver, “The History of Utilitarianism,” Stanford Encyclopedia of Philosophy, 22-Sep-2014. [Online]. Available: https://plato.stanford.edu/entries/utilitarianism-history/. [Accessed: 17-Mar-2021].

[23] D. Brink, “Mill’s Moral and Political Philosophy,” Stanford Encyclopedia of Philosophy, 21-Aug-2018. [Online]. Available: https://plato.stanford.edu/entries/mill-moral-political/. [Accessed: 17-Mar-2021].

[24] “Universal Declaration of Human Rights,” United Nations. [Online]. Available: https://www.un.org/en/universal-declaration-human-rights/index.html. [Accessed: 17-Mar-2021].

[25] “Health Insurance Portability and Accountability Act of 1996 (HIPAA),” Centers for Disease Control and Prevention, 14-Sep-2018. [Online]. Available: https://www.cdc.gov/phlp/publications/topic/hipaa.html. [Accessed: 17-Mar-2021].

[26] “CONFIDENTIAL INFORMATION PROTECTION AND STATISTICAL EFFICIENCY,” U.S. Bureau of Labor Statistics, 2002. [Online]. Available: https://www.bls.gov/bls/cipsea.pdf. [Accessed: 17-Mar-2021].

[27] “General Data Protection Regulation (GDPR) Compliance Guidelines,” GDPR.eu, 2016. [Online]. Available: https://gdpr.eu/. [Accessed: 17-Mar-2021].

[28] S. J. Nass, L. A. Levit, and L. O. Gostin, Beyond the HIPAA privacy rule: enhancing privacy, improving health through research. Washington, D.C.: National Academies Press, 2009.

[29] “COVID-19 Coronavirus Data & Resources,” Wolfram . [Online]. Available: https://www.wolfram.com/covid-19-resources/. [Accessed: 17-Mar-2021].

[30] Spectrum – Information Technologies and Telecommunications, “Jamming Devices are Prohibited in Canada:That’s The Law,” Spectrum management and telecommunications, 12-Sep-2011. [Online]. Available: https://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf10048.html. [Accessed: 17-Mar-2021].