In this study, we investigated the partial face mechanism using EEG to examine the evoked ERP component changes across the face cognition condition and eye-tracking to examine which area of the face the participants focused on. We found two ERP components (N170 and P300; P3a and P3b) evoked as brain responses in face cognition. The amplitudes and latencies of N170 showed no significant difference among all face conditions, whereas the amplitude of P3a and P3b components became smaller, and the latency of P3a and P3b components were extended when compared to the amplitude evoked by full face cognition. In addition, we found that the amplitude of P3a in the condition when the participants recognize faces with the nose cover became larger than the full face cognition. For eye-tracking data, we found that most participants fixed their eyes on the eye area whenever the eyes were presented. We also observed similar participants during the full face, face with nose covered, face with mouth covered, and face with nose and mouth covered cognition tasks. Using the two data, we calculated the canonical correlation among the face cognition conditions and found that the full face cognition, the partial face with the mouth covered, and the partial face with the nose and mouth covered cognition were highly correlated.

From our experiment, we have found two components of ERPs, N170 and P300. The N170 and P3a peaks were found in the frontal area of the brain, whereas P3b was found in the parietal area of the brain. N170 is one of the face characteristic ERPs that can be induced, especially during the cognition task when the target face was shown among the other objects (Joyce and Rossion 2005a; Eimer 2011; Cai et al. 2013). Some studies found significant differences in either latency or amplitude or both of them during partial face cognition (Nemrodov et al. 2014; Rousselet et al. 2014; Ince et al. 2016; Żochowska et al. 2022) whereas we did not find any significant differences in either of the results.

The explanation for the difference in N170 results might be that more factors than face components affect the cognition process, such as presentation time and how the participants were asked to respond during the task. Moreover, the reference electrode we used in our study was different from the previous studies. Eimer (2000) experimented with the reference of the EEG at the tip of the nose. Thierry et al. (2007) has the reference at Cz, whereas Joyce and Rossion (2005b) investigated how N170 and VPP were affected with different references. They found that the reference at the mastoid yielded the smallest amplitude of N170, the nose tip reference yielded the largest N170, and the common average reference yielded the second largest N170. In addition, Leuchs (2019) recommended that the reference at the mastoid be preferable for P300 amplitude. In our previous study (Chanpornpakdi and Tanaka 2023a), we investigated the different ERP components evoked during face cognition tasks; hence, we used the mastoid reference to maximize the number of observed ERP components. As a result, we could not observe significant changes in N170 in the temporal area, which contains P5 and P6 (labeled the temporo-occipital region in Eimer (2011)) when performing the full and partial face cognition tasks. Therefore, we did not consider the temporo-occipital region in this study.

The reason why we found the subcomponents of P300 peak larger at different parts of the brain is that P3a is known for the focal attention, which functional area is located in the frontal lobe, whereas P3b is responsible for the target cognition (Bressler and Ding 2006; Polich and Criado 2006; Polich 2007). In addition, the left inferior frontal cortex is the brain area that processes language as well as memory retrieval (Lundstrom et al. 2005). The P3a evoked might also relate to memory retrieval as the participants tried to recall the face they remembered and tried to retrieve the moment when they were remembering the face. This activity may activate the precuneus in the frontal cortex, which is sensitive to the quantity and quality of the information retrieved (Rugg et al. 1998; Nyberg et al. 2000; Lundstrom et al. 2003, 2005).

The frontal cortex also functions in the visual working memory (Courtney et al. 1998; Chai et al. 2018). Vogel and Machizawa (2004) studied the capacity of the human visual working memory capacity and found that as the number of items increased, the ERP response also became larger, but at some point, it reached a limit. They interpreted that individuals have different memory capacities and found that each has a capacity with an average of about 2.8 items. When we look at the face features, the face contains four components: two eyes (consider the eyebrow as a part of the eye component), one nose, and one mouth. Covering the nose could reduce the number of components into three components almost equal to the capacity that Vogel and Machizawa (2004) have found. This could explain why recognizing the face with the nose covered results in a larger P3a component, not P3b. When we covered both the nose and mouth, we had fewer components, but we could still see a similar level of P3a and P3b peaks with the condition of the partial face with the nose covered. This might be because the eye component is the most important part of face cognition, as we can see when we covered the eye component and asked the participant to recognize the face. Both P3a and P3b dropped significantly. One more reason for this could be neuroplasticity. As we have been living with the mask on for many years, people could learn and get used to recognizing faces with the nose and mouth covered as time passed. This makes us recognize faces with less difficulty and results in a similar level of P3a and P3b even though we have only a few components (only two but important components).

To investigate whether the eye component is the most informative feature in face cognition, we recorded the eye-tracking data along with the EEG. We assumed that the total duration of fixation in the eye AOI would be the longest period among all components. We also expected a similar total fixation duration of AOIs across the full face cognition, partial-face cognition with the nose covered, and partial-face cognition with the nose and mouth covered. These expectation was based on the similar level of accuracy in the target classification results based on ERP in our previous study (Chanpornpakdi and Tanaka 2023c). The results confirmed our assumptions that the participant predominantly focused and fixed their gaze on the eye area when the eyes were presented. These results coincided with our ERP results, which showed that the amplitude decreased when the eye component was covered. It implies that the participants lost some ability to recognize faces and confirmed that people use eye information the most, concluding that eyes play an important role in face cognition (Nguyen and Pezdek 2017; Royer et al. 2018). We also found that some participants still focused on the nose and the fixation position, although the nose was invisible. From these results and the previous studies, we can interpret that people can still perceive the face as a whole and recognize faces using holistic face processing even though some parts of the face are missing. The reason could be because we still presented the eyes and have the visible components in the correct position as people can still recognize faces using holistic processing when the facial features are in the correct position rather than selecting parts of them and identifying the face (Caharel et al. 2006; Calder 2011; de Haas et al. 2016; Hayward et al. 2016; Poltoratski et al. 2021).

Contradictory to the majority, we saw one participant looking at the mouth rather than the eyes when the face with the nose covered was used as a stimulus. There could be two possibilities as to why this person focused on the mouth. This person could have a larger visual field coverage of face-selective regions so that it included the eye area even though the person’s fixation was on the mouth (Poltoratski et al. 2021). Another possibility is that we have selected the target image with the nose and mouth shape that stood out from the rest of the images. This could make the target look different from the Asian face and cause the person to identify the ethnic race rather than the facial identity as people tend to focus on the eyes when identifying a face identity, whereas looking at the mouth when identifying the ethnic (Chakravarthula et al. 2021). From our results, we could interpret that partial face cognition might rely on holistic face processing if we have the important features of the face in the right position and structural face processing depending on the individual. This could give the potential to use partial face cognition to enhance the face cognition task.

From our canonical correlation analysis results, we observed that the full face cognition, the face with the mouth covered, and the face with nose and mouth covered have strong correlations. The reason could be that the face with the nose and mouth covered was the partial face condition that resembles the surgical mask the most. Moreover, The face with the mouth covered is similar to nose commando (Wolff 2020), the condition when people wear a half mask with the nose exposed. This indicates that people can recognize the partial face with mouth covered and the partial face with nose and mouth covered comparably with the full face cognition based on the EEG and eye-tracking data. We can interpret that the cognition mechanism of the masked face and the full face are similar; hence, people can recognize faces correctly even though some features of the face are covered under the mask. Furthermore, our results revealed a strong correlation between the cognition of faces with the mouth covered and faces with both the nose and mouth covered. This finding suggests that the covering of the nose in addition to the mouth does not significantly alter the cognitive processes involved in face cognition as the most crucial information for face cognition was mainly the eyes.

However, our findings revealed the high correlation between the face with only eyes covered and face with only mouth covered. While both eyes and mouth are important features in face cognition, this unexpected similarity raises intriguing questions about the relative importance of these features and the mechanisms underlying partial face recognition. Further research such as the eye gaze order analysis is needed to clarify the factors contributing to this phenomenon.

In our experiment, we used color images and did not remove the remarkable facial features such as moles, this could distract the participants and attract more attention than the major face components we aimed to investigate. In addition, the presentation time was 200 msec, which is considered shorter than the suitable time presentation for face cognition that Lees et al. (2018) suggested. Another limitation of our experiment was that we had a fixed duration for the interstimulus interval (ISI), which could lead the participant to anticipatory effects. To prevent such an effect, we should jitter the ISI randomly to reduce the cue of the stimulus presentation timming and obtain the brain response purely based on the cognitive function.

We also presented the stimuli in the order of face, face with the eyes covered, nose covered, mouth covered, eyes and nose covered, eyes and mouth covered, nose and mouth covered. The reason was that all the faces were unknown to the participants, especially the target face, which might make the task more difficult to recognize the partial face although we have performed the training task. Therefore, we performed the full face at first, followed by the partial face cognition tasks in a systematic way, covering parts from top to bottom. This order could allow the participants to have the same initial exposure to the face, which could be beneficial for standardizing the learning process. However, this fixed order could lead to bias in the results such as the participants’ cognition performance could be influenced by the carry-over information learned in the earlier condition. To reduce such an effect, we asked the participant to rest before proceeding to the next condition. We should consider randomizing the order of the partial face shown after the full face across the participants to minimize the carry-over effect.

Although we have reduced the task to lighten the work burden during the experiment, we have increased the number of trials from 20 trials per image to 100 trials per image to obtain clean EEG data, the workload was still considered to be heavy and could cause the fatigue and reduce the ERP responses. To solve such a problem, we should divide each face cognition condition block into smaller sessions so the participants can work on face cognition on a small batch of face images and get some time to rest between sessions. We also found that the latencies of evoked ERPs were slower than those we found from the experiment in our previous study (Chanpornpakdi and Tanaka 2023c). The reason could be that we created our self-made trigger using photodiodes, resistors, and Arduino to synchronize the EEG and eye-tracking data.

To synchronize the data, we attached the photodiodes to the screen to detect the light changes on the screen, used Arduino software to control Arduino and send the trigger signal to OpenViBE software when there were changes on the screen, then we programmed OpenViBE software to send the external triggers signal to the EEG amplifier using python. All these processes could cause time in accuracy in the trigger due to the frame refresh rate and data transfer, such as delay settings in the controller and external trigger programs. To solve the problem, we need to identify the exact delay by measuring the delay caused by both the Arduino control program and the OpenViBE communication program. We should also consider using the trigger synchronization system sold on the market so that we can minimize time inaccuracy and improve our data quality.

In the future, we plan to analyze our eye-tracking data in more detail to investigate the correlation between the eye gaze area and EEG response and implement a hidden Markov Model (Rabiner and Juang 1986; Rabiner 1989; Chuk et al. 2014) to identify how the eye gaze develops during the experiment and the interaction between the AOIs. Furthermore, other aspects of the EEG, such as energy distribution, frequency domain, and brain network, should be considered in the future. These aspects could also yield meaningful cognitive neurodynamics of the brain during face cognition tasks. In this study, we investigated partial face cognition in a situation where the person learns full face before the task. However, during the pandemic, some people first learned the masked face of a person, and sometimes could not recognize the same person when he removed the mask. This cognitive mechanism also needs to be clarified and compared to the experiment carried out in this study. The least number of face components required in partial face cognition and other partial face cognition such as having only one eye or missing half of the face should also be examined in the future. For future applications of EEG-based RSVP, we intend to apply these results to create a lie detector using a partial face image. We believe that partial face image stimulus could take the revealing concealed face information task to the next level due to the absence of some face features. This stimulus can make the cognition task of the concealed target face difficult to deceive. Another application could be a marketing analyzer using EEGs to investigate customer preference in selecting products.