A review of hybrid EEG-based multimodal human–computer interfaces using deep learning: applications, advances, and challenges

Hartson HR, Hix D. Human computer-interface development—concepts and systems for its management. ACM Comput Surv. 1989;21(1):5–92. https://doi.org/10.1145/62029.62031.

Article  Google Scholar 

Sharma R, Pavlovic VI, Huang TS. Toward multimodal human–computer interface. Proc IEEE. 1998;86(5):853–69. https://doi.org/10.1109/5.664275.

Article  Google Scholar 

Tan D, Nijholt A. Brain-computer interfaces and human–computer interaction. London: Springer; 2010.

Book  Google Scholar 

Usakli AB, Gurkan S. Design of a novel efficient human–computer interface: an electrooculagram based virtual keyboard. IEEE Trans Instrum Meas. 2010;59(8):2099–108. https://doi.org/10.1109/Tim.2009.2030923.

Article  Google Scholar 

Kim KH, Kim HK, Kim JS, Son W, Lee SY. A biosignal-based human interface controlling a power-wheelchair for people with motor disabilities. ETRI J. 2006;28(1):111–4. https://doi.org/10.4218/etrij.06.0205.0069.

Article  Google Scholar 

Teplan M. Fundamentals of EEG measurement. Meas Sci Rev. 2002;2(2):1–11.

Google Scholar 

Shim M, Hwang HJ, Lee SH. Toward practical machine-learning-based diagnosis for drug-naive women with major depressive disorder using EEG channel reduction approach. J Affect Disord. 2023;338:199–206. https://doi.org/10.1016/j.jad.2023.06.007.

Article  Google Scholar 

Shim M, Hwang HJ, Lee SH. Impaired functional cortical networks in the theta frequency band of patients with post-traumatic stress disorder during auditory-cognitive processing. Front Psychiatry. 2022. https://doi.org/10.3389/fpsyt.2022.811766.

Article  Google Scholar 

Shim M, Im CH, Lee SH, Hwang HJ. Enhanced performance by interpretable low-frequency electroencephalogram oscillations in the machine learning-based diagnosis of post-traumatic stress disorder. Front Neuroinform. 2022. https://doi.org/10.3389/fninf.2022.811756.

Article  Google Scholar 

Shim M, Kim D, Lee SH, Hwang HJ. Editorial: AI-based computer-aided diagnosis and prognosis for psychiatric disorders. Front Hum Neurosci. 2022. https://doi.org/10.3389/fnhum.2022.901525.

Article  Google Scholar 

Kim M, Kim M-K, Hwang M, Kim H-Y, Cho J, Kim S-P. Online home appliance control using EEG-based brain–computer interfaces. Electronics. 2019;8(10):1101. https://doi.org/10.3390/electronics8101101.

Article  Google Scholar 

Park S, Cha HS, Im CH. Development of an online home appliance control system using augmented reality and an SSVEP-based brain–computer interface. IEEE Access. 2019;7:163604–14. https://doi.org/10.1109/Access.2019.2952613.

Article  Google Scholar 

Wang HT, Dragomir A, Abbasi NI, Li JH, Thakor NV, Bezerianos A. A novel real-time driving fatigue detection system based on wireless dry EEG. Cogn Neurodyn. 2018;12(4):365–76. https://doi.org/10.1007/s11571-018-9481-5.

Article  Google Scholar 

Memar P, Faradji F. A novel multi-class EEG-based sleep stage classification system. IEEE Trans Neural Syst Rehabil Eng. 2018;26(1):84–95. https://doi.org/10.1109/Tnsre.2017.2776149.

Article  Google Scholar 

Supratak A, Dong H, Wu C, Guo Y. Deepsleepnet: a model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Trans Neural Syst Rehabil Eng. 2017;25(11):1998–2008. https://doi.org/10.1109/TNSRE.2017.2721116.

Article  Google Scholar 

Vidyaratne LS, Iftekharuddin KM. Real-time epileptic seizure detection using EEG. IEEE Trans Neural Syst Rehabil Eng. 2017;25(11):2146–56. https://doi.org/10.1109/Tnsre.2017.2697920.

Article  Google Scholar 

Greene BR, Faul S, Marnane WP, Lightbody G, Korotchikova I, Boylan GB. A comparison of quantitative EEG features for neonatal seizure detection. Clin Neurophysiol. 2008;119(6):1248–61. https://doi.org/10.1016/j.clinph.2008.02.001.

Article  Google Scholar 

Rashid M, et al. Current status, challenges, and possible solutions of EEG-based brain-computer interface: a comprehensive review. Front Neurorobot. 2020. https://doi.org/10.3389/fnbot.2020.00025.

Article  Google Scholar 

Mumtaz W, Rasheed S, Irfan A. Review of challenges associated with the EEG artifact removal methods. Biomed Signal Process. 2021. https://doi.org/10.1016/j.bspc.2021.102741.

Article  Google Scholar 

Jiang J, Zhou ZT, Yin EW, Yu Y, Hu DW. Hybrid brain-computer interface (BCI) based on the EEG and EOG signals. Bio-Med Mater Eng. 2014;24(6):2919–25. https://doi.org/10.3233/Bme-141111.

Article  Google Scholar 

Han CH, Muller KR, Hwang HJ. Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynchronous brain-computer interface. IEEE Trans Neural Syst Rehabil Eng. 2020;28(10):2102–12. https://doi.org/10.1109/Tnsre.2020.3017167.

Article  Google Scholar 

Wu CY, Wang Y, Qiu S, He HG. A bimodal deep learning network based on CNN for fine motor imagery. Cogn Neurodyn. 2024;18(6):3791–804. https://doi.org/10.1007/s11571-024-10159-0.

Article  Google Scholar 

Ji XP, Li Y, Wen P. 3dsleepnet: a multi-channel bio-signal based sleep stages classification method using deep learning. IEEE Trans Neural Syst Rehabil Eng. 2023;31:3513–23. https://doi.org/10.1109/Tnsre.2023.3309542.

Article  Google Scholar 

Lotte F, et al. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J Neural Eng. 2018. https://doi.org/10.1088/1741-2552/aab2f2.

Article  Google Scholar 

Baillet S. Magnetoencephalography for brain electrophysiology and imaging. Nat Neurosci. 2017;20(3):327–39. https://doi.org/10.1038/nn.4504.

Article  Google Scholar 

Cohen D. Magnetoencephalography: detection of the brain’s electrical activity with a superconducting magnetometer. Science. 1972;175(4022):664–6. https://doi.org/10.1126/science.175.4022.664.

Article  Google Scholar 

Singh SP. Magnetoencephalography: basic principles. Ann Indian Acad Neurol. 2014;17:S107–12. https://doi.org/10.4103/0972-2327.128676.

Article  Google Scholar 

Scholkmann F, et al. A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Neuroimage. 2014. https://doi.org/10.1016/j.neuroimage.2013.05.004.

Article  Google Scholar 

Naseer N, Hong KS. fNIRS-based brain-computer interfaces: a review. Front Hum Neurosci. 2015. https://doi.org/10.3389/fnhum.2015.00003.

Article  Google Scholar 

Pinti P, et al. The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Ann NY Acad Sci. 2020;1464(1):5–29. https://doi.org/10.1111/nyas.13948.

Article  Google Scholar 

Hillman EMC. Coupling mechanism and significance of the bold signal: a status report. Annu Rev Neurosci. 2014;37:161–81. https://doi.org/10.1146/annurev-neuro-071013-014111.

Article  Google Scholar 

Griffanti L, et al. ICA-based artefact removal and accelerated fMRI acquisition for improved resting state network imaging. Neuroimage. 2014;95:232–47. https://doi.org/10.1016/j.neuroimage.2014.03.034.

Article  Google Scholar 

Logothetis NK. What we can do and what we cannot do with fMRI. Nature. 2008;453(7197):869–78. https://doi.org/10.1038/nature06976.

Article  Google Scholar 

Choudhury SR, Venkataramanan S, Nemade HB, Sahambi J. Design and development of a novel EOG biopotential amplifier. Int J Bioelectromagn. 2005;7(1):271–4.

Google Scholar 

Tao L, Huang H, Chen C, Feijs L, Hu J, Chen W. Review of electrooculography-based human–computer interaction: recent technologies, challenges and future trends. Connect Health Telemed. 2023. https://doi.org/10.20517/chatmed.2023.05.

Article  Google Scholar 

Kumar D, Poole E. Classification of EOG for human computer interface. In: Proceedings of the Second Joint 24th annual conference and the annual fall meeting of the biomedical Engineering Society] [Engineering in Medicine and Biology, Houston, TX, USA; 2002. p. 64–7. https://doi.org/10.1109/IEMBS.2002.1134390

Ahsan MR, Ibrahimy MI, Khalifa OO. Emg signal classification for human computer interaction: a review. Eur J Sci Res. 2009;33(3):480–501.

Google Scholar 

Liu L, Chen X, Lu ZY, Cao S, Wu D, Zhang X. Development of an EMG-ACC-based upper limb rehabilitation training system. IEEE Trans Neural Syst Rehabil Eng. 2017;25(3):244–53. https://doi.org/10.1109/Tnsre.2016.2560906.

Article  Google Scholar 

Cesqui B, Tropea P, Micera S, Krebs HI. EMG-based pattern recognition approach in post stroke robot-aided rehabilitation: a feasibility study. J Neuroeng Rehabil. 2013. https://doi.org/10.1186/1743-0003-10-75.

Article  Google Scholar 

Fang CM, He BW, Wang YX, Cao J, Gao S. EMG-centered multisensory based technologies for pattern recognition in rehabilitation: state of the art and challenges. Biosens-Basel. 2020;10(8):85. https://doi.org/10.3390/bios10080085.

Article  Google Scholar 

Taborri J, et al. Sport biomechanics applications using inertial, force, and emg sensors: a literature overview. Appl Bionics Biomech. 2020;2020:1–18. https://doi.org/10.1155/2020/2041549.

Article  Google Scholar 

Farina D, Merletti R, Enoka RM. The extraction of neural strategies from the surface emg. J Appl Physiol. 2004;96(4):1486–95. https://doi.org/10.1152/japplphysiol.01070.2003.

Comments (0)

No login
gif