• Clustering outcome across all test subjects for the Reach/Saccade condition

Research

Unobtrusive Neurotechnology and Immersive Human-Computer Interface for Enhanced Learning


Gert Cauwenberghs gert@ucsd.edu (Principal Investigator), Tzyy-Ping Jung (Co-Principal Investigator), Scott Makeig (Co-Principal Investigator), Yu Chi (Co-Principal Investigator), Tim Mullen (Co-Principal Investigator) - August 7, 2017

The increasing prevalence of learning disorders, attention deficits, and lackluster appetite for reading across all walks of life, and particularly among school-age children, poses severe problems to humanity and, in the long run, burdens social and economic development. This Partnership for Innovation Building Innovation Capacity (PFI:BIC) collaborative project tackles the impending threats to humanity of illiteracy and faltering education heads-on by creating a new smart-service human-computer interface (HCI) neurotechnology platform as a highly effective, user-friendly, and fun-to-use tool aiding learning and stimulating cognitive development at home and in the classroom. The immersive HCI neurotechnology will allow directly measuring progress at the cognitive level and providing real-time feedback to guide the user in learning to read more effectively. The project is highly Science, Technology, Engineering and Mathematics (STEM) intensive both in its activities and in the targeted benefits of the developed technology, which extends directly to learning science and mathematics by probing cognitive performance of children while they solve puzzles. The development of unobtrusive neurotechnology further addresses a critical need for practical integrated and modular brain-computer interface (BCI) solutions in HCI promoting widespread consumer and clinical use in the marketplace. The partnership provides opportunities for students to gain practical experience in innovation in the marketplace through internships with the industrial partners.

The central aim is to develop and leverage new HCI technology as a learning coach and personal cognitive development assistant that facilitates learning to read and acquiring other critical skills in cognitive development. The immersive yet unobtrusive HCI technology testbed will comprise a dry-electrode electroencephalography (EEG) BCI, a tablet with touchscreen and integrated camera, and a suite of signal processing algorithms running in the cloud, for monitoring brain and gaze activity in children learning to read, and providing real-time neurofeedback on progress in cognitive performance to promote enhanced learning. The partnership will transition scientific advances of a previous NSF-sponsored UCSD project (NSF EFRI-M3C, ENG-1137279) in studying the distributed dynamics of human motor control, to development of neurofeedback training paradigms for learning enhancement, and to practical deployment on the unobtrusive immersive testbed implemented using Cognionics dry-electrode EEG wireless BCI neurotechnology and Syntrogi real-time cloud-based signal processing software pipelines. The potential for human empowerment by the technology will be demonstrated by evaluating effectiveness in enhancing learning capabilities and cognitive performance in simulated classroom settings and other targeted learning environments.

The lead institution for the project is University of California San Diego (UCSD), with investigators from the Institute for Neural Computation and Department of Bioengineering. The industrial partners in the effort are Syntrogi Inc. (dba Qusp, small business, San Diego CA) and Cognionics, Inc. (small business, San Diego, CA). The project also engages broader context partners Drs. Andrea Chiba and Leanne Chukoskie from the UCSD Temporal Dynamics of Learning Center, Dr. Barbara Moss from San Diego State University Department of Psychology, and Dr. Zewelanji N. Serpell from Virginia Commonwealth University Department of Psychology, in the human case studies and the assessment of the developed HCI technology in diverse learning environments.


PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
https://www.nsf.gov/awardsearch/images/common/bluefade.jpg
 Courellis, Hristos, Mullen, Tim, Poizner, Howard, Cauwenberghs, Gert and Iversen, John R.. "EEG-Based Quantification of Cortical Current Density and Dynamic Causal Connectivity Generalized across Subjects Performing BCI-Monitored Cognitive Tasks," Frontiers in Neuroscience, v.11, 2017. doi:10.3389/fnins.2017.00180  

Chan, Wen-Hsuan and Chiang, Kuan-Jung and Nakanishi, Masaki and Wang, Yu-Te and Jung, Tzyy-Ping. "Evaluating the Performance of Non-Hair SSVEP-Based BCIs Featuring Template-Based Decoding Methods," 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018. doi:10.1109/EMBC.2018.8512662  

Hsu, Sheng-Hsiou and Pion-Tonachini, Luca and Palmer, Jason and Miyakoshi, Makoto and Makeig, Scott and Jung, Tzyy-Ping. "Modeling brain dynamic state changes with adaptive mixture independent component analysis," NeuroImage, v.183, 2018. doi:10.1016/j.neuroimage.2018.08.001  

Paul, A. and Deiss, S.R. and Tourtelotte, D. and Kleffner, M. and Zhang, T. and Cauwenberghs, G.. "Electrode-Skin Impedance Characterization of In-Ear Electrophysiology Accounting for Cerumen and Electrodermal Response," International IEEE/EMBS Conference on Neural Engineering, 2019. 

Ko, Li-Wei and Komarov, Oleksii and Hairston, W. David and Jung, Tzyy-Ping and Lin, Chin-Teng. "Sustained Attention in Real Classroom Settings: An EEG Study," Frontiers in Human Neuroscience, v.11, 2017. doi:10.3389/fnhum.2017.00388  

Hsu, S.-H. and Nakanishi, M. and Chang, C.-Y. and Cauwenberghs, G. and Jung, T.-P.. "Modeling EEG Dynamics of Self-Imagery Emotions: a Pilot Study," International IEEE/EMBS Conference on Neural Engineering, 2019. 

Pion-Tonachini, Luca and Hsu, Sheng-Hsiou and Chang, Chi-Yuan and Jung, Tzyy-Ping and Makeig, Scott. "Online Automatic Artifact Rejection using the Real-time EEG Source-mapping Toolbox (REST)," 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018. doi:10.1109/EMBC.2018.8512191  

Siddharth, S. and Patel, A. and Jung, T-P. and Sejnowski, T.. "A Wearable Multi-modal Bio-sensing System Towards Real-world Applications," IEEE transactions on biomedical engineering, v.66, 2019.