Home Research
People
Publications
Courses Contact

Publications

Use the sub-menu on the left to filter the articles. Click on an item to view the full document.

High-performance neural population dynamics modeling enabled by scalable computational infrastructure

Journal of Open Source Software, 2023
Patel AN, Sedler AR, Huang J, Pandarinath C, Gilja V
Advances in neural interface technology are facilitating parallel, high-dimensional time series measurements of the brain in action. A powerful strategy for analyzing these measurements is to apply unsupervised learning techniques to uncover lower-dimensional latent dynamics that explain much of the variance in the high-dimensional measurements (Cunningham & Yu, 2014; Golub et al., 2018; Vyas et al., 2020). Latent factor analysis via dynamical systems (LFADS) (Pandarinath et al., 2018) provides a deep learning approach for extracting estimates of these latent dynamics from neural population data. The recently developed AutoLFADS framework (Keshtkaran et al., 2022) extends LFADS by using Population Based Training (PBT) (Jaderberg et al., 2017) to effectively and scalably tune model hyperparameters, a critical step for accurate modeling of neural population data. As hyperparameter sweeps are one of the most computationally demanding processes in model development, these workflows should be deployed in a computationally efficient and cost effective manner given the compute resources available (e.g., local, institutionally-supported, or commercial computing clusters). The initial implementation of AutoLFADS used the Ray library (Moritz et al., 2018) to enable support for specific local and commercial cloud workflows. We extend this support, by providing additional options for training AutoLFADS models using local clusters in a container-native approach (e.g., Docker, Podman), unmanaged compute clusters leveraging Ray, and managed compute clusters leveraging KubeFlow and Kubernetes orchestration. As the neurosciences increasingly employ deep learning based models that require compute intensive hyperparameter optimization (Keshtkaran & Pandarinath, 2019; Willett et al., 2021; Yu et al., 2021), standardization and dissemination of computational methods becomes increasingly challenging. Although this work specifically provides implementations of AutoLFADS, the tooling provided demonstrates strategies for employing computation at scale while facilitating dissemination and reproducibility.

Spatiotemporal dynamics of human high gamma discriminate naturalistic behavioral states

PLoS Computational Biology, 2022
Alasfour A, Gabriel P, Jiang X, Shamie I, Melloni L, Thesen T, Dugan P, Friedman D, Doyle W, Devinsky O, Gonda D, Sattar S, Wang S, Halgren E, Gilja V
In analyzing the neural correlates of naturalistic and unstructured behaviors, features of neural activity that are ignored in a trial-based experimental paradigm can be more fully studied and investigated. Here, we analyze neural activity from two patients using electrocorticography (ECoG) and stereo-electroencephalography (sEEG) recordings, and reveal that multiple neural signal characteristics exist that discriminate between unstructured and naturalistic behavioral states such as “engaging in dialogue” and “using electronics”. Using the high gamma amplitude as an estimate of neuronal firing rate, we demonstrate that behavioral states in a naturalistic setting are discriminable based on long-term mean shifts, variance shifts, and differences in the specific neural activity’s covariance structure. Both the rapid and slow changes in high gamma band activity separate unstructured behavioral states. We also use Gaussian process factor analysis (GPFA) to show the existence of salient spatiotemporal features with variable smoothness in time. Further, we demonstrate that both temporally smooth and stochastic spatiotemporal activity can be used to differentiate unstructured behavioral states. This is the first attempt to elucidate how different neural signal features contain information about behavioral states collected outside the conventional experimental paradigm.

High γ Activity in Cortex and Hippocampus Is Correlated with Autonomic Tone during Sleep

eNeuro, 2021
Alasfour A, Jiang X, Gonzalez-Martinez J, Gilja V, Halgren E
Studies in animals have demonstrated a strong relationship between cortical and hippocampal activity, and autonomic tone. However, the extent, distribution, and nature of this relationship have not been investigated with intracranial recordings in humans during sleep. Cortical and hippocampal population neuronal firing was estimated from high γ band activity (HG) from 70 to 110 Hz in local field potentials (LFPs) recorded from 15 subjects (nine females) during nonrapid eye movement (NREM) sleep. Autonomic tone was estimated from heart rate variability (HRV). HG and HRV were significantly correlated in the hippocampus and multiple cortical sites in NREM stages N1–N3. The average correlation between HG and HRV could be positive or negative across patients given anatomic location and sleep stage and was most profound in lateral temporal lobe in N3, suggestive of greater cortical activity associated with sympathetic tone. Patient-wide correlation was related to δ band activity (1–4 Hz), which is known to be correlated with high γ activity during sleep. The percentage of statistically correlated channels was weaker in N1 and N2 as compared with N3, and was strongest in regions that have previously been associated with autonomic processes, such as anterior hippocampus and insula. The anatomic distribution of HRV-HG correlations during sleep did not reproduce those usually observed with positron emission tomography (PET) or functional magnetic resonance imaging (fMRI) during waking. This study aims to characterize the relationship between autonomic tone and neuronal firing rate during sleep and further studies are needed to investigate finer temporal resolutions, denser coverages, and different frequency bands in both waking and sleep.

Local field potentials in a pre-motor region predict learned vocal sequences

PLoS Computational Biology, 2021
Brown D, Chavez J, Nguyen D, Kadwory A, Voytek B, Arneodo E, Gentner T, Gilja V
Neuronal activity within the premotor region HVC is tightly synchronized to, and crucial for, the articulate production of learned song in birds. Characterizations of this neural activity detail patterns of sequential bursting in small, carefully identified subsets of neurons in the HVC population. The dynamics of HVC are well described by these characterizations, but have not been verified beyond this scale of measurement. There is a rich history of using local field potentials (LFP) to extract information about behavior that extends beyond the contribution of individual cells. These signals have the advantage of being stable over longer periods of time, and they have been used to study and decode human speech and other complex motor behaviors. Here we characterize LFP signals presumptively from the HVC of freely behaving male zebra finches during song production to determine if population activity may yield similar insights into the mechanisms underlying complex motor-vocal behavior. Following an initial observation that structured changes in the LFP were distinct to all vocalizations during song, we show that it is possible to extract time-varying features from multiple frequency bands to decode the identity of specific vocalization elements (syllables) and to predict their temporal onsets within the motif. This demonstrates the utility of LFP for studying vocal behavior in songbirds. Surprisingly, the time frequency structure of HVC LFP is qualitatively similar to well-established oscillations found in both human and non-human mammalian motor areas. This physiological similarity, despite distinct anatomical structures, may give insight into common computational principles for learning and/or generating complex motor-vocal behaviors.

Microscale physiological events on the human cortical surface

Cerebral Cortex, 2021
Paulk A, Yang J, Cleary D, Soper D, Halgren M, O’Donnell A, Lee S, Ganji M, Ro Y, Oh H, Hossain L, Lee J, Tchoe Y, Rogers N, Kiliç K, Ryu S, Lee S, Hermiz J, Gilja V, Ulbert I, Fabo D, Thesen T, Doyle W, Devinsky O, Madsen J, Schomer D, Eskander E, Lee J, Maus D, Devor A, Fried S, Jones P, Nahed B, Ben-Haim S, Bick S, Richardson R, Raslan A, Siler D, Cahill D, Williams Z, Cosgrove G, Dayeh S, Cash S
Despite ongoing advances in our understanding of local single-cellular and network-level activity of neuronal populations in the human brain, extraordinarily little is known about their “intermediate” microscale local circuit dynamics. Here, we utilized ultra-high-density microelectrode arrays and a rare opportunity to perform intracranial recordings across multiple cortical areas in human participants to discover three distinct classes of cortical activity that are not locked to ongoing natural brain rhythmic activity. The first included fast waveforms similar to extracellular single-unit activity. The other two types were discrete events with slower waveform dynamics and were found preferentially in upper cortical layers. These second and third types were also observed in rodents, nonhuman primates, and semi-chronic recordings from humans via laminar and Utah array microelectrodes. The rates of all three events were selectively modulated by auditory and electrical stimuli, pharmacological manipulation, and cold saline application and had small causal co-occurrences. These results suggest that the proper combination of high-resolution microelectrodes and analytic techniques can capture neuronal dynamics that lay between somatic action potentials and aggregate population activity. Understanding intermediate microscale dynamics in relation to single-cell and network dynamics may reveal important details about activity in the full cortical circuit.

The Argo: A high channel count recording system for neural recording in vivo

Journal of Neural Engineering, 2021
Sahasrabuddhe K, Khan A, Singh A, Stern T, Ng Y, Tadić A, Orel P, LaReau C, Pouzzner D, Nishimura K, Boergens K, Shivakumar S, Hopper M, Kerr B, Hanna M, Edgington R, McNamara I, Fell D, Gao P, Babaie-Fishani A, Veijalainen S, Klekachev A, Stuckey A, Luyssaert B, Kozai T, Xie C, Gilja V, Dierickx B, Kong Y, Straka M, Sohal H, Angle M
Objective. Decoding neural activity has been limited by the lack of tools available to record from large numbers of neurons across multiple cortical regions simultaneously with high temporal fidelity. To this end, we developed the Argo system to record cortical neural activity at high data rates. Approach. Here we demonstrate a massively parallel neural recording system based on platinum-iridium microwire electrode arrays bonded to a CMOS voltage amplifier array. The Argo system is the highest channel count in vivo neural recording system, supporting simultaneous recording from 65,536 channels, sampled at 32 kHz and 12-bit resolution. This system was designed for cortical recordings, compatible with both penetrating and surface microelectrodes. Main results. We validated this system through initial bench testing to determine specific gain and noise characteristics of bonded microwires, followed by in-vivo experiments in both rat and sheep cortex. We recorded spiking activity from 791 neurons in rats and surface LFP activity from over 30,000 channels in sheep. Significance. These are the largest channel count microwire-based recordings in both rat and sheep. While currently adapted for head-fixed recording, the microwire-CMOS architecture is well suited for clinical translation. Thus, this demonstration helps pave the way for a future high data rate intracortical implant.

Impact of brain surface boundary conditions on electrophysiology and implications for electrocorticography

Frontiers in Neuroscience, 2020
Rogers N, Thuneman M, Devor A, Gilja V
Volume conduction of electrical potentials in the brain is highly influenced by the material properties and geometry of the tissue and recording devices implanted into the tissue. These effects are very large in EEG due to the volume conduction through the skull and scalp but are often neglected in intracranial electrophysiology. When considering penetrating electrodes deep in the brain, the assumption of an infinite and homogenous medium can be used when the sources are far enough from the brain surface and the electrodes to minimize the boundary effect. When the electrodes are recording from the brain's surface the effect of the boundary cannot be neglected, and the large surface area and commonly used insulating materials in surface electrode arrays may further increase the effect by altering the nature of the boundary in the immediate vicinity of the electrodes. This gives the experimenter some control over the spatial profiles of the potentials by appropriate design of the electrode arrays. We construct a simple three-layer model to describe the effect of material properties and geometry above the brain surface on the electric potentials and conduct empirical experiments to validate this model. A laminar electrode array is used to measure the effect of insulating and relatively conducting layers above the cortical surface by recording evoked potentials alternating between a dried surface and saline covering layer, respectively. Empirically, we find that an insulating boundary amplifies the potentials relative to conductive saline by about a factor of 4, and that the effect is not constrained to potentials that originate near the surface. The model is applied to predict the influence of array design and implantation procedure on the recording amplitude and spatial selectivity of the surface electrode arrays.

Stimulus driven single unit activity from micro-electrocorticography

Frontiers in Neuroscience, 2020
Hermiz J, Hossain L, Arneodo E, Ganji M, Rogers N, Vahidi N, Halgren E, Gentner T, Dayeh S, Gilja V
High-fidelity measurements of neural activity can enable advancements in our understanding of the neural basis of complex behaviors such as speech, audition, and language, and are critical for developing neural prostheses that address impairments to these abilities due to disease or injury. We develop a novel high resolution, thin-film micro-electrocorticography (micro-ECoG) array that enables high-fidelity surface measurements of neural activity from songbirds, a well-established animal model for studying speech behavior. With this device, we provide the first demonstration of sensory-evoked modulation of surface-recorded single unit responses. We establish that single unit activity is consistently sensed from micro-ECoG electrodes over the surface of sensorimotor nucleus HVC (used as a proper name) in anesthetized European starlings, and validate responses with correlated firing in single units recorded simultaneously at surface and depth. The results establish a platform for high-fidelity recording from the surface of subcortical structures that will accelerate neurophysiological studies, and development of novel electrode arrays and neural prostheses.

Neural correlates of unstructured motor behaviors

Journal of Neural Engineering, 2019
Gabriel P, Chen K, Alasfour A, Pailla T, Doyle W, Devinsky O, Friedman D, Dugan P, Melloni L, Thesen T, Gonda D, Sattar S, Wang S, Gilja V
Objective. We studied the relationship between uninstructed, unstructured movements and neural activity in three epilepsy patients with intracranial electroencephalographic (iEEG) recordings. Approach. We used a custom system to continuously record high definition video precisely time-aligned to clinical iEEG data. From these video recordings, movement periods were annotated via semi-automatic tracking based on dense optical flow. Main results. We found that neural signal features (8–32 Hz and 76–100 Hz power) previously identified from task-based experiments are also modulated before and during a variety of movement behaviors. These movement behaviors are coarsely labeled by time period and movement side (e.g. 'Idle' and 'Move', 'Right' and 'Left'); movements within a label can include a wide variety of uninstructed behaviors. A rigorous nested cross-validation framework was used to classify both movement onset and lateralization with statistical significance for all subjects. Significance. We demonstrate an evaluation framework to study neural activity related to natural movements not evoked by a task, annotated over hours of video. This work further establishes the feasibility to study neural correlates of unstructured behavior through continuous recording in the epilepsy monitoring unit. The insights gained from such studies may advance our understanding of how the brain naturally controls movement, which may inform the development of more robust and generalizable brain–computer interfaces.

Selective formation of porous Pt nanorods for highly electrochemically efficient neural electrode interfaces

Nano Letters, 2019
Ganji M, Paulk A, Yang J, Vahidi N, Lee S, Liu R, Hossain L, Arneodo E, Thunemann M, Shigyo M, Tanaka A, Ryu S, Lee S, Tchoe Y, Marsala M, Devor A, Cleary D, Martin J, Oh H, Gilja V, Gentner T, Fried S, Halgren E, Cash S, Dayeh S
The enhanced electrochemical activity of nanostructured materials is readily exploited in energy devices, but their utility in scalable and human-compatible implantable neural interfaces can significantly advance the performance of clinical and research electrodes. We utilize low-temperature selective dealloying to develop scalable and biocompatible one-dimensional platinum nanorod (PtNR) arrays that exhibit superb electrochemical properties at various length scales, stability, and biocompatibility for high performance neurotechnologies. PtNR arrays record brain activity with cellular resolution from the cortical surfaces in birds and nonhuman primates. Significantly, strong modulation of surface recorded single unit activity by auditory stimuli is demonstrated in European Starling birds as well as the modulation of local field potentials in the visual cortex by light stimuli in a nonhuman primate and responses to electrical stimulation in mice. PtNRs record behaviorally and physiologically relevant neuronal dynamics from the surface of the brain with high spatiotemporal resolution, which paves the way for less invasive brain–machine interfaces.

Correlation Structure in Micro-ECoG Recordings is Described by Spatially Coherent Components

PLoS Computational Biology, 2019
Rogers N, Hermiz J, Ganji M, Kaestner E, Kılıç K, Hossain L, Thunemann M, Cleary D, Carter B, Barba D, Devor A, Halgren E, Dayeh S, Gilja V
Electrocorticography (ECoG) is becoming more prevalent due to improvements in fabrication and recording technology as well as its ease of implantation compared to intracortical electrophysiology, larger cortical coverage, and potential advantages for use in long term chronic implantation. Given the flexibility in the design of ECoG grids, which is only increasing, it remains an open question what geometry of the electrodes is optimal for an application. Conductive polymer, PEDOT:PSS, coated microelectrodes have an advantage that they can be made very small without losing low impedance. This makes them suitable for evaluating the required granularity of ECoG recording in humans and experimental animals. We used two-dimensional (2D) micro-ECoG grids to record intra-operatively in humans and during acute implantations in mouse with separation distance between neighboring electrodes (i.e., pitch) of 0.4 mm and 0.2/0.25 mm respectively. To assess the spatial properties of the signals, we used the average correlation between electrodes as a function of the pitch. In agreement with prior studies, we find a strong frequency dependence in the spatial scale of correlation. By applying independent component analysis (ICA), we find that the spatial pattern of correlation is largely due to contributions from multiple spatially extended, time-locked sources present at any given time. Our analysis indicates the presence of spatially structured activity down to the sub-millimeter spatial scale in ECoG despite the effects of volume conduction, justifying the use of dense micro-ECoG grids.

Use of Accelerometry for Long Term Monitoring of Stroke Patients

IEEE Journal of Translational Engineering in Health and Medicine, 2019
Lucas A, Hermiz J, Labuzetta J, Arabadzhi Y, Karanjia N, Gilja V
Stroke patients are monitored hourly by physicians and nurses in an attempt to better understand their physical state. To quantify the patients' level of mobility, hourly movement (i.e. motor) assessment scores are performed, which can be taxing and time-consuming for nurses and physicians. In this paper, we attempt to find a correlation between patient motor scores and continuous accelerometer data recorded in subjects who are unilaterally impaired due to stroke. The accelerometers were placed on both upper and lower extremities of four severely unilaterally impaired patients and their movements were recorded continuously for 7 to 14 days. Features that incorporate movement smoothness, strength, and characteristic movement patterns were extracted from the accelerometers using time-frequency analysis. Support vector classifiers were trained with the extracted features to test the ability of the long term accelerometer recordings in predicting dependent and antigravity sides, and significantly above baseline performance was obtained in most instances (P <; 0.05). Finally, a leave-one-subject-out approach was carried out to assess the generalizability of the proposed methodology, and above baseline performance was obtained in two out of the three tested subjects. The methodology presented in this paper provides a simple, yet effective approach to perform long term motor assessment in neurocritical care patients.

Autoencoders for learning template spectrograms in electrocorticographic signals

Journal of Neural Engineering, 2019
Pailla T, Miller K, Gilja V
Objective. Current brain-computer interface (BCI) studies demonstrate the potential to decode neural signals obtained from structured and trial-based tasks to drive actuators with high performance within the context of these tasks. Ideally, to maximize utility, such systems will be applied to a wide range of behavioral settings or contexts. Thus, we explore the potential to augment such systems with the ability to decode abstract behavioral contextual states from neural activity. Approach. To demonstrate the feasibility of such context decoding, we used electrocorticography (ECoG) and stereo-electroencephalography (sEEG) data recorded from the cortical surface and deeper brain structures, respectively, continuously across multiple days from three subjects. During this time, the subjects were engaged in a range of naturalistic behaviors in a hospital environment. Behavioral contexts were labeled manually from video and audio recordings; four states were considered: engaging in dialogue, rest, using electronics, and watching television. We decode these behaviors using a factor analysis and support vector machine (SVM) approach. Main results. We demonstrate that these general behaviors can be decoded with high accuracies of 73% for a four-class classifier for one subject and 71% and 62% for a three-class classifier for two subjects. Significance. To our knowledge, this is the first demonstration of the potential to disambiguate abstract naturalistic behavioral contexts from neural activity recorded throughout the day from implanted electrodes. This work motivates further study of context decoding for BCI applications using continuously recorded naturalistic activity in the clinical setting.

Coarse behavioral context decoding

Journal of Neural Engineering, 2019
Alasfour A, Gabriel P, Jiang X, Shamie X, Melloni L, Thesen T, Dugan P, Friedman D, Doyle W, Devinsky O, Gonda D, Sattar S, Wang S, Halgren E, Gilja V
Objective. Current brain-computer interface (BCI) studies demonstrate the potential to decode neural signals obtained from structured and trial-based tasks to drive actuators with high performance within the context of these tasks. Ideally, to maximize utility, such systems will be applied to a wide range of behavioral settings or contexts. Thus, we explore the potential to augment such systems with the ability to decode abstract behavioral contextual states from neural activity. Approach. To demonstrate the feasibility of such context decoding, we used electrocorticography (ECoG) and stereo-electroencephalography (sEEG) data recorded from the cortical surface and deeper brain structures, respectively, continuously across multiple days from three subjects. During this time, the subjects were engaged in a range of naturalistic behaviors in a hospital environment. Behavioral contexts were labeled manually from video and audio recordings; four states were considered: engaging in dialogue, rest, using electronics, and watching television. We decode these behaviors using a factor analysis and support vector machine (SVM) approach. Main results. We demonstrate that these general behaviors can be decoded with high accuracies of 73% for a four-class classifier for one subject and 71% and 62% for a three-class classifier for two subjects. Significance. To our knowledge, this is the first demonstration of the potential to disambiguate abstract naturalistic behavioral contexts from neural activity recorded throughout the day from implanted electrodes. This work motivates further study of context decoding for BCI applications using continuously recorded naturalistic activity in the clinical setting.

Monolithic and Scalable Au Nanorod Substrates Improve PEDOT–Metal Adhesion and Stability in Neural Electrodes

Advanced Healthcare Materials, 2018
M Ganji, L Hossain, A Tanaka, M Thunemann, Halgren E, Gilja V, Devor A, Dayeh S
Poly(3,4‐ethylenenedioxythiophene) or PEDOT is a promising candidate for next‐generation neuronal electrode materials but its weak adhesion to underlying metallic conductors impedes its potential. An effective method of mechanically anchoring the PEDOT within an Au nanorod (Au‐nr) structure is reported and it is demonstrated that it provides enhanced adhesion and overall PEDOT layer stability. Cyclic voltammetry (CV) stress is used to investigate adhesion and stability of spin‐cast and electrodeposited PEDOT. The Au‐nr adhesion layer permits 10 000 CV cycles of coated PEDOT film in phosphate buffered saline solution without delamination nor significant change of the electrochemical impedance, whereas PEDOT coating film on planar Au electrodes delaminates at or below 1000 cycles. Under CV stress, spin‐cast PEDOT on planar Au delaminates, whereas electroplated PEDOT on planar Au encounters surface leaching/decomposition. After 5 weeks of accelerated aging tests at 60 °C, the electrodeposited PEDOT/Au‐nr microelectrodes demonstrate a 92% channel survival compared to only 25% survival for spin‐cast PEDOT on planar films. Furthermore, after a 10 week chronic implantation onto mouse barrel cortex, PEDOT/Au‐nr microelectrodes do not exhibit delamination nor morphological changes, whereas the conventional PEDOT microelectrodes either partially or fully delaminate. Immunohistochemical evaluation demonstrates no or minimal response to the PEDOT implant.

Patient-specific pose estimation in clinical environments

IEEE Journal of Translational Engineering in Health and Medicine, 2018
Chen K, Gabriel P, Alasfour A, Gong C, Doyle W, Devinsky O, Friedman D, Dugan P, Melloni L, Thesen T, Gonda D, Sattar S, Wang S, Gilja V
Reliable posture labels in hospital environments can augment research studies on neural correlates to natural behaviors and clinical applications that monitor patient activity. However, many existing pose estimation frameworks are not calibrated for these unpredictable settings. In this paper we propose a semi-automated approach for improving upper-body pose estimation in noisy clinical environments, whereby we adapt and build around an existing joint tracking framework to improve its robustness to environmental uncertainties. The proposed framework uses subject-specific convolutional neural network (CNN) models trained on a subset of a patient’s RGB video recording chosen to maximize the feature variance of each joint. Furthermore, by compensating for scene lighting changes and by refining the predicted joint trajectories through a Kalman filter with fitted noise parameters, the extended system yields more consistent and accurate posture annotations when compared to two state-of-the-art generalized pose tracking algorithms for three hospital patients recorded in two research clinics.

Sub-millimeter ECoG pitch in human enables higher fidelity cognitive neural state estimation

NeuroImage, 2018
Hermiz J, Rogers N, Kaestner E, Ganji M, Cleary DR, Carter B, Barba D, Dayeh S, Halgren E, Gilja V
Electrocorticography (ECoG), electrophysiological recording from the pial surface of the brain, is a critical measurement technique for clinical neurophysiology, basic neurophysiology studies, and demonstrates great promise for the development of neural prosthetic devices for assistive applications and the treatment of neurological disorders. Recent advances in device engineering are poised to enable orders of magnitude increase in the resolution of ECoG without comprised measurement quality. This enhancement in cortical sensing enables the observation of neural dynamics from the cortical surface at the micrometer scale. While these technical capabilities may be enabling, the extent to which finer spatial scale recording enhances functionally relevant neural state inference is unclear. We examine this question by employing a high-density and low impedance 400 μm pitch microECoG (μECoG) grid to record neural activity from the human cortical surface during cognitive tasks. By applying machine learning techniques to classify task conditions from the envelope of high-frequency band (70–170Hz) neural activity collected from two study participants, we demonstrate that higher density grids can lead to more accurate binary task condition classification. When controlling for grid area and selecting task informative sub-regions of the complete grid, we observed a consistent increase in mean classification accuracy with higher grid density; in particular, 400 μm pitch grids outperforming spatially sub-sampled lower density grids up to 23%. We also introduce a modeling framework to provide intuition for how spatial properties of measurements affect the performance gap between high and low density grids. To our knowledge, this work is the first quantitative demonstration of human sub-millimeter pitch cortical surface recording yielding higher-fidelity state estimation relative to devices at the millimeter-scale, motivating the development and testing of μECoG for basic and clinical neurophysiology as well as towards the realization of high-performance neural prostheses.

Development and translation of PEDOT:PSS microelectrodes for intraoperative monitoring

Advanced Functional Materials, 2017
Ganji M, Kaestner E, Hermiz J, Rogers N, Tanaka A, Cleary D, Lee SH, Snider J, Halgren M, Cosgrove GR, Carter BS, Barba D, Uguz I, Malliaras GG, Cash SS, Gilja V, Halgren E, Dayeh SA
Recording neural activity during neurosurgical interventions is an invaluable tool for both improving patient outcomes and advancing our understanding of neural mechanisms and organization. However, increasing clinical electrodes' signal-to-noise and spatial specificity requires overcoming substantial physical barriers due to the compromised metal electrochemical interface properties. The electrochemical properties of poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS) based interfaces surpass those of current clinical electrocorticography electrodes. Here, robust fabrication process of PEDOT:PSS microelectrode arrays is demonstrated for safe and high fidelity intraoperative monitoring of human brain. PEDOT:PSS microelectrodes measure significant differential neural modulation under various clinically relevant conditions. This study reports the first evoked (stimulus-locked) cognitive activity with changes in amplitude across pial surface distances as small as 400 µm, potentially enabling basic neurophysiology studies at the scale of neural micro-circuitry.

Clinical translation of a high-performance neural prosthesis

Nature Medicine, 2015
Gilja V*, Pandarinath C, Blabe CH, Nuyujukian P, Simeral JD, Sarma AA, Sorice BL, Perge JA, Jarosiewicz B, Hochberg LR, Henderson JM, Shenoy KV
Neural prostheses have the potential to improve the quality of life of individuals with paralysis by directly mapping neural activity to limb- and computer-control signals. We translated a neural prosthetic system previously developed in animal model studies for use by two individuals with amyotrophic lateral sclerosis who had intracortical microelectrode arrays placed in motor cortex. Measured more than 1 year after implant, the neural cursor-control system showed the highest published performance achieved by a person to date, more than double that of previous pilot clinical trial participants.

Neural population dynamics in human motor cortex during movements in people with ALS

eLife, 2015
Pandarinath C, Gilja V, Blabe CH, Nuyujukian P, Sarma AA, Sorice BL, Eskandar EN, Hochberg LR, Henderson JM, Shenoy KV
The prevailing view of motor cortex holds that motor cortical neural activity represents muscle or movement parameters. However, recent studies in non-human primates have shown that neural activity does not simply represent muscle or movement parameters; instead, its temporal structure is well-described by a dynamical system where activity during movement evolves lawfully from an initial pre-movement state. In this study, we analyze neuronal ensemble activity in motor cortex in two clinical trial participants diagnosed with Amyotrophic Lateral Sclerosis (ALS). We find that activity in human motor cortex has similar dynamical structure to that of non-human primates, indicating that human motor cortex contains a similar underlying dynamical system for movement generation.

Assessment of brain machine interfaces from the perspective of people with paralysis

Journal of Neural Engineering, 2015
Blabe CH, Gilja V, Chestek CA, Shenoy KV, Anderson KD, Henderson JM
Objective. One of the main goals of brain–machine interface (BMI) research is to restore function to people with paralysis. Currently, multiple BMI design features are being investigated, based on various input modalities (externally applied and surgically implantable sensors) and output modalities (e.g. control of computer systems, prosthetic arms, and functional electrical stimulation systems). While these technologies may eventually provide some level of benefit, they each carry associated burdens for end-users. We sought to assess the attitudes of people with paralysis toward using various technologies to achieve particular benefits, given the burdens currently associated with the use of each system. Approach. We designed and distributed a technology survey to determine the level of benefit necessary for people with tetraplegia due to spinal cord injury to consider using different technologies, given the burdens currently associated with them. The survey queried user preferences for 8 BMI technologies including electroencephalography, electrocorticography, and intracortical microelectrode arrays, as well as a commercially available eye tracking system for comparison. Participants used a 5-point scale to rate their likelihood to adopt these technologies for 13 potential control capabilities. Main Results. Survey respondents were most likely to adopt BMI technology to restore some of their natural upper extremity function, including restoration of hand grasp and/or some degree of natural arm movement. High speed typing and control of a fast robot arm were also of interest to this population. Surgically implanted wireless technologies were twice as 'likely' to be adopted as their wired equivalents. Significance. Assessing end-user preferences is an essential prerequisite to the design and implementation of any assistive technology. The results of this survey suggest that people with tetraplegia would adopt an unobtrusive, autonomous BMI system for both restoration of upper extremity function and control of external devices such as communication interfaces.

Comparison of spike sorting and thresholding of voltage waveforms for intracortical brain–machine interface performance

Journal of Neural Engineering, 2015
Christie BP, Tat DM, Irwin ZT, Gilja V, Nuyujukian P, Foster JD, Ryu SI, Shenoy KV, Thompson DE, Chestek CA
Objective. For intracortical brain–machine interfaces (BMIs), action potential voltage waveforms are often sorted to separate out individual neurons. If these neurons contain independent tuning information, this process could increase BMI performance. However, the sorting of action potentials ('spikes') requires high sampling rates and is computationally expensive. To explicitly define the difference between spike sorting and alternative methods, we quantified BMI decoder performance when using threshold-crossing events versus sorted action potentials. Approach. We used data sets from 58 experimental sessions from two rhesus macaques implanted with Utah arrays. Data were recorded while the animals performed a center-out reaching task with seven different angles. For spike sorting, neural signals were sorted into individual units by using a mixture of Gaussians to cluster the first four principal components of the waveforms. For thresholding events, spikes that simply crossed a set threshold were retained. We decoded the data offline using both a Naïve Bayes classifier for reaching direction and a linear regression to evaluate hand position. Main results. We found the highest performance for thresholding when placing a threshold between −3 and −4.5 × Vrms. Spike sorted data outperformed thresholded data for one animal but not the other. The mean Naïve Bayes classification accuracy for sorted data was 88.5% and changed by 5% on average when data were thresholded. The mean correlation coefficient for sorted data was 0.92, and changed by 0.015 on average when thresholded. Significance. For prosthetics applications, these results imply that when thresholding is used instead of spike sorting, only a small amount of performance may be lost. The utilization of threshold-crossing events may significantly extend the lifetime of a device because these events are often still detectable once single neurons are no longer isolated.

Self-recalibrating classifiers for intracortical brain computer interfaces

Journal of Neural Engineering, 2014
Bishop WE, Chestek CA, Gilja V, Nuyujukian P, Foster JD, Ryu SI, Shenoy KV, Yu BM
Objective. Intracortical brain-computer interface (BCI) decoders are typically retrained daily to maintain stable performance. Self-recalibrating decoders aim to remove the burden this may present in the clinic by training themselves autonomously during normal use but have only been developed for continuous control. Here we address the problem for discrete decoding (classifiers).Approach. We recorded threshold crossings from 96-electrode arrays implanted in the motor cortex of two rhesus macaques performing center-out reaches in 7 directions over 41 and 36 separate days spanning 48 and 58 days in total for offline analysis.Main results. We show that for the purposes of developing a self-recalibrating classifier, tuning parameters can be considered as fixed within days and that parameters on the same electrode move up and down together between days. Further, drift is constrained across time, which is reflected in the performance of a standard classifier which does not progressively worsen if it is not retrained daily, though overall performance is reduced by more than 10% compared to a daily retrained classifier. Two novel self-recalibrating classifiers produce a ~15% increase in classification accuracy over that achieved by the non-retrained classifier to nearly recover the performance of the daily retrained classifier.Significance. We believe that the development of classifiers that require no daily retraining will accelerate the clinical translation of BCI systems. Future work should test these results in a closed-loop setting.

Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas

Journal of Neural Engineering, 2013
Chestek CA, Gilja V, Blabe CH, Foster BL, Shenoy KV, Parvizi J, Henderson JM
Objective. Brain-machine interface systems translate recorded neural signals into command signals for assistive technology. In individuals with upper limb amputation or cervical spinal cord injury, the restoration of a useful hand grasp could significantly improve daily function. We sought to determine if electrocorticographic (ECoG) signals contain sufficient information to select among multiple hand postures for a prosthetic hand, orthotic, or functional electrical stimulation system.Approach. We recorded ECoG signals from subdural macro- and microelectrodes implanted in motor areas of three participants who were undergoing inpatient monitoring for diagnosis and treatment of intractable epilepsy. Participants performed five distinct isometric hand postures, as well as four distinct finger movements. Several control experiments were attempted in order to remove sensory information from the classification results. Online experiments were performed with two participants.Main results. Classification rates were 68%, 84% and 81% for correct identification of 5 isometric hand postures offline. Using 3 potential controls for removing sensory signals, error rates were approximately doubled on average (2.1×). A similar increase in errors (2.6×) was noted when the participant was asked to make simultaneous wrist movements along with the hand postures. In online experiments, fist versus rest was successfully classified on 97% of trials; the classification output drove a prosthetic hand. Online classification performance for a larger number of hand postures remained above chance, but substantially below offline performance. In addition, the long integration windows used would preclude the use of decoded signals for control of a BCI system.Significance. These results suggest that ECoG is a plausible source of command signals for prosthetic grasp selection. Overall, avenues remain for improvement through better electrode designs and placement, better participant training, and characterization of non-stationarities such that ECoG could be a viable signal source for grasp control for amputees or individuals with paralysis.

A high-performance neural prosthesis enabled by control algorithm design

Nature Neuroscience, 2012
Gilja V*, Nuyujukian P*, Chestek CA, Cunningham JP, Yu BM, Fan JM, Churchland MM, Kaufman MT, Kao JC, Ryu SI, Shenoy KV
Neural prostheses translate neural activity from the brain into control signals for guiding prosthetic devices, such as computer cursors and robotic limbs, and thus offer individuals with disabilities greater interaction with the world. However, relatively low performance remains a critical barrier to successful clinical translation; current neural prostheses are considerably slower, with less accurate control, than the native arm. Here we present a new control algorithm, the recalibrated feedback intention-trained Kalman filter (ReFIT-KF) that incorporates assumptions about the nature of closed-loop neural prosthetic control. When tested in rhesus monkeys implanted with motor cortical electrode arrays, the ReFIT-KF algorithm outperformed existing neural prosthetic algorithms in all measured domains and halved target acquisition time. This control algorithm permits sustained, uninterrupted use for hours and generalizes to more challenging tasks without retraining. Using this algorithm, we demonstrate repeatable high performance for years after implantation in two monkeys, thereby increasing the clinical viability of neural prostheses.

Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex

Journal of Neural Engineering, 2011
Chestek CA, Gilja V, Nuyujukian P, Foster JD, Fan JM, Kaufman MT, Churchland MM, Rivera-Alvidrez Z, Cunningham JP, Ryu SI, Shenoy KV
Cortically-controlled prosthetic systems aim to help disabled patients by translating neural signals from the brain into control signals for guiding prosthetic devices. Recent reports have demonstrated reasonably high levels of performance and control of computer cursors and prosthetic limbs, but to achieve true clinical viability, the long-term operation of these systems must be better understood. In particular, the quality and stability of the electrically-recorded neural signals require further characterization. Here, we quantify action potential changes and offline neural decoder performance over 382 days of recording from four intracortical arrays in three animals. Action potential amplitude decreased by 2.4% per month on average over the course of 9.4, 10.4, and 31.7 months in three animals. During most time periods, decoder performance was not well correlated with action potential amplitude (p > 0.05 for three of four arrays). In two arrays from one animal, action potential amplitude declined by an average of 37% over the first 2 months after implant. However, when using simple threshold-crossing events rather than well-isolated action potentials, no corresponding performance loss was observed during this time using an offline decoder. One of these arrays was effectively used for online prosthetic experiments over the following year. Substantial short-term variations in waveforms were quantified using a wireless system for contiguous recording in one animal, and compared within and between days for all three animals. Overall, this study suggests that action potential amplitude declines more slowly than previously supposed, and performance can be maintained over the course of multiple years when decoding from threshold-crossing events rather than isolated action potentials. This suggests that neural prosthetic systems may provide high performance over multiple years in human clinical trials.

Challenges and opportunities for next-generation intra-cortically based neural prostheses

IEEE Transactions on Biomedical Engineering, 2011
Gilja V, Chestek CA, Diester I, Henderson JM, Deisseroth K, Shenoy KV
Neural prosthetic systems aim to help disabled patients by translating neural signals from the brain into control signals for guiding computer cursors, prosthetic arms, and other assistive devices. Intracortical electrode arrays measure action potentials and local field potentials from individual neurons, or small populations of neurons, in the motor cortices and can provide considerable information for controlling prostheses. Despite several compelling proof-of-concept laboratory animal experiments and an initial human clinical trial, at least three key challenges remain which, if left unaddressed, may hamper the translation of these systems into widespread clinical use. We review these challenges: achieving able-bodied levels of performance across tasks and across environments, achieving robustness across multiple decades, and restoring able-bodied quality proprioception and somatosensation. We also describe some emerging opportunities for meeting these challenges. If these challenges can be largely or fully met, intracortically based neural prostheses may achieve true clinical viability and help increasing numbers of disabled patients.

A closed-loop human simulator for investigating the role of feedback-control in brain-machine interfaces

Journal of Neurophysiology, 2011
Cunningham JP, Nuyujukian P, Gilja V, Chestek CA, Ryu SI, Shenoy KV
Neural prosthetic systems seek to improve the lives of severely disabled people by decoding neural activity into useful behavioral commands. These systems and their decoding algorithms are typically developed "offline", using neural activity previously gathered from a healthy animal, and the decoded movement is then compared with the true movement that accompanied the recorded neural activity. However, this offline design and testing may neglect important features of a real prosthesis, most notably the critical role of feedback control, which enables the user to adjust neural activity while using the prosthesis. We hypothesize that understanding and optimally designing high-performance decoders require an experimental platform where humans are in closed-loop with the various candidate decode systems and algorithms. It remains unexplored the extent to which the subject can, for a particular decode system, algorithm, or parameter, engage feedback and other strategies to improve decode performance. Closed-loop testing may suggest different choices than offline analyses. Here we ask if a healthy human subject, using a closed-loop neural prosthesis driven by synthetic neural activity, can inform system design. We use this online prosthesis simulator (OPS) to optimize "online" decode performance based on a key parameter of a current state-of-the-art decode algorithm, the bin width of a Kalman filter. First, we show that offline and online analyses indeed suggest different parameter choices. Previous literature and our offline analyses agree that neural activity should be analyzed in bins of 100- to 300-ms width. OPS analysis, which incorporates feedback control, suggests that much shorter bin widths (25-50 ms) yield higher decode performance. Second, we confirm this surprising finding using a closed-loop rhesus monkey prosthetic system. These findings illustrate the type of discovery made possible by the OPS, and so we hypothesize that this novel testing approach will help in the design of prosthetic systems that will translate well to human patients.

Autonomous head-mounted electrophysiology systems for freely-behaving primates

Current Opinion in Neurobiology, 2010
Gilja V*, Chestek CA*, Nuyujukian P, Foster J, Shenoy KV
Recent technological advances have led to new light-weight battery-operated systems for electrophysiology. Such systems are head mounted, run for days without experimenter intervention, and can record and stimulate from single or multiple electrodes implanted in a freely behaving primate. Here we discuss existing systems, studies that use them, and how they can augment traditional, physically restrained, 'in-rig' electrophysiology. With existing technical capabilities, these systems can acquire multiple signal classes, such as spikes, local field potential, and electromyography signals, and can stimulate based on real-time processing of recorded signals. Moving forward, this class of technologies, along with advances in neural signal processing and behavioral monitoring, have the potential to dramatically expand the scope and scale of electrophysiological studies.

HermesD: A high-rate long-range wireless transmission system for simultaneous multichannel neural recording applications

IEEE Transactions on Biomedical Circuits and Systems, 2010
Miranda H, Gilja V, Chestek C, Shenoy KV, Meng TH
HermesD is a high-rate, low-power wireless transmission system to aid research in neural prosthetic systems for motor disabilities and basic motor neuroscience. It is the third generation of our "Hermes systems" aimed at recording and transmitting neural activity from brain-implanted electrode arrays. This system supports the simultaneous transmission of 32 channels of broadband data sampled at 30 ks/s, 12 b/sample, using frequency-shift keying modulation on a carrier frequency adjustable from 3.7 to 4.1 GHz, with a link range extending over 20 m. The channel rate is 24 Mb/s and the bit stream includes synchronization and error detection mechanisms. The power consumption, approximately 142 mW, is low enough to allow the system to operate continuously for 33 h, using two 3.6-V/1200-mAh Li-SOCl2 batteries. The transmitter was designed using off-the-shelf components and is assembled in a stack of three 28 mm ? 28-mm boards that fit in a 38 mm ? 38 mm ? 51-mm aluminum enclosure, a significant size reduction over the initial version of HermesD. A 7-dBi circularly polarized patch antenna is used as the transmitter antenna, while on the receiver side, a 13-dBi circular horn antenna is employed. The advantages of using circularly polarized waves are analyzed and confirmed by indoor measurements. The receiver is a stand-alone device composed of several submodules and is interfaced to a computer for data acquisition and processing. It is based on the superheterodyne architecture and includes automatic frequency control that keeps it optimally tuned to the transmitter frequency. The HermesD communications performance is shown through bit-error rate measurements and eye-diagram plots. The sensitivity of the receiver is -83 dBm for a bit-error probability of 10-9. Experimental recordings from a rhesus monkey conducting multiple tasks show a signal quality comparable to commercial acquisition systems, both in the low-frequency (local field potentials) and u- per-frequency bands (action potentials) of the neural signals. This system can be easily scaled up in terms of the number of channels and data rate to accommodate future generations of Hermes systems.

Methods for estimating neural firing rates and their application to brain-machine interfaces

Neural Networks, special issue on brain-machine interfaces, 2004
Cunningham JP, Gilja V, Ryu SI, Shenoy KV
Neural spike trains present analytical challenges due to their noisy, spiking nature. Many studies of neuroscientific and neural prosthetic importance rely on a smoothed, denoised estimate of a spike train's underlying firing rate. Numerous methods for estimating neural firing rates have been developed in recent years, but to date no systematic comparison has been made between them. In this study, we review both classic and current firing rate estimation techniques. We compare the advantages and drawbacks of these methods. Then, in an effort to understand their relevance to the field of neural prostheses, we also apply these estimators to experimentally gathered neural data from a prosthetic arm-reaching paradigm. Using these estimates of firing rate, we apply standard prosthetic decoding algorithms to compare the performance of the different firing rate estimators, and, perhaps surprisingly, we find minimal differences. This study serves as a review of available spike train smoothers and a first quantitative comparison of their performance for brain-machine interfaces.

HermesC: Low-power wireless neural recording system for freely moving primates

IEEE TNSRE, special issue on wireless neurotechnology, 2009
Chestek CA*, Gilja V*, Nuyujukian P, Kier R, Solzbacher F, Ryu SI, Harrison RA, Shenoy KV
Neural prosthetic systems have the potential to restore lost functionality to amputees or patients suffering from neurological injury or disease. Current systems have primarily been designed for immobile patients, such as tetraplegics functioning in a rather static, carefully tailored environment. However, an active patient such as amputee in a normal dynamic, everyday environment may be quite different in terms of the neural control of movement. In order to study motor control in a more unconstrained natural setting, we seek to develop an animal model of freely moving humans. Therefore, we have developed and tested HermesC-INI3, a system for recording and wirelessly transmitting neural data from electrode arrays implanted in rhesus macaques who are freely moving. This system is based on the integrated neural interface (INI3) microchip which amplifies, digitizes, and transmits neural data across a approximately 900 MHz wireless channel. The wireless transmission has a range of approximately 4 m in free space. All together this device consumes 15.8 mA and 63.2 mW. On a single 2 A-hr battery pack, this device runs contiguously for approximately six days. The smaller size and power consumption of the custom IC allows for a smaller package (51 x 38 x 38 mm (3)) than previous primate systems. The HermesC-INI3 system was used to record and telemeter one channel of broadband neural data at 15.7 kSps from a monkey performing routine daily activities in the home cage.

Wireless neural recording with single low-power integrated circuit

IEEE TNSRE, special issue on wireless neurotechnology, 2009
Harrison RR, Kier RJ, Chestek CA, Gilja V, Nuyujukian P, Ryu SI, Gregor B, Solzbacher F, Shenoy KV
We present benchtop and in vivo experimental results from an integrated circuit designed for wireless implantable neural recording applications. The chip, which was fabricated in a commercially available 0.6- mum 2P3M BiCMOS process, contains 100 amplifiers, a 10-bit analog-to-digital converter (ADC), 100 threshold-based spike detectors, and a 902-928 MHz frequency-shift-keying (FSK) transmitter. Neural signals from a selected amplifier are sampled by the ADC at 15.7 kSps and telemetered over the FSK wireless data link. Power, clock, and command signals are sent to the chip wirelessly over a 2.765-MHz inductive (coil-to-coil) link. The chip is capable of operating with only two off-chip components: a power/command receiving coil and a 100-nF capacitor.

Factor-analysis methods for higher-performance neural prostheses

Journal of Neurophysiology, 2009
Santhanam G, Yu BM, Gilja V, Afshar A, Ryu SI, Sahani M, Shenoy KV
Neural prostheses aim to provide treatment options for individuals with nervous-system disease or injury. It is necessary, however, to increase the performance of such systems before they can be clinically viable for patients with motor dysfunction. One performance limitation is the presence of correlated trial-to-trial variability that can cause neural responses to wax and wane in concert as the subject is, for example, more attentive or more fatigued. If a system does not properly account for this variability, it may mistakenly interpret such variability as an entirely different intention by the subject. We report here the design and characterization of factor-analysis (FA)-based decoding algorithms that can contend with this confound. We characterize the decoders (classifiers) on experimental data where monkeys performed both a real reach task and a prosthetic cursor task while we recorded from 96 electrodes implanted in dorsal premotor cortex. The decoder attempts to infer the underlying factors that comodulate the neurons' responses and can use this information to substantially lower error rates (one of eight reach endpoint predictions) by 150 ms, although still advantageous at shorter timescales, that Gaussian-based algorithms performed better than the analogous Poisson-based algorithms and that the FA algorithm is robust even with a limited amount of training data. We propose that FA-based methods are effective in modeling correlated trial-to-trial neural variability and can be used to substantially increase overall prosthetic system performance.

Toward optimal target placement for neural prosthetic devices

Journal of Neurophysiology, 2008
Cunningham JP, Yu BM, Gilja V, Ryu SI, Shenoy KV
Neural prosthetic systems have been designed to estimate continuous reach trajectories (motor prostheses) and to predict discrete reach targets (communication prostheses). In the latter case, reach targets are typically decoded from neural spiking activity during an instructed delay period before the reach begins. Such systems use targets placed in radially symmetric geometries independent of the tuning properties of the neurons available. Here we seek to automate the target placement process and increase decode accuracy in communication prostheses by selecting target locations based on the neural population at hand. Motor prostheses that incorporate intended target information could also benefit from this consideration. We present an optimal target placement algorithm that approximately maximizes decode accuracy with respect to target locations. In simulated neural spiking data fit from two monkeys, the optimal target placement algorithm yielded statistically significant improvements up to 8 and 9% for two and sixteen targets, respectively. For four and eight targets, gains were more modest, as the target layouts found by the algorithm closely resembled the canonical layouts. We trained a monkey in this paradigm and tested the algorithm with experimental neural data to confirm some of the results found in simulation. In all, the algorithm can serve not only to create new target layouts that outperform canonical layouts, but it can also confirm or help select among multiple canonical layouts. The optimal target placement algorithm developed here is the first algorithm of its kind, and it should both improve decode accuracy and help automate target placement for neural prostheses.

Signal processing challenges for neural prostheses

IEEE Signal Processing Magazine, special issue on brain-computer interfaces, 2008
Linderman MD, Santhanam G, Kemere CT, Gilja V, O'Driscoll S, Yu BM, Afshar A, Ryu SI, Shenoy KV, Meng TH
Cortically controlled prostheses are able to translate neural activity from the cerebral cortex into control signals for guiding computer cursors or prosthetic limbs. While both noninvasive and invasive electrode techniques can be used to measure neural activity, the latter promises considerably higher levels of performance and therefore functionality to patients. The process of translating analog voltages recorded at the electrode tip into control signals for the prosthesis requires sophisticated signal acquisition and processing techniques. In this article we briefly review the current state-of-the-art in invasive, electrode-based neural prosthetic systems, with particular attention to the advanced signal processing algorithms that enable that performance. Improving prosthetic performance is only part of the challenge, however. A clinically viable prosthetic system will need to be more robust and autonomous and, unlike existing approaches that depend on multiple computers and specialized recording units, must be implemented in a compact, implantable prosthetic processor (IPP). In this article we summarize recent results which indicate that state-of-the-art prosthetic systems can be implemented in an IPP using current semiconductor technology, and the challenges that face signal processing engineers in improving prosthetic performance, autonomy and robustness within the restrictive constraints of the IPP.

Electrical signals propagate unbiased in cortex

Neuron, 2007
Gilja V, Moore T
The greater spatial coherence of local field potentials (LFPs) compared with that of spiking activity has been attributed to frequency-dependent propagation of signals through the cortical medium. However, in this issue of Neuron, Logothetis and colleagues show that signal propagation within cortex is largely unbiased across different frequencies, thus suggesting a more functional and interpretable basis of LFP coherence.

HermesB: A continuous neural recording system for freely behaving primates

IEEE Transactions in Biomedical Engineering, 2007
Santhanam G*, Linderman MD*, Gilja V, Afshar A, Ryu SI, Meng TH, Shenoy KV
Chronically implanted electrode arrays have enabled a broad range of advances in basic electrophysiology and neural prosthetics. Those successes motivate new experiments, particularly, the development of prototype implantable prosthetic processors for continuous use in freely behaving subjects, both monkeys and humans. However, traditional experimental techniques require the subject to be restrained, limiting both the types and duration of experiments. In this paper, we present a dual-channel, battery-powered neural recording system with an integrated three-axis accelerometer for use with chronically implanted electrode arrays in freely behaving primates. The recording system called HermesB, is self-contained, autonomous, programmable, and capable of recording broadband neural (sampled at 30 kS/s) and acceleration data to a removable compact flash card for up to 48 h. We have collected long-duration data sets with HermesB from an adult macaque monkey which provide insight into time scales and free behaviors inaccessible under traditional experiments. Variations in action potential shape and root-mean square (RMS) noise are observed across a range of time scales. The peak-to-peak voltage of action potentials varied by up to 30% over a 24-h period including step changes in waveform amplitude (up to 25%) coincident with high acceleration movements of the head. These initial results suggest that spike-sorting algorithms can no longer assume stable neural signals and will need to transition to adaptive signal processing methodologies to maximize performance. During physically active periods (defined by head-mounted accelerometer), significantly reduced 5-25-Hz local field potential (LFP) power and increased firing rate variability were observed. Using a threshold fit to LFP power, 93% of 403 5-min recording blocks were correctly classified as active or inactive, potentially providing an efficient tool for identifying different behavioral contexts in prosthetic applications. These results demonstrate the utility of the HermesB system and motivate using this type of system to advance neural prosthetics and electrophysiological experiments.

Single-neuron stability during repeated reaching in macaque premotor cortex

Journal of Neuroscience, 2007
Chestek CA*, Batista AP*, Santhanam G, Yu BM, Afshar A, Cunningham JP, Gilja V, Ryu SI, Churchland MM, Shenoy KV
Some movements that animals and humans make are highly stereotyped, repeated with little variation. The patterns of neural activity associated with repeats of a movement may be highly similar, or the same movement may arise from different patterns of neural activity, if the brain exploits redundancies in the neural projections to muscles. We examined the stability of the relationship between neural activity and behavior. We asked whether the variability in neural activity that we observed during repeated reaching was consistent with a noisy but stable relationship, or with a changing relationship, between neural activity and behavior. Monkeys performed highly similar reaches under tight behavioral control, while many neurons in the dorsal aspect of premotor cortex and the primary motor cortex were simultaneously monitored for several hours. Neural activity was predominantly stable over time in all measured properties: firing rate, directional tuning, and contribution to a decoding model that predicted kinematics from neural activity. The small changes in neural activity that we did observe could be accounted for primarily by subtle changes in behavior. We conclude that the relationship between neural activity and practiced behavior is reasonably stable, at least on timescales of minutes up to 48 h. This finding has significant implications for the design of neural prosthetic systems because it suggests that device recalibration need not be overly frequent, It also has implications for studies of neural plasticity because a stable baseline permits identification of nonstationary shifts.

Unsupervised Channel Compression Methods in Motor Prostheses Design

IEEE EMBC, 2021
Alothman A, Gilja V
The development of high performance brain machine interfaces (BMIs) requires scaling recording channel count to enable simultaneous recording from large populations of neurons. Unfortunately, proposed implantable neural interfaces have power requirements that scale linearly with channel count. To facilitate the design of interfaces with reduced power requirements, we propose and evaluate an unsupervised-learning-based compressed sensing strategy. This strategy suggests novel neural interface architectures which compress neural data by methodically combining channels of spiking activity. We develop an entropy-based compression strategy that models the population of neurons as being generated from a lower dimensional set of latent variables and aims to minimize the loss of information in the latent variables due to compression. We evaluate compressed features by inferring the latent variables from these features and measuring the accuracy with which the activity of held out neurons and arm movements can be estimated. We apply these methods to different cortical regions (PMd and M1) and compare the proposed compression methods to a random projections strategy often employed for compressed sensing and to a supervised regression based channel dropping strategy traditionally applied in BMI applications.

Affective response to volitional input perturbations in obstacle avoidance and target tracking games

IEEE EMBC, 2021
Patel AN, Chau G, Chang C, Sun A, Huang J, Jung TP, Gilja V
We present the use of two game-like tasks, Catnip and Dinorun, to explore affective responses to volitional control perturbations. We analyze behavioral and physiological measures with the self-assessment manikin (SAM), pupillometry, and electroencephalography (EEG) responses to provide intratrial emotional state as well as inter-trial correlates with selfreported survey responses. We find that subject gameplay characteristics significantly correlate with valence and dominance scores for both games, and that perturbations to the games produce a measurable decrease in response scores for Dinorun. During perturbation events, pupillometry analysis reveals considerable SAM-agnostic dilation, with stronger responses in more rigid trialized event structures. Furthermore, analyses of neural activity from central and parietal regions demonstrate significant measurable evoked responses to perturbed events across the majority of subjects for both games. By introducing perturbations, this set of experiments and analyses inform and enable further studies of affective responses to the loss of volitional control during engaging, game-like tasks.

Improving Classification Accuracy in Cortical Surface Recordings Using ICA-Based Features

IEEE SMC, 2018
Estrin S, Martinez-Cancino R, Makeig S, Gilja V
Performance of classifiers on electrophysiological signals are often affected by volume conduction, thus compromising their reliability and classification accuracy. This issue is usually incorrectly overlooked when dealing with electrocorticography (ECoG) recordings. Here we propose that preprocessing ECoG signals using Independent Component Analysis (ICA) can improve classification performance. To test this hypothesis we use ECoG signals measured from the cortical surface of an epileptic subject. ECoG signals from subtemporal cortex were recorded while a series of face and house images were displayed briefly. We compare the performance of house versus face classifiers using features extracted from the recorded signals versus their independent components (ICs). We show that classification accuracy based on IC features is preserved when the channels with the highest single channel classification accuracy are removed from the analysis. Hence, features of independent signal spaces derived by ICA decomposition may improve the robustness and reliability of signal-based Brain-Computer Interface (BCI) classifiers.

Patient-specific pose estimation in a clinical environment

SoCal Machine Learning Symposium, 2017
Chen K, Gabriel PG, Alasfour A, Doyle WK, Devinsky O, Friedman D, Thesen T, Gilja V
We demonstrate a method for improving automatic upper-body pose estimation from hours of RGB video recordings of a subject in a clinical environment. Our semi-automated approach uses a patient-specific ConvNet model trained on a 15-minute subset of postures taken from manually annotated video to estimate the same patient's postures on an additional 120 minutes of video. By including temporal constraints and adapting to scene lighting changes, the proposed framework yields higher labeling consistency for increased spatial tolerances compared to similar methods for a single subject recorded in a clinical setting.

Neural correlates to automatic behavior estimations from RGB-D video in epilepsy unit

IEEE EMBC, 2016
Gabriel PG, Doyle WK, Devinsky O, Friedman D, Thesen T, Gilja V
To augment neural monitoring, a minimally intrusive multi-modal capture system was designed and implemented in the epilepsy clinic. This system provides RGB-D audio-video synchronized with patient electrocorticography (ECoG), which records neural activity across cortex. We propose an automated approach to studying the human brain in a naturalistic setting. We demonstrate coarse functional mapping of ECoG electrodes correlated to contralateral arm movements. Motor electrode mapping was generated by analyzing continuous movement data recorded over several hours from epilepsy patients in hospital rooms. From these recordings we estimate the kinematics of patient hand movement behaviors using computer vision algorithms. We compare movement behaviors to neural data collected from ECoG, specifically high-γ (70-110 Hz) spectral features. We present a functional map of electrode responses to natural arm movements, generated using a statistical test. We demonstrate that our approach has the potential to aid in the development of automated functional brain mapping using continuous video and neural recordings of patients in clinical settings.

Decoding speech using the timing of neural signal modulation

IEEE EMBC, 2016
Jiang W, Pailla T, Dichter B, Chang EF, Gilja V
Brain-machine interfaces (BMIs) have great potential for applications that restore and assist communication for paralyzed individuals. Recently, BMIs decoding speech have gained considerable attention due to their potential for high information transfer rates. In this study, we propose a novel decoding approach based on hidden Markov models (HMMs) that uses the timing of neural signal changes to decode speech. We tested the decoder’s performance by predicting vowels from electrocorticographic (ECoG) data of three human subjects. Our results show that timing-based features of ECoG signals are informative of vowel production and enable decoding accuracies significantly above the level of chance. This suggests that leveraging the temporal structure of neural activity to decode speech could play an important role towards developing high-performance, robust speech BMIs.

ECoG data analyses to inform closed-loop BCI experiments for speech-based prosthetic applications

IEEE EMBC, 2016
Pailla T, Jiang W, Dichter B, Chang EF, Gilja V
Brain Computer Interfaces (BCIs) assist individuals with motor disabilities by enabling them to control prosthetic devices with their neural activity. Performance of closed-loop BCI systems can be improved by using design strategies that leverage structured and task-relevant neural activity. We use data from high density electrocorticography (ECoG) grids implanted in three subjects to study sensorymotor activity during an instructed speech task in which the subjects vocalized three cardinal vowel phonemes. We show how our findings relate to the current understanding of speech physiology and functional organization of human sensory-motor cortex. We investigate the effect of behavioral variations on parameters and performance of the decoding model. Our analyses suggest experimental design strategies.

Hidden-Markov factor analysis as a spatiotemporal model for electrocorticography

IEEE EMBC, 2016
Omigbodun AO, Doyle WK, Devinsky O, Friedman D, Thesen T, Gilja V
We present a new approach to extracting low-dimensional neural trajectories that summarize the electrocorticographic (ECoG) signals recorded with high-channel-count electrode arrays implanted subdurally. In our approach, Hidden-Markov Factor Analysis (HMFA), a finite set of factor analyzers are used to model the relationship between the high-dimensional ECoG neural space and a low-dimensional latent neural space; the factor analyzers at different time points are in turn linked together with a hidden Markov model. The recorded ECoG signals were band-pass filtered such that our analysis was focused on a sub-band (76-100Hz) of high gamma. HMFA affords the quantization of the ECoG neural space and dimensionality reduction in a common probabilistic space. We applied this method to the ECoG recordings of 2 subjects who responded with button presses to audiovisual stimuli in an experimental task. Using a goodness-of-fit metric that measures how well the ECoG activity of each electrode can be predicted by all the other electrodes, we found that HMFA performed best when compared with Gaussian-Process Factor Analysis (GPFA) and other related spatiotemporal modeling techniques. In contradistinction to HMFA, GPFA and the other techniques integrate temporal smoothing with dimensionality reduction. We believe that this method will provide a powerful tool for relating high-channel-count ECoG signals to the perception and behavior of subjects.

A clinic compatible, open source electrophysiology system

IEEE EMBC, 2016
Hermiz J, Rogers N, Kaestner E, Ganji M, Cleary D, Snider J, Barba D, Dayeh S, Halgren E, Gilja V
Open source electrophysiology (ephys) recording systems have several advantages over commercial systems such as customizability and affordability enabling more researchers to conduct ephys experiments. Notable open source ephys systems include Open-Ephys, NeuroRighter and more recently Willow, all of which have high channel count (64+), scalability, and advanced software to develop on top of. However, little work has been done to build an open source ephys system that is clinic compatible, particularly in the operating room where acute human electrocorticography (ECoG) research is performed. We developed an affordable (< $10,000) and open system for research purposes that features power isolation for patient safety, compact and water resistant enclosures and 256 recording channels sampled up to 20ksam/sec, 16-bit. The system was validated by recording ECoG with a high density, thin film device for an acute, awake craniotomy study at the University of California, San Diego, Thornton Hospital Operating Room.

Using accelerometers in the neurological ICU to monitor unilaterally motor impaired patients

Neurology, 2016
LaBuzetta J, Hermiz J, Gilja V, Karanjia N
Objective: We explore the feasibility of automating motor function scoring by using continuous accelerometer measurements to identify characteristics present in both accelerometer recordings and motor exam scores. Background: The neurological motor exam provides important information about a patient’s clinical status in the ICU, but the frequency at which it needs to be performed often makes it arduous for both healthcare providers and patients. Methods: Patients with unilateral motor impairment (defined as a score of 0-2 on “weak” side compared with 3-5 on “normal” side) in the neurological ICU were considered for this study. The Axivity AX3 accelerometer was used to measure three-dimensional accelerations for up to 14 days in each patient. These unobtrusive devices were placed on all 4 limbs using standard hospital bands, and required no active maintenance. Acceleration signals were processed to identify movement events that lasted ≥0.5msec. Movement events per hour were summated and compared between “weak” and “normal” limbs, and average movement count per hour was compared against motor strength score. Results: Three subjects were recruited; mean age 44years, 67[percnt] male. Two patients were diagnosed with ischemic strokes, and one with hemorrhagic stroke. One subject was impaired on the left hemibody; two were right-side impaired. The mean recording duration was 10 days (range 6-14 days). We found that there were significantly more movements in each of the normal upper and lower extremities as compared to their respective weak upper and lower extremities for all 3 subjects (P < 0.05, one-tail, two sample Kolmogorov-Smirnov test). Moreover, mean movement counts per hour appeared to correlate with objective provider motor scoring. Conclusions: Continuous monitoring of limb activity with accelerometers is feasible and can be used to identify motor asymmetry after neurological injury. It is unclear if accelerometer monitoring can be used to assess nuanced motor function.

A brain machine interface control algorithm designed from a feedback control perspective

Proc. of the 34th Annual International Conference IEEE EMBS, 2012
Gilja V*, Nuyujukian P*, Chestek CA, Cunningham JP, Yu BM, Fan JM, Churchland MM, Kaufman MT, Ryu SI, Shenoy KV
We present a novel brain machine interface (BMI) control algorithm, the recalibrated feedback intention-trained Kalman filter (ReFIT-KF). The design of ReFIT-KF is motivated from a feedback control perspective applied to existing BMI control algorithms. The result is two design innovations that alter the modeling assumptions made by these algorithms and the methods by which these algorithms are trained. In online neural control experiments recording from a 96-electrode array implanted in M1 of a macaque monkey, the ReFIT-KF control algorithm demonstrates large performance improvements over the current state of the art velocity Kalman filter, reducing target acquisition time by a factor of two, while maintaining a 500 ms hold period, thereby increasing the clinical viability of BMI systems.

Monkey models for brain-machine interfaces: The need for maintaining diversity

Proc. of the 33rd Annual International Conference IEEE EMBS, 2011
Nuyujukian P*, Fan JM*, Gilja V, Kalanithi PS, Chestek CA, Shenoy KV
Brain-machine interfaces (BMIs) aim to help disabled patients by translating neural signals from the brain into control signals for guiding prosthetic arms, computer cursors, and other assistive devices. Animal models are central to the development of these systems and have helped enable the successful translation of the first generation of BMIs. As we move toward next-generation systems, we face the question of which animal models will aid broader patient populations and achieve even higher performance, robustness, and functionality. We review here four general types of rhesus monkey models employed in BMI research, and describe two additional, complementary models. Given the physiological diversity of neurological injury and disease, we suggest a need to maintain the current diversity of animal models and to explore additional alternatives, as each mimic different aspects of injury or disease.

Spiking neural network decoder for brain-machine interfaces

Proc. of the 5th International IEEE EMBS Conference on Neural Engineering, 2011
Dethier J, Gilja V, Nuyujukian P, Elassaad S, Shenoy KV, Boahen K
We used a spiking neural network (SNN) to decode neural data recorded from a 96-electrode array in premotor/motor cortex while a rhesus monkey performed a point-to-point reaching arm movement task. We mapped a Kalman-filter neural prosthetic decode algorithm developed to predict the arm's velocity on to the SNN using the Neural Engineering Framework and simulated it using Nengo, a freely available software package. A 20,000-neuron network matched the standard decoder's prediction to within 0.03% (normalized by maximum arm velocity). A 1,600-neuron version of this network was within 0.27%, and run in real-time on a 3GHz PC. These results demonstrate that a SNN can implement a statistical signal processing algorithm widely used as the decoder in high-performance neural prostheses (Kalman filter), and achieve similar results with just a few thousand neurons. Hardware SNN implementations - neuromorphic chips - may offer power savings, essential for realizing fully-implantable cortically controlled prostheses.

Neural prosthetic systems: Current problems and future directions

Proc. of the 31st Annual International Conf. of the IEEE EMBS, 2009
Chestek C*, Cunningham JP*, Gilja V, Nuyujukian P, Ryu SI, Shenoy KV
By decoding neural activity into useful behavioral commands, neural prosthetic systems seek to improve the lives of severely disabled human patients. Motor decoding algorithms, which map neural spiking data to control parameters of a device such as a prosthetic arm, have received particular attention in the literature. Here, we highlight several outstanding problems that exist in most current approaches to decode algorithm design. These include two problems that we argue will unlikely result in further dramatic increases in performance, specifically spike sorting and spiking models. We also discuss three issues that have been less examined in the literature, and we argue that addressing these issues may result in dramatic future increases in performance. These include: non-stationarity of recorded waveforms, limitations of a linear mappings between neural activity and movement kinematics, and the low signal to noise ratio of the neural data. We demonstrate these problems with data from 39 experimental sessions with a non-human primate performing reaches and with recent literature. In all, this study suggests that research in cortically-controlled prosthetic systems may require reprioritization to achieve performance that is acceptable for a clinically viable human system.

HermesD: A high-rate long-range wireless transmission system for multichannel neural recording applications

Proc. of the IEEE International Symposium on Circuits and Systems (ISCAS), 2009
Miranda H, Gilja V, Chestek C, Shenoy KV, Meng TH
HermesD is a high-rate, low-power wireless transmission system to aid research in neural prosthetic systems for motor disabilities and basic motor neuroscience. It is the third generation of our "Hermes systems" aimed at recording and transmitting neural activity from brain-implanted electrode arrays. This system supports the simultaneous transmission of 32 channels of broadband data sampled at 30 ks/s, 12 b/sample, using frequency-shift keying modulation on a carrier frequency adjustable from 3.7 to 4.1 GHz, with a link range extending over 20 m. The channel rate is 24 Mb/s and the bit stream includes synchronization and error detection mechanisms. The power consumption, approximately 142 mW, is low enough to allow the system to operate continuously for 33 h, using two 3.6-V/1200-mAh Li-SOCl2 batteries. The transmitter was designed using off-the-shelf components and is assembled in a stack of three 28 mm ? 28-mm boards that fit in a 38 mm ? 38 mm ? 51-mm aluminum enclosure, a significant size reduction over the initial version of HermesD. A 7-dBi circularly polarized patch antenna is used as the transmitter antenna, while on the receiver side, a 13-dBi circular horn antenna is employed. The advantages of using circularly polarized waves are analyzed and confirmed by indoor measurements. The receiver is a stand-alone device composed of several submodules and is interfaced to a computer for data acquisition and processing. It is based on the superheterodyne architecture and includes automatic frequency control that keeps it optimally tuned to the transmitter frequency. The HermesD communications performance is shown through bit-error rate measurements and eye-diagram plots. The sensitivity of the receiver is -83 dBm for a bit-error probability of 10(-9). Experimental recordings from a rhesus monkey conducting multiple tasks show a signal quality comparable to commercial acquisition systems, both in the low-frequency (local field potentials) and upper-frequency bands (action potentials) of the neural signals. This system can be easily scaled up in terms of the number of channels and data rate to accommodate future generations of Hermes systems.

A wireless neural interface for chronic recording

Proc. of the IEEE Biomedical Circuits and Systems Conference, special session B3L-A "Revolutionizing Prosthetics Lecture" (talk), 2008
Harrison RR, Kier RJ, Kim S, Rieth L, Warren DJ, Ledbetter NM, Clark GA, Solzbacher F, Chestek CA, Gilja V, Nuyujukian, Ryu SI, Shenoy KV
A primary goal of the Integrated Neural Interface Project (INIP) is to develop a wireless, implantable device capable of recording neural activity from 100 micromachined electrodes. The heart of this recording system is a low-power integrated circuit that amplifies 100 weak neural signals, detects spikes with programmable threshold-crossing circuits, and returns these data via digital radio telemetry. The chip receives power, clock, and command signals through a coil-to-coil inductive link. Here we report that the isolated integrated circuit successfully recorded and wirelessly transmitted digitized electrical activity from peripheral nerve and cortex at 15.7 kS/s. The chip also simultaneously performed accurate on-chip spike detection and wirelessly transmitted the spike threshold-crossing data. We also present preliminary successful results from full system integration and packaging.

Wireless neural signal acquisition with single low-power integrated circuit

Proc. of the IEEE International Symposium on Circuits and Systems (ISCAS) (talk), 2008
Harrison RR, Kier RJ, Greger B, Solzbacher F, Chestek CA, Gilja V, Nuyujukian P, Ryu SI, Shenoy KV
We present experimental results from an integrated circuit designed for wireless neural recording applications. The chip, which was fabricated in a 0.6-mum 2P3M BiCMOS process, contains 100 amplifiers and a 10-bit ADC and 902-928 MHz FSK transmitter. Neural signals from one amplifier are sampled by the ADC at 15.7 kSps and telemetered over the FSK wireless data link. Power, clock, and command signals are sent to the chip wirelessly over a 2.765-MHz inductive (coil-to-coil) link. The chip is capable of operating with only two off-chip components: a power receive coil and a 100-nF capacitor.

HermesC: RF wireless low-power neural recording system for freely behaving primates

Proc. of the IEEE International Symposium on Circuits and Systems (ISCAS) (talk), 2008
Chestek CA, Gilja V, Nuyujukian P, Ryu SI, Kier RJ, Solzbacher F, Harrison RR, Shenoy KV
Neural prosthetics for motor systems is a rapidly growing field with the potential to provide treatment for amputees or patients suffering from neurological injury and disease. To determine whether a physically active patient such as an amputee can take advantage of these systems, we seek to develop an animal model of freely moving humans. Therefore, we have developed and tested HermesC, a system for recording neural activity from electrode arrays implanted in rhesus monkeys and transmitting this data wirelessly. This system is based on the integrated neural interface (INI) microchip, which amplifies, digitizes, and transmits neural data across a ~900 MHz wireless channel. The wireless transmission has a range of ~4 m in free space. All together, this device consumes 11.7 mA from a 4.0 V lithium ion battery pack for a total of 46.8 mW. To test the performance, the device was used to record and telemeter one channel of broadband neural data at 15.7 kSps from one monkey doing various physical activities in a home cage, such as eating, climbing and swinging. The in-band noise of the recorded neural signal is 34 muVrms, which is low enough to allow the detection of neural units on an active electrode. This system can be readily upgraded to use future generations of the INI chip, with circuits providing 96 channels of programmable threshold crossing event data.

A factor-analysis decoder for high-performance neural prostheses

Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2008
Santhanam G, Yu BM, Gilja V, Ryu SI, Afshar A, Sahani M, Shenoy KV
Increasing the performance of neural prostheses is necessary for assuring their clinical viability. One performance limitation is the presence of correlated trial-to-trial variability that can cause neural responses to wax and wane in concert as the subject is, for example, more attentive or more fatigued. We report here the design and characterization of a Factor- Analysis-based decoding algorithm that is able to contend with this confound. We characterize the decoder (classifier) on a previously reported dataset where monkeys performed both a real reach task and a prosthetic cursor movement task while we recorded from 96 electrodes implanted in dorsal pre- motor cortex. In principle, the decoder infers the underlying factors that co-modulate the neurons' responses and can use this information to function with reduced error rates (1 of 8 reach target prediction) of up to ~75% (~20% total prediction error using independent Gaussian or Poisson models became ~5%). Such Factor-Analysis based methods appear to be effective when attempting to combat directly unobserved trial-by-trial neural variabiliy.

An autonomous, broadband, multi-channel neural recording system for freely behaving primates

Proc. of the 28th Annual International Conf. of the IEEE EMBS, 2006
Linderman MD, Gilja V, Santhanam G, Afshar A, Ryu SI, Meng TH, Shenoy KV
Successful laboratory proof-of-concept experiments with neural prosthetic systems motivate continued algorithm and hardware development. For these efforts to move beyond traditional fixed laboratory setups, new tools are needed to enable broadband, multi-channel, long duration neural recording from freely behaving primates. In this paper we present a dualchannel, battery powered, neural recording system with integrated 3-axis accelerometer for use with chronically implanted electrode arrays. The recording system, called HermesB, is selfcontained, autonomous, programmable and capable of recording broadband neural and head acceleration data to a removable compact flash card for up to 48 hours.

Multiday electrophysiological recordings from freely behaving primates

Proc. of the 28th Annual International Conference of the IEEE EMBS (talk), 2006
Gilja V, Linderman MD, Santhanam G, Afshar A, Ryu SI, Meng TH, Shenoy KV
Continuous multiday broadband neural data provide a means for observing effects at fine timescales over long periods. In this paper we present analyses on such data sets to demonstrate neural correlates for physically active and inactive time periods, as defined by the response of a head-mounted accelerometer. During active periods, we found that 5-25 Hz local field potential (LFP) power was significantly reduced, firing rate variability increased, and firing rates have greater temporal correlation. Using a single threshold fit to LFP power, 93% of the 403 5 minute blocks tested were correctly classified as active or inactive (as labeled by thresholding each block's maximal accelerometer magnitude). These initial results motivate the use of such data sets for testing neural prosthetics systems and for finding the neural correlates of natural behaviors.

Neural recording stability of chronic electrode arrays in freely behaving primates

Proc. of the 28th Annual International Conf. of the IEEE EMBS (talk), 2006
Linderman MD, Gilja V, Santhanam G, Afshar A, Ryu SI, Meng TH, Shenoy KV
Chronically implanted electrode arrays have enabled a broad range of advances, particularly in the field of neural prosthetics. Those successes motivate development of prototype implantable prosthetic processors for long duration, continuous use in freely behaving subjects. However, traditional experimental protocols have provided limited information regarding the stability of the electrode arrays and their neural recordings. In this paper we present preliminary results derived from long duration neural recordings in a freely behaving primate which show variations in action potential shape and RMS noise across a range of time scales. These preliminary results suggest that spike sorting algorithms can no longer assume stable neural signals and will need to transition to adaptive signal processing methodologies to maximize performance.

Imaging through Windansee electrode arrays reveals a small fraction of local neurons following surface MUA

bioRxiv, 2022
Thunemann M, Hossain L, Ness TV, Rogers N, Lee K, Lee SH, Kiliç K, Oh H, Economo MN, Gilja V, Einevoll GT, Dayeh SA, Devor A
Prior studies have shown that neuronal spikes can be recorded with microelectrode arrays placed on the cortical surface. However, the etiology of these spikes remains unclear. Because the top cortical layer (layer 1) contains very few neuronal cell bodies, it has been proposed that these spikes originate from neurons with cell bodies in layer 2. To address this question, we combined two-photon calcium imaging with electrophysiological recordings from the cortical surface in awake mice using chronically implanted PEDOT:PSS electrode arrays on transparent parylene C substrate. Our electrode arrays (termed Windansee) were integrated with cortical windows offering see-through optical access while also providing measurements of local field potentials (LFP) and multiunit activity (MUA) from the cortical surface. To enable longitudinal data acquisition, we have developed a mechanical solution for installation, connectorization, and protection of Windansee devices aiming for an unhindered access for high numerical aperture microscope objectives and a lifetime of several months while worn by a mouse. Contrary to the common notion, our measurements revealed that only a small fraction of layer 2 neurons from the sampled pool (~13%) faithfully followed MUA recorded from the surface above the imaging field-of-view. Surprised by this result, we turned to computational modeling for an alternative explanation of the MUA signal. Using realistic modeling of neurons with back-propagating dendritic properties, we computed the extracellular action potential at the cortical surface due to firing of local cortical neurons and compared the result to that due to axonal inputs to layer 1. Assuming the literature values for the cell/axon density and firing rates, our modeling results show that surface MUA due to axonal inputs is over an order of magnitude larger than that due to firing of layer 2 pyramidal neurons. Thus, a combination of surface MUA recordings with two-photon calcium imaging can provide complementary information about the input to a cortical column and the local circuit response. Cortical layer I plays an important role in integration of a broad range of cortico-cortical, thalamocortical and neuromodulatory inputs. Therefore, detecting their activity as MUA while combining electrode recording with two-photon imaging using optically transparent surface electrode arrays would facilitate studies of the input/output relationship in cortical circuits, inform computational circuit models, and improve the accuracy of the next generation brain-machine interfaces.

Cellular Calcium Activity at Depth Predicted from Surface Potential Recordings using Ultra-high Density Transparent Graphene Arrays

bioRxiv, 2023
R Mehrdad, Kim J, Liu X, Ren C, Alothman A, De-Eknamkul C, Wilson M, Cubukcu E, Gilja V, Komiyama T, Kuzum D
Recording brain activity with high spatial and high temporal resolution across deeper layers of cortex has been a long-sought methodology to study how neural information is coded, stored, and processed by neural circuits and how it leads to cognition and behavior. Electrical and optical neural recording technologies have been the key tools in neurophysiology studies toward a comprehensive understanding of the neural dynamics. The advent of optically transparent neural microelectrodes has facilitated multimodal experiments combining simultaneous electrophysiological recordings from the brain surface with optical imaging and stimulation of neural activity. A remaining challenge is to scale down electrode dimensions to single -cell size and increase the density to record neural activity with high spatial resolution across large areas to capture nonlinear neural dynamics at multiple spatial and temporal scales. Here, we developed microfabrication techniques to create transparent graphene microelectrodes with ultra-small openings and a large, completely transparent recording area. We achieved this by using long graphene microwires without any gold extensions in the field of view. To overcome the quantum capacitance limit of graphene and scale down the microelectrode diameter to 20 μm, we used Pt nanoparticles. To prevent open circuit failure due to defects and disconnections in long graphene wires, we employed interlayer doped double layer graphene (id-DLG) and demonstrated cm-scale long transparent graphene wires with microscale width and low resistance. Combining these two advances, we fabricated high-density microelectrode arrays up to 256 channels. We conducted multimodal experiments, combining recordings of cortical potentials with high-density transparent arrays with two-photon calcium imaging from layer 1 (L1) and layer 2/3 (L2/3) of the V1 area of mouse visual cortex. High-density recordings showed that the visual evoked responses are more spatially localized for high-frequency bands, particularly for the multi-unit activity (MUA) band. The MUA power was found to be strongly correlated with the cellular calcium activity. Leveraging this strong correlation, we applied dimensionality reduction techniques and neural networks to demonstrate that single-cell (L2/3) and average (L1 and L2/3) calcium activities can be decoded from surface potentials recorded by high-density transparent graphene arrays. Our high-density transparent graphene electrodes, in combination with multimodal experiments and computational methods, could lead to the development of minimally invasive neural interfaces capable of recording neural activity from deeper layers without requiring depth electrodes that cause damage to the tissue. This could potentially improve brain computer interfaces and enable less invasive treatments for neurological disorders.

Sequence Transfer Learning for Neural Decoding

bioRxiv, 2017
Elango V, Patel AN, Miller K, Gilja V
A fundamental challenge in designing brain-computer interfaces (BCIs) is decoding behavior from time-varying neural oscillations. in typical applications, decoders are constructed for individual subjects and with limited data leading to restrictions on the types of models that can be utilized. currently, the best performing decoders are typically linear models capable of utilizing rigid timing constraints with limited training data. Here we demonstrate the use of Long Short-Term Memory (LSTM) networks to take advantage of the temporal information present in sequential neural data collected from subjects implanted with electrocorticographic (ECoG) electrode arrays performing a finger flexion task. our constructed models are capable of achieving accuracies that are comparable to existing techniques while also being robust to variation in sample data size. Moreover, we utilize the LSTM networks and an affine transformation layer to construct a novel architecture for transfer learning. We demonstrate that in scenarios where only the affine transform is learned for a new subject, it is possible to achieve results comparable to existing state-of-the-art techniques. The notable advantage is the increased stability of the model during training on novel subjects. Relaxing the constraint of only training the affine transformation, we establish our model as capable of exceeding performance of current models across all training data sizes. Overall, this work demonstrates that LSTMS are a versatile model that can accurately capture temporal patterns in neural data and can provide a foundation for transfer learning in neural decoding.

Brain machine interface

US8792976 B2, 2014
Gilja V, Nuyujukian P, Chestek CA, Cunningham JP, Yu BM, Fan JM, Ryu SI, Shenoy KV
Artificial control of a prosthetic device is provided. A brain machine interface contains a mapping of neural signals and corresponding intention estimating kinematics (e.g. positions and velocities) of a limb trajectory. The prosthetic device is controlled by the brain machine interface. During the control of the prosthetic device, a modified brain machine interface is developed by modifying the vectors of the velocities defined in the brain machine interface. The modified brain machine interface includes a new mapping of the neural signals and the intention estimating kinematics that can now be used to control the prosthetic device using recorded neural brain signals from a user of the prosthetic device. In one example, the intention estimating kinematics of the original and modified brain machine interface includes a Kalman filter modeling velocities as intentions and positions as feedback.

Analysis of Behavioral and Autonomic States in Unstructured Multiday Human Intracranial Electrophysiology

University of California, San Diego, 2020
Alasfour, Abdulwahab A.
Systems neuroscience and neural engineering have relied mainly on the task-based experimental paradigm to understand brain activity. While this method has proved fruitful, it fails to capture the full variability of unstructured and naturalistic neural activity. In this thesis, we explore the value of investigating multiday unstructured intracranial electrophysiology recordings in humans. Using stereotactic-electroencephalography (sEEG) and electrocorticography (ECoG) electrodes, we analyze days of neural recordings to investigate how internal and external states, such as autonomic tone and behavior, correlate to neural activity. Firstly, we determine whether coarsely labeled unstructured behavioral contexts or states are discriminable in the neural activity space. Subjects were not instructed to perform any task; therefore, only spontaneous behaviors were analyzed. Controls to determine whether temporal correlations and time of day effects impact the separability of behavioral states were investigated, concluding that both the time of day and behavior have a combined effect on neural activity. Secondly, once we determined that these behavioral states are separable, the cause of this separability was further investigated. In other words, what neural signal characteristics are responsible for our ability to decode abstract behavioral states? Both long term signal characteristics and spatiotemporal dynamics contribute information regarding naturalistic behavior, showing that outside the lab, neural activity has multiple axes of variability that could be used to discriminate between different states. In the final section of this work, we investigate the neural correlates to autonomic tone during sleep, leveraging multiple days of unstructured neural activity to make physiological conclusions regarding the connection between the central and autonomic nervous systems.

Characterizing Unstructured Motor Behaviors in the Epilepsy Monitoring Unit

University of California, San Diego, 2019
Gabriel, Paolo
Key advancements in recording hardware, data computation, clinical care, and cognitive science continue to drive new possibilities in how humans and machines can interact directly through thought. Neural data analyses with these advancements has progressed neuroscience research in functional brain mapping and brain-computer interfaces (BCIs). Much of our knowledge about BCIs is informed by data collected through carefully controlled experiments. Constraining BCI experiments with structured paradigms allows researchers to collect a high number of consistent data in a short amount of time, while also controlling for external confounds. Very little is currently known about how well these task-based relationships extend to daily life, in part because collecting data outside of the lab is challenging. To further understand natural brain activity, we must study more complex behaviors in more environmentally relevant settings. The results of this dissertation address three general challenges to studying neural correlates to unstructured behaviors. First, we continuously monitored unstructured human movements in the epilepsy monitoring unit using a video sensor synchronized to clinical intracortical electrodes. Second, we annotated unstructured behaviors from these video using both manual and computer vision methods. Finally, analyzed neural features with respect to unstructured human movements, and evaluated the performance of features identified in previous task-based studies. The preliminary nature of this work means that a majority of our demonstrations are whether the continuous paradigm can be leveraged, how one might go about leveraging it, and evaluations that tie our results back to earlier task-based studies. Our advances here motivate future works that focus more intently on what types of behaviors and neural signal features to explore.

Deep learning methods for Electrocorticographic data analyses

University of California, San Diego, 2019
Pailla, Tejaswy
Electrocorticography (ECoG) records brain activity from the cortical surface. ECoG data analyses has led to significant advancements in neuroscientific research, particularly in two major domains: functional mapping to understand the cortical organization of human brain; and brain-machine interfaces (BMIs) that decode intent from neural data. Designing high performance BMIs is an active area of interest. A discrete BMI design primarily involves decoding specific targets from features extracted from ECoG data. Majority of ECoG based research studies use spectral features i.e. powers in specific frequency bands; which are based on empirical observations. However, given the non-stationarity and variability of neural signals, features extracted in a data driven way could lead to more robust BMIs. In addition to efficient feature extraction and decoder training, the choice of targets presented to BMI user can greatly affect the bit-rate or throughput of the BMI.

Continuous Human Pose Estimation Using Long Short-Term Memory and Particle Filter

University of California, San Diego, 2019
Gong, Chenghao
Estimating human pose in a continuous time series has many practical applications. For example, imagine that some time in the future robot would like to interact with human beings, for that robot to meaningfully interact with a human it needs to interpret and anticipate human movements and gestures. Acquiring continuous human pose estimates can also inform specific applications like brain-machine interface; specifically, we can use accounts of human pose data across time to study the relationship between neural signals and human pose. In this thesis, we will focus our work on the continuous human pose estimation in the clinical environment.

A Study on Human Pose Data Anomaly Detection

University of California, San Diego, 2019
Zhang, Haotian
Identifying anomalous human pose data is crucial to many emerging data-driven artificial intelligence systems. For instance, patient behavior monitoring systems can analyze patient behavior based on patient movement and pose predictions. Although pose tracking methods have improved over the years, anomalous pose estimates, even if infrequent, can result in troublesome events, such as error information on the patient behaviors, which can lead to false diagnosis and requires human labor intensive processes to identify those anomalous poses. This cost could be mitigated by correcting or identifying anomalous pose estimates in an automated fashion. Thus, we present an anomaly analysis framework for clinical human pose estimates to address these concerns.

Continuous behavior acquisition in clinical environments

University of California, San Diego, 2018
Chen, Kenny J.
Continuous behavioral labels of hospital patients provide quantitative data that can be informative for both research studies and clinical applications. An analysis of neural correlates to natural behavioral labels extracted from pose estimates, for example, could enable more robust brain-machine prostheses for those with limb loss or motor impairment. Automated patient motion analysis could also provide additional insight to clinicians during motor scoring assessments and seizure classification for a better-informed diagnosis. Likewise, continuous patient safety monitoring enabled by posture annotations could detect potential bed falls or other injury risks and quickly alert nurses to administer preventive measures. Such scenarios rely on consistent and accurate patient posture tracking in clinical environments. While many existing pose estimation frameworks are effective when subjects are located in uncluttered settings, clinical environments can provide several visual challenges that these general frameworks are not calibrated for. In this thesis, we propose a semi-automated approach for improving upper-body pose estimation in noisy clinical environments, whereby we adapt and build around an existing joint tracking framework to improve its robustness to environmental uncertainties. The proposed framework uses subjectspecific convolutional neural network (CNN) models trained on a subset of a patient’s RGB video recording chosen to maximize the feature variance of each joint. Furthermore, by compensating for scene lighting changes and by refining the predicted joint trajectories through a Kalman filter with fitted noise parameters, the expanded framework can yield more consistent posture annotations in these settings when compared to general methods. The perspectives gained from this work provide better insight in developing a practical pose estimation framework for researchers and clinicians in these environments.

Electrocortiography on the micron scale: Single units and enhanced neural state estimation

University of California, San Diego, 2018
Hermiz, John
Electrocorticography on the micron scale (micro-ECoG) is an emerging neural sensing modality that provides a high-resolution view of the brain. Micron scale electrodes measure electrical potential propagated to the brain surface from local and distant current sources. Moreover, electrodes spatially sample the surface of the brain at the micron scale over potentially large regions providing both high resolution and large coverage. Micro-ECoG can be likened to HD monitors, whereas classical ECoG grids are more like Hex LED displays. In this dissertation, I demonstrate the value of micro-ECoG both in animal model and in humans. I developed a suite of neural acquisition tools (NACQ), which was used to record from human subjects intraoperatively and in the epilepsy monitoring unit for research purposes. These tools provide an affordable alternative to commercial systems and a safer alternative to existing open source systems. I demonstrate that micro-ECoG electrode can sense physiologically relevant features, including single unit activity in songbird. These physiological features measured from micro-ECoG are compared to gold standard probes including penetrating laminar silicon shanks in songbird and clinical ECoG strips in human. Finally, I explored theoretical and empirical instances in which a high density grid of electrodes outperforms sub-sampled lower density grids in discrete neural state estimation. Empirically, I show that when controlling for area and selecting task informative sub-regions of the complete grid, we observed a consistent increase in mean binary classification accuracy with higher grid density; in particular, 400 μm pitch grids outperforming spatially sub-sampled lower density grids up to 23%. Micro-ECoG is a promising neural sensing modality that may lead to new neuroscientific discoveries and neuroengineering achievements. For example, it may uncover novel neural dynamics from cortical columns or intricate cortical wave patterns important from neural information processing. Micro-ECoG may lead to the development of a high-bandwidth brain machine interface that not only restores abilities of disabled individuals, but augments and enhances abilities of able-bodied people. Neuroscientists and neurotechnologists are poised to make major advances in neuroscience and neuroengineering with the advent of micro-ECoG.

Finding structure in electrocorticographic neural signals for brain-machine interface applications

University of California, San Diego, 2017
Omigbodun, Akinyinka O.
Electrocorticography (ECoG), also known as intracranial electroencephalography (iEEG), is the practice of recording electrical potentials on the cerebral cortex via electrodes placed on the exposed brain surface. ECoG has been a critical component of epilepsy medical treatment protocols involving neurosurgery for more than half a century. More recently, ECoG has emerged as a promising recording modality for brain-machine interfaces and neuroscience research. The BRAIN Initiative is representative of a renewed and concerted effort to push the boundaries of possibility in medical care and technology, and to expand our understanding of brain function. Concomitant with this new drive is a need for techniques that address the challenges posed by high-channel count ECoG signal analysis as well as by neural data collection limited due to the invasiveness of ECoG. In this dissertation, we introduce the use of a discrete-state based probabilistic method for modeling ECoG-derived signals, and contrast this method with previously existing analogous probabilistic models without a discrete component. We then explore a class of discrete-state based probabilistic models, and identify spatial and temporal model constraints that were advantageous in the analysis of a high-channel count ECoG dataset. Finally, we introduce another probabilistic model that we use for unsupervised learning of ECoG trial spatiotemporal structure, and clustering of the ECoG trials in a data-limited context.

Towards practical neural prosthetic interfaces

University of California, San Diego, 2017
Patel, Aashish N.
The connection between our brain and our muscles via the peripheral nerves enables us to communicate and act upon the world. Individuals suffering from disease and injury that affect these connections have limited treatment options and often rely upon prostheses. A fundamental challenge for such prostheses is the development of a control system that is capable of interpreting desired intentions accurately. Existing control schemes utilize activation by other functional systems using clever mechanical linkages or electromyography signal based control. Increasingly, neural interfaces are being considered for these applications due to their potential ability to restore behavioral function without co-opting existing motor functions. Implanted electrode based neural interfaces, such as electrocorticography (ECoG) hold promise for providing high signal to noise ratio measurements that are stable over long time periods while electroencephalography (EEG) provides convenient, non-invasive scalp surface measurements. Although these techniques provide powerful approaches to measuring neural signals, their application currently yields limited performance in neural interfacing applications.

Sequence learning for brain computer interfaces

University of California, San Diego, 2017
Elango, Venkatesh
A fundamental challenge in designing brain-computer interfaces (BCIs) is decoding behavior accurately from time-varying neural oscillations. Studies using BCIs to function as communication prosthesis have demonstrated the plausibility of using these systems for recording neural signals over the long term as well as the ability to decode user intention from these signals. In most scenarios, the decoder used in a BCI is trained specifically for a subject and also has to be trained for every session of use with limited training data. Given these dataset size constraints, the class of decoding algorithms typically explored have restricted complexity, often limited to linear models that process neural signals within a fixed duration. However, such constraints can limit the practicality and usability of BCIs.

Minimally intrusive gaze detection in clinical environments

University of California, San Diego, 2015
Wang, Yuchen
Motivated by the Electronic Health Record (EHR) system's demand of capturing patients' multimodal activities and the wide application of gaze detection, we develop a minimally intrusive gaze detection system with Microsoft Kinect sensor and test its performance in a simulated clinical environment. Traditional methods require either a close distance between the camera and the user or a fixed head pose which may severely interrupt the clinical workflow and the interaction between the physician and the patient. Compared with the traditional methods, our system allows a wider range of detection, while achieving an accuracy around 70%.

Simulation of a scalable electrochemical immunosignaturing biosensor array for disease diagnosis

University of California, San Diego, 2015
Au, Anthony
The ability to detect diseases during early progression greatly impacts the effectiveness of treatments, especially for cancers. Current research focuses on discovering specific biomarkers associated with the disease, but these are difficult to discover and present in very low concentrations. Conversely, immunosignaturing leverages the immense amplification provide by the immune system to examine antibody patterns on a random array of peptides. This thesis explores a new electrochemical detection method instead of the traditional optical detection for use with the immunosignaturing chip. I developed a software simulation in order to investigate the parameters of the system. The first part of the simulation tracked the concentration changes of the molecules of interest. These concentrations of molecules drove the electrochemical discharges modeled in the next part of the simulation. The efficacy of the simulated discharges was determined by comparison with experimental discharge data. The first two parts of the simulation showed that crosstalk occurs with adjacent non-active sensors and the time delay before it happens largely depends on array geometry and sensor capacitance. The last part of the simulation explored the ability of this method to discern various diseases. Classification of a transformed optical data yielded similar classification accuracy compared to the original optical data. A mock end to end simulation demonstrated high accuracy as well. This thesis outlines a few approaches for implementation of the physical device, while laying out the framework to further explore parameter variations and disease classification.

Towards clinically viable neural prosthetic systems

Stanford University, 2010
Gilja, Vikash
By restoring the ability to move and communicate with the world, brain machine interfaces (BMIs) offer the potential to improve quality of life for people suffering from spinal cord injury, stroke, or neurodegenerative diseases, such as amyotrophic lateral sclerosis (ALS). BMIs attempt to translate measured neural signals into the users intentions and, subsequently, control a computer or actuator. Recently, compelling examples of intra-cortical BMIs have been demonstrated in tetraplegic patients. Although these studies provide a powerful proof-of-concept, clinical viability is impeded by limited performance and robustness over short (hours) and long (days) timescales. We address performance and robustness over short time periods by approaching BMIs as a systems level design problem. We identify key components of the system and design a novel BMI from a feedback control perspective. In this perspective, the brain is the controller of a new plant, defined by the BMI, and the actions of this BMI are witnessed by the user. This simple perspective leads to design advances thatresult in significant qualitative and quantitative performance improvements. Throughonline closed loop experiments, we show that this BMI is capable of producing continuousendpoint movements that approach native limb performance and can operate continuously for hours. We also demonstrate how this system can be operated acrossdays by a bootstrap procedure with the potential to eliminate an explicit recalibration step. To examine the use of BMIs over longer timescales, we develop new electrophysiology tools that allow for continuous multi-day neural recording. Through application of this technology, we measure the signal acquisition stability (and instability) of the electrode array technology used in current BMI clinical trials. We also demonstrate how these systems can be used to study BMI decoding over longer time periods. In this demonstration, we present a simple methodology for switching BMI systems on and off at appropriate times. The algorithms and methods demonstrated can be run with existing low power application specific integrated circuits (ASICs), with a defined path towards the development of a fully implantable neural interface system. We believe that these advances are a step towards clinical viability and, with careful user interface design, neural prosthetic systems can be translated into real world solutions.
Hi

Article Unavailable

This document is currently unavailable publicly, please check back later!

© 2023 TNEL | Gilja Group