Shulou(Shulou.com)06/01 Report--

How to use BCI to carry out brain imagination handwriting for text output, many novices are not very clear about this, in order to help you solve this problem, the following small series will explain in detail for everyone, there are people who need this aspect can learn, I hope you can gain something.

Brain-computer interfaces (BCI) can help patients who have lost the ability to move or speak regain their ability to communicate. A major focus of BCI research to date has been restoring body muscle motor skills, such as tapping input with touch and grip or 2D computer cursors. However, highly dexterous behaviors such as handwriting or touch-typing may require faster communication speeds.

Input Chinese characters by thinking

Earlier reports said that the University of Hong Kong Chinese successfully developed a "brain-computer interface" system, which can convert brain waves into traditional Chinese characters, so that patients who are paralyzed and unable to speak have the opportunity to "open the heart window." The system was exhibited for the first time at the Hong Kong Central Library and attracted a large number of curious members of the public.

The patient wears a wireless EEG receiver with 16 contact surfaces. Staring at a computer screen, the five strokes flash alternately. The patient thinks about the stroke to write, and the receiver receives instructions to select the stroke and write the Chinese character stroke by stroke.

The seemingly magical "idea input method" is actually very simple. James said that when people see the strokes they want to type flash, they will produce psychological stimulation, when the brain will release brain waves. For example, if you want to type a horizontal stroke, when the "-" on the screen lights up, the brain releases brain waves. The system captures the brain waves and feeds them back to the computer software to complete the input of a horizontal stroke. Then input other strokes in turn. Depending on the association function of the software, words or phrases will appear continuously on the screen for users to choose.

A 2013 paper found that different parts of the brain respond differently in combination when people process different words and information. It is an article published in Nature that when we humans think of a word or text, different brain regions of our brain have a different correspondence. There are also simple applications being developed that allow the brain to enter a simple number directly, with a lot of training.

introduction

In one study, researchers demonstrated an intracortical brain-computer interface that can decode imaginary written actions from neural activity in the motor cortex and translate them into text in real time using a new recursive neural network decoding method. With this BCI, participants in the study (with paralyzed hands) typed faster than any other BCI: 90 characters per minute, 99% accuracy, and universal auto-correction.

Such typing speeds are comparable to the typing speeds of able-bodied smartphones in the age group of participants in the project (115 characters per minute) and significantly close the gap between BCI-enabled typing speeds and smart typing speeds.

The researchers say the project's findings open up a new approach to BCI and demonstrate the feasibility of accurately decoding, fast, dexterous movements years after paralysis.

experimental process

After extensive research into the field of BCI, researchers have found that motor intentions (such as grabbing or moving a computer cursor) in the motor cortex are still encoded by neurons after paralysis. Dexterous motor skills such as handwriting also remained unchanged. The researchers tested this by recording neural activity from two microelectrode arrays in the central forward hand "knob" region while participant T5 in the project experiment attempted to handwrite single letters and symbols (Figure 1A). T5 has high spinal cord injury, paralysis from neck down. We instruct T5 to "try" to write as if his hands were not paralyzed (and imagine him holding a pen on a piece of checkered paper).

The researchers used principal component analysis to reduce the recorded neural activity (multiunit threshold crossing rates) to the first three dimensions containing the largest variance (Figure 1B). Although the peak and trough times of neural activity varied from experiment to experiment (probably due to fluctuations in writing speed), neural activity appeared strong and repeatable.

The researchers asked participant T5 to try writing one character at a time according to instructions on a computer screen, as shown in Figure 1A, below which the panel describes what is displayed on the screen according to a timeline. And by adjusting the timing of neural activity to eliminate repetitive variations in writing speed, the researchers found that the pattern of activity for each letter was consistent.

1. Neural coding for handwriting

Figure 1 above shows neural coding for handwriting.

(A)Participant T5 attempted to handwrite one character at a time following instructions on a computer screen.

(B)Neural activity is shown for the first 3 principal components (PCs) of three example letters (d, e, and m) and 27 repetitions ("trials") of each letter. Color scales are normalized separately in each panel for visualization.

(C)Eliminates repetitive variations in writing speed by adjusting the timing of neural activity. In the illustration above C, the example temporal warping function is shown as the letter "m" and is relatively close to the identity line (the warping function for each trial is plotted with a different colored line).

The experiment shows the decoded pen tracks of 31 test characters:26 lowercase letters, commas, apostrophes, question marks, slashes (~), and greater than signs (>), as shown in Figure D below. The expected 2D tip velocity is linearly decoded from neural activity by cross-validation (each character is displayed). The decoded velocities were averaged throughout the trial and integrated to calculate the pen trajectory (orange circles indicate the start of the trajectory).

(E)Two-dimensional visualization of neural activity using t-SNE. Each circle is a separate trial (27 trials for each of the 31 characters).

1. Neural coding for handwriting

In this study, the researchers designed a set of decoding process, the algorithm schematic diagram is shown in Figure A. First, neural activity (multi-threshold crossing) is time-binned (20 ms binning) and smoothed at each electrode. A recurrent neural network (RNN) then converts this neural population time series (xt) into a probabilistic time series (pt-d) describing the likelihood of each character and the likelihood of any new character beginning. The RNN has a one-second output delay (d), which gives it time to observe the complete character before proceeding to recognition. Finally, the character probabilities are thresholded to produce "raw output" for real-time use (when the probability of "new character" exceeds a threshold at time t, the character is most likely to be emitted at time t+0.3). In an offline retrospective analysis, character probabilities were combined with a large-vocabulary language model to decode the text participants were most likely to write (the researchers used a custom 50,000-word bidirectional character model).

Figure 2. Real-time neural decoding of handwriting attempts

In Figure B above, two real-time sample experiments are shown showing that RNNs are able to decode easy-to-understand text from untrained sentences. Errors are highlighted in red and spaces are indicated by ">." (C)Error rate (edit distance) and typing speed are displayed for 5 days, with 4 blocks per stage, each block containing 7-10 sentences (each block is represented by a circle). This was more than twice as fast as the second-fastest intracortical BCI7.

The researchers analyzed the spatiotemporal patterns of neural activity corresponding to 16 handwritten characters (lasting 1 second) and 16 handwritten linear movements (lasting 0.6 seconds), as shown in Figure A, and found spatiotemporal neural patterns by averaging all trials (aftertime-warping to align the trials in time) for a given movement. The neural activity is then resampled to balance the duration of each set of actions (otherwise the duration of the linear motion would be shorter), resulting in a 192 x 100 matrix of each action (192 electrodes and 100 time steps), as shown in Figure 3B.

Figure (C) above calculates pairwise Euclidean distances between neural patterns for each group, revealing larger nearest neighbor distances (but not average distances) for characters. Each circle represents a movement and the bar height represents the average. (D)A larger nearest neighbor distance makes characters easier to classify than straight lines. Noise is measured in standard deviation and matches the magnitude of the distance. (E)The spatial dimensions of characters and lines are similar, but the temporal dimensions of characters are twice as high, indicating that more complex temporal patterns constitute an increase in nearest neighbor distance and better classification performance. Error bars show 95% CI (bootstrap percentile method). Dimension is defined as the participation ratio, which is approximately equal to the number of dimensions required to explain 80% of the variables. (F,G, H) A toy example illustrates intuitively how the increased time dimension makes neural trajectories more separable. Four neural trajectories are drawn (N1 and N2 are two hypothetical neurons whose activity is restricted to one spatial dimension, the unit diagonal). By adding a curve to allow the trajectory to change over time (increasing the time dimension from 1 to 2), larger nearest neighbor distances (G) and better classification (H) can be achieved.

research conclusions

These results suggest that time-varying motion patterns, such as handwritten letters, are fundamentally easier to decode than point-to-point motion, and therefore higher communication rates can be achieved. This concept can be applied more broadly to improving any BCI to enable discrete choices between a set of options (by associating those options with gestures that change over time, rather than simple actions). Using the principle of maximizing the nearest neighbor distance between motions, a set of trajectories can be optimized for classification purposes (as was done previously when optimizing target positions).

The researchers explored this accordingly and designed an alphabet that is theoretically easier to classify than letters in the Latin alphabet (Figure 4). The researchers 'findings reveal a flaw in Latin letters from a neural decoding perspective: a large number of redundant letters are written in similar ways (most letters begin with downward strokes or counterclockwise rotations).

Did reading the above help you? If you still want to have further understanding of related knowledge or read more related articles, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.