Abstract
Researchers
have shown that the firing patterns of multiple cells simultaneously recorded
from the motor cortex can be used to predict arm trajectories in primates.
These signals could make a good control signal for an arm
Introduction/Background
Restoring arm
function to quadriplegics requires a reliable control signal to indicate the
desired arm trajectory. Electrode arrays implanted in the motor cortex can
detect the individual firing patterns of many cells simultaneously. When enough
cells are recorded together, complex 3D arm trajectories can accurately be
predicted from these signals. Most off-line studies predict that many hundreds
or even thousands of cells need to be recorded to accurately predict complex 3D
movements[1,2]. Electrode technology is currently unable to obtain and hold
such large numbers of cells for long periods of time.
However, the
actual number of cells needed to provide a practical upper limb
By using a
virtual reality system, we can simulate the experience of a
cortically-controlled, arm prosthetic system, and determine if sufficient
control can be achieved using the small numbers of cortical units typically
recorded with today’s electrodes.
Methods
Two macaques
were trained in a virtual 3D center-out reaching task. Their left arm was
restrained while their right arm was free to move. The animals could not see
their real arms but instead saw a virtual arm cursor (2 cm diameter yellow
sphere) which moved with their wrist movement in 3D space. The animals could
also see one central start target and eight virtual targets positioned in a
cube formation (2.4 cm diameter green spheres). In the task, the central start
target would appear. Once the monkey moved the virtual arm cursor (VAC) to the
center target, one of the eight corner targets would appear in a random order.
The animal would then move the VAC to that corner target to receive a reward.
There were no physical targets to touch in the workspace–only virtual ones. The
virtual 3D stereo images were created using an SGI Octane with Performer
software. The images were projected through a NuVision light polarizing shutter
screen (96Hz) onto a mirror suspended at 45º in front of the animal’s face.
Differently polarized filters were placed in front of each eye. Wrist position
was acquired at 100Hz with an Optotrak 3020 motion tracking system (Northern
Digital Inc). We could observe no perceptible lag between the actual and
virtual wrist movements.
Once the
animals were proficient in this task, they were chronically implanted with 64
stainless steel microwire electrodes in the motor and premotor cortices. Each
day, eight minutes of cortical and movement data were acquired during the task
described above. Spike times of the individual cortical units were acquired in
real-time using a Plexon data acquisition system and RASPUTIN software. Each
unit's preferred direction was determined by regressing firing rate with wrist
velocity[3]. The task was then modified to alternate between eight
arm-controlled and eight brain-controlled center-to-target movements. During
the brain-controlled sets, the VAC movement was controlled by the cortical
signals instead of by the actual wrist movement. The control signal was a
population vector[1] calculated online in real time. Each cell’s contribution
to the population vector was also scaled by its level of directional tuning (F
value from its preferred direction regression equation raised to the 0.8
power). Additionally, the X, Y and Z components were appropriately scaled to
compensate for any differences in the quantity of cells contributing to each
component.
Three
experimental conditions were examined. The control case when the VAC was linked
to wrist movement, the brain-controlled experiment described above, and an
offline population vector analysis of the control case.
The
percentage of brain-controlled center-to-target movements that successfully hit
their target (%BC_hit) and the percentage of offline predicted trajectories of
the control set that successfully hit their target (%OPT_hit) were calculated
for each set and for each day. This experiment continued for 32 days in monkey
‘L’, and 40 days in monkey ‘M’. Matlab (MathWorks) was used to organize the
data and compile statistics.
Results
Table 1 shows summary
data on the number and quality of cells recorded over the course of the
experiment. It also shows summary data comparing the mean %BC_hit with
%OPT_hit. Paired t-tests showed this difference to be highly significant
(p<2x10-16).
Figure 1 shows 2D projections of the wrist trajectories
during arm-controlled trials (A), the offline predicted trajectories of those
arm-controlled trials (B), and the VAC trajectories during brain-controlled
trials (C). These were from a day when activity from 17 cortical units were
used with a mean R2 of 0.77 (R2 from regressing each
unit’s firing rate per target as a function of target direction).
|
Data
Type |
‘L’ |
‘M’ |
‘L+M’ |
|
# Cortical Units |
18 ± 4 |
18 ± 3 |
18 ± 4 |
|
Mean R2
of Cortical Units |
0.63 ± 0.07 |
0.64 ± 0.09 |
0.64 ± 0.08 |
|
%BC_hit |
52 ± 14 |
46 ± 18 |
49 ± 17 |
|
%OPT_hit |
31 ± 11 |
23 ± 5 |
27 ± 9 |
Table
1: Mean and Standard Deviation of Daily Statistics by Animal (‘L’,‘M’ or both).
‘Mean R2’ is the R2 from regressing each unit’s average
firing rate per target as a function of target direction then averaging R2s
daily over all units.
Figure 2 shows a plot of the difference in percentage of
targets hit under the brain-controlled and offline-predicted conditions
(%BC_hit – %OPT_hit) as a function of the number of days each animal had done
the experiment.
Taking %BC_hit data across all days and comparing them by
set showed the mean %BC_hit increased significantly from the first to the third
set (p< 0.03) but not significantly after that.
Figure 1: 2D projections of 3D trajectories to each of
the eight targets in the three experimental conditions. The thick straight
lines go from the center start position to the intended targets. Thin lines
are the individual trajectories. Asterisks indicate when individual
trajectories have successfully hit the target. Upper two rows are to
proximal targets. Bottom two rows are to distal targets. Actual wrist
trajectories during the control sets (non-brain-controlled) are shown in A.
B shows the offline predicted trajectories of the trials shown in A. C
shows trajectories of the virtual arm cursor during the brain control
trials.
Figure 2: Improvement by day in percentage of targets hit in the brain-controlled case over the offline predicted trajectory case.
Discussion/Conclusions
In this
experiment the typical number of units used was an order of magnitude lower
than the literature says is needed to accurately predict trajectories. As
expected, the offline predictions of the arm-controlled trajectories were very
poor. However, in the brain-controlled trials, the animals used visual feedback
to correct for errors in the trajectories. Initially, the animals would move
their real arm toward the virtual target position in space. The small numbers
of units used in the experiments were an imperfect predictor of the intended
movement. Therefore, the VAC would often drift away from the intended
trajectory. The animals would then compensate by altering their real arm
movements. The accompanying changes in the cortical activity would redirect the
VAC to head toward the target again. On occasion, the animals would stop moving
their arm completely and move the VAC just by thinking about it, but neither animal
learned to do this consistently. The animals usually explored the space with
their arm during the initial brain-controlled trials, and learned that day’s
mapping between their arm/brain activity and the VAC trajectory. Each day the
number and modulation properties of the cortical units would change. The
significant increase in %BC_hit from the first to the third set shows the
animals learned each day’s new mapping fairly quickly.
Both animals
improved their brain-control skills over the course of the experiment
(increased average %BC_hit by 0.6% per day, p=0.0013). This could happen by
coincidence if the number and/or quality of units recorded from day-to-day
improved. If this were so, we would expect the %OPT_hit to also improve. Since
the %OPT_hit decreased slightly over the course of the experiment, the
improvement seen in %BC_hit from day to day must be due to learning. Figure 2
shows this improvement in brain control over the offline predictions (average
improvement of about 1% per day, p<1x10-8)
The
population vector used in these experiments is a very simple prediction
algorithm that is easy to compute in real-time. Other algorithms which take
correlations and history into consideration have been shown to more accurately
predict trajectories[4]. We are currently investigating whether the more
complicated algorithms will improve brain-control, or if they will simply make
the mapping more difficult to learn.
These animals
experienced brain-control for less than two hours per day during the
experiment. The rest of the day they were free to used their arms normally. If
a prosthetic system was used continuously throughout the day, we would expect
the rate of learning to be much faster.
It is unknown
how the cortical re-mapping after paralysis will affect the ability to learn
brain-control of an FES system. There may be a limited time window after
paralysis in which implantation and training will be successful.
As with any
new technology, the risks have to be weighed against the benefits before it is
used in humans. This research shows the benefit of using even small numbers of
cortical signals is much greater than previous offline studies suggest.
References
[1] Georgopoulos A.P. et al. (1988)
Primate Motor Cortex and Free Arm Movements to Visual Targets in 3D Space. II.
Coding of Direction of Movement by Neural Population. J. Neurosci. 8(8):
2928-2937
[2] Wessberg J. et al. (2000) Real-Time
Prediction of hand Trajectory by Ensembles of Cortical Neurons in Primates.
Nature 408: 361-365
[3] Schwartz A.B. et al. (1988) Primate
Motor Cortex and Free Arm Movements to Visual Targets in 3D Space. I. Relations
Between Single Cell Discharge and Direction of Movement. J. Neurosci. 8(8):
2913-2927
[4] Issacs RE. et al. (2000) Work
Toward Real-Time Control of a Cortical Neural Prosthesis. IEEE Trans. Rehabil.
Eng. 8(2): 196-198
Acknowledgments: This research was supported by: The Whitaker Foundation, PEO Scholarship Fund, and PHS contracts #N01-NS-6-2347 #N01-NS-9-2321.