Multisensory integration for human detection of abnormalities during active steering

Authors & Affiliations

Brandon G. Rasman [1]

[1] Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands

Abstract

Accurate perception of self-motion is crucial for safe and meaningful interaction with the surrounding world. While driving a vehicle, for instance, the human brain constantly monitors how the vehicle (and the human body) moves in relation to one’s steering actions. If unexpected behavior occurs, such as skidding on a slippery road, the brain must detect the abnormality and respond accordingly to keep the vehicle on the desired path and avoid an accident. A widely-held theory in neuroscience implies that this is achieved by the brain combining multiple cues of sensory feedback and comparing the resulting estimates to neural predictions of expected feedback (1-3). However, the relative contributions of different sensory modalities (i.e., weightings) and the manner in which they are combined (i.e., statistically optimal or biased neural integration) during active steering remain poorly understood (4). This lack of understanding is both surprising and concerning, given the prevalence of human-driven motor vehicles in our societies. This project will reveal how humans integrate different sensory cues to form a stable perception of active steering and detect abnormal behavior. Using a sophisticated human-controlled driving platform, we will manipulate the dynamics of steering control (simulate skidding on road) and assess human ability to perceive and correct for disturbances to driving. Steering dynamics will be manipulated under different sensory conditions (e.g., with or without vision). This approach will facilitate the quantification of self-motion perception derived from different sensory modalities and their combination. Numerical models of multisensory integration will be applied to describe the neural principles of cue combination during active steering. This project will provide crucial knowledge in understanding how the human brain monitors self-motion while steering and will lay the foundation for future scientific and technical developments relevant for safe driving.

Project Updates (March 2024 - November 2024)

(1) Programming steering simulator for experiments

From previous studies in our lab, there was existing code for running the steering simulator in a closed-loop control mode (where the participant controls the motion of the platform by rotating the steering wheel). While the code provided a foundation, it needed to be modified for the present study.
A summary of needs for the new code:
- Ability to manipulate the control dynamics of the steering simulator. Such manipulations include: changing the control gain, imposing control delays, inducing varying levels of friction/viscosity.
- Updates to improve smoothness and timing precision of steering simulator control.
- Integration of visual stimuli (provided via OLED screen) with motion platform. In some conditions, participants perform steering movements with an OLED screen providing visual cues (other conditions are performed in complete darkness).
- Automation for experiments. Specifically the experiments involve a perceptual-motor task - where participants have to both perform a steering motor task and simultaneously report their perception of self-motion. Here, I am employing a psychophysical design, where participants must perform many trial repetitions with differing stimuli magnitudes such that a reliable perceptual threshold for each participant can be estimated. These experiments are time consuming. Therefore, automated code to implement specific trial parameters and iterate to the next trials is needed.
- Integration of auditory beeps and tones to alert participants when trials begin and end.
Numerous iterations of code have been written and tested. Through numerous tests, pilot experiments (see next sections) and updates, the computer scripts are ready for data collection.

Figure 1: A simplified depiction of the steering simulator and its closed-loop control. Participants sit in a car seat and control the motion of the platform by rotating a steering wheel. The platform moves in one dimension (left-right translation). Visual stimuli/feedback (not depicted in image) are integrated with the platform to create an immersive steering simulation. The dynamics of the steering simulation are dictated by customized computer code, where the steering wheel angle is taken as an input signal and transformed into motion of the platform and visual environment. These dynamics can be manipulated, such as through changes in: the control gain, control delays, friction, and viscosity. Participants receive feedback of their experienced self-motion through various human sensory systems (i.e., vestibular, visual, somatosensory). The simulation can be modified such that participants don't receive any visual feedback (i.e., steering in complete darkness) or don't experience any real physical motion (i.e., chair position remains fixed, but steering wheel controls motion in visual environment). This facilitates the ability to examine human perception and steering control performance in different sensory environments.

Visual stimuli

Integrated with the motion platform is a 77-inch OLED screen used to provide visual cues in the experiments. The screen is positioned centrally in front of the sled track at viewing distance of ~1.70 m. Visual stimuli are presented on the screen with a resolution of 1920 × 1080 pixels and a refresh rate of 120 Hz. Visual stimuli presented on the display screen consist of a field of 400 randomly positioned dots, a “home” rectangular area, a “go” line, and a “target” rectangular area. More details are provided in sections below.

(2) Pilot testing: Selecting the experimental manipulation of steering control

As mentioned above, the control dynamics of the steering simulator can be manipulated in a variety of ways (gain change, imposed delay, viscosity). Recall, one of the aims of this study is to evoke abnormal (or unexpected) steering/navigation behavior and determine what information individuals rely on to perceive and correct for these abnormalities. The idea is that after becoming familiar to the baseline settings of the steering simulator, any changes to these baseline settings should be evoke unexpected steering behavior. This is because the brain will use the prior experience of steering in the simulator to make predictions of what should happen with a given steering action, and compare this prediction to actual behavior. Naturally, changes on a small scale may go unperceived, whereas larger scale changes should be clearly perceived.

Through numerous pilot experimentation, data analysis and considerations, we have decided to use an imposed control delay as the experimental manipulation. Delays have several advantages in evoking unexpected behavior during closed-loop control. First, delays already exist in the control loop, in the form of biological delays (i.e., sensory detection, neural transmission, neural processing time and motor command execution) and delays inherent in the instrument being controlled (i.e. delays between a steering wheel rotation and motion of the vehicle). Indeed, when we drive a vehicle we are always overcoming both biological delays and vehicle related delays. Experimentally, we can simply increase the delay magnitude by storing the steering wheel signal in a time buffer before it is used to drive steering motion. Here, we will impose delays ranging from 0 - 400 milliseconds. The expectation is that small delays will result in small errors in steering movement and may go undetected at the cognitive level, whereas larger delays will evoke large errors in steering motion and be clearly detected as unexpected self-motion. By imposing various delays, we will determine human perceptual sensitivity to detecting unexpected self-motion and further determine how humans correct their steering movement in response to the evoked errors.

Figure 2: Delays are imposed into the steering simulation to evoke unexpected steering self-motion. After participants have trained and familiarized to steering the simulator under its baseline control (like getting accustomed to a new vehicle), the brain will rely on that experience to predict what self-motion should arise from steering movements. When delays are imposed into the simulation, the brain will compare the predicted self-motion with the actual self-motion. Large delays should evoke large errors, allowing the brain to determine that unexpected self-motion has occurred and must be corrected for.

Imposing control delays

Experimentally, delays can be imposed between the participant-controlled velocity command signal (derived from wheel angle) and the resulting steering self-motion (i.e. sensory feedback). In the steering stimulation, the baseline delay between the steering wheel rotation and experienced self-motion (either linear translation of motion platform or linear motion of visual stimuli) is ~25 milliseconds. For the experiments, additional delays can be programmatically added to the steering simulation. Delays are imposed by buffering the velocity command signal recording such that signals driving the simulated steering self-motion can be delivered based on participant steering actions up to 400 ms in the past. Participants will be naïve to the implementation of the delay, and will simply be informed that “in some trials, the steering simulation will change, such that your steering actions may result in unexpected or abnormal movements.”

(3) Pilot testing: Choosing a steering task

Recall that this study is focused on assessing both self-motion perception and motor performance when steering control changes (i.e. with imposed delays), and determining how individuals rely on different sensory cues to perceive unexpected behavior. As described in the section above, imposed delays will be used as the experimental manipulation to investigate: (1) perceptual sensitivity to detecting manipulations of steering control with different available sensory feedback, (2) the sensorimotor information participants rely on for these detections, and (3) how delays influence the steering motor task. Therefore, it is important to design a steering task that participants can repeatedly perform with different imposed delays (baseline - 400 milliseconds; see below) and under different sensory conditions (i.e., inertial, visual, combined; see below). The aim is to have a task that is somewhat repeatable under baseline control conditions such that participants can have a reliable concept of what “normal” control feels like. Through piloting, I have designed the following steering task. Participants are asked to navigate to a remembered target location (briefly presented on the visual display screen) in ~2.5 s and return to the starting home position in ~2.5 s, resulting in a total desired movement time of ~5 s. Auditory tones (400 Hz, 300 ms duration) are provided to aid participants in performing the movement in the appropriate timeframe. Specifically, after reaching the “go” position, the first tone played (cueing the participant to start the movement) followed by the second tone 2.5 s later (cueing that the target should have been reached), followed by the third tone 2.5 s later (cueing that the participant should be reaching the home position).

Figure 3: Simplified depiction of the visual steering environment and target presentation. Participants begin every trial on the left side of the track at the “home" position. Before the trial begins, the home position (green rectangle) and target position (red rectangle) are briefly presented on the OLED screen for 2 seconds. The targets then disappear and the participants have to steer to the remembered target (in ~2.5 seconds), turnaround and steer back to the home position (in another ~2.5 seconds). Auditory tones (played via headphones) are provided to assist participants with the timing of the steering movements. During the steering task, participants either steer with a field of dots presented or in complete darkness (see Sensorimotor conditions).

(4) Pilot testing: Sensorimotor conditions

Participants will perform steering movements in three sensorimotor environments: inertial cues only, visual cues only, inertial and visual cues. During the inertial cues only condition, participants will perform steering actions (turn steering wheel) that physically moves the motion platform in complete darkness. Therefore, all the sensory information participants receive will be related to other sensors encoding self-motion (e.g., vestibular and somatosensory receptors). During the visual cues only condition, participants will perform steering actions that result in no physical motion of platform, but experience virtual motion on the visual display via movements of the dots presented. Finally, the inertial and visual cues condition represents a combined version of the other two conditions, where participants receive both inertial and visual information (dots presented on screen) of self-motion while performing steering movements. Evaluating both steering performance and perceptual behavior under these different conditions allow for the assessment of multisensory integration for the perceptual-motor steering task.

Figure 4: Depiction of three sensorimotor conditions. In all conditions, each trial begins with the targets briefly appearing on the screen (see Figure 3). In the Inertial only condition, after the targets disappear all visual cues are removed and participants must perform the steering task in complete darkness. In the visual only condition, the participants don't experience any physical motion (the motion platform remains stationary), but their steering actions result in optic flow via motion of dots on the screen. In the combined condition (Inertial + Visual), the participants receive all sensory information. Their steering actions result in physical motion of the platform and they view dots on the screen that periodically flicker (like a starry night). This results in both inertial motion cues (vestibular and somatosensory) and visual motion cues (optic flow).

(5) Pilot testing: Steering task performance in baseline control and delayed control

Below are some traces from pilot testing. Here, the participant performs many trial repetitions of the steering task as the steering simulation operates in either baseline control (i.e., baseline delay of ~25 ms) or with an additional imposed delay (50 - 400 ms). Traces are presented for the Inertial+Visual condition, the Inertial Only condition, and the Visual Only condition. Each figure depicts position and velocity.

Figure 5: Position and velocity traces of motion platform from pilot testing in the Inertial+Visual condition. Here, many trial repetitions were performed with the steering simulator operating with its baseline delay (~25 ms, black lines) and additional delays (50 - 400 ms, red lines). As described above, the task was to steer rightward to a remembered target and then steer leftward back to the starting area. The target region (defined by the red rectangle in Figure 3) is defined by the boundaries of the horizontal solid lines.

Figure 6: Position and velocity traces of motion platform from pilot testing in the Inertial Only condition. Here, many trial repetitions were performed with the steering simulator operating with its baseline delay (~25 ms, black lines) and additional delays (50 - 400 ms, red lines). As described above, the task was to steer rightward to a remembered target and then steer leftward back to the starting area. The target region (defined by the red rectangle in Figure 3) is defined by the boundaries of the horizontal solid lines. Note that in this Inertial Only condition, the targets are briefly presented, followed by the screen turning dark. The task is to steer to the remembered target and return to the home area in complete darkness (thus relying on inertial sensory cues only).

Figure 7: Position and velocity traces of motion platform from pilot testing in the Visual Only condition. Here, many trial repetitions were performed with the steering simulator operating with its baseline delay (~25 ms, black lines) and additional delays (50 - 400 ms, red lines). As described above, the task was to steer rightward to a remembered target and then steer leftward back to the starting area. The target region (defined by the red rectangle in Figure 3) is defined by the boundaries of the horizontal solid lines. Note that in this Visual Only condition, the motion platform (i.e., chair on a linear sled track) never moves. Instead, the participants steer their “virtual" self. Motion cues are provided by linear velocity of the dots presented on the screen being coupled to the steering wheel angle. Therefore, participant steer through virtual space (thus relying on visual sensory cues only).

(6) Pilot testing: Perceptual task and performance

Recall that in addition to performing the steering movement, participants must report whether they perceive their steering control to be expected/normal or unexpected/abnormal. They are not informed that delays are being imposed into the steering simulator. They are simply told to perform the steering task and report how they perceive their steering control.

For the experiment, various delays are imposed into the simulation: (0 (sham), 50, 100, 150, 200, 250, 300, 350, 400). Each delay level is repeated numerous times. By recording participant's perceptual response on a trial-by-trial basis (perceived as expected OR perceived as unexpected), we can generate psychometric functions - sigmoidal curves fit to the perceptual data that help describe perceptual behavior.

Figure 8: A psychometric function generated from pilot data in the Inertial+Visual condition. From the perceptual responses given at each delay level, the proportion of correct detections can be calculated and then plotted. A sigmoidal curve is fit to the data (using a cumulative distribution function) to provide a psychometric function - a curve which described perceptual sensitivity to a given stimulus (in this case, detecting unexpected steering behavior when delays are imposed into closed-loop control).

Psychometric functions are generated for all three sensorimotor condition.

Figure 9: Psychometric functions generated from pilot data obtained from the Inertial+Visual (purple), Inertial only (red), and Visual only (blue) conditions.

From these perceptual data and psychometric functions, we can then test whether the perceptual system is following the rules of optimal multisensory integration. Specifically, does the perceptual behavior in the combined case (Inertial+Visual) look like the statistically-optimal combination of perceptual behavior obtained in the Inertial only and Visual only conditions. Optimal integration dictates that the uncertainty (i.e. variance) in the combined cue estimate is reduced by weighting each individual cue relative to its reliability. Mathematically, if a visual cue has a variance of $σ^2visual$ and the inertial cue has a variance of $σ^2inertial$ , then the variance of the combined estimate is the given by:

$σ^2combined = (σ^2visual * σ^2inertial) / (σ^2visual + σ^2inertial)$ .

One way to test for this is to generate probability density functions for the different data collected in the different experimental conditions. A probability density function for the theoretical optimal integration can be generated following the logic of optimal integration.

Figure 10: Probability density functions generated from pilot psychometric data. Colored lines represent functions generated from pilot experimental data in the three sensorimotor conditions. The black dotted line represents the statistically-optimal combination of Inertial Only and Visual Only conditions.

The analysis of single cue and combined cue psychometric functions with imposed delays provides an indication of how precise (i.e. noisy) different sensory cues are to perceive unexpected self-motion in these delayed steering movements. Comparing the distributions to the theoretical optimal combination reveals whether the observer (the participant in this case) optimally combines sensory cues or biases towards a certain cue for perception.

However, the analysis thus far does not provide any insight into the physical signals (i.e. kinematics: position, velocity, acceleration) that are being relied upon for perception. Recall that every trial involves the participant steering towards a remembered target and then steering back to the starting position. Therefore, on a trial-by-trial basis, we can extract kinematic variables (e.g. peak position, peak velocity, peak acceleration) associated with the trial and couple those variables to whether the participant perceived unexpected self-motion (‘1’ response) or did not (‘0’ response). Doing this over many trials, we build an array where each binary perceptual response (1 or 0) is associated with kinematic variables.

Figure 11: Kinematics from a sample steering trial. Position, velocity and acceleration are plotted. On a trial-by-trial basis, variables can be extracted from these kinematics (e.g., peak position, peak velocity) and be associated with the reported perceptual response (1 or 0).

Another straightforward variable to compute is the steering position error relative to the remembered target (i.e. difference between peak position and target position).

Figure 12: Position relative to target location in sample trial. The black trace represents position of the steering simulator and the red line represents the target location. Note that the target is briefly presented before the trial and then disappears, so the participant steers to a remembered target location. On a trial-by-trial basis, the target error (peak position – target position) can be calculated. In this trial, the participant slightly undershoots the target.

In addition to extracting variables from kinematics, we can also compute error signals related to the delays that are imposed into the closed loop steering control. Here, we make the assumption that after practicing to control the steering simulator in its baseline settings (i.e. no additional delay), the human brain builds a predictive model of how steering motor commands (turning the steering wheel) results in self-motion (changes in position, velocity, acceleration). Under this assumption, when an additional delay (either small or large) is imposed into the steering simulator, the resulting self-motion will be erroneous relative to the brain’s predicted self-motion. Using this framework, we can calculate error signals between the steering simulator’s actual behaviour and that which is derived from the steering wheel if no delay was added. We will define these errors as “prediction errors”, and they can be computed for each kinematic signal (e.g., velocity) of interest.

Figure 13: Actual and predicted velocity from a 350-millisecond delay sample trial. The solid black trace represents actual velocity of the steering simulator (i.e., 350 ms delay is imposed between wheel and simulator motion) and the dashed black trace represents the predicted velocity (i.e., that which would arise if no delay was imposed into the simulation). The red line represents the absolute error (difference) between the two signals. On a trial-by-trial basis, different prediction error variables (i.e., peak predicted position error, peak predicted velocity error) can be calculated and associated with the reported perception.

Just as we do for the kinematic variables, we can extract different prediction errors (e.g., peak position error, peak velocity error) for each trial. Thus, we add variables to our array, with each row representing a trial and being associated with a binary perceptual response and numerous prediction error and kinematic variables. With all of these extracted variables related to the steering behaviour, we can perform an analysis to determine which variables can best explain whether or a not a participant perceive unexpected motion.

Receiver Operating Characteristic (ROC) curves are useful ways to evaluate the performance of a binary classification model (e.g. what we have here with the perceptual task). The idea is to determine how well a given variable (e.g., peak prediction error) does at predicting whether the participant will perceive unexpected self-motion (1) or not (0). Here, we plot the true positive rate (sensitivity) against the false positive rate (1-specificity) at different decision threshold settings.

The true positive (or hit) rate measures the proportion of actual positives correctly identified by the model.

The False positive (or false alarm) rate measures the proportion of actual negatives that are incorrectly classified as positives.

The classification threshold determines the cutoff point for deciding whether a prediction is positive or negative. Varying this threshold allows for the generation of different points on the curve.

Figure 14: Sample Receiver Operating Characteristic (ROC) curve. The ROC curve was generated from participant data. The red line represents the ROC curve using peak position prediction error as the variable used for the classification model. The diagonal dashed line represents random guess behaviour.

The ROC curve helps demonstrate the trade-off between sensitivity and specificity. A model that performs well will have a curve that is closer to the top left corner of the graph (indicating high true positive, low false positive rates). The diagonal line represents random guessing, where true positive rate = false positive rate. Thus, the further the ROC curve is from this guessing line, the better the model. To obtain a quantification of the model performance, we can calculate area under the curve (AUC). And AUC of 1.0 would indicate a perfect classifier, where an area of AUC indicates random guessing.

For this analysis, we can generate an ROC curve for each extracted variable (prediction errors, target errors, kinematics) and determine which variable best predicts a participant’s perceptual detection behaviour. We can also compare how different variables perform in different sensory conditions of steering. ROC curves plotted using the responses of 5 participants are presented below.

Figure 15: Receiver Operating Characteristic (ROC) curves for different extracted variables in the Inertial+Visual condition.

Figure 16: Receiver Operating Characteristic (ROC) curves for different extracted variables in the Visual condition.

Figure 17: Receiver Operating Characteristic (ROC) curves for different extracted variables in the Inertial condition.

There is a quite a bit to interpret from these ROC curves. A few important summary points include:

Across all three sensory conditions, prediction errors (solid lines) perform better at modelling a participant’s perceptual behaviour than their kinematic counterparts (dotted lines)
The Inertial+Visual condition have the best performing curves (closest to the top right). This would be confirmed by comparing AUC values.
Certain signals become more important for perception than others, depending on the sensory condition. For example, in the Visual condition, peak acceleration error does not perform as well as peak position error or peak velocity error. In the Inertial condition, we observe that the peak acceleration error performs much better in relation to its counterparts. This can be rationalized based on our understanding of physiology and perception. Notably, the visual system is known to be more sensitive to position and velocity signals. In contrast, in the inertial conditions (where physical motion occurs), the vestibular system plays a strong role in perception. The vestibular system is known to be sensitive to velocity and acceleration. Therefore, the differences in the ROC curves across conditions can be explained using our knowledge of different sensory systems.

With this analysis, we can determine which behavioural steering signals (prediction errors, kinematics) are most important for stable and accurate perception in different sensory conditions.

Data collection is ongoing. The next update will provide updated plots and additional analysis.

__________

June 2025 Update

Study 2: Adaptation to novel sensorimotor delays during continuous steering control

General background & interest

In addition to understanding the principles of how individuals perceive their self-motion (e.g., expected vs unexpected motion) during steering activities (Study 1), it is also of interest to reveal how humans can adapt their neural control of steering when faced with novel dynamics. Study 2 explores this topic.
Building off of Study 1, additional control loop delays are imposed into the steering simulator as participants perform a continuous steering task (following a moving target).
Sensorimotor adaptation of steering control is examined by performing a frequency-based system identification approach

Introduction

Effective control during continuous motor tasks is predicated on the ability to overcome outdated estimates of self-motion that arise from delayed sensory feedback within the sensorimotor loop. Humans and other animals compensate for their physiologically-inherent sensorimotor delays by predicting future self-motion estimates and engaging efficient motor control policies. If sensorimotor delays lengthen, the execution of continuous movements suffers (Gifford and Lyman, 1967; Miall et al. 1985; Smith et al. 1960), with longer delays destabilizing motor control and eliciting large errors (Miall & Jackson, 2006; Rasman et al. 2021). Through prolonged exposure to long sensorimotor delays, however, humans can adapt their motor behaviour to reduce errors and regain some stability in movement control (Foulkes et al. 2000; Cunningham et al. 2001; Miall and Jackson, 2006; Rasman et al. 2021; Rasman et al. 2024). Revealing the mechanisms underlying this form of motor learning are of fundamental interest in movement neuroscience.

Mechanisms associated with learning to control self-motion with novel sensorimotor delays can be investigated by experimentally imposing delays into the control loop of a continuous motor task. This has often been implemented in visuomotor upper-limb experiments, where individuals use a hand-held controller (e.g, steering wheel, joystick) to steer their position (e.g., cursor on screen) and attempt to follow a moving target (Foulkes et al. 2000; Cunningham et al. 2001; Miall & Jackson, 2006). After participants perform the task in baseline conditions, delays are introduced between their motor actions and cursor motion, and participants practice performing the task under these novel sensorimotor conditions (Foulkes 2000; Cunningham et al. 2001; Miall and Jackson 2006). Following training, participants can be again exposed to baseline conditions (additional experimental delays removed) to test for the presence of aftereffects (Foulkes 2000; Miall & Jackson, 2006) – a signature of sensorimotor adaptation found across a variety of motor learning experiments (Shadmehr et al. 2010; Krakauer et al. 2019).

As an individual practices to track a target with novel sensorimotor delays in the control loop, the nervous system may recalibrate its motor controller. One possibility is for the brain to learn the new temporal relationships between motor commands and sensory feedback and adjust its predictions regarding the timing of self-motion that arises from self-generated actions. With this form of learning, individuals may shift the timing of their motor commands. However, no clear evidence of temporal shifts in sensorimotor responses have been observed from previous investigations of adaptation to novel sensorimotor delays during continuous control tasks (Foulkes 2000; Cunningam et al. 2001; Miall & Jackson, 2006; Rasman et al. 2021; Rasman et al. 2024). Another possible solution to accommodate imposed delays in the control loop is for the nervous system to adjust its sensorimotor gain across specific frequencies bandwidths. As the net delay in a closed-loop control task increases, overall performance progressively degrades, but the impact of delays will vary across frequencies because the phase shift imposed by the delay is proportional to frequency. At low frequencies, the induced phase lag is relatively small and the control system should be able to respond low-frequency inputs effectively, whereas the same delay induces a greater phase lag at higher frequencies and push the system to instability. Consequently, with additional delays imposed into the control loop, the sensorimotor controller may exhibit low-pass filtering, with a lower frequency cut-off and sharper attenuation of control gain at high frequencies as the imposed delay increases.

Another factor that may affect adaptation to new control loop delays is the sensory modalities providing relevant feedback during the task. Naturally, visual feedback of self-motion is fundamental when performing a visuomotor tracking task. In real-world activities, however, often there are additional sensory modalities providing useful feedback. When steering a vehicle, for instance, the brain not only receives visual inflow regarding self-motion but also inertial sensory cues (vestibular, somatosensory) associated with the experienced physical motion. Given that each sensory system operates with distinct temporal dynamics and frequency bandwidths, it is possible they will have differing contributions to sensorimotor delay compensation. For example, the visual system introduces neural delays of ~100-150 ms, whereas vestibular feedback has much shorter latencies of ~10-20 ms (Shlag & Shlag-Rey, 2002; De Winkel et al. 2013; Diaz-Artiles & Karmali 2021). Consequently, a condition where multiple sensory channels provide self-motion feedback may have more robust tracking performance and adaptation to delays compared to a condition with only visual feedback.

Here, we examined sensorimotor adaptation to additional delays imposed into the control loop of a target tracking task. Participants sat in a steering simulator and were tasked to track a moving target (presented on a screen) by rotating a steering wheel (Figure 1). As the protocol progressed, we gradually increased the loop delay in the steering simulation, where every trial had an additional 70 ms delays imposed between the steering wheel and experienced self-motion (baseline delay of 30 ms; maximum delay of 380 ms). Following the training trials with imposed delays, the simulator was then returned to its baseline delay setting to examine the presence and characteristics of any aftereffects. The target followed a sum-of-sines signal composed of frequencies ranging from 0.1 – 2.05 Hz, resulting in a target trajectory that was quick and unpredictable. Importantly, this sum-of-sines design of the target motion facilitated us to use a frequency-based system identification approach to characterize changes in participants’ sensorimotor controllers during the learning procedure. Additionally, we performed this experiment with two groups of participants in different sensorimotor conditions: real motion, where the steering actions physically moved the motion platform, and virtual motion, where steering actions only resulted in motion of a cursor on the screen. This allowed us to compare and contrast learning characteristics when different sensory feedback channels are available.

We hypothesized that as the delay increased, tracking performance would degrade (as observed in previous studies). We anticipated that if participants learned to predict for the novel delays between motor commands and self-motion, they would reduce tracking errors within a trial, bringing tracking performance closer to baseline. We further hypothesized that such a predictive adaptation in the neural controller would result in shifts in the timing of steering actions. If, however, adaptation to delays is only achieved through a tuning of sensorimotor gain, we then hypothesized that within-trial learning would be modest (not reaching baseline levels) and adaptation of the controller would be characterized by higher frequencies being attenuated with increasing sensorimotor delays. Finally, in regards to the different sensorimotor conditions, we hypothesized that if inertial cues (vestibular, somatosensory) provide a significant contribution to performing and adapting tracking behaviour, the real motion condition would exhibit superior reduction of error through training. Additionally, because inertial sensory cues have shorter latencies than visual feedback, we expected that compared to the virtual motion condition, the real motion condition would demonstrate less pronounced attenuation of gain at higher frequencies.

Introduction

Motivation

For humans to safely drive a vehicle, the brain must monitor how the vehicle (and the human body) moves in relation to one’s steering actions. If unexpected behavior occurs, such as skidding on a slippery road, the driver must detect the abnormal motion to respond accordingly, keep the vehicle on the desired path and avoid an accident. There is a poor understanding of how different sensory and motor cues are integrated for the brain to detect and correct for abnormal steering behavior. This project will investigate how humans use different sensory and motor information to disambiguate between normal and abnormal self-motion during active steering, and maintain stable steering performance.

Impact/Significance

How humans combine different sensory and motor information to form accurate estimates of self-motion is a fundamental question in neuroscience.
To develop and update safe driving systems, a fundamental understanding of human perception during driving activities (i.e., steering) is required.
This work will provide knowledge regarding how humans combine sensory and motor cues to monitor and updates estimates of self-motion during active steering.
The findings will have important implications for designing both human-controlled and computer-controlled (i.e., autonomous) driving systems.

Hypotheses/Aims

Through behavioral experiments and subsequent analyses, the following aims will be addressed:

Characterize how humans rely on different sensory cues (i.e., visual, vestibular) to detect and correct for abnormalities during active steering.
Determine how different sensory cues are combined (i.e., weighted and integrated) for human perception during active steering.
Determine how different kinematic signals (position, velocity, acceleration) are relied upon to discriminate between normal and abnormal steering behaviors.

Materials/Methods

Note: The general design of this study has been formed. However, the specific experimental parameters (e.g., sensory conditions, stimulus magnitude levels, number of trials) are subject to change based on pilot experimentation. Pilot experiments will be conducted in the first 2 months of this project and the experimental protocol will be finalized by month 3. Consequently, the descriptions in this section are intentionally brief.

Experimental set-up: Steering simulator

Experiments will be conducted using a custom-designed steering simulator (Figure. 1). Participants are fastened in a car seat mounted atop a linear motion platform and use a steering wheel to control their translational self-motion along the interaural axis (i.e., translation in the left-right direction). A visual display system is integrated with the motion platform to provide independent manipulation of visual feedback. Together, the steering simulator provides a closed-loop steering control representing a simplified version of vehicle driving – where the human is the controller and delivers commands by rotating the steering wheel which then moves the vehicle based on programmed dynamics. As the movement dynamics (physical motion of platform, virtual motion provided from visual display) are defined through computer programs, this allows for the manipulation of the steering environment (i.e., sensory feedback and control dynamics).

Figure 2: Steering simulator. Participants sit in a car seat and control the motion of the platform by turning the steering wheel. A visual display screen is situated in front of the motion platform to provide different forms of visual feedback.

In the experiments, participants will be tasked to perform a steering movement (e.g., similar to making a driving turn at an intersection). Through the steering simulation, I will systematically manipulate the sensory feedback and control dynamics in certain conditions. This allows for me (the experimenter) to induce transitions between the baseline steering environment (i.e., baseline control dynamics and sensory feedback) and altered steering environments (i.e., novel control dynamics, manipulated sensory feedback). For instance, to evoke abnormal steering dynamics, the control gain of the steering simulator (i.e., how self-motion velocity changes with steering wheel turns) can be decreased or enhanced (5). Through these capabilities, I will explore how individuals detect and correct for novel (i.e., abnormal) steering behaviors.

Participants will perform steering movements in three sensory environments: inertial cues only, visual cues only, inertial and visual cues. During the inertial cues only condition, participants will perform steering actions (turn steering wheel) that physically move the motion platform in complete darkness and with the eyes closed. Therefore, all the sensory information participants receive will be related to other sensors encoding self-motion (e.g., vestibular and somatosensory receptors). During the visual cues only condition, participants will perform steering actions that only move a virtual representation of the motion platform on a visual display. Finally, the inertial and visual cues condition represents a combined version of the other two conditions, where participants receive both inertial and visual information of self-motion while performing steering movements.

Study type

This is a behavioral experiment with human participants. All participants will perform three behavioral experiments in a laboratory (repeated-measures design).

Data

Registration prior to creation of data: As of the date of submission of this research plan for preregistration, the data have not yet been collected, created, or realized.

Data collection procedures

Participants will be recruited through advertisements at Radboud University campus. Participants will be compensated with gift cards, food vouchers and/or course credits (for students) for participating. Participants will be aged 19-40 years old.

Sample size

The target sample size is 20 participants. Each participant will perform three experiments (inertial cues, visual cues, inertial and visual cues), each ~2.5 hours in duration. Each experiment being conducted on a different day.

Processing and Analysis Plan

Participant perceptual responses recorded from different experimental conditions will be used to generated psychometric curves – sigmoidal functions used to characterize perceptual sensitivity and thresholds to different stimuli (6, 7). The psychometric functions obtained from these behavioral experiments will then be compared to those generated based on optimal cue integration theory (6, 7, 8).

Participant steering behavior (i.e., steering wheel and motion platform position and motion) will be processed to determine how individuals adjust their steering to abnormal control dynamics. For instance, the steering simulator velocity profiles will be compared between normal and abnormal control conditions.

Budget

The funding from this grant will be used for the following items:

Publication costs: After completion, this research will be published in an open-access journal. Publication costs in these journals are typically 1500-3000 USD.
Conference costs: I will present the results of this project at a neuroscience research conference (e.g. Neural Control of Movement). Funding will be used for conference registration and travel costs.
Participant compensation: Remaining funds will be used for participant gift cards.

If awarded the grant, ResearchHub will be acknowledged as a funder in both the manuscript publication and conference presentations.

References

Cullen KE. Vestibular processing during natural self-motion: implications for perception and action. Nature reviews Neuroscience. 2019; 20, 346-363.
Ernst MO, Banks MS. Humans integrate visual and haptic information in a statistically optimal fashion. Nature. 2002; 415(6870):429-33.
Friston K. The free-energy principle: a unified brain theory? Nature reviews Neuroscience. 2010; 11(2):127-38.
Rineau AL, Bringoux L, Sarrazin JC, Berberian B. Being active over one's own motion: Considering predictive mechanisms in self-motion perception. Neuroscience and biobehavioral reviews. 2023; 146:105051.
van Helvert M.J.L., Selen L.P.J., van Beers RJ., Medendorp W.P. Predictive steering: integration of artificial motor signals in self-motion estimation. Journal of Neurophysiology. 2022; 128:1395-1408.
ter Horst AC, Koppen M, Selen L.P.J, Medendorp W.P. Reliability-based weighting of visual and vestibular cues in displacement estimation. PLoS ONE. 2015; 10(12): e0145015.
Ernst MC, Bülthoff HH. Merging the senses into a robust percept. Trends in Cognitive Sciences. 2004; 8(4).
Dakin CJ, Rosenberg A. Gravity estimation and verticality perception. Handbook in Clinical Neurology. 2018; 159: 43-59.

Multisensory integration for human detection of abnormalities during active steering

Preprint

Preprint

Preprint

Preprint

Multisensory integration for human detection of abnormalities during active steering

Preprint

Preprint