fmri-gesture-reconstruction

name: fmri-gesture-reconstruction description: "fMRI2GES: Dual brain decoding alignment framework for co-speech gesture reconstruction from fMRI signals. Maps brain responses to external stimuli and decodes gestural behavior through dual-alignment brain decoding. Activation: fMRI gesture reconstruction, brain-to-gesture decoding, fMRI2GES, co-speech gesture brain decoding, dual brain decoding alignment."

fMRI Gesture Reconstruction (fMRI2GES)

Dual brain decoding alignment framework for reconstructing co-speech gestures from fMRI signals, advancing brain-to-behavior decoding beyond traditional speech/motor paradigms.

Metadata

• Source: arXiv:2512.01189
• Authors: Chunzheng Zhu, Jialin Shao, Jianxin Lin, Yijun Wang, Jing Wang, Jinhui Tang
• Published: 2025-11-30
• Categories: Not specified in abstract

Core Methodology

Key Innovation

Extends brain decoding beyond speech/text reconstruction to gesture reconstruction — decoding the physical gestural behavior that accompanies speech from fMRI signals. Uses dual alignment strategy to improve decoding fidelity.

Technical Framework

1. Dual Brain Decoding Alignment

   • Aligns fMRI signals to both gesture kinematics and speech features
   • Two-stage alignment: brain-to-gesture + brain-to-speech cross-modal mapping
   • Leverages shared neural representations between speech and gesture production

2. Gesture Reconstruction Pipeline

   • Input: fMRI time series during speech-with-gesture tasks
   • Intermediate: Aligned latent representations
   • Output: Reconstructed gesture sequences (motion trajectories)

3. Cross-Modal Learning

   • Joint training on gesture and speech decoding objectives
   • Shared encoder captures multimodal neural representations
   • Separate decoders for gesture kinematics and speech features

Applications

• Brain-computer interfaces for gesture communication
• Understanding neural basis of co-speech gesture production
• Neurorehabilitation for speech-gesture coordination deficits
• Multimodal brain decoding research

Implementation Guide

Prerequisites

• fMRI data with gesture annotation
• Motion capture or gesture tracking data
• Deep learning framework for sequence modeling

Step-by-Step

1. Collect fMRI data during natural speech-with-gesture tasks
2. Annotate gesture sequences (kinematics, timing, type)
3. Build dual-alignment encoder for fMRI-to-gesture mapping
4. Train with joint gesture + speech decoding objectives
5. Evaluate gesture reconstruction quality against ground truth
6. Analyze neural regions contributing to gesture decoding

Pitfalls

• fMRI temporal resolution limits gesture reconstruction precision
• Requires synchronized fMRI and motion capture data
• Gesture annotation is labor-intensive
• Cross-modal alignment may be sensitive to individual variability
• Limited generalizability across subjects without adaptation

Related Skills

• brain-dit-fmri-foundation-model
• brain-to-speech-prosody-feature-engineering
• eeg2vision-multimodal-eeg-framework-2d-visual
• visual-imagery-decoding-fmri