mrine-multiscale-realtime-neural-decoding - SKILL.md Agent Skill

name: mrine-multiscale-realtime-neural-decoding description: Multiscale Recurrent Inference Network for Encoding (MRINE) - real-time nonlinear latent factor modeling for multimodal neural activity decoding with different timescales and missing samples version: 1.0.0 author: Eray Erturk, Maryam M. Shanechi arxiv_id: 2512.12462 paper_title: Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference published_date: 2025-12-13 conference: NeurIPS 2025 github: https://github.com/ShanechiLab/mrine keywords: [neural decoding, real-time inference, multimodal neural activity, multiscale dynamics, latent factors, brain-computer interface, missing data handling] tags: [neuroscience, machine-learning, neural-decoding, real-time, multimodal, dynamical-systems]

MRINE: Multiscale Recurrent Inference Network for Real-time Neural Decoding

Overview

MRINE (Multiscale Recurrent Inference Network for Encoding) is a novel framework for real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities with different timescales, distributions, and missing samples. Published at NeurIPS 2025.

Key Innovation: Addresses the challenge of aggregating information across neural modalities (spiking activity, field potentials) that have different sampling rates and probabilistic distributions while enabling real-time recursive decoding.

Core Components

1. Multiscale Encoder

Purpose: Nonlinearly aggregates information after learning within-modality dynamics
Handles: Different timescales (sampling rates) across modalities
Features: Missing sample resilience - can operate with incomplete data streams
Mechanism: Modality-specific encoding that respects temporal dynamics

2. Multiscale Dynamical Backbone

Purpose: Extracts multimodal temporal dynamics
Enables: Real-time recursive decoding (critical for BCI applications)
Architecture: Dynamical system that captures cross-modal interactions
Advantage: Online inference without requiring full batch processing

3. Modality-Specific Decoders

Purpose: Account for different probabilistic distributions across modalities
Design: Tailored to discrete spiking vs. continuous field potential distributions
Output: Unified decoded target variables from heterogeneous inputs

Methodology

Mathematical Framework

The MRINE framework models multimodal neural activity as:

Z_t = f(X_t^{(1)}, X_t^{(2)}, ..., X_t^{(M)})

Where:

Z_t = latent state at time t
X_t^{(m)} = neural activity from modality m
f = nonlinear aggregation function

Key features:

Recursive estimation: Kalman-like update with nonlinear dynamics
Missing data handling: Probabilistic imputation within the dynamical framework
Timescale alignment: Temporal interpolation across modalities with different sampling rates

Training Process

Modality-specific encoding: Learn dynamics for each neural modality independently
Cross-modal aggregation: Train multiscale encoder to combine information
Target decoding: Learn decoders for specific task variables (movement, behavior, etc.)
Online adaptation: Recursive updates during real-time inference

Applications

Brain-Computer Interfaces (BCI)

Real-time movement decoding from intracranial recordings
Motor imagery classification with multimodal EEG/ECoG
Adaptive neural prosthetics control

Neuroscience Research

Cross-modal neural correlation analysis
Dynamical state estimation from heterogeneous recordings
Missing data robustness in chronic recordings

Clinical Applications

Motor rehabilitation monitoring
Seizure prediction from multimodal signals
Consciousness state decoding

Implementation Details

Architecture Specifications

Encoder depth: Modality-specific RNN/Transformer layers
Backbone type: Nonlinear state-space model
Decoder heads: Task-specific linear/nonlinear projections
Missing data mask: Binary indicator per modality per timestep

Computational Requirements

Training: Requires full multimodal datasets with alignment
Inference: Real-time capable (sub-millisecond latency achievable)
Memory: Efficient recurrent updates, no batch history needed
Hardware: GPU-accelerated training, CPU-sufficient inference

Key Hyperparameters

τ_m = timescale parameter for modality m
λ_missing = missing data handling coefficient
α_aggregation = cross-modal aggregation weight
β_nonlinearity = nonlinear activation strength

Experimental Validation

Dataset Performance

The framework was validated on three distinct multiscale brain datasets:

Motor cortex recordings: Spiking + LFP decoding for movement trajectories
Hippocampal activity: Multi-region ensemble decoding for memory tasks
Clinical iEEG: Wideband + spike decoding for seizure prediction

Benchmarks

Linear methods: PCA, CCA, Linear Dynamical Systems
Nonlinear baselines: RNNs, Transformers, Deep Encoders
Multimodal baselines: Cross-modal attention, multimodal VAEs

Results: MRINE consistently outperformed benchmarks in:

Real-time decoding accuracy (+15-25% vs. linear methods)
Missing data robustness (maintained performance with 30% missing samples)
Cross-modal integration (better than single-modality or simple concatenation)

Advantages over Existing Methods

vs. Linear Dynamical Systems

Nonlinear latent factors capture richer dynamics
Better performance on complex neural manifolds
Adaptive to distributional differences across modalities

vs. Deep Neural Networks (RNN/Transformer)

Real-time recursive inference (no batch processing needed)
Explicit timescale handling (not just temporal convolutions)
Missing sample robustness (not handled by standard architectures)

vs. Standard Multimodal Fusion

Distribution-aware decoding (Gaussian vs. Poisson vs. others)
Timescale-aware aggregation (not just temporal alignment)
Dynamical consistency (state-space structure)

Implementation Workflow

Step 1: Data Preparation

# Align multimodal neural recordings
spiking_data = load_spike_trains(channel_ids, sampling_rate=1000)
lfp_data = load_lfp(channel_ids, sampling_rate=250)

# Create missing data masks
missing_mask_spiking = detect_missing_samples(spiking_data)
missing_mask_lfp = detect_missing_samples(lfp_data)

Step 2: Model Configuration

from mrine import MRINEncoder, MRINEDecoder

# Configure modality-specific encoders
encoder_spike = MRINEncoder(
    modality_type='spiking',
    distribution='poisson',
    timescale=1.0,  # 1ms resolution
    hidden_dim=128
)

encoder_lfp = MRINEncoder(
    modality_type='continuous',
    distribution='gaussian',
    timescale=4.0,  # 4ms resolution (250Hz)
    hidden_dim=64
)

# Create multiscale backbone
backbone = MRINEBackbone(
    encoders=[encoder_spike, encoder_lfp],
    latent_dim=32,
    dynamical_type='nonlinear_rnn'
)

# Create decoder for target variable
decoder = MRINEDecoder(
    target_type='continuous',  # movement trajectory
    output_dim=3  # x, y, z coordinates
)

Step 3: Training

# Train on aligned multimodal data
mrine_model = MRINE(backbone, decoder)

mrine_model.train(
    neural_data=[spiking_data, lfp_data],
    target_data=movement_trajectories,
    missing_masks=[missing_mask_spiking, missing_mask_lfp],
    epochs=100,
    batch_size=256
)

Step 4: Real-time Inference

# Online decoding with recursive updates
def realtime_decode(current_spike, current_lfp, missing_flags):
    latent_state = mrine_model.update_latent(
        current_spike, current_lfp,
        missing_flags,
        recursive=True  # Use previous state
    )
    
    decoded_target = mrine_model.decode_target(latent_state)
    return decoded_target

Key Insights & Principles

1. Timescale Matters

Different neural modalities capture different temporal dynamics:

Spiking: Fast, discrete events (millisecond resolution)
LFP/ECoG: Slower, continuous oscillations (hundreds of Hz)
EEG: Even slower population dynamics

Implication: Simple concatenation or temporal interpolation fails - need dynamical alignment.

2. Missing Data is Common

Real-world neural recordings frequently have:

Channel dropouts (electrode failure)
Artifact rejection periods
Intermittent wireless transmission gaps

Implication: Decoding must be robust to incomplete data without catastrophic failure.

3. Distributional Heterogeneity

Spiking ≈ Poisson, LFP ≈ Gaussian, others may be different:

Linear methods assume Gaussianity
Deep networks ignore distributional differences
MRINE explicitly models per-modality distributions

Implication: Better statistical modeling improves decoding accuracy.

4. Real-time Requirement

BCI and clinical monitoring need sub-second latency:

Batch processing (Transformers) has latency overhead
Standard RNNs don't handle timescale differences
MRINE designed for online recursive inference

Implication: Architecture must support causal, online updates.

Extensions & Future Work

Potential Extensions

Adaptive timescale learning: Automatically discover optimal timescales per modality
Non-stationarity handling: Adapt to changing neural dynamics over time
Multi-task decoding: Decode multiple target variables simultaneously
Uncertainty quantification: Bayesian extensions for confidence estimates

Research Directions

Causal discovery: Use MRINE latent factors to infer cross-modal causal relationships
Transfer learning: Pre-train on large datasets, adapt to individual subjects
Model compression: Optimize for edge deployment in implantable devices
Interpretability: Visualize latent dynamics and cross-modal interactions

Limitations & Considerations

Current Limitations

Training complexity: Requires aligned multimodal datasets (data collection challenge)
Hyperparameter sensitivity: Timescale parameters need tuning per application
Modality assumptions: Currently tested on spiking + LFP, extension to EEG/MEG needs validation
Computational cost: Training is expensive, though inference is efficient

Deployment Considerations

Hardware constraints: Edge deployment may require model quantization
Calibration: Individual subject variation requires per-subject training/fine-tuning
Drift adaptation: Long-term recordings may need periodic retraining
Safety margins: Clinical applications need robustness guarantees

References

Paper: Erturk, E., & Shanechi, M. M. (2025). Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference. NeurIPS 2025. arXiv:2512.12462
Code: https://github.com/ShanechiLab/mrine
Related Work: Shanechi Lab's prior work on neural decoding and dynamical modeling

Activation Triggers

Use this skill when:

Implementing real-time neural decoding systems
Handling multimodal neural recordings (spiking + LFP + EEG/ECoG)
Dealing with missing data in neural recordings
Building brain-computer interfaces with multiple signal types
Researching cross-modal neural dynamics
Comparing linear vs. nonlinear decoding methods

Keywords: real-time decoding, multimodal neural, missing data, neural decoding, multiscale dynamics, BCI, latent factors, MRINE, Shanechi Lab, timescale alignment

Quick Reference

Method Name: MRINE (Multiscale Recurrent Inference Network for Encoding)
Innovations: Real-time + multiscale + missing data + distribution-aware
Performance: +15-25% vs. baselines on three brain datasets
Availability: Open-source code on GitHub
Conference: NeurIPS 2025