iphoneme-brain-to-text-als-conformerxl

name: iphoneme-brain-to-text-als-conformerxl description: "iPhoneme brain-to-text communication system for ALS using ConformerXL phoneme decoder with gaze-assisted interface. Achieves 92.14% phoneme accuracy (7.86% PER) and 73.39% word accuracy on T15 intracranial EEG dataset. 180ms latency on CPU. Activation: brain-to-text, speech BCI, phoneme decoding, Conformer, ALS, intracranial EEG, iEEG."

iPhoneme: Brain-to-Text Communication for ALS Using ConformerXL Decoding

arXiv: 2604.16441
Published: 2026-04-07
Authors: Yoonmin Cha, Dawit Chun, Sung Park
Categories: cs.SD, cs.AI, cs.CL

Problem

Speech BCIs for ALS face two critical challenges:

1. Neural decoding accuracy limits practical deployment
2. Input interface design suffers from Midas touch problem (unintended selections in eye-tracking)

Despite transformative potential for 173,000-232,500 ALS patients worldwide, high-performance speech BCIs demonstrated in only 22-31 patients globally.

Core System: iPhoneme

Component 1: ConformerXL Phoneme Decoder (192.9M parameters)

Architecture

• Temporal Prenet: Multi-scale dilated convolutions + bidirectional GRU
  • Handles neural jitter correction across temporal scales
  • Dilated convolutions capture long-range temporal dependencies
• Temporal Subsampling: Reduces sequence length for CTC training stability
• 12 Encoder Blocks with Pre-RMSNorm stabilization
  • Conformer architecture combining CNN + self-attention
  • Pre-RMSNorm instead of Post-LayerNorm for training stability

Training

• Optimizer: AdamW with cosine scheduling
• Loss: CTC (Connectionist Temporal Classification) for alignment-free phoneme prediction
• 6-gram phoneme language model trained on 3.1M sequences
• WFST beam search (beam=128) for decoding

Component 2: Gaze-Assisted Phoneme Input Interface

Chorded Gaze-Plus-Silent-Speech Paradigm

• Replaces traditional dwell-time selection
• Chorded input: Combines gaze direction with silent speech attempt
• Mitigates Midas touch problem through multi-modal verification
• Enables more efficient phoneme input rate

Key Results

T15 Dataset (256-channel intracranial EEG)

• ~3% above prior state-of-the-art
• Real-time operation on standard CPU hardware

Technical Details

Data

• T15 dataset: 45 sessions, 8,071 trials
• 256-channel intracranial EEG from speech motor cortex regions
• Intracranial (iEEG/ECoG) signals — higher SNR than scalp EEG

Phoneme Language Model

• 6-gram model trained on 3.1M phoneme sequences
• Integrated via Weighted Finite-State Transducer (WFST)
• Beam search with beam width = 128 for efficient decoding

Neural Jitter Correction

• Temporal prenet with multi-scale dilated convolutions handles timing variability
• Bidirectional GRU captures forward/backward temporal context
• Critical for handling non-deterministic neural response timing

Reusable Methodology

1. ConformerXL for Neural Signal Decoding

# Architecture pattern
Input → TemporalPrenet(dilated_conv + BiGRU) 
     → Subsampling 
     → 12x ConformerBlock(Pre-RMSNorm)
     → CTC Loss

2. Gaze-Assisted Interface Design

• Chorded paradigm: gaze_direction + silent_speech → phoneme selection
• Dual verification prevents unintended inputs
• Applicable to other BCI modalities

3. Phoneme-Level Brain-to-Text Pipeline

1. Record iEEG from speech motor cortex
2. Temporal preprocessing with jitter correction
3. ConformerXL phoneme prediction
4. WFST beam search with language model
5. Phoneme-to-text conversion

Applications

• ALS communication: Primary target for speech restoration
• Locked-in syndrome: Brain-to-text for completely paralyzed patients
• Speech neuroprosthetics: General speech BCI applications
• Real-time BCI: 180ms latency enables conversational use

Datasets

• T15: 256-channel intracranial EEG
  • 45 recording sessions
  • 8,071 trials total
  • Speech motor cortex coverage

Key Innovations

1. ConformerXL adaptation for neural signal phoneme decoding (192.9M params)
2. Multi-scale temporal prenet for neural jitter correction
3. Chorded gaze-plus-silent-speech interface replacing dwell-time
4. CPU real-time operation at 180ms latency
5. State-of-the-art phoneme (92.14%) and word (73.39%) accuracy

Limitations

• Requires intracranial EEG (invasive) — not applicable to non-invasive BCI
• Performance on limited patient population
• Language model trained on English phonemes only
• 192.9M parameters — large model size

Related Skills

• brain-to-speech-prosody-feature-engineering: Brain-to-speech synthesis
• brain-to-speech-transformer-reconstruction: Speech reconstruction from brain signals
• eeg-foundation-model-adapters: EEG foundation models with adaptation
• neural-population-decoding: Neural population decoding methods