memory-uncertainty-relation-recurrent-networks

name: memory-uncertainty-relation-recurrent-networks description: > Memory Uncertainty Relation in random recurrent networks: inequality bounding short-term memory from below as an uncertainty relation between memory capacity and state-space fluctuations. Defines harmonic memory as an analytically tractable lower bound achieved by optimal readout weights. triggers: - short-term memory recurrent networks - reservoir computing memory capacity - harmonic memory - uncertainty relation neural dynamics - state-space fluctuations - random recurrent network memory - echo state network capacity - dynamical systems memory bounds category: ai_collection tags: - recurrent-networks - reservoir-computing - memory-capacity - neural-dynamics - information-theory - dynamical-systems - cs-NE

Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks

Overview

Paper: "Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks"
Authors: Taichi Haruna, Kohei Nakajima
arXiv: 2605.24628
Published: 2026-05-23
Categories: nlin.AO, cond-mat.dis-nn, cs.NE

Core Contribution

This work establishes a fundamental uncertainty-type inequality for short-term memory in dynamical systems:

$$\text{STM}(\mathcal{S}) \cdot \text{Fluct}(\mathcal{S}) \geq C$$

Where:

• STM: Short-term memory capacity measure
• Fluct: Size of state-space fluctuations induced by input signals
• C: A positive constant (lower bound)

This is analogous to Heisenberg's uncertainty principle — you cannot simultaneously minimize both memory and fluctuations below the bound.

Key Definitions

Short-Term Memory (STM) Capacity

Following Jaeger (2001), STM measures how well a reservoir can recall past inputs:

$$\text{STM} = \sum_{k=1}^{\infty} r^2(s_t, u_{t-k})$$

Where $r^2$ is the coefficient of determination between the state $s_t$ and a delayed input $u_{t-k}$.

Harmonic Memory

The harmonic memory $H(\mathcal{S})$ is the suboptimal lower bound achievable by the best linear readout weight:

$$H(\mathcal{S}) = \frac{\left(\sum_k \text{cov}(s_t, u_{t-k})\right)^2}{\text{var}(s_t)}$$

This is always achievable (tight) — making it a constructive lower bound.

State-Space Fluctuation

Measured as the variance of the reservoir state induced by the input process:

$$\text{Fluct}(\mathcal{S}) = \text{Tr}[\text{Var}(s_t)]$$

Main Results

Theorem 1: Memory Uncertainty Inequality

For any reservoir system $\mathcal{S}$ with state process $s_t$ driven by input $u_t$:

$$\text{STM}(\mathcal{S}) \geq H(\mathcal{S}) \geq \frac{C}{\text{Fluct}(\mathcal{S})}$$

Interpretation: To have high memory capacity, you need either large fluctuations OR the system must be close to the harmonic optimum.

When Equality Holds

1. Exact equality: When the state is a simple linear function of past inputs
2. Asymptotic equality: When reservoir spectral radius → 1 (edge of chaos)
3. Strict inequality: For nonlinear reservoirs with complex internal dynamics

Effect of State-Space Regularization

Adding L2 regularization to reservoir states:

• Reduces fluctuations
• May decrease memory capacity proportionally
• The ratio STM/Fluct is conserved — the bound is "tight" under regularization

Implementation Guide

Measuring STM in a Reservoir

import numpy as np
from sklearn.linear_model import LinearRegression

def measure_stm(states, inputs, max_delay=50):
    """Measure short-term memory capacity of a reservoir.
    
    Args:
        states: (T, N) reservoir states
        inputs: (T,) input time series
        max_delay: maximum delay to consider
    
    Returns:
        stm: total STM capacity
        r2_per_delay: R² for each delay
    """
    T, N = states.shape
    r2_per_delay = []
    
    for k in range(1, max_delay + 1):
        # Align states with delayed inputs
        s = states[k:, :]
        u_delayed = inputs[:T - k].reshape(-1, 1)
        
        # Fit linear readout
        reg = LinearRegression().fit(s, u_delayed)
        r2 = reg.score(s, u_delayed)
        r2_per_delay.append(max(0, r2))  # Clamp to [0, 1]
    
    stm = sum(r2_per_delay)
    return stm, r2_per_delay


def measure_harmonic_memory(states, inputs, max_delay=50):
    """Compute harmonic memory lower bound.
    
    Args:
        states: (T, N) reservoir states (use single-neuron for simplicity)
        inputs: (T,) input time series
    
    Returns:
        harmonic_memory: lower bound on STM
    """
    T, N = states.shape
    
    # Sum of covariances: sum_k cov(s_t, u_{t-k})
    cov_sum = 0
    for k in range(1, min(T // 2, 200)):
        cov = np.cov(states[k:, 0], inputs[:T - k])[0, 1]
        cov_sum += cov
    
    # Variance of state
    state_var = np.var(states[:, 0])
    
    if state_var < 1e-10:
        return 0.0
    
    return (cov_sum ** 2) / state_var


def measure_fluctuation(states):
    """Measure state-space fluctuation (trace of covariance)."""
    return np.trace(np.cov(states.T))

Reservoir with Tunable Spectral Radius

class EchoStateNetwork:
    """Simple Echo State Network for memory capacity experiments."""
    
    def __init__(self, N=100, spectral_radius=0.9, input_scaling=0.1, 
                 density=0.1, seed=42):
        rng = np.random.RandomState(seed)
        
        # Random recurrent weights
        W = rng.randn(N, N)
        # Sparse connectivity
        mask = rng.uniform(0, 1, (N, N)) < density
        W *= mask
        # Scale to desired spectral radius
        sr = np.max(np.abs(np.linalg.eigvals(W)))
        self.W = spectral_radius * W / sr
        
        # Input weights
        self.W_in = input_scaling * rng.randn(N, 1)
        
        self.N = N
        self.spectral_radius = spectral_radius
    
    def run(self, inputs, washout=100):
        """Run reservoir on input sequence.
        
        Args:
            inputs: (T,) input time series
            washout: initial steps to discard
        
        Returns:
            states: (T - washout, N) reservoir states
        """
        T = len(inputs)
        states = np.zeros((T, self.N))
        x = np.zeros(self.N)
        
        for t in range(T):
            x = np.tanh(self.W @ x + self.W_in.flatten() * inputs[t])
            states[t] = x
        
        return states[washout:]
    
    def memory_profile(self, inputs, max_delay=100):
        """Full memory capacity profile."""
        states = self.run(inputs, washout=200)
        stm, r2s = measure_stm(states, inputs[200:], max_delay)
        harmonic = measure_harmonic_memory(states, inputs[200:], max_delay)
        fluct = measure_fluctuation(states)
        
        return {
            'stm': stm,
            'harmonic_memory': harmonic,
            'fluctuation': fluct,
            'r2_profile': r2s,
            'memory_uncertainty_product': stm * fluct,
        }

Analyzing the Uncertainty Bound

import matplotlib.pyplot as plt

def analyze_memory_uncertainty(spectral_radii=np.linspace(0.5, 0.99, 20)):
    """Analyze how the memory-fluctuation trade-off changes with spectral radius."""
    
    results = []
    rng = np.random.RandomState(0)
    inputs = rng.randn(2000)
    
    for sr in spectral_radii:
        esn = EchoStateNetwork(N=100, spectral_radius=sr)
        profile = esn.memory_profile(inputs)
        profile['spectral_radius'] = sr
        results.append(profile)
    
    # Plot
    fig, axes = plt.subplots(1, 3, figsize=(15, 4))
    
    sr_vals = [r['spectral_radius'] for r in results]
    stms = [r['stm'] for r in results]
    flucts = [r['fluctuation'] for r in results]
    products = [r['memory_uncertainty_product'] for r in results]
    
    axes[0].plot(sr_vals, stms, 'b-o', label='STM')
    axes[0].set_xlabel('Spectral Radius')
    axes[0].set_ylabel('STM Capacity')
    axes[0].set_title('Memory vs Spectral Radius')
    
    axes[1].plot(sr_vals, flucts, 'r-o', label='Fluctuation')
    axes[1].set_xlabel('Spectral Radius')
    axes[1].set_ylabel('State Fluctuation')
    axes[1].set_title('Fluctuation vs Spectral Radius')
    
    axes[2].plot(sr_vals, products, 'g-o', label='STM × Fluct')
    axes[2].axhline(y=min(products), color='k', linestyle='--', label='Lower Bound')
    axes[2].set_xlabel('Spectral Radius')
    axes[2].set_ylabel('STM × Fluctuation')
    axes[2].set_title('Uncertainty Product (should be ≥ bound)')
    axes[2].legend()
    
    plt.tight_layout()
    return results, fig

Theoretical Implications

For Reservoir Computing Design

Connection to Physics

• Analogous to Heisenberg uncertainty: ΔxΔp ≥ ℏ/2
• Here: STM × Fluct ≥ C
• The "harmonic memory" is the minimum uncertainty state — like a coherent state in QM

Practical Guidelines

1. Choose spectral radius near 1 for best memory efficiency
2. Use harmonic memory as a diagnostic: if STM ≈ H, the reservoir is close to optimal
3. Regularization trade-off: L2 regularization reduces fluctuations but proportionally reduces memory
4. Nonlinearity: Strong nonlinear activation (tanh saturation) increases the STM/H gap

Applications

1. Reservoir computing optimization: Tune spectral radius to the harmonic optimum
2. Memory-efficient RNNs: Design architectures that approach the harmonic bound
3. Cognitive modeling: Memory capacity bounds as a model for working memory limitations
4. Anomaly detection: Compare measured STM vs harmonic bound to detect sub-optimal network states

Pitfalls

• Washout period: Always discard initial transients when measuring STM
• Input statistics matter: The bound depends on input process statistics
• Finite-time estimation: Need long time series for accurate STM estimation (T ≥ 10 × max_delay)
• Nonlinear reservoirs: The harmonic bound may be loose for highly nonlinear systems

Citation

@article{haruna2026memory,
  title={Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks},
  author={Haruna, Taichi and Nakajima, Kohei},
  journal={arXiv preprint arXiv:2605.24628},
  year={2026}
}