Characterizing State Space Model and
Hybrid Language Model Performance
with Long Context

Abstract

Why SSMs for Long-Context Inference?

At short sequence lengths, heavily optimized Transformer models such as Qwen2.5 outperform SSMs. However, as context length grows, the quadratic memory and compute cost of attention creates an insurmountable bottleneck. The figure below encapsulates this performance crossover on an RTX 4090, comparing Qwen2.5-0.5B (Transformer) and Mamba2-780m (SSM) across representative prefill and decode workloads.

Figure 1: TTFT (a, b) and TPOT (c, d) scaling comparison of Qwen2.5-0.5B (Transformer) and Mamba2-780m (SSM). While Qwen is faster (1.9×) at shorter sequence lengths (<8K), Mamba2's superior scaling provides a significant performance advantage (2.65×–3×) at longer contexts for both prefill and decode stages (generation length 256, batch size 1).

Architectures & Accuracy–Latency Trade-offs

We evaluate three primary model families: (1) Transformers, which use scaled dot-product attention with KV-cache and O(n²) complexity; (2) SSMs (Mamba, Mamba-2), which use a selective state mechanism with O(n) complexity and constant O(1) memory during generation; and (3) Hybrid models (Zamba2, Hymba, Falcon-H1), which interleave attention and SSM layers to combine accuracy and efficiency.

Figure 2: (a) Basic building block of Transformers: a scaled dot-product attention module; (b) a S6 block showing the fundamental computation of SSMs; (c) Overview of auto-regressive generation (prefill & decode) in LLMs.

While pure SSMs achieve the fastest inference, they sacrifice accuracy on knowledge-intensive tasks. Hybrid models like Falcon-H1 bridge this gap — achieving accuracy on par with or exceeding Transformer baselines while maintaining a 2.8× TTFT speedup.

Figure 3: Accuracy–latency efficiency frontier of Transformer (Qwen2.5-1.5B), SSM (Mamba2-1.3B), and Hybrid (Falcon-H1-1.5B) models at a 57K-token prefill stage on RTX 4090. Hybrid models achieve Pareto-optimal balance: Falcon-H1 matches/surpasses Transformer accuracy with 2.8× lower TTFT.

SSM-Scope: Characterization Framework

SSM-Scope is an open-source, backend-agnostic performance characterization framework for language models. It takes a workload configuration (model registry entry, sequence lengths, batch sizes) and profiling configuration as input, and produces end-to-end and operator-level breakdowns for latency, memory, and energy metrics.

Figure 4: Overall flow of the SSM-Scope characterization framework. The framework includes a flexible model registry, data preprocessing module, and a performance profiling engine that captures operator-level breakdown via PyTorch Profiler or TensorRT's trtexec.

Evaluation Platforms

Jetson Orin Nano evaluated in MAXN power mode with 16 GB NVMe swap enabled.

Evaluated Models

Memory Footprint & Scaling Limits

We characterize peak GPU memory across sequence lengths from 1K to over 220K tokens. Transformers (Qwen2.5, Llama3.2) reach their out-of-memory (OOM) limits at 57K–65K tokens on the RTX 4090 due to the growing KV-cache. By contrast, SSMs carry no KV-cache and their activation memory scales linearly at a much lower rate, allowing Mamba and Mamba-2 (~790M) to run at up to 220K tokens within 24 GB. Falcon-H1 extends to ~164K tokens, far beyond same-accuracy Transformer models. On the edge Jetson platform, SSMs and hybrids again show clear advantage, supporting contexts >16K where Transformers cannot operate.

Figure 5: Memory footprint of the prefill stage for Transformer, SSM, and Hybrid models on (a) consumer GPU (RTX 4090) and (b) edge GPU (Jetson Orin Nano). Stacked bars show weight, KV-cache, and activation memory. Bars stop when OOM is hit. SSMs operate at up to 4× longer context than the best-performing Transformer.

Energy Consumption & End-to-End Throughput

We measure total inference energy (J) and end-to-end token throughput (tokens/s) across contexts from 1K to 57K tokens on the RTX 4090, for three representative models: Qwen2.5-0.5B (Transformer), Mamba2-780m (SSM), and Falcon-H1-0.5B (Hybrid).

At short sequences (<16K), Transformers benefit from highly optimized GEMM kernels and consume comparable or less energy. However, beyond this threshold the quadratic compute burden reverses the trend dramatically. At 57K tokens, Qwen2.5 consumes 1,492 J — approximately 4× more than Mamba-2 (370 J). The hybrid Falcon-H1 (613 J) offers a balanced profile: 2.4× more energy-efficient than Transformer while maintaining higher accuracy. For throughput, Mamba2 achieves 2.64× and Falcon-H1 achieves 1.54× the throughput of Qwen2.5 at 32K sequence length.

Figure 6: (a) Inference energy consumption (log scale) and (b) end-to-end throughput of Qwen2.5-0.5B, Mamba2-780m, and Falcon-H1-0.5B over sequence length on RTX 4090. SSMs and hybrids achieve dramatically better energy efficiency and throughput at long contexts.

Operator-Level Performance Analysis

To understand the runtime composition of SSMs and hybrid models, we classify all operators into three categories: SSM-specific kernels (e.g., mamba_inner_fn, mamba_split_conv1d_scan_combined_fn), GEMM-based operators (e.g., matmul, linear), and non-GEMM operators (normalization, memory layout, activations). We profile latency breakdown across sequence lengths from 256 to 131K tokens on both the consumer and edge GPUs.

Figure 7: Latency breakdown of SSMs by operator class on RTX 4090 for (a) Mamba-130M (first generation) and (b) Mamba-2 (second generation). As sequence length increases, GEMM operators grow while SSM-specific kernels remain a near-constant dominant share. Mamba-2's larger state dimensions (64 vs. 16) make its selective scan operator more expensive.

Key Takeaways from Operator Analysis

Citation

If you find SSM-Scope or this work useful in your research, please consider citing:

@article{mitra2025characterizing,
  title   = {Characterizing State Space Model (SSM) and SSM-Transformer Hybrid
             Language Model Performance with Long Context Length},
  author  = {Mitra, Saptarshi and Karami, Rachid and Xu, Haocheng and
             Huang, Sitao and Kwon, Hyoukjun},
  journal = {arXiv preprint arXiv:2507.12442},
  year    = {2025}
}