Mixed Precision Training Stabilizes Neural ODEs
Researchers demonstrate a framework that reduces memory use by 50% and speeds up neural ODE training 2x by carefully mixing low and high precision arithmetic.
Source: arxiv/cs.LG · Elena Celledoni, Brynjulf Owren, Lars Ruthotto, Tianjiao Nicole Yang · open original ↗
Mixed precision training for neural ODEs reduces memory and computation while maintaining accuracy through selective precision management.
- — Standard mixed precision fails for neural ODEs due to accumulated roundoff errors in iterative solvers.
- — Framework uses low precision for network velocity evaluation and intermediate state storage.
- — High precision accumulation for gradients and solutions prevents numerical instability.
- — Custom dynamic adjoint scaling addresses gradient growth across time steps.
- — Achieves 50% memory reduction and up to 2x speedup on image classification and generative tasks.
- — Open-source PyTorch package (rampde) provides drop-in replacement for existing code.
- — Explicit ODE solvers paired with custom backpropagation enable the precision switching strategy.
Astrobobo tool mapping
- Knowledge Capture Document the precision strategy (where low/high precision is used) and the adjoint scaling technique for your team's ODE training playbook.
- Focus Brief Summarize the three key innovations (low-precision velocity, high-precision accumulation, dynamic adjoint scaling) as a one-page reference for code review.
- Reading Queue Queue related papers on gradient checkpointing and other memory-reduction techniques to compare trade-offs with this mixed-precision approach.
Frequently asked
- Neural ODEs solve differential equations iteratively over many time steps. Roundoff errors from low-precision arithmetic accumulate across these iterations, causing numerical instability. Standard mixed precision, which only protects weights, does not account for the error growth in the solution trajectory itself. This framework adds high-precision accumulation of gradients and solutions to mitigate that problem.
cite ▸
APA
Elena Celledoni, Brynjulf Owren, Lars Ruthotto, Tianjiao Nicole Yang. (2026, May 3). Mixed Precision Training Stabilizes Neural ODEs. Astrobobo Content Engine (rewrite of arxiv/cs.LG). https://astrobobo-content-engine.vercel.app/article/mixed-precision-training-stabilizes-neural-odes-46f8d7
MLA
Elena Celledoni, Brynjulf Owren, Lars Ruthotto, Tianjiao Nicole Yang. "Mixed Precision Training Stabilizes Neural ODEs." Astrobobo Content Engine, 3 May 2026, https://astrobobo-content-engine.vercel.app/article/mixed-precision-training-stabilizes-neural-odes-46f8d7. Based on "arxiv/cs.LG", https://arxiv.org/abs/2510.23498.
BibTeX
@misc{astrobobo_mixed-precision-training-stabilizes-neural-odes-46f8d7_2026,
author = {Elena Celledoni, Brynjulf Owren, Lars Ruthotto, Tianjiao Nicole Yang},
title = {Mixed Precision Training Stabilizes Neural ODEs},
year = {2026},
url = {https://astrobobo-content-engine.vercel.app/article/mixed-precision-training-stabilizes-neural-odes-46f8d7},
note = {Astrobobo rewrite of arxiv/cs.LG, https://arxiv.org/abs/2510.23498},
}