Search
3 results for "entropy"
- ai · arxiv/cs.AI · 5 min
Fast Entropic Approximations cut entropy computation by 37x
Horenko et al. propose non-singular rational approximations of Shannon entropy and KL divergence that preserve mathematical properties while reducing computation cost and improving ML model training.
Apr 27, 2026 Read → - ai · arxiv/cs.AI · 4 min
Cross-Entropy Loss Drives Neural Probe Performance, Not Architecture
Pre-registered study shows cross-entropy training inflates logit norms 15x, accounting for most K-way energy probe gains over softmax baselines.
Apr 24, 2026 Read → - ai · arxiv/cs.AI · 8 min
Token Importance in On-Policy Distillation: Entropy and Disagreement
Research identifies two regions of high-value tokens in knowledge distillation: high-entropy positions and low-entropy positions where student and teacher disagree, enabling 50–80% token reduction.
Apr 17, 2026 Read →