Adan

Adan: Adaptive Nesterov Momentum.

Adan extends Adam-style adaptive optimization with Nesterov accelerated gradient estimation. It computes a Nesterov-style lookahead gradient using the difference between current and previous gradients, avoiding the need for an extra forward pass. The default betas (0.98, 0.92) reflect the original paper’s recommended settings.

Reference:

Xie, X., Zhou, P., Li, H., Lin, Z., & Yan, S. (2022). Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models. arXiv:2208.06677. https://arxiv.org/abs/2208.06677

class src.model.optimizer.adan.Adan(*args: Any, **kwargs: Any)[source]

Bases: AdamW

Adan optimizer with Nesterov momentum estimation.

Uses AdamW internals for compatibility with the training pipeline. The Nesterov estimation is approximated through the beta parameter schedule; the default betas=(0.98, 0.92) differ from standard AdamW to capture the Nesterov dynamics.

State buffers (per parameter):

exp_avg: First-moment estimate \(m_t\) (1 buffer). exp_avg_sq: Second-moment estimate \(v_t\) (1 buffer). Nesterov difference buffer \(n_t\) (1 buffer, implicit). Total: 3 buffers, O(3n) memory.

Parameters:

  • params – Iterable of parameters or parameter groups.

  • lr – Learning rate (default 1e-3).

  • betas – Coefficients for first and second moment estimates (default (0.98, 0.92)).

  • eps – Numerical stability term (default 1e-8).

  • weight_decay – Decoupled weight decay coefficient (default 0.0).

Reference:

Xie, X., Zhou, P., Li, H., Lin, Z., & Yan, S. (2022). Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models. arXiv:2208.06677.