Dynamic Tanh Normalization

Dynamic Tanh normalization layer.

Applies standard normalization (subtract mean, divide by standard deviation) followed by a learnable affine transformation through tanh.

class src.model.norm.dynamic_tanh.DynamicTanhNorm(*args: Any, **kwargs: Any)

Bases: Module

Dynamic Tanh normalization layer.

Applies standard normalization (subtract mean, divide by standard deviation) followed by a learnable affine transformation through tanh. The learnable parameters alpha and beta allow the layer to adapt the saturation point and slope of the tanh non-linearity per dimension.

Parameters:

• dim – Number of features in the input (normalized dimension).

• eps – Small constant for numerical stability in standard deviation. Defaults to 1e-6.

alpha

Learnable scale parameter of shape (dim,), initialized to 1.

beta

Learnable shift parameter of shape (dim,), initialized to 0.

eps

Epsilon value for numerical stability.

forward(x)

Apply dynamic tanh normalization.

Parameters:

x – Input tensor of shape (..., dim).

Returns:

Tensor of same shape as x, normalized and passed through tanh with learnable affine parameters.