Exponential Activation Functions

Exponential (ELU-based) activation functions.

Implements the exponential-linear family surveyed in arXiv:2109.14545 §5: ELU and its scaled, parametric, deformable, and combined variants.

class src.model.activation_function.exponential.CELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Continuously Differentiable ELU: x if x >= 0 else alpha * (e^{x/alpha} - 1).

Range (-alpha, inf). Unlike ELU, CELU is once-differentiable at 0 for any alpha. Reference: Barron (2017), arXiv:1704.07483; survey §5.

Parameters:: alpha – Saturation value for negative inputs. Default: 1.0.

alpha: The negative-saturation constant.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply CELU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.EELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Elastic ELU: ELU with learnable alpha and a positive-region slope beta.

beta * max(0, x) + alpha * min(0, e^x - 1). Reference: survey §5.

Parameters:

dim – Number of features.
alpha_init – Initial negative saturation. Default: 1.0.
beta_init – Initial positive slope. Default: 1.0.

alpha: Learnable negative saturation of shape (dim,).

beta: Learnable positive slope of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Elastic ELU elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.ELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Exponential Linear Unit: x if x >= 0 else alpha * (e^x - 1).

Range (-alpha, inf). Smooth negative saturation; mean closer to zero than ReLU. Reference: Clevert et al. (2016), arXiv:1511.07289; Lederer §2.3.2; survey §5.

Parameters:: alpha – Saturation value for negative inputs. Default: 1.0.

alpha: The negative-saturation constant.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply ELU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.ELiSH(*args: Any, **kwargs: Any)[source]

Bases: Module

Exponential Linear Sigmoid SquasHing.

x * sigmoid(x) for x >= 0 and (e^x - 1) / (1 + e^{-x}) for x < 0 (Swish positive branch, ELU-gated negative branch). Reference: Basirat & Roth (2019), survey §5.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply ELiSH elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.FELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Fast ELU: like ELU but alpha is learnable and clamped to [0, 1].

A numerically cheaper variant that restricts the saturation constant. Reference: survey §5.

Parameters:

dim – Number of features.
alpha_init – Initial saturation value. Default: 1.0.

alpha: Learnable saturation of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Fast ELU elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.HardELiSH(*args: Any, **kwargs: Any)[source]

Bases: Module

Hard ELiSH: ELiSH with a hard-sigmoid approximation.

x * relu6(x + 3) / 6 for x >= 0 and (e^x - 1) * relu6(x + 3) / 6 otherwise. Reference: Basirat & Roth (2019), survey §5.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply HardELiSH elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.MPELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Multi-Parametric ELU: ELU with learnable alpha and a PReLU-like slope beta.

max(0, x) + beta * min(0, alpha * (e^x - 1)). Generalizes both ELU and PReLU. Reference: survey §5.

Parameters:

dim – Number of features.
alpha_init – Initial ELU saturation. Default: 1.0.
beta_init – Initial negative slope. Default: 1.0.

alpha: Learnable saturation of shape (dim,).

beta: Learnable negative slope of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Multi-Parametric ELU elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.PDELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Parametric Deformable ELU.

x if x >= 0 else alpha * (e^{x/alpha} - 1) + (1 - alpha) * x. Learnable alpha interpolates between CELU and the identity on the negative branch. Reference: survey §5.

Parameters:

dim – Number of features.
alpha_init – Initial deformation in (0, 1]. Default: 1.0.

alpha: Learnable deformation parameter of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply PDELU elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.PELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Parametric ELU: alpha/beta * x if x >= 0 else alpha * (e^{x/beta} - 1).

Both alpha and beta are learnable, generalizing ELU. Reference: Trottier et al. (2017), arXiv:1605.09332; survey §5.

Parameters:

dim – Number of features. alpha/beta have shape (dim,).
alpha_init – Initial positive saturation. Default: 1.0.
beta_init – Initial positive scale. Default: 1.0.

alpha: Learnable saturation parameter of shape (dim,).

beta: Learnable scale parameter of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Parametric ELU elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.PREU(*args: Any, **kwargs: Any)[source]

Bases: Module

Parametric Rectified Exponential Unit.

max(0, x) + alpha * min(0, e^x - 1) with learnable alpha (negative branch) and beta (positive-branch slope). Range (-alpha, inf). Reference: survey §5.

Parameters:

dim – Number of features.
alpha_init – Initial negative saturation. Default: 1.0.
beta_init – Initial positive slope. Default: 1.0.

alpha: Learnable negative saturation of shape (dim,).

beta: Learnable positive slope of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply PREU elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.SELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Scaled Exponential Linear Unit.

scale * x if x >= 0 else scale * alpha * (e^x - 1) with fixed alpha ~= 1.6733 and scale ~= 1.0507. Induces self-normalizing activations that keep mean 0 and variance 1. Reference: Klambauer et al. (2017), arXiv:1706.02515; Lederer §2.3.2; survey §5.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply SELU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.exponential.SoftExp(*args: Any, **kwargs: Any)[source]

Bases: Module

Soft Exponential: interpolates between exponential, linear, and logarithmic.

For alpha < 0: -(log(1 - alpha*(x + alpha))) / alpha; alpha == ``: ``x; alpha > 0: (exp(alpha * x) - 1) / alpha + alpha. alpha is learnable. Reference: survey §5, Godfrey & Gashler (2015).

Parameters:

dim – Number of features.
alpha_init – Initial interpolation parameter. Default: 0.0.

alpha: Learnable interpolation parameter of shape (dim,).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Soft Exponential elementwise (per-channel).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape.