Common Activation Functions

Common elementwise activation functions.

Groups the classical textbook activation functions: the sigmoid/tanh family, the smooth softplus family, and the core rectifier / gating units used in the feed-forward network. These are stateless (no learnable parameters).

Formulations follow Lederer (arXiv:2101.09957) and the survey by Dubey et al. (arXiv:2109.14545).

class src.model.activation_function.common.Arctan(*args: Any, **kwargs: Any)[source]

Bases: Module

Inverse tangent: arctan(x).

Range (-pi/2, pi/2). Monotonic, smooth, bounded. Reference: Lederer §2.1.2.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply arctan elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with arctan(x) applied elementwise.

class src.model.activation_function.common.Elliott(*args: Any, **kwargs: Any)[source]

Bases: Module

Elliott activation (fast sigmoid approximation): x / (1 + |x|).

Alias of Softsign; the survey (arXiv:2109.14545) lists it under the name “Elliott”. Range (0, 1) for the one-sided variant, but the implemented two-sided form matches the standard elliottsig.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the Elliott activation elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x / (1 + |x|) applied elementwise.

class src.model.activation_function.common.GELU(*args: Any, **kwargs: Any)[source]

Bases: Module

Gaussian Error Linear Unit (exact): x * Phi(x) = x * 0.5 * (1 + erf(x / sqrt(2))).

Range (-0.17, inf). Smooth, probabilistic. Used in BERT, GPT-2/3. Reference: Hendrycks & Gimpel (2016), arXiv:1606.08415; survey §7.2.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the exact GELU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x * Phi(x) applied elementwise.

class src.model.activation_function.common.GELUTanh(*args: Any, **kwargs: Any)[source]

Bases: Module

GELU with the tanh approximation.

0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3))).

Faster than exact GELU; the approximation used by the original GPT-2 and BERT implementations.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the tanh-approximated GELU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with the tanh-approx GELU applied.

class src.model.activation_function.common.Identity(*args: Any, **kwargs: Any)[source]

Bases: Module

Identity / linear activation: f(x) = x.

Range (-inf, inf). Useful as a no-op activation. Reference: Lederer §2.2.1.

forward(x: torch.Tensor) → torch.Tensor[source]

Return the input unchanged.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: The same tensor x.

class src.model.activation_function.common.Mish(*args: Any, **kwargs: Any)[source]

Bases: Module

Mish: x * tanh(softplus(x)) = x * tanh(log(1 + e^x)).

Non-monotonic, smooth, self-regularized. Reported to outperform Swish on several benchmarks. Reference: Misra (2019), arXiv:1908.08681; survey §7.1.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Mish elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x * tanh(softplus(x)) applied.

class src.model.activation_function.common.ReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Rectified Linear Unit: max(0, x).

Range [0, inf). Reference: Nair & Hinton (2010); Lederer §2.2.2.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply ReLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with max(0, x) applied elementwise.

class src.model.activation_function.common.SiLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Sigmoid Linear Unit (SiLU / Swish with beta=1): x * sigmoid(x).

Range (-0.278, inf). Smooth, non-monotonic. Default FFN activation in Llama, PaLM, and this codebase. Reference: Elfwing et al. (2018), Ramachandran et al. (2017) arXiv:1710.05941; survey §3.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply SiLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x * sigmoid(x) applied.

class src.model.activation_function.common.Sigmoid(*args: Any, **kwargs: Any)[source]

Bases: Module

Logistic sigmoid: 1 / (1 + e^{-x}).

Range (0, 1). Monotonic, smooth, bounded. Reference: Lederer §2.1.1.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the logistic sigmoid elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with 1 / (1 + e^{-x}) applied elementwise.

class src.model.activation_function.common.Softplus(*args: Any, **kwargs: Any)[source]

Bases: Module

Softplus: log(1 + e^x).

Smooth approximation of ReLU; an antiderivative of the logistic sigmoid. Range [0, inf). Reference: Lederer §2.3.1, Glorot et al. (2011).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply softplus elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with log(1 + e^x) applied elementwise.

class src.model.activation_function.common.Softsign(*args: Any, **kwargs: Any)[source]

Bases: Module

Softsign (Elliott activation): x / (1 + |x|).

Range (-1, 1). Once-differentiable at 0, smooth elsewhere. Cheaper than tanh. Reference: Lederer §2.1.4 (also called elliottsig).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the softsign elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x / (1 + |x|) applied elementwise.

class src.model.activation_function.common.Swish(*args: Any, **kwargs: Any)[source]

Bases: Module

Parametric Swish: x * sigmoid(beta * x) with fixed beta.

Range (-c, inf) where c ~= 0.278 / beta. Non-monotonic, smooth. When beta = 1 this reduces to SiLU. Large beta approaches ReLU. Use SwishTrainable for a learnable beta.

Parameters:: beta – Fixed positive slope. Default: 1.0.

beta: The (fixed) sigmoid slope.

Reference: Ramachandran et al. (2017), arXiv:1710.05941; survey §6.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the parametric Swish elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x * sigmoid(beta * x) applied.

class src.model.activation_function.common.Tanh(*args: Any, **kwargs: Any)[source]

Bases: Module

Hyperbolic tangent: (e^x - e^{-x}) / (e^x + e^{-x}) = 2 sigmoid(2x) - 1.

Range (-1, 1). Monotonic, smooth, bounded. Reference: Lederer §2.1.3.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply the hyperbolic tangent elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with tanh applied elementwise.