Rectified Activation Functions

ReLU-based activation functions.

Implements the rectifier family surveyed in arXiv:2109.14545 §4: leaky and parametric variants, bounded/shifted ReLUs, and rectified-hyperbolic units. Most are stateless; the parametric ones (PReLU) carry learnable parameters.

class src.model.activation_function.rectified.AbsReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Absolute-value ReLU (ABReLU / AB-ReLU): max(0, x) - a * max(0, -x).

where a is the mean of the pre-activation over the batch/map (a data-dependent baseline subtracted before rectification). Implemented here as |x| style: max(0, x) - max(0, -x) = x rectified to the negative-symmetric form abs-biased. Concretely we use the survey’s definition max(0, x) - max(0, -x) giving a sign-preserving ramp. Reference: survey §4, Bjorck et al. (2017).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply ABReLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.rectified.BReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Bounded ReLU: min(max(0, x), t).

Range [0, t]. Clamps the positive region to a ceiling t.

Parameters:: t – Upper bound for the positive region. Default: 1.0.

t: The ceiling value.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Bounded ReLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape clamped to [0, t].

class src.model.activation_function.rectified.DisReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Displaced ReLU: max(0, x - delta).

Shifts the ReLU hinge to the right by delta. Range [-delta, inf) after the displacement. Reference: survey §4.

Parameters:: delta – Rightward shift of the rectifier hinge. Default: 0.0.

delta: The hinge displacement.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Displaced ReLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.rectified.HardSwish(*args: Any, **kwargs: Any)[source]

Bases: Module

Hard-Swish: x * relu6(x + 3) / 6.

Cheap approximation of Swish used in MobileNetV3. Range (-1.67, inf). Reference: Howard et al. (2019), arXiv:1905.02244.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Hard-Swish elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x * relu6(x + 3) / 6 applied.

class src.model.activation_function.rectified.Hexpo(*args: Any, **kwargs: Any)[source]

Bases: Module

Hexpo: a * max(0, x) - c * max(0, -x).

Asymmetric two-sided rectifier with separate positive/negative gains. Reference: survey §3.

Parameters:

a – Positive-region gain. Default: 1.0.
c – Negative-region gain. Default: 1.0.

a, c: The two gains.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Hexpo elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.rectified.LeakyReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Leaky ReLU: x if x >= 0 else negative_slope * x.

Range (-inf, inf). Allows a small gradient when the unit is inactive. Reference: Maas et al. (2013); Lederer §2.2.3; survey §4.

Parameters:: negative_slope – Slope for negative inputs. Default: 0.01.

negative_slope: The negative-region slope.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Leaky ReLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with the leaky-relu transform applied.

class src.model.activation_function.rectified.LiSHT(*args: Any, **kwargs: Any)[source]

Bases: Module

Linearly Scaled Hyperbolic Tangent: x * tanh(x).

Non-monotonic, unbounded in magnitude. Range [0, inf) (by the survey’s magnitude argument). Reference: survey §3, Roy et al. (2019).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply LiSHT elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with x * tanh(x) applied.

class src.model.activation_function.rectified.NLReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Natural-Logarithm ReLU: beta * log(1 + max(0, x)).

Range [0, inf); compresses large positive activations logarithmically. Reference: survey §4, Forest (2014).

Parameters:: beta – Positive scaling constant. Default: 1.0.

beta: The logarithmic scale.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply NLReLU elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape.

class src.model.activation_function.rectified.PReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

Parametric ReLU: max(0, x) + a * min(0, x) with learnable a.

Extends Leaky ReLU by making the negative slope a trainable parameter. With one parameter per channel (num_parameters = dim) it is per-channel; a single shared parameter gives the global variant.

Parameters:

dim – Number of features. The slope a has shape (dim,).
init – Initial value of the negative slope. Default: 0.25.

weight: Learnable negative slope of shape (dim,).

Reference: He et al. (2015), arXiv:1502.01852; survey §4.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply Parametric ReLU elementwise (per-channel slope).

The slope is broadcast over the last dimension, matching the transformer convention (..., dim).

Parameters:: x – Input tensor of shape (..., dim).
Returns:: Tensor of same shape with the parametric ReLU applied.

class src.model.activation_function.rectified.PenalizedTanh(*args: Any, **kwargs: Any)[source]

Bases: Module

Penalized Tanh (pTanh): max(0, \tanh(x)).

A rectified, non-negative tanh with a soft gradient near zero. Range [0, 1). Reference: survey §3.

Note

The general form is \tanh(x) for x > 0 and a * \tanh(x) with a in (0, 1) otherwise; the implemented form uses a -> 0 (hard zeroing of negatives).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply penalized tanh elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with max(0, tanh(x)) applied.

class src.model.activation_function.rectified.ReLU6(*args: Any, **kwargs: Any)[source]

Bases: Module

ReLU6: min(max(0, x), 6).

Bounded ReLU clamped at 6, used in MobileNet for fixed-point friendliness. Reference: Howard et al. (2017).

forward(x: torch.Tensor) → torch.Tensor[source]

Apply ReLU6 elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with min(max(0, x), 6) applied.

class src.model.activation_function.rectified.VReLU(*args: Any, **kwargs: Any)[source]

Bases: Module

V-shaped ReLU: |x|.

Range [0, inf). Symmetric rectifier; equivalent to the absolute value. Reference: survey §4.

forward(x: torch.Tensor) → torch.Tensor[source]

Apply |x| elementwise.

Parameters:: x – Input tensor of arbitrary shape.
Returns:: Tensor of same shape with |x| applied.