HoPE Positional Encoding

Hybrid Positional Encoding (HoPE).

Implements hyperbolic positional encoding over consecutive dimension pairs with monotonic exponential damping. Unlike RoPE which uses trigonometric rotation (sin/cos), HoPE uses hyperbolic functions (sinh/cosh) combined with an exponential damping factor that decays with position distance. A layer-dependent scale factor modulates the angles, making deeper layers sensitive to longer-range positions.

HoPE is used as the positional encoding for TitanAttention blocks.

class src.model.embeddings.hope.HoPE(*args: Any, **kwargs: Any)[source]

Bases: Module

Hybrid Positional Encoding with hyperbolic rotation and exponential damping.

Applies a hyperbolic transformation to each pair of adjacent dimensions. The angle for pair i at position p in layer l is:

theta_i(p, l) = p * exp(-log(base) * i / (pair_dim - 1)) * (1 + 0.05 * l)

The transformation uses cosh and sinh (instead of cos and sin) multiplied by an exponential damping factor:

damping(p, l) = exp(-damping * (1 + 0.05 * l) * p)

This provides monotonic decay of positional influence with distance while the layer-dependent scaling allows deeper layers to attend over longer ranges.

Parameters:

  • head_dim – Dimensionality of each attention head. Must be even for proper pairing.

  • base – Base for the geometric progression of inverse frequencies. Defaults to 10000.0.

  • damping – Damping coefficient controlling the rate of exponential decay with position. Defaults to 0.01.

head_dim

Total head dimensionality.

pair_dim

Number of dimension pairs (head_dim // 2).

base

Base frequency for inverse frequency computation.

damping

Damping coefficient for exponential position decay.

forward(x: torch.Tensor, logical_layer_idx: int = 0) → torch.Tensor[source]

Apply hyperbolic positional encoding.

Parameters:

  • x – Input tensor of shape (batch, heads, seq_len, head_dim).

  • logical_layer_idx – Logical layer index used to compute the layer-dependent angle scaling factor. Defaults to 0.

Returns:

Tensor of same shape as x with hyperbolic positional encoding applied. If pair_dim == 0, returns x unchanged.