The Most Complete Breakdown of LLM Positional Encoding: Absolute, Relative, Rotary, and No Positional Encoding at All

About 3565 wordsAbout 12 min

positional-encoding transformer llm hung-yi-lee

2026-07-22

Abstract: Transformer's Self-Attention itself does not perceive token order — shuffling the tokens of "the cat ate the fish" produces exactly the same Attention output. This article systematically traces the complete technical evolution of Positional Embedding: from the clock-hand analogy of Sinusoidal absolute positional encoding, to ALiBi and T5 relative positional biases, to the mathematical principles of RoPE (Rotary Position Embedding), and finally exploring Train Short Test Long extension methods and the "no positional encoding" NoPE/DroPE approaches.

01 A Basic Question: How Does Transformer Know the Order of Tokens?

The original Transformer has no way to consider the order of input. How does it process input tokens?

Input tokens become Embeddings, which then become inputs to a Layer. Each layer contains Self-Attention, which outputs another set of tokens.

But there's a problem: Self-Attention itself does not consider the order of input tokens.

If I swap the positions of Token A and Token C and compute Self-Attention again, you'll find that the output $O_D$ is completely unaffected.

So we must find a way to "inject" positional information into the Transformer. This is what Positional Embedding solves.

This article covers:

Absolute Positional Embedding (Sinusoidal)
Relative Positional Embedding (ALiBi, T5)
RoPE (Rotary Position Embedding)
Train Short, Test Long (Position Interpolation, Frequency-Based Methods, YaRN, Dynamic Scaling, LongRoPE)
No Positional Embedding (NoPE, DroPE)

02 Absolute Positional Embedding

The earliest Positional Embedding idea was straightforward: assign a special Embedding to each position. This Embedding represents positional information.

We use $P_0$ through $P_3$ to represent the special Embedding for each position, then add this Embedding to our vectors.

When tokens change order, their Embeddings become different, and the final Self-Attention output also differs.

Sinusoidal Positional Embedding

Reference: https://arxiv.org/abs/1706.03762

This Positional Embedding existed from the birth of the Transformer.

Let $d$ represent the length of the Embedding ( $d = 128 \text{ or } 256$ ).

We use $\boldsymbol{p}[i]$ to represent its values.

How does Sinusoidal Position Embedding construct all Embeddings? The formula is:

\begin{align} \boldsymbol{p_k}[2i] &= \sin\left(\frac{k}{10000^{2i/d}}\right) \\ \boldsymbol{p_k}[2i+1] &= \cos\left(\frac{k}{10000^{2i/d}}\right) \end{align}

Visual Understanding

If we take $\boldsymbol{p_0}[0]$ through $\boldsymbol{p_{49}}[0]$ , we'll see a Sine function.

Next, $\boldsymbol{p_0}[1]$ through $\boldsymbol{p_{49}}[1]$ form a Cosine function.

Next, $\boldsymbol{p_0}[10]$ through $\boldsymbol{p_{49}}[10]$ also form a Sine function, but with a larger period.

Plotting all Position Embeddings would look like this:

Clock Hand Analogy: Second Hand, Minute Hand, Hour Hand

We can view Sinusoidal Positional Embedding differently: even dimensions use Sine, odd dimensions use Cosine. So we can imagine each pair $2i$ and $2i+1$ as a 2D vector (a clock hand on a 2D plane). This hand rotates as $k$ increases — when $k$ rises, the hand rotates counterclockwise.

The rotation period is:

\frac{k}{10000^{2i/d}} = 2\pi

The number of $k$ needed for one full rotation:

k = 2\pi \cdot 10000^{2i/d}

If $d = 128$ , then $i = 0, ..., \frac{d}{2} - 1 = 0, ..., 63$ .

Different $i$ values have different periods:

$i$ value	$k$ value
$i = 0$ (dimensions 0, 1)	6.3
$i = 32$ (dimensions 64, 65)	628.3
$i = 63$ (dimensions 126, 127)	54410.1

The first two dimensions rotate fastest — about 6.3 tokens per cycle. Dimensions 10-11 rotate more slowly. Dimensions 100-101 barely change within the first 6 tokens.

Every two dimensions form a hand. The fastest is the second hand, slower is the minute hand, slowest is the hour hand. With dimension 128, we have 64 hands. The Transformer uses these 64 hands and their positions to determine each token's location.

Using clock hands to represent position seems reasonable, but why did the authors choose this approach? Because they wanted Positional Embedding to handle Relative Position.

Relative Positioning is Crucial!

Relative Position means: in the sentence "the cat ate the fish," cat and fish are two tokens apart. When Transformer considers which token should follow fish, it looks back at cat. Fish's attention score for cat is 0.7.

If I insert many tokens before cat — say 1000 tokens — but the relative position between cat and fish remains unchanged, we want fish's attention on cat to remain unchanged. Even if fish is at position 1004 and cat at 1001, fish's attention score for cat remains 0.7.

But if cat is at position 1 and fish at position 1004 — very far apart — we want fish's attention score for cat to be small.

Sinusoidal Positional Embedding was chosen because it has special properties that help with Relative Position.

\boldsymbol{p_{k+r}} = M_r \boldsymbol{p_k}

Where $\boldsymbol{p_k}$ represents position $k$ and $\boldsymbol{p_{k+r}}$ represents position $k+r$ .

This formula is independent of $k$ — it only depends on the relative position between $k+r$ and $k$ :

\begin{align} \boldsymbol{p_{4}} &= M_3 \boldsymbol{p_{1}} \\ \boldsymbol{p_{14}} &= M_3 \boldsymbol{p_{11}} \\ \boldsymbol{p_{104}} &= M_3 \boldsymbol{p_{101}} \end{align}

Let's examine how Sinusoidal Positional Embedding satisfies the Relative Position condition.

Position $k$ formula:

\begin{align} \boldsymbol{p_k}[2i] &= \sin\left(\frac{k}{10000^{2i/d}}\right) \\ \boldsymbol{p_k}[2i+1] &= \cos\left(\frac{k}{10000^{2i/d}}\right) \end{align}

Position $k+r$ formula:

\begin{align} \boldsymbol{p_{k+r}}[2i] &= \sin\left(\frac{k+r}{10000^{2i/d}}\right) \\ \boldsymbol{p_{k+r}}[2i+1] &= \cos\left(\frac{k+r}{10000^{2i/d}}\right) \end{align}

Using sine and cosine addition formulas:

\begin{align} \sin(a+b) &= \sin(a)\cos(b) + \cos(a)\sin(b) \\ \cos(a+b) &= \cos(a)\cos(b) - \sin(a)\sin(b) \end{align}

Expand $\boldsymbol{p_{k+r}}$ :

\begin{align} \boldsymbol{p_{k+r}}[2i] &= \sin\left(\frac{k+r}{10000^{2i/d}}\right) \\ &= \boldsymbol{p_k}[2i] \cos\left(\frac{r}{10000^{2i/d}}\right) + \boldsymbol{p_k}[2i+1] \sin\left(\frac{r}{10000^{2i/d}}\right) \end{align}

\begin{align} \boldsymbol{p_{k+r}}[2i+1] &= \cos\left(\frac{k+r}{10000^{2i/d}}\right) \\ &= \boldsymbol{p_k}[2i+1] \cos\left(\frac{r}{10000^{2i/d}}\right) - \boldsymbol{p_k}[2i] \sin\left(\frac{r}{10000^{2i/d}}\right) \end{align}

In matrix form:

\begin{bmatrix} \boldsymbol{p_{k+r}}[2i] \\ \boldsymbol{p_{k+r}}[2i + 1] \end{bmatrix} = \begin{bmatrix} \cos\left(\frac{r}{10000^{2i/d}}\right) & \sin\left(\frac{r}{10000^{2i/d}}\right) \\ -\sin\left(\frac{r}{10000^{2i/d}}\right) & \cos\left(\frac{r}{10000^{2i/d}}\right) \end{bmatrix} \begin{bmatrix} \boldsymbol{p_{k}}[2i] \\ \boldsymbol{p_{k}}[2i + 1] \end{bmatrix}

The rotation matrix is denoted as $M_{r,i}$ .

How does this design affect Self-Attention? Using $p_n$ and $p_m$ for positions, the attention score $a$ is:

\begin{align} a &= \boldsymbol{q_B} \cdot \boldsymbol{k_A} \\ &= (W_q (\boldsymbol{x_{B}} + \boldsymbol{p_m}))^T W_k (\boldsymbol{x_A} + \boldsymbol{p_n}) \\ &= \boldsymbol{x_{B}} W_q^T W_k \boldsymbol{x_A} + \boldsymbol{x_{B}}^T W_q^T W_k \boldsymbol{p_n} + \boldsymbol{p_m}^T W_q^T W_k \boldsymbol{x_A} + \boldsymbol{p_m}^T W_q^T W_k \boldsymbol{p_n} \end{align}

The term $\boldsymbol{p_m}^T W_q^T W_k \boldsymbol{p_n}$ shows absolute position. Converting to relative position:

\begin{align} \boldsymbol{p_m}^T W_q^T W_k \boldsymbol{p_n} &= (M_{m-n} \boldsymbol{p_n})^T W_q^T W_k \boldsymbol{p_n} \\ &= (\boldsymbol{p_n})^T M_{m-n} W_q^T W_k \boldsymbol{p_n} \end{align}

Where $M_{m-n}$ only depends on the relative position, but the overall formula still contains absolute position correlations.

03 Relative Positional Embedding

Since using Absolute Positional Embedding to achieve relative positioning is circuitous, a natural question arises: can we directly modify the Attention computation based on relative position?

ALiBi (Attention with Linear Biases)

Reference: https://arxiv.org/abs/2108.12409

ALiBi's approach: discard Positional Embedding entirely. Compute Attention scores without positional information, then subtract $b(m-n)$ — the relative distance between positions $m$ and $n$ . This makes Attention smaller for distant tokens. The constant $b$ is manually set and can differ per Attention Head.

ALiBi outperforms Sinusoidal methods across the board, especially on longer sequences.

T5's Learnable Relative Bias

Reference: https://arxiv.org/abs/1910.10683

T5 uses a learnable parameter for the bias $b_{m-n}$ instead of manually setting it. However, the hand-crafted ALiBi actually outperformed the trained T5 approach.

ALiBi eventually gave way to stronger methods, but its lesson remains: Relative Position matters greatly.

RoPE: Rotary Position Embedding

Reference: https://arxiv.org/abs/2104.09864

RoPE is one of today's most popular methods. Instead of adding a positional embedding, it applies a rotation to $k$ and $q$ vectors to encode positional information.

The Attention weight is then:

a = (\boldsymbol{k^n_A})^T \boldsymbol{q^m_B} = \boldsymbol{(k_A)}^T R_{m-n} \boldsymbol{q_B}

$R_{m-n}$ only depends on the relative position between A and B, directly encoding relative position into the attention computation.

Key advantage: It doesn't affect the Attention computation process itself — only $q$ and $k$ are modified, and the transformed $k$ can be stored in KV Cache.

Rotation as Position

Consider only the first two dimensions of $k$ and $q$ :

These two dimensions form a vector on a 2D plane.

Adding positional information means rotating this vector. To encode position $n$ , rotate the first two dimensions of $k$ by $n\theta$ :

Similarly for $q$ , rotate by $m\theta$ :

RoPE applies this rotation to every pair of dimensions:

Each pair of dimensions uses a different rotation angle $n\theta_i$ , where:

\theta_i = \frac{1}{10000^{2i/d}}

with $i = 0, 1, ..., \frac{d}{2} - 1$ .

RoPE's Key Property: Relative Position Invariance

If we move $k$ from position $n$ to $n+r$ and $q$ from $m$ to $m+r$ , their relative position stays the same, and the Attention value remains unchanged:

\boldsymbol{k^n_A} \cdot \boldsymbol{q^m_B} = \boldsymbol{k^{n+r}_A} \cdot \boldsymbol{q^{m+r}_B}

Suppose cat is at position 1 and fish at position 3. We add position 1 to cat's $k$ and position 3 to fish's $k$ . Fish gets its Query, adds position to get $\boldsymbol{q^3_B}$ , then computes attention $a$ with cat's $k$ .

Adding many tokens before cat and fish doesn't affect their Embeddings. If cat is at 101 and fish at 103, the attention value is identical to when cat was at 1 and fish at 3.

Why does RoPE achieve this? Comparing original $k$ and $k$ with position $n$ :

If $k$ is moved to position $n+r$ , it is rotated by $(n+r)\theta$ :

The angle between $\begin{bmatrix} \boldsymbol{k^n_A}[0] \\ \boldsymbol{k^n_A}[1] \end{bmatrix}$ and $\begin{bmatrix} \boldsymbol{k^{n+r}_A}[0] \\ \boldsymbol{k^{n+r}_A}[1] \end{bmatrix}$ is $r\theta$ . Same for $\boldsymbol{q_B}$ :

The dot product of $\begin{bmatrix} \boldsymbol{k^n_A}[0] \\ \boldsymbol{k^n_A}[1] \end{bmatrix}$ and $\begin{bmatrix} \boldsymbol{q^m_B}[0] \\ \boldsymbol{q^m_B}[1] \end{bmatrix}$ equals that of $\begin{bmatrix} \boldsymbol{k^{n+r}_A}[0] \\ \boldsymbol{k^{n+r}_A}[1] \end{bmatrix}$ and $\begin{bmatrix} \boldsymbol{q^{m+r}_B}[0] \\ \boldsymbol{q^{m+r}_B}[1] \end{bmatrix}$ , because equal rotation doesn't change the inner product.

RoPE Mathematical Derivation

Let's examine why RoPE only depends on the relative position of $k$ and $q$ , using the first two dimensions.

Rotating $k$ by $n\theta$ :

\begin{bmatrix} \boldsymbol{k^n_A}[0] \\ \boldsymbol{k^n_A}[1] \end{bmatrix}= \begin{bmatrix} \cos(n\theta) & -\sin(n\theta) \\ \sin(n\theta) & \cos(n\theta) \end{bmatrix} \begin{bmatrix} \boldsymbol{k_A}[0] \\ \boldsymbol{k_A}[1] \end{bmatrix}

Rotating $q$ by $m\theta$ :

\begin{bmatrix} \boldsymbol{q^m_B}[0] \\ \boldsymbol{q^m_B}[1] \end{bmatrix}= \begin{bmatrix} \cos(m\theta) & -\sin(m\theta) \\ \sin(m\theta) & \cos(m\theta) \end{bmatrix} \begin{bmatrix} \boldsymbol{q_B}[0] \\ \boldsymbol{q_B}[1] \end{bmatrix}

Computing the dot product:

\begin{align} \begin{bmatrix} \boldsymbol{k^n_A}[0] \\ \boldsymbol{k^n_A}[1] \end{bmatrix} \cdot \begin{bmatrix} \boldsymbol{q^m_B}[0] \\ \boldsymbol{q^m_B}[1] \end{bmatrix} &= \begin{bmatrix} \boldsymbol{k_A}[0] \\ \boldsymbol{k_A}[1] \end{bmatrix}^{\!T} \begin{bmatrix} \cos(n\theta) & -\sin(n\theta) \\ \sin(n\theta) & \cos(n\theta) \end{bmatrix}^{\!T} \begin{bmatrix} \cos(m\theta) & -\sin(m\theta) \\ \sin(m\theta) & \cos(m\theta) \end{bmatrix} \begin{bmatrix} \boldsymbol{q_B}[0] \\ \boldsymbol{q_B}[1] \end{bmatrix} \\[1em] &= \begin{bmatrix} \boldsymbol{k_A}[0] \\ \boldsymbol{k_A}[1] \end{bmatrix}^{\!T} \begin{bmatrix} \cos((m-n)\theta) & -\sin((m-n)\theta) \\ \sin((m-n)\theta) & \cos((m-n)\theta) \end{bmatrix} \begin{bmatrix} \boldsymbol{q_B}[0] \\ \boldsymbol{q_B}[1] \end{bmatrix} \end{align}

The rotation matrices combine: $-n\theta$ (transpose) $+ m\theta = (m-n)\theta$ .

Replacing $\boldsymbol{k^n_A}$ with $\boldsymbol{k^{n+r}_A}$ and $\boldsymbol{q^m_B}$ with $\boldsymbol{q^{m+r}_B}$ :

\begin{align} \begin{bmatrix} \boldsymbol{k^{n+r}_A}[0] \\ \boldsymbol{k^{n+r}_A}[1] \end{bmatrix} \cdot \begin{bmatrix} \boldsymbol{q^{m+r}_B}[0] \\ \boldsymbol{q^{m+r}_B}[1] \end{bmatrix} &= \begin{bmatrix} \boldsymbol{k_A}[0] \\ \boldsymbol{k_A}[1] \end{bmatrix}^{\!T} \begin{bmatrix} \cos(((m+r)-(n+r))\theta) & -\sin(((m+r)-(n+r))\theta) \\ \sin(((m+r)-(n+r))\theta) & \cos(((m+r)-(n+r))\theta) \end{bmatrix} \begin{bmatrix} \boldsymbol{q_B}[0] \\ \boldsymbol{q_B}[1] \end{bmatrix} \end{align}

Since $((m+r)-(n+r))\theta = (m-n)\theta$ , the result is identical — confirming relative position invariance.

RoPE Implementation Details

RoPE rotates $k$ and $q$ separately before computing Attention:

This is equivalent to multiplying $q$ by $R_{m-n}$ , computing only $q$ 's rotation:

In practice, the first approach is preferred. With the first method, both position-encoded $k$ and $q$ can be stored in KV Cache. With the second, $q$ would need to multiply by different $R_{m-n}$ each time. RoPE's compatibility with KV Cache is key to its popularity.

RoPE vs ALiBi: An Important Distinction

Many misunderstand RoPE — thinking like ALiBi, farther $k$ and $q$ have smaller Attention weights. But RoPE does not guarantee this.

This isn't necessarily a weakness. RoPE can do something ALiBi cannot: skip intermediate tokens to attend directly to earlier ones. In ALiBi, farther tokens always have smaller attention. With RoPE, rotation angles can bring certain positions into alignment regardless of distance.

For example, "my cat" and "his dog" — "cat" should attend to "my" and "dog" to "his," not to the middle word "the." RoPE enables this nuanced attention.

If $q$ and $k$ differ by $2\theta$ , but $q$ at $n+1$ and $k$ at $n$ differ by just $\theta$ , and $q$ at $n+2$ and $k$ at $n$ might have the same angle — attention scores can actually increase despite greater distance, allowing the model to skip intermediate tokens.

04 Train Short, Test Long

Reference: https://amaarora.github.io/posts/2025-09-21-rope-context-extension.html

Can we train on short sequences but handle very long sequences during testing without failure?

This is crucial for Agents that need to run indefinitely with very large context windows.

During training, we may not have very long corpora. The goal is positional encoding that supports Train Short, Test Long.

Intuitively, during training we only see $N$ tokens with positions $1, 2, ..., N$ .

When testing with $LN$ tokens, can we just assign positions up to $LN$ ?

Can RoPE handle this? It seems not.

(From: https://arxiv.org/abs/2108.12409)

When testing with very long sequences:

Sinusoidal: fails immediately on longer sequences.
RoPE: slightly better than Sinusoidal, but also degrades with length.
ALiBi: the only one that holds up, because it uses hand-crafted rather than learned parameters — proving surprisingly robust.

Why Does RoPE Fail on Long Sequences?

During training, the maximum rotation angle seen is $N\theta$ . Testing at $2N\theta$ presents an angle the model has never seen, and RoPE "freaks out."

Position Interpolation

References:

Solution: don't assign rotation angles beyond $N$ .

With $LN$ tokens, position numbers don't have to be $1$ to $LN$ — they can be $\frac{1}{L}$ to $N$ :

This is Position Interpolation, though it still requires fine-tuning.

Frequency-Based Approach

RoPE processes dimensions in pairs, and Position Interpolation treats all dimensions identically. Frequency-Based Approach asks: can different dimensions be treated differently?

When extending from $N$ to $LN$ , can we multiply by a function that depends on both the extension factor and the dimension?

Principle: $\theta_0$ is high-frequency, larger $i$ means lower frequency. High-frequency vectors can stay unchanged; low-frequency vectors need compression.

Every two dimensions form a rotating pointer. For the first two dimensions, the pointer rotates very fast. With $N=128$ , $\theta_0$ may have seen the entire plane many times over, while $\theta_{32}$ has barely covered 1/4.

For $\theta_0$ , positions beyond $N$ are fine — it's seen it all. But for $\theta_{32}$ , extending $N$ to 256 means encountering an unseen rotation angle.

NTK-aware Scaling

f(L, i) = \left(\frac{1}{L}\right)^{\frac{2i}{d-2}}

Meaning:

$i=0$ : $f(L,0)=1$ , no compression at highest frequency.
$i=\frac{d}{2}-1$ : $f(L, \frac{d}{2}-1)=\frac{1}{L}$ , maximum compression at lowest frequency.

Reference: https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

YaRN

YaRN (Yet another RoPE extensionN method). Reference: https://arxiv.org/abs/2309.00071

Keep some low-frequency scaling unchanged, keep some high-frequency scaling unchanged, only modify the middle frequencies.

Dynamic Scaling

Previous methods handle long sequences, but performance on short sequences may degrade.

Reference: https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/

Dynamic Scaling: use different treatments for different sequence lengths. For short sequences, don't modify — performance stays good since training saw these lengths.

One variant: don't modify positions at the beginning of the sequence, only apply scaling after a certain proportion.

Dynamic Scaling combined with Position Interpolation or NTK outperforms both alone.

Frequency-Based + Dynamic: LongRoPE

Frequency-Based determines the compression function; Dynamic Scaling determines where compression begins. LongRoPE uses Evolutionary Search to find optimal strategies.

Reference: https://arxiv.org/abs/2402.13753

This achieves remarkable results — models handling up to 2048K input length.

05 No Positional Embedding?!

Does Self-Attention Really Have No Positional Information?

Single-layer Self-Attention has no positional information. But multi-layer Self-Attention is different.

The first layer captures relationships between tokens. The second layer sees these relationships embedded — "cat relates to 'ate'" and "fish relates to 'ate'" produce different representations. So "cat ate fish" and "fish ate cat" yield different final Embeddings even without Positional Embedding.

NoPE: No Positional Embedding

Reference: https://arxiv.org/pdf/2305.19466

Experiments show that No Position Embedding works surprisingly well. On Copy tasks, models with Positional Embedding degraded on long sequences, while NoPE maintained high accuracy.

However, comparing RoPE vs NoPE during training — NoPE underperforms RoPE.

We need Positional Embedding because it helps during training.

DroPE: Dropping Positional Embedding Late in Training

Start training with Positional Embedding, then remove it near the end. Loss spikes briefly but recovers quickly. Training without Position Embedding can match RoPE's loss.

DroPE outperforms RoPE + YaRN on extended contexts. Training at 2K context, then removing positional embedding enables good performance at inference contexts far beyond 2K.

Reference: https://arxiv.org/pdf/2512.12167

References

Copyright

License under:Kstheme