Cellular Anatomy of PostgreSQL: The 8KB Page & Heap Tuple Layout¶

1. Introduction: The Fundamental Cell of Storage¶

In biology, the basic unit of living matter is the biological cell. In PostgreSQL, the fundamental structural unit of all table data, indexes, and materialized views is the 8,192-byte Block (Page).

Every operation in PostgreSQL—whether it is a simple SELECT id FROM users WHERE id = 1 or a massive multi-terabyte analytical join—ultimately operates by reading, modifying, locking, and flushing individual 8KB pages.

PostgreSQL does not interact with records or rows directly on disk. It interacts with Pages through the Buffer Manager (src/backend/storage/buffer/bufmgr.c).

2. Anatomical Blueprint of an 8KB Page¶

Located in src/include/storage/bufpage.h. Every page has an exact, byte-aligned physical layout:

+-----------------------------------------------------------------------------------+
| PageHeaderData (24 Bytes)                                                         |
| 00..07: pd_lsn (8 bytes)     - Log Sequence Number of last WAL byte modifying page|
| 08..09: pd_checksum (2 bytes)- 16-bit CRC Checksum of page contents               |
| 10..11: pd_flags (2 bytes)   - Status flags (Page full, all-visible, etc.)        |
| 12..13: pd_lower (2 bytes)   - Byte offset to END of Line Pointers array          |
| 14..15: pd_upper (2 bytes)   - Byte offset to START of newest Tuple payload       |
| 16..17: pd_special (2 bytes) - Byte offset to START of Special Space              |
| 18..19: pd_pagesize_version  - Page layout version (currently 4) & page size      |
| 20..23: pd_prune_xid (4 bytes)- Oldest unpruned XID on page                       |
+-----------------------------------------------------------------------------------+
| ItemIdData Array (Line Pointers: 4 bytes each)                                    |
| [ ItemId 1 (4B) ] ---> (Points to Tuple 1 at byte offset 8120)                    |
| [ ItemId 2 (4B) ] ---> (Points to Tuple 2 at byte offset 8040)                    |
| [ ItemId 3 (4B) ] ---> (Points to Tuple 3 at byte offset 7950)                    |
| (grows DOWNWARDS toward pd_lower)                                                 |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|                              FREE SPACE HOLE                                      |
|                        Size = (pd_upper - pd_lower)                               |
|                                                                                   |
+-----------------------------------------------------------------------------------+
| Tuple Heap Data (grows UPWARDS from pd_upper toward pd_lower)                     |
| ... [ Tuple 3 Payload: 90 Bytes ] (offset 7950..8039)                             |
| ... [ Tuple 2 Payload: 80 Bytes ] (offset 8040..8119)                             |
| ... [ Tuple 1 Payload: 72 Bytes ] (offset 8120..8191)                             |
+-----------------------------------------------------------------------------------+
| Special Space (0 to N bytes, at end of page: used exclusively by B-Tree/GiST)     |
+-----------------------------------------------------------------------------------+

3. Dissecting `PageHeaderData` Byte-by-Byte¶

typedef struct PageHeaderData
{
    /* 0..7 */   XLogRecPtr      pd_lsn;           /* LSN: Next byte after last byte of WAL record for last change */
    /* 8..9 */   uint16          pd_checksum;      /* Page CRC-16 checksum */
    /* 10..11 */ uint16          pd_flags;         /* Flag bits */
    /* 12..13 */ LocationIndex   pd_lower;         /* Offset to start of free space */
    /* 14..15 */ LocationIndex   pd_upper;         /* Offset to end of free space */
    /* 16..17 */ LocationIndex   pd_special;       /* Offset to start of special space */
    /* 18..19 */ uint16          pd_pagesize_version;
    /* 20..23 */ TransactionId   pd_prune_xid;     /* Oldest XMAX of any deleted tuple on page */
} PageHeaderData;

The Biological Purpose of Key Fields:¶

pd_lsn (The Page's Heartbeat):
Every time a backend modifies any byte on this page, it records the 64-bit Log Sequence Number (LSN) of the WAL record into pd_lsn.
Why it matters: During crash recovery, the recovery engine inspects a WAL record. If wal_record.lsn <= page.pd_lsn, Postgres knows the change is already on disk and skips it. If wal_record.lsn > page.pd_lsn, it replays the change.
pd_lower and pd_upper (The Free Space Squeeze):
Line pointers grow downwards ($0 \rightarrow 8192$).
Tuples grow upwards ($8192 \rightarrow 0$).
Available free space is strictly: $$\text{FreeSpace} = \text{pd\_upper} - \text{pd\_lower}$$
When $\text{pd\_lower} == \text{pd\_upper}$, the page is saturated.
pd_prune_xid (The Opportunistic Vacuum Trigger):
Records the oldest transaction that deleted a row on this page.
When a simple SELECT reads the page, if pd_prune_xid is older than the current transaction horizon, the backend process cleans up dead tuples inline (HOT pruning) without waiting for Autovacuum!

4. The Line Pointer: `ItemIdData` (The Physical Nerve Ending)¶

Located in src/include/storage/itemid.h. Every row is physically referenced by an ItemIdData. It is exactly 32 bits (4 bytes) wide:

typedef struct ItemIdData
{
    unsigned    lp_off:15,          /* Offset to tuple payload (0..32767) */
                lp_flags:2,         /* State of line pointer (0..3) */
                lp_len:15;          /* Byte length of tuple payload (0..32767) */
} ItemIdData;

The 4 Line Pointer Flags (`lp_flags`):¶

LP_UNUSED (0): Unused line pointer. Free to be allocated for a new row.
LP_NORMAL (1): Normal live tuple. lp_off points to valid tuple payload.
LP_REDIRECT (2): The HOT (Heap-Only Tuples) Link. Points to another ItemId on the same page. When an update occurs without modifying indexed columns, the old index entry points to ItemId 1, which redirects to ItemId 2.
LP_DEAD (3): The tuple bytes have been deleted and reclaimed, but the line pointer must remain because external B-Tree indexes still point to this (block, offset) CTID.

5. The Cellular Anatomy of a Heap Tuple (`HeapTupleHeaderData`)¶

Located in src/include/access/htup_details.h. Preceding every user row of data is a mandatory 23-byte header (padded to 24 bytes on 64-bit architectures):

+-------------------------------------------------------------------------------+
| t_choice.t_heap.t_xmin (4 bytes)  | Transaction ID that created this tuple    |
+-----------------------------------+-------------------------------------------+
| t_choice.t_heap.t_xmax (4 bytes)  | Transaction ID that deleted/updated tuple |
+-----------------------------------+-------------------------------------------+
| t_choice.t_heap.t_field3 (4 bytes)| t_cid (Command ID) OR t_xvac (Vacuum XID) |
+-----------------------------------+-------------------------------------------+
| t_ctid (6 bytes)                  | Physical address of this tuple OR pointer |
|                                   | to newer tuple version (Block, Offset)    |
+-----------------------------------+-------------------------------------------+
| t_infomask2 (2 bytes)             | Column count (11 bits) + HOT status bits  |
+-----------------------------------+-------------------------------------------+
| t_infomask (2 bytes)              | Visibility flags, NULL bitmap flag, etc.  |
+-----------------------------------+-------------------------------------------+
| t_hoff (1 byte)                   | Header offset: Distance to actual user data|
+-----------------------------------+-------------------------------------------+
| NULL BITMAP (Variable size)       | 1 bit per column (1 = NOT NULL, 0 = NULL) |
+-----------------------------------+-------------------------------------------+
| USER DATA (Columns)               | Aligned user values                       |
+-------------------------------------------------------------------------------+

The Secrets of `t_infomask` (The Visibility Status Bits):¶

To avoid checking the commit log (pg_xact / CLOG) on disk for every single tuple, PostgreSQL caches transaction states directly in t_infomask:

HEAP_XMIN_COMMITTED (0x0100): The inserting transaction is committed. Visible!
HEAP_XMIN_INVALID (0x0200): The inserting transaction aborted. Invisible!
HEAP_XMAX_COMMITTED (0x0400): The deleting transaction is committed. Dead!
HEAP_XMAX_INVALID (0x0800): The deleting transaction aborted. Alive!
HEAP_XMIN_FROZEN (0x0300): Tuple is frozen (immune to XID wraparound).
HEAP_HASNULL (0x0001): Tuple contains at least one NULL column. A NULL bitmap follows the header.
HEAP_HASVARWIDTH (0x0002): Tuple contains variable-length data (text, varchar, jsonb).

6. The Cellular Organ of Memory: `BufferDesc` (The Buffer Header)¶

When an 8KB page is loaded from disk into shared_buffers, it is assigned a Buffer Descriptor (BufferDesc). Located in src/include/storage/buf_internals.h:

typedef struct BufferDesc
{
    BufferTag       tag;            /* RelFileNode (DB OID, Table OID) + Block Number */
    int             buf_id;         /* Buffer ID (0 .. N-1 in shared_buffers) */

    /* Atomic State (32-bit Integer) */
    pg_atomic_uint32 state;

    int             freeNext;       /* Index of next buffer on free list */
    LWLock          content_lock;   /* LWLock protecting the 8KB page data */
} BufferDesc;

Anatomy of `BufferDesc.state` (The 32-bit Atomic Heartbeat):¶

The state integer packs 4 distinct fields:

[Bits 0 .. 15] : REFCOUNT (Pin Count: Number of active backends reading this page)
[Bits 16 .. 18]: USAGECOUNT (Clock Sweep weight: 0 to 5)
[Bit 19]       : BM_LOCKED (Spinlock bit protecting this descriptor)
[Bit 20]       : BM_DIRTY (1 = Page modified in RAM, must write to disk before evicting)
[Bit 21]       : BM_VALID (1 = Contains valid data read from disk)
[Bit 22]       : BM_TAG_VALID (1 = Tag correctly maps to a relation block)
[Bit 23]       : BM_IO_IN_PROGRESS (1 = A backend is currently reading/writing this page)
[Bit 24]       : BM_JUST_DIRTIED (1 = Modified recently)

The Lifecycle of Reading a Row (The "Pinning" Ritual):¶

Backend looks up BufferTag (DB_OID, Table_OID, Block_Num) in the shared memory Hash Table.
If found, it atomically increments REFCOUNT by 1. This is a PIN.
While REFCOUNT > 0, the Clock Sweep algorithm is forbidden from evicting this page.
The backend acquires LWLockAcquire(buf->content_lock, LW_SHARED).
It reads the tuple bytes.
It releases content_lock.
It atomically decrements REFCOUNT. This is an UNPIN.
It increments USAGECOUNT (up to a maximum of 5).

7. The Clock Sweep Eviction Algorithm in Code¶

When shared_buffers is exhausted and a new page must be read from disk, StrategyGetBuffer() sweeps through descriptors:

/* Pseudocode representation of src/backend/storage/buffer/freelist.c */
BufferDesc *
StrategyGetBuffer(void)
{
    for (;;)
    {
        BufferDesc *buf = &BufferDescriptors[clock_hand];

        /* Advance clock hand circularly */
        clock_hand = (clock_hand + 1) % NBuffers;

        uint32 state = pg_atomic_read_u32(&buf->state);

        /* Rule 1: If page is pinned by ANY backend, we CANNOT evict it */
        if (BUF_STATE_GET_REFCOUNT(state) > 0)
            continue;

        /* Rule 2: If page has usage count > 0, give it a second chance */
        if (BUF_STATE_GET_USAGECOUNT(state) > 0)
        {
            /* Decrement usage count */
            pg_atomic_fetch_sub_u32(&buf->state, BUF_USAGECOUNT_ONE);
            continue;
        }

        /* Rule 3: Found a victim! Usage count == 0 and Refcount == 0 */
        if (state & BM_DIRTY)
        {
            /* Page is dirty! Must flush to disk before reusing! */
            SyncUnencryptedBuffer(buf);
        }

        return buf; /* Reclaim this buffer for new page */
    }
}

Cellular Anatomy of PostgreSQL: The 8KB Page & Heap Tuple Layout¶

1. Introduction: The Fundamental Cell of Storage¶

2. Anatomical Blueprint of an 8KB Page¶

3. Dissecting PageHeaderData Byte-by-Byte¶

The Biological Purpose of Key Fields:¶

4. The Line Pointer: ItemIdData (The Physical Nerve Ending)¶

The 4 Line Pointer Flags (lp_flags):¶

5. The Cellular Anatomy of a Heap Tuple (HeapTupleHeaderData)¶

The Secrets of t_infomask (The Visibility Status Bits):¶

6. The Cellular Organ of Memory: BufferDesc (The Buffer Header)¶

Anatomy of BufferDesc.state (The 32-bit Atomic Heartbeat):¶