--- title: "PostgreSQL Internals - Module 1: Process & Memory Architecture" description: "Learn how PostgreSQL manages backend processes, the postmaster, shared buffers, and memory contexts. A deep dive into PG's process and memory model." author: "Samba Siva" published: 2026-03-28T10:18:47.318Z updated: 2026-11-03T12:53:01.960Z tags: - "PostgreSQL" - "Technical" - "PGInternals" url: "https://sambasivareddy.in/blog/postgresql-internals-module-1-process-memory-architecture" --- # PostgreSQL Internals - Module 1: Process & Memory Architecture PostgreSQL is a widely known open-source relational database, but it's true strength lies beneath the surface, in the carefully engineered internals that makes the database reliable, better performer, and supports extensibility. Today, we are going to learn about the Core module i.e **_Process & Memory Architecture_**, where we explore how postgres spans a new Backend Processes and Memory to them. * * * ## Part 1: How PostgreSQL Starts Up When you run **"pg\_ctl start"**, here's exactly what happens: **The Postmaster Process** The Postmaster is the first and most important process. It is the parent of all other PostgreSQL processes. Its responsibilities: - Read postgresql.conf and pg\_hba.conf. - Allocates shared memory (the buffer pool, WAL buffers, lock tables, etc.) - Listens on a TCP port (default 5432) or a Unix socket. - Forks background worker processes. - Accepts incoming client connections and forks a backend for each. ```markdown pg_ctl start ▼ postmaster (pid 1) ├── checkpointer ├── background writer ├── WAL writer ├── autovacuum launcher ├── stats collector └── [backend] ← forked per client connection ``` You can verify this yourself by running: ```shell ps aux | grep postgres You'll see output like: ----------------------- postgres 1234 ... postgres: postmaster postgres 1235 ... postgres: checkpointer postgres 1236 ... postgres: background writer postgres 1237 ... postgres: walwriter postgres 1238 ... postgres: autovacuum launcher postgres 1239 ... postgres: stats collector postgres 1240 ... postgres: backend (mydb myuser 127.0.0.1) ← your connection ``` * * * ## Part 2: What happens When a Client Connects This is the sequence of how the client connections establishes and then querying works: ![Process](https://pub-b8d5ca13188446a08ac9941fcca1304e.r2.dev/Process.png) Once the backend is forked, **the postmaster is completely out of the loop**. The client communicates directly with its dedicated backend process for that entire session, and the postmaster never sees your SQL queries. * * * ## Part 3: Every Background Process ### Checkpointer What it does: Periodically flushes all dirty pages from the shared buffers to disk and writes a checkpoint record to WAL. Why it matters: Without checkpoints, crash recovery would have to replay the entire WAL history from the beginning. A checkpoint creates a safe "restart point". When it runs: - Every **_checkpoint\_timeout_** seconds (default: 5 minutes) - When WAL size exceeds **_max\_wal\_size_** (default: 1GB) - On **_CHECKPOINT_** command - On clean shutdown ```sql -- See last checkpoint info SELECT * FROM pg_control_checkpoint(); -- Output includes: -- checkpoint_lsn = WAL location of last checkpoint -- prior_lsn = previous checkpoint location -- redo_lsn = where recovery would start from -- checkpoint_time = when it happened ``` ### Background Writer (BGWriter) What it does: Proactively writes dirty buffers from shared buffer pool to disk, before the checkpointer needs to. Why it matters: When a backend needs a free buffer and none are available, it would have to write a dirty page itself, by stalling your query. BGWriter prevents this by keep cleaning the buffers available. How it works: - Sleeps for **_bgwriter\_delay_** ms (default: 200ms) - Wakes up, scans the buffer pool using a clock-sweep algorithm. - Writes up to **_bgwriter\_lru\_maxpages_** dirty buffers (default: 100) - Goes back to sleep ``` -- Monitory BGWrite activity SELECT buffers_clean, -- buffers written by bgwriter maxwritten_clean, -- time bgwriter hit its page limit buffers_backend, -- buffers written by backends buffers_checkpoint -- buffers written by checkpoint FROM pg_stat_bgwriter; ``` > Note: If buffers\_backend is high, your BGWriter can't keep up. Increase bgwriter\_lru\_maxpages or decrease bgwriter\_delay ### WAL Writer What it does: Flushes WAL records from WAL buffers (in shared memory) to WAL files on disk. Why it matters: COMMIT must guarantee durability. WAL must be on disk before COMMIT returns to the client. The WAL Writer batches these flushes efficiently. How it works: - Wakes up every **_wal\_writer\_delay_** ms (default: 200ms) - Or immediately when **_wal\_writer\_flush\_after_** bytes accumulate (default: 1MB) - Calls **_fysnc()_** or **_fdatasync()_** to ensure durability ### Autovacuum Launcher + Workers What it does: The launcher wakes up every autovacuum\_naptime (default: 1min), checks pg\_stat\_user\_tables, and forks worker processes for tables that need vacuuming. Why it matters: Because of MVCC, dead tuples accumulate. Without vacuum, tables bloat and critically transaction ID wrap-around can occur. ``` -- See which tables autovacuum is targeting SELECT relname, n_dead_tup, n_live_tup, last_autovacuum, last_autoanalyze FROM pg_stat_user_tables ORDER BY n_dead_tup DESC; ``` ### Stats Collector Collects runtime statistics and writes them to files in **_"$PGDATA/pg\_stat\_tmp/"_**. This feeds all the **_"pg\_stat\_\*"_** views you query. * * * ## Part 4: Memory Architecture PostgreSQL Memory is split into two categories: ### Shared Memory - Allocated once at the startup, visible to all processes. - Example: - Shared Buffer Pool (shared\_buffers) - default: 128MB - WAL Buffers (wal\_buffers) - default: 16MB - CLOG Buffers (commit status) - Lock Table (pg\_locks) - Sub Trans Buffers - Proc Array (active transactions) **shared\_buffers** shared\_buffers is the single most important memory parameter. This is PostgreSQL's **page cache** - every data page read from disk is loaded here, and every page write goes here first before eventually hitting disk. How it works internally: 1. Backend needs page (table=orders, block=42) 2. Looks up buffer pool hash table: is block 42 in memory? 3. Cache hit: return pointer to buffer - no disk I/O 4. Cache miss: find a free buffer, read page from disk into it and return pointer. **The Clock-Sweep Eviction Algorithm:** ![Clock Sweep](https://pub-b8d5ca13188446a08ac9941fcca1304e.r2.dev/clock_sweep_blog_banner.svg) PostgreSQL doesn't use LRU. It uses a clock sweep: - Each buffer has a usage\_count (0-5) - A "clock hand" sweeps through buffers - If usage\_count > 0: decrement it, skip - If usage\_count == 0: evict this buffer (write to disk first if dirty) - Each time a buffer is accessed, its usage\_count is incremented ``` -- See what's currently in the buffer pool CREATE EXTENSION pg_buffercache; SELECT c.relname, count(*) AS buffers, round(count(*) * 8.0 / 1024, 2) AS size_mb, round(100.0 * count(*) / (SELECT count(*) FROM pg_buffercache), 2) AS pct FROM pg_buffercache b JOIN pg_class c ON c.relfilenode = b.relfilenode WHERE b.isdirty IS NOT NULL GROUP BY c.relname ORDER BY buffers DESC LIMIT 20; ``` **WAL Buffers** WAL records are first written into WAL buffers (circular buffer in shared memory). The WAL writer flushes them to disk. Size is wal\_buffers (default: auto-tuned to ~1/32 of shared\_buffers, min 64kB, max 16MB). Critical rule: WAL must hit disk before the data page it protects. This is the "write-ahead" guarantee. **CLOG (Commit Log)** The CLOG stores **2 bits per transaction**: - 00 = IN\_PROGRESS - 01 = COMMITTED - 10 = ABORTED - 11 = SUB\_COMMITTED When a **backend checks whether a tuple is visible, it looks at xmin/xmax and then checks CLOG to see if that transaction committed or aborted**. This is fundamental to MVCC visibility. CLOG is stored in **_$PG\_DATA/pg\_xact_** on disk, cached in shared memory. **Process Array (ProcArray)** A shared memory array with one slot per active backend. Each slot holds: - Backend PID - Current transaction ID (xid) - Current snapshot info This is what makes **_pg\_stat\_activity_** work, and it's what MVCC uses to compute snapshots (which transactions are "in flight" right now). ### Local Memory - Memory will be allocated per backend not for all. - Example: - work\_mem: Uses for sorts, hash joins, etc. - maintenance\_work\_mem: For VACUUM. CREATE INDEX - temp\_buffers: For temp tables **work\_mem:** Used for **each sort or hash operation** in a query. The catch: a single query can use work\_mem **many times** — once per sort node, once per hash join, etc. ``` -- If a query has 5 sort operations and work_mem = 4MB -- That query can use up to 20MB of memory -- When work_mem is exceeded, PostgreSQL spills to disk (temp files) -- You can detect this: SELECT query, temp_blks_read, temp_blks_written FROM pg_stat_statements WHERE temp_blks_written > 0 ORDER BY temp_blks_written DESC; ``` **maintenance\_work\_mem:** Used by: - VACUUM: for collecting dead tuple IDs - CREATE INDEX: for sorting index entries - ALTER TABLE ... ADD FOREIGN KEY: for constraint validation. Can be set much higher than work\_mem since only a few maintenance ops run at once. Setting it to 256MB–1GB for CREATE INDEX can dramatically speed things up. **temp\_buffers:** Used for temporary tables that are created with "CREATE TEMP TABLE". Separate from the main buffer pool. Default is 8MB. * * * ## Part 5: The pg\_hba.conf & Authentication Flow When a client connects, the backend process: 1. Read pg\_hba.conf top-to-bottom 2. Find the first matching rule (by connection type, database, user, IP) 3. Applies that authentication method (trust, md5, scram-sha-256, peer, cert, etc..) 4. If auth passes: loads session parameters from postgresql.conf and role settings. 5. Sends ReadyForQuery to client ``` # pg_hba.conf format: # TYPE DATABASE USER ADDRESS METHOD host mydb alice 192.168.1.0/24 scram-sha-256 host all all 0.0.0.0/0 md5 local all all peer ``` * * * ## Conclusion Understanding PostgreSQL's process and memory architecture isn't just theoritical, it directly changes how you debug and tune your database. When a query spills to disk, you now know it's work\_mem that ran out. When you see buffer\_backends climbing in pg\_stat\_writer, you know your BGWriter can't keep pace. When a COMMIT feels slow, you know the WAL Writer is the one making the durability guarantee happen before control returns to you. PostgreSQL isn't magic. It's a carefully engineered system where every process has a job, every memory region has a purpose, and every design decision has a reason. Once you see it clearly, you stop guessing and start reasoning. * * * ### **What's Next → Module 2: Page Layout & Storage Internals** Everything we discussed — shared buffers, BGWriter, Checkpointer — revolves around one thing: **pages**. PostgreSQL reads, writes, and caches data in 8KB pages. But what's actually inside one? In Module 2, we'll crack open a live page using "pageinspect" and read it byte by byte: - How PostgreSQL lays out a heap page — the header, line pointers, and tuples - What a tuple header actually contains (xmin, xmax, infomask, ctid) - How dead tuples accumulate in the same page after updates - How to use pageinspect extension to inspect the real pages in your databases. * * * > Subscribe to the newsletter and get notified via email whenever a new blog goes live.