Dr. Ibrar Ahmed

HomePostgreSQLArticle

PostgreSQL Mechanics

Amazon Aurora PostgreSQL Database: What Happens When It Fails?

Dr. Ibrar Ahmed11 min readFrom the lecture notes

What you will learn

  • Backup on Aurora works differently, and it's genuinely better.
  • Aurora gives you four kinds of endpoint, and each has one job.
  • Now let's simulate losing an availability zone, and watch the promise from earlier hold.
  • Minor version upgrades try something clever, called zero downtime patching.
The production request pathSystem sketch
ApplicationRetry-aware clientStable endpointRouting layerPostgreSQLWriter and replicas
01

The failure model

Managed doesn't mean unbreakable. But on Aurora, the thing that breaks is compute. Your data sits in a separate storage service that the instance failure never touched. So when the writer dies, what happens next depends on one question. Do you have a replica? With one, Aurora promotes it, usually in under thirty seconds. Without one, Aurora has to build you a new writer, and that takes minutes. Lose a whole availability zone and writes keep going. We'll build all of it, break it, and measure it.

So who owns what? On Aurora the line moved down. AWS now runs the whole storage layer. Six copies, the write quorum, segment repair, continuous backup. All of it, invisible to you. Everything above that is still yours. Your schema, your queries, your parameters, how the app reconnects. And one thing hasn't changed. The rds_superuser role is powerful, but it is not a real superuser, and there is still no shell.

Here's the idea everything else follows from. On Aurora, compute and storage are two different services. The instance has no volume of its own. And the writer doesn't push pages down to a disk. It ships redo log records to the storage service, and storage builds the pages. Now look at the layout. Your cluster volume is cut into ten gibibyte pieces called protection groups. Every one of those pieces is stored six times. Two copies in each of three availability zones. Not six copies of your database. Six copies of every slice.

Six copies buys you a specific guarantee, so let's be precise. A write is acknowledged when four of the six copies have it. A read needs three. Now lose an availability zone. Two copies are gone, four are left, and four of six is still met. Your writes never stop. Lose that zone plus one more copy and you have three. Still readable, still repairable, but no longer writable. That's the honest edge. And repair happens per ten gibibyte segment, from the surviving peers, in parallel.

Notice what you never configured. There is no volume type, no size, no provisioned throughput. The cluster volume grows on its own, in ten gibibyte steps, up to two hundred and fifty six tebibytes. Most articles still say one hundred and twenty eight, and that number is out of date. And unlike RDS, this volume shrinks. Dynamic resize hands space back after you delete data, which is something RDS storage has never done. Sources: Aurora compute and storage are decoupled; an instance failure does not affect the cluster volume, With no replica, Aurora creates a replacement DB instance, which takes materially longer.

02

Cluster Create

Let's create it. And the first difference is the API you call. On RDS you create a DB instance. On Aurora you create a cluster, and then you create instances inside it. Engine, Aurora PostgreSQL. Notice there is no storage size field anywhere on this page. But there is a choice that decides your bill. Aurora Standard charges you per storage request. I O Optimized doesn't, and costs more per hour. We'll come back to that. Your subnet group needs at least two availability zones. Use three where you can. Aurora's storage keeps six copies across three either way.

Networking is the same discipline as any production database. The cluster sits in private subnets, and that subnet group has to span at least two zones. The security group allows five four three two from your application only, never from anywhere. Force TLS with the cluster parameter, and verify the certificate from the client. IAM authentication gives you short lived tokens instead of stored passwords. And public accessibility stays off.

Now prove it. Connect to the cluster endpoint with verify-full, so the certificate is actually checked. Ask the server whether it's in recovery. False means you reached the writer. Ask for the Aurora version, and you get the engine build underneath. One rule before we go on. There is an instance endpoint for every instance, and it must never appear in your application config.

Aurora splits parameters into two groups, and that trips people up. The cluster parameter group holds settings that must agree across the cluster. Logical replication, force SSL. The DB parameter group is per instance. Shared buffers, work memory, max connections. Which gives you something RDS can't do. You can tune a reader differently from the writer. Static parameters still need a reboot to take effect.

Serverless version two changes how you buy compute. Capacity is measured in Aurora capacity units. One unit is roughly two gibibytes of memory plus matching CPU and network. You set a floor and a ceiling, and Aurora resizes in place, in seconds, with no failover. It can go all the way down to zero on recent PostgreSQL versions, and pause. But be honest about the trade. Waking up costs about fifteen seconds, and at steady load this is more expensive than a provisioned instance. Sources: AWS operates the distributed storage layer, replication, repair and backup; the customer owns schema, queries, parameters and client behaviour, The cluster volume is divided into 10 GiB protection groups, each replicated six ways with two copies in each of three Availability Zones.

Observe and verify
bash
aws rds describe-orderable-db-instance-options \--engine postgres \--engine-version "$ENGINE_VERSION" \--region "$AWS_REGION" \--query 'OrderableDBInstanceOptions[].DBInstanceClass' \--output text | tr '\t' '\n' | sort -u
Run the failure drill
bash
aws rds reboot-db-instance --db-instance-identifier "$SINGLE_ID"; }
03

Backups and recovery

Backup on Aurora works differently, and it's genuinely better. It's continuous, and it happens inside the storage layer. Not a job running on your instance. So there is no backup window, no snapshot pause, and nothing to schedule around. Retention runs one to thirty five days, and you can land on any second inside it. One honest note on cost. You're billed on how much the volume changes. So a write heavy database is expensive to keep for thirty five days.

Point in time recovery is still not an undo button. And on Aurora there's an extra thing to understand. Restore doesn't rewind your cluster. It builds a new one. New volume, new writer, new endpoints. Which means your application is pointed at the old cluster until you move it. Check the earliest restorable time before you promise anyone anything. And you choose the storage billing mode again at restore.

Two things here, and one of them is a warning. First, cloning. Aurora can hand you a full writable copy of a terabyte cluster in minutes, because it copies nothing. The clone shares the same pages, and you only pay for what diverges. That's the best pre-migration tool Aurora has. Now the warning. People come to Aurora expecting Backtrack, that rewind the cluster in seconds feature. Watch what happens when you ask for it. Backtrack is Aurora MySQL only. On PostgreSQL, your undo is point in time recovery into a new cluster.

Now add readers. And this is where the shared volume pays off. An Aurora replica doesn't get a copy of your data. It attaches to the same volume the writer is already using. So adding a reader adds compute, and nothing else. No second copy, no storage bill, no replication stream to fall behind. You can run up to fifteen of them. Every one of them is reading pages the writer just wrote.

Replica lag changes unit, and that's the whole lesson. On RDS you watch lag in seconds. On Aurora you watch it in milliseconds. Typically ten to twenty, and usually well under a hundred. Because nothing is being shipped. The pages are already there. What you're actually measuring is how fast a reader applies and caches them. So the causes are different, and so are the fixes. Sources: Aurora writes redo log records to the storage layer, which materializes pages, rather than writing data pages from the instance, Lost segments are repaired from peer copies per protection group.

Observe and verify
bash
out=$(PGCONNECT_TIMEOUT=2 psql -h "$host" -d "$db" -tA -F'|' -c \"SELECT inet_server_addr(), pg_is_in_recovery(), clock_timestamp();" 2>/dev/null) || out=""
04

Endpoints

Aurora gives you four kinds of endpoint, and each has one job. The cluster endpoint always points at the writer. The reader endpoint spreads connections across your readers. Custom endpoints let you name your own subset of instances. And instance endpoints are for debugging, only. Let's prove the reader is read only. Ask it whether it's in recovery, and it says true. Try to insert, and PostgreSQL refuses, because it's a read only transaction.

One warning about the reader endpoint. It is not a load balancer. It's DNS, and it round robins per new connection. It never rebalances the connections you already have. So a long lived pool resolves once, and can pin every single connection onto one reader. Three readers, and one of them is doing all the work. Custom endpoints isolate reader groups. The real fix is recycling connections, or an RDS Proxy endpoint.

When the writer dies, Aurora promotes a reader. And because the volume already holds every acknowledged commit, promotion isn't data recovery. It's a role change, plus a cache that has to warm up. Which reader wins? You decide, with promotion tiers, zero through fifteen. Lowest number wins. If two readers tie, the larger instance wins. So set them deliberately. Otherwise Aurora will happily promote your smallest analytics reader into production.

Let's simulate a writer failure. One command triggers the failover. Watch the probe. Writes fail for a few seconds, the cluster endpoint moves, and the promoted reader starts taking writes. AWS documents under sixty seconds, and often under thirty when a replica is there to promote. Every acknowledged commit survived, and here's why. The volume was never in doubt. Four of six copies had every one of those commits before the writer died. In flight transactions still aborted. The old writer comes back as a reader.

Now the same test, on a cluster with one instance. No reader, so nothing to promote. Watch how different this is. Aurora has to provision a replacement writer, and that is minutes of downtime, not seconds. AWS documents that replacement as typically under ten minutes. So be clear about what you're buying. Fast failover comes from having a promotable reader, not from Aurora by itself. Sources: Failover typically under 60s, often under 30s when an Aurora Replica is available for promotion, Aurora PostgreSQL clusters support a maximum volume size of 256 TiB, raised from 128 TiB.

05

Storage Fault

Now let's simulate losing an availability zone, and watch the promise from earlier hold. Two of the six copies are gone. Four remain, four of six is still the write quorum, and look at the probe. Not one failed write. Meanwhile the volume rebuilds the missing segments from surviving peers, ten gibibytes at a time, in parallel. That's why recovery is minutes and not hours. The blast radius of a bad disk is one segment.

Everything so far has been one region. Aurora Global Database extends the same trick across regions. The storage layer replicates, not the engine, so there's no logical replication to break. You get up to ten secondary regions, sixteen readers in each, and lag that's typically under a second. A planned failover is managed and loses nothing. An unplanned promotion is faster, but you accept a small gap. Secondary regions are read only. Write forwarding sends their writes to the primary. And you pay for a second region you hope never to use.

RDS Proxy works with Aurora, and it does two useful things. It pools connections, so thousands of clients become a handful of database sessions. And it gives you read only proxy endpoints that route to your readers. Use it mainly for connection management. But don't dismiss the failover benefit. It bypasses DNS caching and holds application connections through a promotion.

Here's the proxy failure mode nobody warns you about. Multiplexing only works while a session stays clean. The moment a client does something session specific, the proxy pins that connection to one backend, and stops sharing it. Set a parameter, prepare a statement, create a temp table, listen on a channel, take a session advisory lock. All of those pin. Watch pinning climb, and your effective pool collapses. PostgreSQL has no pinning filters to turn this off.

What you watch on Aurora is different, because some of these numbers are money. Volume bytes used is your storage bill. Volume read and write counters are your request bill, if you're on Standard. Aurora replica lag, in milliseconds. Capacity unit utilization, if you went serverless. And the buffer cache hit ratio, which on Aurora is not just latency. A cache miss is a billable read. Shared buffers is now a cost parameter. Sources: Aurora uses a 4 of 6 write quorum and a 3 of 6 read quorum across six copies in three AZs, Aurora tolerates loss of an entire AZ without losing write availability, and AZ plus one additional copy without losing read availability.

SQL verification
sql
SET application_name = 'pin-test';PREPARE get_time AS SELECT now();CREATE TEMP TABLE session_data (id integer);LISTEN application_event;SELECT pg_advisory_lock(1001);   -- session-level lock pins; txn-level (pg_advisory_xact_lock) does NOT
Run the failure drill
bash
aws rds reboot-db-instance --db-instance-identifier "$MULTI_ID" --force-failover; }
06

Maintenance Zdp

Minor version upgrades try something clever, called zero downtime patching. Aurora waits for a quiet moment, preserves your connections, and swaps the engine underneath them. When it works, nobody notices. But read the words best effort carefully. A long running transaction, open temporary tables, an active logical replication slot. Any of those, and Aurora falls back to an ordinary restart. So you can plan for probably zero downtime. You cannot promise zero.

Major version upgrades are a different problem, and Aurora has a good answer. A blue green deployment clones your cluster, upgrades the copy, and keeps it in sync with logical replication. You test the green cluster properly, with real traffic shapes, for as long as you like. Then you switch over, and AWS documents that as typically under a minute. Your endpoints follow. The old cluster stays around, in case you need it.

Before you decide, here's what Aurora is not. There is no operating system and no shell. rds_superuser is not a superuser. You get the extensions AWS supports, and no others. There is no Backtrack on PostgreSQL, so no rewind. There are no storage knobs at all. No volume type, no throughput setting, no filesystem, so your only lever is doing fewer reads and writes. Max connections is bound by a formula on instance memory. And nothing here protects you from a delete you wrote yourself.

Now the part that decides whether Aurora fits. Compute is the obvious line, and readers multiply it. Storage is per gibibyte of what you use. And then there are requests. On Aurora Standard, every storage read and write is metered, and that line has no ceiling. A plan that flips to a sequential scan now costs you money, every time it runs. I O Optimized removes that risk. You pay about thirty percent more per hour, and no request charges. Once requests pass about a quarter of your Aurora spend, switch. And know the direction. Dropping back to Standard is available any time. Going the other way is once every thirty days.

So here's the checklist. One volume, six copies, three zones. At least one replica, or you don't actually have fast failover. Promotion tiers set on purpose. Endpoints routed properly, with custom endpoints for analytics. Clone before you migrate, and remember there is no rewind. And pick your storage billing mode deliberately. Then three questions decide Aurora against RDS. How available does this need to be. How much read scale do you need. And how predictable does the bill need to be. Sources: rds_superuser is not a true PostgreSQL superuser and there is no OS access, The cluster volume grows automatically in 10 GiB increments with no customer-provisioned size or IOPS.

Run the failure drill
bash
aws rds failover-db-cluster --db-cluster-identifier "$CLUSTER_ID"; }