Dr. Ibrar Ahmed

HomePostgreSQLArticle

PostgreSQL Mechanics

Amazon RDS PostgreSQL Database: What Happens When It Fails?

Dr. Ibrar Ahmed10 min readFrom the lecture notes

What you will learn

  • Managed doesn't mean it never breaks.
  • Status says available, so let's connect.
  • Lag is how far a replica trails the primary, and CloudWatch reports it as ReplicaLag, in seconds.
  • A Multi-AZ DB cluster isn't just more nodes.
  • Every choice here has a price, so let's read the bill.
The production request pathSystem sketch
ApplicationRetry-aware clientStable endpointRouting layerPostgreSQLWriter and replicas
01

The failure model

Managed doesn't mean it never breaks. When the writer goes away, every open connection goes with it. What happens next depends on the architecture you picked. Single-AZ has no standby, so recovery means a restart, or a replacement. AWS documents sixty to one hundred twenty seconds for a Multi-AZ DB instance, and under thirty-five for a Multi-AZ DB cluster. Sockets never migrate. So we build each pattern, break it, and measure what the application sees.

So who owns what? AWS takes the host operating system, engine patching, automated backups, storage, and the failover machinery. Everything above that is yours. The data model, your queries, how you handle connections, parameter tuning, security groups, and when updates get applied. You get rds_superuser. Powerful, but not unlimited. There's no shell on the host, and some parameters stay locked. Know that line, and RDS stops surprising you.

The simplest deployment is one DB instance, in one availability zone. One compute instance runs PostgreSQL, attached EBS storage holds your data, and backups go to S3. Look at what's missing. There's no standby to take over. If that instance fails, RDS has to launch a replacement, attach storage, and recover. That takes time. Fine for development, or anything that tolerates downtime, but real traffic needs more. We start here, then add redundancy.

Let's create it in the console. Engine, PostgreSQL, version eighteen point four, and that's only an example, so confirm what your Region offers. Instance class db.t3.medium, storage gp3, one hundred gigabytes. Multi-AZ stays off for this first build, and we set the master username and password. The VPC, subnet group, and security group decide who reaches this database. Public accessibility, disabled. That one matters. Automated backups on, seven day retention, then review and create. Provisioning runs a few minutes while AWS allocates compute, storage, and networking. Sources: Managed RDS still fails on host/AZ loss; existing connections terminate; recovery time depends on deployment choice, Shared responsibility: AWS runs host/OS/storage/failover; you own schema, params, security, reconnect.

Observe and verify
bash
aws rds describe-orderable-db-instance-options \--engine postgres \--engine-version "$ENGINE_VERSION" \--region "$AWS_REGION" \--query 'OrderableDBInstanceOptions[].DBInstanceClass' \--output text | tr '\t' '\n' | sort -u
Run the failure drill
bash
aws rds failover-db-cluster --db-cluster-identifier "$CLUSTER_ID"; }
02

Connect and verify

Status says available, so let's connect. psql, with the endpoint hostname RDS handed us. SSL is on by default. First thing I check is the version. Then pg_is_in_recovery. It comes back false, and that's what I want to see. This server isn't recovering, it's the primary. Pull the availability zone from RDS metadata, and there it is. One instance, one zone, taking writes. The foundation is ready for data.

AWS runs the storage subsystem, but type and size are your call. GP3 gives a tunable baseline, IOPS and throughput set separately. IO2 targets higher durability and sustained performance. Autoscaling grows the volume when free space stays under ten percent for at least five minutes, with no storage change in the previous six hours. It only grows, never shrinks. Watch FreeStorageSpace. Run out, and the instance goes storage-full, and it can go unavailable. Provisioned IOPS need matching throughput and instance class.

There's no config file here. Settings live in parameter groups, and the default group is read-only, so you create your own. Then it splits: dynamic parameters like work_mem apply right away, static ones like shared_buffers wait for a reboot, and a few stay protected. RDS-specific ones expose what you can change, so enabling rds.logical_replication moves wal_level for you. Static changes stay pending until you reboot, by hand.

Three things decide access. Subnet group, security group, authentication. The subnet group picks which zones can host the database. The security group is your firewall, and it should allow the app tier on the database port, nothing else. Private subnets keep it off the internet. IAM authentication issues short-lived tokens instead of passwords, and SSL protects the traffic. When connections fail, check those rules first. That's usually it. Sources: Single-AZ = one instance, one AZ, EBS volume; genuine host failure triggers replacement, db instance class is an example; verify per Region/version.

Observe and verify
bash
out=$(PGCONNECT_TIMEOUT=2 psql -h "$host" -d "$db" -tA -F'|' -c \"SELECT inet_server_addr(), pg_is_in_recovery(), clock_timestamp();" 2>/dev/null) || out=""
Run the failure drill
bash
aws rds reboot-db-instance --db-instance-identifier "$SINGLE_ID"; }
03

Backups and recovery

Automated backups are two parts. A daily snapshot, plus transaction logs to S3 about every five minutes. Retention, one to thirty-five days, sets how far back you can go. On single-AZ, expect brief IO suspension during the backup window. Multi-AZ takes it from the standby, so the primary never feels it. Those logs give you point-in-time recovery, to almost any second in the window. Manual snapshots stay until you delete them. Storage in the regional allowance is included, past that you're billed. Keep that window away from your busy hours.

Here's the part people get wrong. Point-in-time recovery doesn't rewind your database. It builds a brand new instance, restored to the timestamp you name. RDS takes the nearest snapshot, then replays transaction logs to that exact second. New instance, new endpoint, so the application has to be repointed. And unless you say otherwise, the restore uses default network and parameter settings. It's a recovery tool, not an undo button. Test it before the day you need it.

Read replicas move read traffic off the primary, using asynchronous streaming replication. Up to fifteen per primary, in-Region or cross-Region, each with a read-only endpoint. A replica can lag, so a write you just made may not be there yet. Delete the source, and same-Region replicas are promoted to standalone instances, newest writes at risk. Use them for read scale. Promotion helps in a disaster. Neither replaces a backup.

Now the read replica endpoint. pg_is_in_recovery comes back true, so this one's receiving WAL. A select works fine, it's reading committed data. Watch what happens when I insert. Rejected. Cannot execute INSERT in a read-only transaction. That's not a setting somebody flipped, PostgreSQL is in recovery mode, and no parameter changes that while replication is active. Your application has to know the difference: pools and load balancers send writes to the primary, reads to the replicas. Sources: PostgreSQL 18.4 is an example minor; confirm availability in your Region, Fields shown (backup retention, encryption, deletion protection, log exports, Insights) are create-time settings.

04

Replica Lag

Lag is how far a replica trails the primary, and CloudWatch reports it as ReplicaLag, in seconds. Milliseconds to a few seconds under write load? Normal. Large lag is different. Readers can't see recent writes, and an app reading its own writes gets stale data. Usual causes. A write burst, network trouble, or a long query blocking replay. Alarm on sustained lag. Growth under load is expected. Growth that never stops is not.

Let's break it. A controlled reboot on the single-AZ instance. That stops and restarts the database service on the same instance. No standby here, so nothing takes over while it comes back. Console says rebooting, existing connections are gone, and new attempts get connection refused. Then the service returns, and the same instance accepts connections again. How long were we down? Tens of seconds to several minutes, depending on the recovery work, unavailable the whole time. Single-AZ has no automatic failover. Fast recovery in production means Multi-AZ.

A Multi-AZ DB instance adds a synchronous standby in a different availability zone. One endpoint covers both. A write doesn't commit until both have it, and yes, that costs latency. In return, the standby holds every committed transaction. You can't read from it. It exists for one job. Failover. When the primary fails, RDS promotes the standby and updates the DNS record. Connections die, applications reconnect, and DNS caching decides how fast they find the new address. Sixty to one hundred twenty seconds, typically. Treat that window as downtime.

Reboot, with force failover. The primary shuts down, and every active connection terminates with it. In-flight transactions abort. But anything already acknowledged is safe, because the synchronous standby had it first. The standby is promoted, and RDS repoints the DNS record to it. Some clients keep retrying the old address for a moment. That's caching, not a bug. And we land inside the documented sixty to one hundred twenty second range. The old primary comes back later, as the new standby. Sources: Autoscaling triggers require free space < 10% sustained 5 minutes and 6 hours since last storage modification, Parameter groups; static vs dynamic; default group read-only; rds.logical_replication sets wal_level=logical.

Run the failure drill
bash
aws rds reboot-db-instance --db-instance-identifier "$MULTI_ID" --force-failover; }
05

Multi-AZ Cluster

A Multi-AZ DB cluster isn't just more nodes. It's a different architecture. One writer, two readable standbys, across three availability zones. The transaction log is forwarded to the readers, and a commit needs at least one to acknowledge. This isn't Aurora, though it borrows ideas. Two endpoints, writer and reader. On failover a reader is promoted, and because it already holds recent data, you're under thirty-five seconds, typically. Faster than the DB instance.

Two endpoints, two jobs. The writer endpoint always points at the current writer. Use it for every write, and for reads that need the latest data. The reader endpoint balances connections across the readers, and that's where read queries belong, if they tolerate a little lag. Let's prove it. On the writer endpoint, pg_is_in_recovery is false. On the reader endpoint, true. And a write sent to the reader endpoint fails. So routing is your job, not the cluster's. One pool for the writer, a separate one for the readers.

Same test, on the cluster. We trigger failover on the writer. The API shuts the writer down, and connections to it terminate. A reader begins promotion, and because log forwarding already gave it recent transactions, there's little to catch up on. The writer endpoint DNS moves to the new writer. Watch the reader endpoint too, connections there can fail briefly while the cluster reorganizes. Under thirty-five seconds, in the documented typical case. Anything reconnecting to the writer endpoint reaches the new primary. The former writer returns as a reader, so the cluster is three instances again.

RDS Proxy sits between the application and the database. Your app connects to the proxy, the proxy keeps a pool to the database, and many clients share fewer backend connections. During failover, it absorbs the connection failures and reconnects to the new primary. Idle connections usually survive. An active query or transaction can still fail, and you retry it. It pays off for connection storms and short-lived connections. Long, steady connections gain less. Sources: Multi-AZ DB instance failover typically 60-120s; Multi-AZ DB cluster failover typically under 35s, pg_is_in_recovery()=f confirms writable primary; SSL/TLS on by default.

06

Session Pinning

And here's the catch. Pinning. When a session needs backend state to stay put, the proxy binds your connection to one database connection, and multiplexing is gone. What causes it? Prepared statements, SET commands, advisory locks, temporary tables, some sequence operations. CloudWatch shows how often it happens, and the proxy logs show why. High pinning means you're paying for proxy capacity and getting little of the benefit. So watch the session state your code leaves behind, and use the extended query protocol carefully.

Diagnosis starts in CloudWatch Database Insights. CPU, memory, storage, IOPS, connections, and the one that explains things, database load by wait event. Performance Insights redirects here after July thirty-first, twenty twenty-six. Top SQL shows your heaviest queries, high IO wait points at storage, and high CPU with no IO wait points at compute. Observe first, then change one thing.

Maintenance happens in the window you configure. Minor version upgrades can be automatic, or you drive them. Major upgrades are explicit, and always tested first. Blue-Green deployments give you a parallel environment for exactly that, test before you cut over. Now the limits worth writing down. Maximum connections follows instance memory. Storage never shrinks after you scale up. Some parameter changes need a reboot. Pending maintenance shows in the console, though some applies immediately, window or not. Watch the AWS Health Dashboard, and plan maintenance instead of reacting to it.

One more truth before you ship this. Managed PostgreSQL isn't PostgreSQL on a server you own. There's no operating system login, and no shell. Your account is rds_superuser, powerful, but not a true superuser. Extensions come from the catalog RDS supports, and that's the list. You can't touch the filesystem. Some parameters stay locked. AWS owns the failover internals, so you tune around them, never through them. And two things RDS will never do for you. It won't retry your application's transactions. And it won't save you from logical corruption you caused yourself. Learn those edges here, not in production. Sources: rds_superuser is not a true superuser; no OS access, gp3 baseline + configurable IOPS/throughput; autoscaling grows, never shrinks; storage-full state.

SQL verification
sql
SET application_name = 'pin-test';PREPARE get_time AS SELECT now();CREATE TEMP TABLE session_data (id integer);LISTEN application_event;SELECT pg_advisory_lock(1001);   -- session-level lock pins; txn-level (pg_advisory_xact_lock) does NOT
07

Cost Anatomy

Every choice here has a price, so let's read the bill. You pay for instance hours, and Multi-AZ or a cluster multiplies them by the standby and readers. You pay for allocated storage, provisioned IOPS and throughput, and backups beyond the allowance. Then read replicas, cross-Region transfer, RDS Proxy, monitoring, each adds a line. Roughly speaking. Single-AZ is cheapest, least available. A Multi-AZ instance costs about double, for a standby you can't read. A cluster costs more, and its two extra instances serve reads. Cost structure, not a quote.

You've watched every pattern built and broken. Single-AZ for development. Multi-AZ DB instance for production, sixty to one hundred twenty second failover. Multi-AZ DB cluster for faster failover and readable standbys. Replicas for read scale, proxy for connections, Insights for truth. And the edges: no OS access, no automatic transaction retry. Three questions decide it: how available, how much read scale, how much work you own. Next episode, Amazon Aurora PostgreSQL, compute and storage pulled apart, six copies, three zones. Subscribe, and I'll see you in the next build.