Bare-Metal K3s Cluster with PostgreSQL HA

The client had six on-premise servers all configured by hand. Each box ran a different Ubuntu version with different kernel parameters, inconsistent firewall rules, and SSH configurations that varied from password auth on some to key-only on others. Every time they needed a new environment, an engineer spent two to three days SSHing into boxes and running commands from a shared Google Doc that was perpetually outdated. PostgreSQL replication was set up manually and had silently broken on the standby twice without anyone noticing until a failover was actually needed. When the failover failed, they lost four hours of data because the last backup was a manual pg_dump from six days prior. There was no monitoring for replication lag, no backup integrity verification, and no consistent security baseline across servers. One server had no firewall at all. Another had root SSH login enabled with a password.

The Ansible project is organized into seven reusable roles: base-os, security, k3s-server, k3s-agent, postgresql-primary, postgresql-standby, and nginx-proxy. The base-os role runs first on every server, normalizing the Ubuntu version, setting kernel parameters via sysctl for network performance tuning, configuring NTP synchronization, setting timezone to UTC, and ensuring consistent package sources. The security role configures UFW with default-deny ingress and explicit allow rules for required ports only from trusted CIDR ranges, installs fail2ban with a three-attempt SSH threshold and ten-minute ban with permanent ban after five recurrences, hardens SSH to key-only authentication with root login disabled and idle timeout at three hundred seconds, and enables unattended-upgrades for automatic security patches with a reboot window outside business hours. The k3s-server role installs K3s in HA mode with embedded etcd across three control plane nodes, configuring automatic TLS certificate rotation and generating the kubeconfig for remote access. The postgresql-primary role installs PostgreSQL, configures WAL archiving to a local archive directory, sets up streaming replication parameters, creates the replication user, and configures pg_hba.conf for standby connections. The postgresql-standby role runs pg_basebackup from the primary to initialize the data directory, configures recovery.conf for streaming replication, and sets up a cron job that queries pg_stat_replication on the primary every five minutes and sends a Slack webhook alert if replication lag exceeds thirty seconds. Daily backups use pg_dump piped through gzip, stored on an NFS mount with thirty-day retention enforced by a find-and-delete cron job. Each backup runs pg_restore --list against the dump file to verify integrity before marking success. The nginx-proxy role installs Nginx with reverse proxy configuration, runs certbot for TLS certificates with automatic renewal via systemd timer, and configures HTTP to HTTPS redirect. Rolling updates use an Ansible playbook that cordons and drains each node sequentially, applies the update, and uncordons before proceeding to the next node.

Automated the provisioning and hardening of a production K3s Kubernetes cluster with high-availability PostgreSQL on bare-metal servers using idempotent Ansible playbooks. The stack bootstraps from a single command in under thirty minutes. K3s runs in HA mode with embedded etcd across three control plane nodes. PostgreSQL uses streaming replication with WAL archiving for point-in-time recovery. Automated pg_basebackup initializes standby nodes without manual intervention. A cron job monitors replication lag every five minutes and fires Slack alerts if lag exceeds thirty seconds. Security hardening includes UFW firewall with default-deny, fail2ban for SSH brute-force protection, key-only SSH authentication with root login disabled, and automatic security updates via unattended-upgrades. Daily backups run pg_dump to an NFS share with thirty-day retention and integrity verification.

Eliminated manual server provisioning entirely. What previously took two to three days of an engineer SSHing into boxes and running commands from a shared Google Doc now completes in under thirty minutes from a single ansible-playbook command. PostgreSQL replication had silently broken twice before without detection. The automated lag monitoring caught a replication break within five minutes during the first week of deployment. Backup integrity checks caught a corrupted dump file on day twelve that would have gone unnoticed until a restore was needed. The hardened security posture blocked over four hundred SSH brute-force attempts in the first month via fail2ban.