Runbook

Operational reference for the tunnel. Most failures recover on their own through systemd Restart=on-failure and the ASG EC2 health check; this chapter is the decision path and the manual procedures.

On-call quick reference

• Primary edge view: the Cloudflare Zero Trust dashboard shows each tunnel’s connector health. New Relic (account 497659) shows the host and whether cloudflared is running as a process.
• First question on a tunnel alert: is the daemon down or the instance down? If cloudflared died but the box is fine, systemd restarts it within seconds. If the instance failed EC2 status checks, the ASG replaces it from the AMI in a few minutes and the bootstrap re-fetches credentials.
• Access: Session Manager only, no SSH. Use aws ssm start-session or an aws ssm send-command.

Recovery scenarios

Scenario	Detection	Expected recovery	Operator action
cloudflared process crash	Connector drops in Zero Trust; process gone in New Relic	systemd restarts cloudflared within seconds	None unless it flaps repeatedly
Transient edge disconnect	Brief connector drop	cloudflared auto-reconnects	None
Instance failure	EC2 status-check failure	ASG replaces the instance from the AMI; bootstrap re-fetches creds; tunnel returns in a few minutes	Watch the replacement; if it fails to come up, run the manual recovery below
Bad AMI rolled	New instances fail to connect after a refresh	Roll back: re-point ami-id-latest to the previous AMI and refresh	See Deployment
Tunnel credentials rotated or revoked	Connector will not establish	Re-render via the cloudflare-tunnels module and instance-refresh	See Deployment
Aurora endpoint change	cloudflared logs show connection errors	If the writer DNS flipped, no action; if an endpoint was renamed, update the ingress in the cloudflare-tunnels module	Apply the module
Full environment rebuild	DR drill or catastrophic loss	terraform apply recreates the ASG, launch template, SG, and IAM; the Cloudflare tunnel and its secret persist	Apply the three modules; no need to recreate the Cloudflare tunnel

Manual recovery

When the ASG cannot bring up a healthy instance, work down this list, smallest reversible step first: confirm the subnet has free addresses; confirm the instance role can read /adventive/cloudflared/<env> and /adventive/newrelic/license-key from Secrets Manager; if the AMI is suspect, roll back the ami-id-latest pointer and refresh; if credentials were rotated, re-render and refresh. Host-side, an SSM send-command checking systemctl is-active cflared-bootstrap cloudflared newrelic-infra and the boot logs (journalctl -u cflared-bootstrap) pinpoints where boot failed.

Raise an environment to high availability

Two replicas across AZs is a change to that environment’s entry in the cflared-asg environments map, not a redesign. Add a second-AZ subnet to the entry’s subnet_ids and raise desired_capacity to 2 (and min_healthy_percentage to 50 so rolling refreshes keep one replica serving). Plan and apply the module. Both replicas run the same tunnel, Cloudflare registers both connectors and load-balances, and no AMI rebuild, credential change, or ingress change is needed. Confirm two healthy connectors in the Zero Trust dashboard, then run the failover drill (set one replica unhealthy, confirm the other carries traffic and the ASG replaces the terminated instance).

When to do it (production)

Flip production to two replicas when any one holds: a customer SLA is tied to a tunnel consumer; a revenue-critical real-time path ships through the tunnel; production sees two unplanned tunnel outages in a 90-day window; sustained P95 CPU on the prd host exceeds 40 percent; or leadership declares a consumer Tier 1.

Checking a tunnel host with Systems Manager

Tunnel hosts take no SSH and expose no inbound ports; all host access is through AWS Systems Manager. Every command below runs from the local sandbox that sources .cowork-env. Instances carry the tags adv:project=cloudflare-tunnel, adv:env=<env>, and adv:role=tunnel-runtime, so you can target them by tag without first looking up instance IDs.

List the running hosts for an environment:

cd ~/Repositories/GitHub/Adventive/adventive-platform-infra && . ~/Documents/Claude/.cowork-env && aws ec2 describe-instances --filters "Name=tag:adv:project,Values=cloudflare-tunnel" "Name=tag:adv:env,Values=prd" "Name=instance-state-name,Values=running" --query "Reservations[].Instances[].InstanceId" --output text --region us-east-1

Run the host health check across every host in an environment at once. It reports the three services, the live tunnel-connection count, and that the New Relic config was rendered:

cd ~/Repositories/GitHub/Adventive/adventive-platform-infra && . ~/Documents/Claude/.cowork-env && aws ssm send-command --document-name "AWS-RunShellScript" --targets "Key=tag:adv:project,Values=cloudflare-tunnel" "Key=tag:adv:env,Values=prd" --comment "cflared host health check" --parameters 'commands=["systemctl is-active cflared-bootstrap cloudflared newrelic-infra","curl -s http://127.0.0.1:2000/metrics | grep cloudflared_tunnel_ha_connections","grep -q enable_process_metrics /etc/newrelic-infra.yml && echo newrelic-config-ok"]' --region us-east-1 --query "Command.CommandId" --output text

Read the results with the command ID printed by the previous step:

cd ~/Repositories/GitHub/Adventive/adventive-platform-infra && . ~/Documents/Claude/.cowork-env && aws ssm list-command-invocations --command-id COMMAND_ID --details --region us-east-1 --query "CommandInvocations[].{Instance:InstanceId,Status:Status,Output:CommandPlugins[0].Output}" --output json

A healthy host returns active for all three services, a cloudflared_tunnel_ha_connections line at 1 or more, and newrelic-config-ok.

Open an interactive shell on one host when you need to read logs directly:

cd ~/Repositories/GitHub/Adventive/adventive-platform-infra && . ~/Documents/Claude/.cowork-env && aws ssm start-session --target INSTANCE_ID --region us-east-1

Inside the session, journalctl -u cflared-bootstrap shows the boot-time credential fetch and config render and journalctl -u cloudflared shows the tunnel connection. The bootstrap is a oneshot that runs once at boot: after rotating a secret or changing the AMI, replace the host with an instance refresh rather than expecting the running host to re-read.

Verification checklist

Run after standing up, rebuilding, or rolling an environment.

1. Image Builder build AVAILABLE; AMI ID present in /adventive/cloudflared/ami-id-latest.
2. terraform plan on each touched module reports no unexpected changes after apply.
3. Cloudflare Zero Trust shows the adv-cflared-<env> tunnel with the expected connector count for more than 10 minutes.
4. A throwaway Worker on the environment’s Hyperdrive resource runs SELECT 1 against Aurora successfully.
5. Host check (SSM send-command): cflared-bootstrap, cloudflared, newrelic-infra all active; /etc/newrelic-infra.yml has the license key and enable_process_metrics: true; curl -s http://127.0.0.1:2000/metrics returns cloudflared_tunnel_ha_connections at 1 or more.
6. The host appears in New Relic under Infrastructure with cloudflared in its process list.
7. Self-heal drill: stop cloudflared on the host and confirm systemd restarts it; separately, set the ASG instance health to Unhealthy and confirm the ASG replaces it from the AMI.