Servers that failed to launch, then fixed themselves
My first terraform apply of the web tier failed. The tempting fix was to tear it down and start again. The right fix was one command.
- Area
- AWS · Auto Scaling
- Project
- AWS cloud homelab
What I saw
terraform apply failed at aws_autoscaling_group.app, and Terraform marked the group as tainted, meaning it would destroy and recreate it on the next apply. The Auto Scaling activity history said: Authentication Failure. Launching EC2 instance failed.
Why it happened
IAM is eventually consistent. Terraform created the servers’ new instance profile and, seconds later, the Auto Scaling group that uses it. The group tried to launch servers before the new profile had reached EC2.
How I found it
Instead of re-running anything, I read the group’s Activity tab: two failed launches, then two successful ones about a minute later. AWS had retried on its own and the group was already healthy. Only Terraform’s record still said it was broken.
The fix
terraform untaint aws_autoscaling_group.app told Terraform the group was fine after all. terraform plan then showed no changes.
Lesson: new IAM roles take a moment to propagate. Read the activity history before rebuilding something that may already have recovered.