Operations

Chapter

Chapter 13: Troubleshooting & Debugging

The Friction

✕ Cryptic Errors: Error: 403 Forbidden (With no context).
✕ Ghost Resources: Terraform fails but resource was created.
✕ Dependency Loops: Circular logic preventing plan generation.

The R8way Solution

✓ Structured Debugging: Step-by-step guides for common failures.
✓ State Surgery: Runbooks for terragrunt state rm/import.
✓ Log Aggregation: Centralized logs for deeper analysis.

13.1 Common Issues & Solutions

State Locking Errors

Error:
The error Error: Error acquiring the state lock occurs when another process is holding the DynamoDB lock.

Code

Error: Error acquiring the state lock
LockID: iac-state-bucket-123456789/dev/dev-1/vpc/terraform.tfstate

Causes:

Another operation in progress
Previous operation crashed (stuck lock)
CI/CD pipeline conflict

Solutions:
1. Wait 5-10 minutes (locks auto-expire)
2. Check for active operations:

Code

aws dynamodb scan --table-name iac-state-lock-table

Manually release stuck lock:

Code

aws dynamodb delete-item \
  --table-name iac-state-lock-table \
  --key '{"LockID": {"S": "iac-state-bucket-ACCOUNT/dev/dev-1/vpc/terraform.tfstate"}}'

Prevention:

Use component-specific operations
Implement lock timeout in CI/CD
Monitor for stuck locks

Permission Denied

Error:
The error Error: AccessDenied: Access Denied indicates insufficient IAM permissions.

Code

Error: AccessDenied: Access Denied

Causes:

IAM role lacks permissions
Wrong AWS credentials
Incorrect role ARN

Solutions:
1. Verify IAM role:

Code

# Check role ARN in env.hcl
cat live/dev/env.hcl

# Verify role exists
aws iam get-role --role-name IaCExecutionRole

Check permissions:

Code

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject",
    "dynamodb:PutItem",
    "dynamodb:GetItem",
    "dynamodb:DeleteItem",
    "ec2:*",
    "eks:*",
    "rds:*"
  ],
  "Resource": "*"
}

Verify credentials:

Code

aws sts get-caller-identity

Module Not Found

Error:
The error Error: Failed to download module happens when Terraform cannot locate the source.

Code

Error: Failed to download module

Causes:

Module source path incorrect
Network connectivity issues
Module version unavailable

Solutions:
1. Verify module source:

Code

# Check _env/vpc.hcl
terraform {
  source = "${get_parent_terragrunt_dir()}/../../modules/vpc"
}

Re-run scaffold:

Code

./scripts/scaffold.sh live/dev/dev-1

Re-initialize:

Code

cd live/dev/dev-1/vpc
terragrunt init

Dependency Failures

Error:
The error Error: Error reading dependency output occurs when a dependent module hasn't been applied.

Code

Error: Error reading dependency output

Causes:

Dependency not deployed
Dependency outputs missing
Mock outputs incorrect

Solutions:
1. Deploy dependency first:

Code

# Deploy VPC first
./scripts/apply.sh live/dev/dev-1 vpc

# Then deploy EKS
./scripts/apply.sh live/dev/dev-1 eks

Check dependency outputs:

Code

cd live/dev/dev-1/vpc
terragrunt output

Verify mock outputs:

Code

# Check _env/eks.hcl
dependency "vpc" {
  mock_outputs = {
    vpc_id = "vpc-00000000"
    private_subnets = ["subnet-000000", "subnet-000001"]
  }
}

Terraform Crashes

Error:
The error Error: Unexpected error occurred suggests a provider bug or state corruption.

Code

Error: Unexpected error occurred

Causes:

Resource conflicts
State corruption
Provider bugs

Solutions:
1. Check Terraform logs:

Code

export TF_LOG=DEBUG
terragrunt plan

Refresh state:

Code

terragrunt refresh

Import missing resources:

Code

terragrunt import aws_vpc.example vpc-12345678

13.2 Debugging Techniques

Verbose Logging

Enable Debug Logging:

Code

export TF_LOG=DEBUG
export TF_LOG_PATH=./terraform.log
terragrunt plan

Log Levels:

TRACE: Most verbose
DEBUG: Detailed
INFO: Normal (default)
WARN: Warnings only
ERROR: Errors only

State Inspection

List Resources:

Code

cd live/dev/dev-1/vpc
terragrunt state list

Show Resource:

Code

terragrunt state show module.vpc.aws_vpc.this[0]

State Format:

Code

terragrunt state show -json | jq '.'

Dependency Graphs

Generate Graph:

Code

cd live/dev/dev-1
terragrunt graph | dot -Tpng > graph.png

View Dependencies:

Code

terragrunt output -json | jq 'keys'

Plan Output Analysis

Save Plan:

Code

terragrunt plan -out=tfplan
terragrunt show -json tfplan | jq '.'

Analyze Changes:

Code

terragrunt show -json tfplan | \
  jq '.resource_changes[] | select(.change.actions[] == "create")'

13.3 Error Message Reference

Common Error Messages

Error	Cause	Solution
Error acquiring state lock	Lock exists	Wait or release lock
Access Denied	IAM permissions	Check role permissions
Module not found	Source path wrong	Re-scaffold and init
Dependency not found	Dependency not deployed	Deploy dependency first
Invalid variable	Missing required var	Check values.yaml
Resource already exists	State mismatch	Import or refresh
Timeout	Operation too slow	Check AWS service status

13.4 Recovery Procedures

Corrupted State

Symptoms:

Terraform can't read state
Resources exist but not in state
Plan shows unexpected changes

Recovery:
1. List state versions:

Code

aws s3api list-object-versions \
  --bucket iac-state-bucket-ACCOUNT \
  --prefix dev/dev-1/vpc/terraform.tfstate

Download previous version:

Code

aws s3api get-object \
  --bucket iac-state-bucket-ACCOUNT \
  --key dev/dev-1/vpc/terraform.tfstate \
  --version-id <VERSION_ID> \
  recovered-state.tfstate

Verify recovered state:

Code

terraform show recovered-state.tfstate

Upload recovered state:

Code

aws s3 cp recovered-state.tfstate \
  s3://iac-state-bucket-ACCOUNT/dev/dev-1/vpc/terraform.tfstate

Failed Deployments

Symptoms:

Partial resource creation
Resources in inconsistent state
Error during apply

Recovery:
1. Check current state:

Code

terragrunt state list

Review failed resources:

Code

terragrunt show | grep -A 10 "Error"

Fix issues and re-apply:

Code

# Fix values.yaml or module
./scripts/plan.sh live/dev/dev-1 vpc
./scripts/apply.sh live/dev/dev-1 vpc

Clean up orphaned resources (if needed):

Code

# Import missing resources
terragrunt import aws_vpc.example vpc-12345678

# Or remove from state
terragrunt state rm aws_vpc.example

Orphaned Resources

Symptoms:

Resources exist in AWS but not in state
Plan shows resources to be created (already exist)

Recovery:
1. Import resources:

Code

terragrunt import aws_vpc.example vpc-12345678

Or remove resources manually:

Code

# Delete in AWS Console or CLI
aws ec2 delete-vpc --vpc-id vpc-12345678

13.5 Chapter Summary

Executive Summary

★
Common Issues
State locking errors
Permission denied
Module not found
Dependency failures
Terraform crashes
★
Debugging Techniques
Verbose logging
State inspection
Dependency graphs
Plan analysis
★
Recovery Procedures
Corrupted state recovery
Failed deployment recovery
Orphaned resource cleanup
★
Best Practices
Always plan before apply
Backup state files
Monitor for stuck locks
Document recovery procedures