Service Discovery ForceNew: Operational Rule¶
Part of the BigRAE deploy and operations docs; start at the documentation home.
Observed ForceNew Attribute¶
The Terraform AWS provider marks health_check_custom_config.failure_threshold
inside aws_service_discovery_service as a ForceNew attribute. Changing this
field causes Terraform to delete the existing Cloud Map service and create a new
one in its place.
This list is not exhaustive. The ForceNew set is determined by the provider version in use; a provider upgrade may introduce additional ForceNew attributes on this resource type.
Why It Matters¶
ECS uses Cloud Map to route traffic between services. When ECS tasks are
registered in a Cloud Map service and Terraform attempts to delete that service,
the AWS API rejects the delete with a ResourceInUse error. The apply fails
mid-way, leaving infrastructure in a partial state.
Operational Rule¶
Any Terraform plan that contains a delete action on
aws_service_discovery_service requires ECS task deregistration before the
apply runs:
- Scale affected ECS services to
desiredCount=0. - Wait for
runningCountto reach 0. - Wait for Cloud Map
list-instancesto return zero instances. - Run
terraform apply. - Restore each ECS service to its original
desiredCount.
Steps 1-3 are the drain phase; step 5 is the restore phase.
Implementation¶
scripts/handle_sd_replacement/handle_sd_replacement.py automates this:
# Before terraform apply (drain)
python3 scripts/handle_sd_replacement/handle_sd_replacement.py \
drain --tf-plan tfplan --tf-outputs-from <terraform-dir>
# After terraform apply (restore)
python3 scripts/handle_sd_replacement/handle_sd_replacement.py \
restore --snapshot sd_replacement_snapshot.json
The script reads the saved plan via terraform show -json, determines which ECS
services correspond to the affected Cloud Map services using the
_SD_SUFFIX_TO_OUTPUT_KEY mapping in the source file, and writes a
sd_replacement_snapshot.json with the original desired counts. The restore
step is idempotent: if the snapshot is empty or absent, it exits 0 without
calling AWS.
The deploy pipeline in .github/workflows/_shifter-platform.yml runs both
phases automatically around the terraform apply step.