Error when instance changed that has EBS volume attached
Problem
This is the specific error I get from terraform: [code block] We are building out some infrastructure in EC2 using terraform (v0.6.0). I'm currently working out our persistent storage setup. The strategy I'm planning is to have the root volume of every instance be ephemeral, and to move all persistent data to a separate EBS volume (one persistent volume per instance). We want this to be as automated as possible of course. Here is a relevant excerpt from our terraform config: [code block] And mount.sh: [code block] As you can see, this: - Provisions an instance to run Rundeck (http://rundeck.org/) - Provisions an EBS volume based off of a snapshot. The snapshot in this case is just an empty ext4 partition. - Attaches the voulme to the instance - Mounts the volume inside the instance, and then creates some directories to store data in This works fine the first time it's run. But any time we: - make a change to the instance configuration (i.e. change the value of var.aws_ami_rundeck) or - make a change to the provisioner config of the volume attachment resource Terraform then tries to detach the extant volume from the instance, and this task fails every time. I believe this is because you are meant to unmount the ebs volume from inside the instance before detaching the volume. The problem is, I can't work out how to get terraform to unmount the volume inside the instance _before_ trying to detach the volume. It's almost like I need a provisioner to run before the resou
Error Output
error I get from terraform:
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Implement Pre-Detach Unmount Script for EBS Volumes in Terraform
Terraform attempts to detach the EBS volume from the EC2 instance without ensuring that the volume is unmounted first. This leads to a failure because the volume must be unmounted from the operating system before it can be detached from the instance. The current configuration does not allow for running a script to unmount the volume before the detachment occurs.
Awaiting Verification
Be the first to verify this fix
- 1
Modify Terraform Configuration to Include Unmount Script
Add a provisioner to the EC2 instance that runs a script to unmount the EBS volume before Terraform attempts to detach it. This ensures that the volume is properly unmounted from the operating system.
hclprovisioner "remote-exec" { inline = [ "sudo umount /mnt/my-ebs-volume" ] } - 2
Update Volume Attachment Resource
Ensure that the volume attachment resource in Terraform is configured to depend on the unmount script. This can be done using the 'depends_on' attribute to ensure the unmount script runs before the volume is detached.
hclresource "aws_ebs_volume_attachment" "my_volume_attachment" { instance_id = aws_instance.my_instance.id volume_id = aws_ebs_volume.my_volume.id depends_on = [null_resource.unmount_script] } - 3
Create Null Resource for Unmount Script Execution
Define a null resource in Terraform that will trigger the unmount script. This resource will be executed before the EBS volume is detached, ensuring that the volume is unmounted first.
hclresource "null_resource" "unmount_script" { provisioner "remote-exec" { inline = [ "sudo umount /mnt/my-ebs-volume" ] } } - 4
Test Configuration Changes
Run 'terraform apply' to test the new configuration. Verify that the EBS volume is unmounted successfully before detachment and that the instance configuration changes are applied without errors.
bashterraform apply
Validation
Confirm that the EBS volume is unmounted successfully before Terraform attempts to detach it. Check the Terraform output for any errors during the apply process. Additionally, verify that the instance configuration changes are applied without issues.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep