How does UTM in AWS HA work?

Hi Justin,

When a primary UTM EC2 instance fails in AWS, the DNS records and Elastic IP should automatically be re-assigned to the secondary UTM EC2 instance. The only time this does not happen, is if there is a manual termination of the EC2 instance and the Elastic IP is set to be released.

The configuration required to handle this is within the CloudFormation template, as the UTM instances do not know they are part of an HA pair. All HA configuration within the UTM's web admin is turned off by default in the AWS HA setup as the architecture of AWS does not allow for the traditional HA setup used by the UTM.

Within the CloudFormation template is a section of parameters that handles Auto Scaling Groups and launch configurations. By using Auto Scaling Groups and launch configurations, the existing Elastic IP is able to be disassociated from the terminated EC2 instance and re-associated with the new EC2 instance that is brought up.

For more information on how the configuration is done on AWS:
docs.aws.amazon.com/.../autoscalingsubnets.html
docs.aws.amazon.com/.../vpc-ip-addressing.html

The CloudFormation template logic that makes the two EC2 UTM instances part of the Auto Scaling Group:
"UTMLaunchConfiguration": {
"Type" : "AWS::AutoScaling::LaunchConfiguration",
"Properties" : {
"AssociatePublicIpAddress" : true,
"IamInstanceProfile" : { "Ref" : "UTMInstanceProfile" },
"ImageId" : {
"Fn::If": [
"UseRegionMap",
{ "Fn::FindInMap": [ "RegionMap", { "Ref": "AWS::Region" }, { "Ref": "LicenseType" } ] },
{ "Ref": "AMI" }
]
},
"InstanceType" : "m3.medium",
"KeyName" : { "Ref" : "KeyName" },
"BlockDeviceMappings" : [{
"DeviceName" : "/dev/sda",
"Ebs" : { "VolumeSize" : "100" }
}],
"SecurityGroups" : [
{ "Ref": "UTMSecurityGroup" },
{ "Ref": "TrustedNetworkGroup" },
{ "Ref": "UntrustedGroup" }
],
"UserData" : { "Fn::Base64" : { "Fn::Join" : [ "", [
"#!/bin/bash\n",
"date +'UserData start %c' > /tmp/user_data.log\n",
"echo 'version: \"0.1.0\"' >> /etc/cloud/user_data.yml\n",
"echo 'instance_role: \"ha_standalone\"' >> /etc/cloud/user_data.yml\n",
"echo 'deployment_type: \"ha_warm_standby\"' >> /etc/cloud/user_data.yml\n",
"echo 'license_pool: \"", { "Ref": "LicensePool" }, "\"' >> /etc/cloud/user_data.yml\n",
"echo 'region: \"", { "Ref": "AWS::Region" }, "\"' >> /etc/cloud/user_data.yml\n",

"cc=`/usr/local/bin/confd-client.plx country_name_to_code \"",{ "Ref": "Country" },"\"`\n",
"echo \"{ hostname => '", { "Ref": "Hostname" }, "', organization => '", { "Ref": "Organization"}, "', city => '", { "Ref" : "City" },"', country => '$cc', email => '", { "Ref": "Email" }, "', password => '", { "Ref": "AdminPassword" }, "' }\" > /var/confd/var/storage/setup.ph\n",

"/etc/init.d/confd restart\n",
"sleep 5\n",
"echo 'elastic_ip: \"",
{ "Fn::If" : [
"AllocateElasticIP",
{ "Ref" : "IPAddress" },
{ "Ref" : "ExistingElasticIP" }
]
},"\"' >> /etc/cloud/user_data.yml\n",
"echo 's3_bucket: \"",
{ "Fn::If" : [
"CreateS3Bucket",
{ "Ref" : "S3Bucket" },
{ "Ref" : "ExistingS3Bucket" }
]
},"\"' >> /etc/cloud/user_data.yml\n",

"echo 'stack_name: \"", { "Ref": "AWS::StackName"} ,"\"' >> /etc/cloud/user_data.yml\n",

"date +'UserData confd config start %c' >> /tmp/user_data.log\n",

"echo '{' > /tmp/user_data.config\n",
"echo '\"cloudwatch\" => { \"status\" => 1 },' >> /tmp/user_data.config\n",
"echo '\"confd\" => { \"backup\" => 1, \"backup_interval\" => 300, \"restore\" => 1, \"restore_done\" => 0 },' >> /tmp/user_data.config\n",
"echo '\"instance_role\" => \"ha_standalone\",' >> /tmp/user_data.config\n",
"echo '\"postgres\" => { \"archive_timeout\" => 300, \"backup\" => 1, \"base_backup_interval\" => 3600, \"restore\" => 1 },' >> /tmp/user_data.config\n",
"echo '\"syslog\" => { \"backup\" => 1, \"restore\" => 1, \"restore_period\" => 8 },' >> /tmp/user_data.config\n",
"echo '\"s3_bucket\" => \"",
{ "Fn::If" : [
"CreateS3Bucket",
{ "Ref" : "S3Bucket" },
{ "Ref" : "ExistingS3Bucket" }
]
},"\",' >> /tmp/user_data.config\n",

"echo '\"stack_name\" => \"", { "Ref": "AWS::StackName"},"\",' >> /tmp/user_data.config\n",

"echo '\"elastic_ip\" => \"",
{ "Fn::If" : [
"AllocateElasticIP",
{ "Ref" : "IPAddress" },
{ "Ref" : "ExistingElasticIP" }
]
}, "\",' >> /tmp/user_data.config\n",

"echo '\"trusted_network\" => \"", { "Ref": "TrustedNetwork"},"\"' >> /tmp/user_data.config\n",
"echo '}' >> /tmp/user_data.config\n",

"/usr/local/bin/confd-client.plx -noquote -stdin set \\'ha\\' \\'aws\\' < /tmp/user_data.config >> /tmp/user_data.log\n",

"date +'UserData confd config end %c' >> /tmp/user_data.log\n",

"date +'UserData ha aws start %c' >> /tmp/user_data.log\n",
"/etc/init.d/ha_aws start\n",

"date +'UserData awslogs agent setup start %c' >> /tmp/user_data.log\n",
"/usr/local/bin/awslogs-agent-setup.py -n -r ", { "Ref" : "AWS::Region" }, " -c /etc/cloud/awslogs.conf >> /tmp/user_data.log\n",
"date +'UserData awslogs agent setup end %c' >> /tmp/user_data.log\n",

"/usr/local/bin/confd-client.plx trigger ha_aws\n",

"exit 0\n"
] ] } }
}
},


Thanks,

Ted

Hi Justin,

When a primary UTM EC2 instance fails in AWS, the DNS records and Elastic IP should automatically be re-assigned to the secondary UTM EC2 instance. The only time this does not happen, is if there is a manual termination of the EC2 instance and the Elastic IP is set to be released.

The configuration required to handle this is within the CloudFormation template, as the UTM instances do not know they are part of an HA pair. All HA configuration within the UTM's web admin is turned off by default in the AWS HA setup as the architecture of AWS does not allow for the traditional HA setup used by the UTM.

Within the CloudFormation template is a section of parameters that handles Auto Scaling Groups and launch configurations. By using Auto Scaling Groups and launch configurations, the existing Elastic IP is able to be disassociated from the terminated EC2 instance and re-associated with the new EC2 instance that is brought up.

For more information on how the configuration is done on AWS:
docs.aws.amazon.com/.../autoscalingsubnets.html
docs.aws.amazon.com/.../vpc-ip-addressing.html

The CloudFormation template logic that makes the two EC2 UTM instances part of the Auto Scaling Group:
"UTMLaunchConfiguration": {
"Type" : "AWS::AutoScaling::LaunchConfiguration",
"Properties" : {
"AssociatePublicIpAddress" : true,
"IamInstanceProfile" : { "Ref" : "UTMInstanceProfile" },
"ImageId" : {
"Fn::If": [
"UseRegionMap",
{ "Fn::FindInMap": [ "RegionMap", { "Ref": "AWS::Region" }, { "Ref": "LicenseType" } ] },
{ "Ref": "AMI" }
]
},
"InstanceType" : "m3.medium",
"KeyName" : { "Ref" : "KeyName" },
"BlockDeviceMappings" : [{
"DeviceName" : "/dev/sda",
"Ebs" : { "VolumeSize" : "100" }
}],
"SecurityGroups" : [
{ "Ref": "UTMSecurityGroup" },
{ "Ref": "TrustedNetworkGroup" },
{ "Ref": "UntrustedGroup" }
],
"UserData" : { "Fn::Base64" : { "Fn::Join" : [ "", [
"#!/bin/bash\n",
"date +'UserData start %c' > /tmp/user_data.log\n",
"echo 'version: \"0.1.0\"' >> /etc/cloud/user_data.yml\n",
"echo 'instance_role: \"ha_standalone\"' >> /etc/cloud/user_data.yml\n",
"echo 'deployment_type: \"ha_warm_standby\"' >> /etc/cloud/user_data.yml\n",
"echo 'license_pool: \"", { "Ref": "LicensePool" }, "\"' >> /etc/cloud/user_data.yml\n",
"echo 'region: \"", { "Ref": "AWS::Region" }, "\"' >> /etc/cloud/user_data.yml\n",

"cc=`/usr/local/bin/confd-client.plx country_name_to_code \"",{ "Ref": "Country" },"\"`\n",
"echo \"{ hostname => '", { "Ref": "Hostname" }, "', organization => '", { "Ref": "Organization"}, "', city => '", { "Ref" : "City" },"', country => '$cc', email => '", { "Ref": "Email" }, "', password => '", { "Ref": "AdminPassword" }, "' }\" > /var/confd/var/storage/setup.ph\n",

"/etc/init.d/confd restart\n",
"sleep 5\n",
"echo 'elastic_ip: \"",
{ "Fn::If" : [
"AllocateElasticIP",
{ "Ref" : "IPAddress" },
{ "Ref" : "ExistingElasticIP" }
]
},"\"' >> /etc/cloud/user_data.yml\n",
"echo 's3_bucket: \"",
{ "Fn::If" : [
"CreateS3Bucket",
{ "Ref" : "S3Bucket" },
{ "Ref" : "ExistingS3Bucket" }
]
},"\"' >> /etc/cloud/user_data.yml\n",

"echo 'stack_name: \"", { "Ref": "AWS::StackName"} ,"\"' >> /etc/cloud/user_data.yml\n",

"date +'UserData confd config start %c' >> /tmp/user_data.log\n",

"echo '{' > /tmp/user_data.config\n",
"echo '\"cloudwatch\" => { \"status\" => 1 },' >> /tmp/user_data.config\n",
"echo '\"confd\" => { \"backup\" => 1, \"backup_interval\" => 300, \"restore\" => 1, \"restore_done\" => 0 },' >> /tmp/user_data.config\n",
"echo '\"instance_role\" => \"ha_standalone\",' >> /tmp/user_data.config\n",
"echo '\"postgres\" => { \"archive_timeout\" => 300, \"backup\" => 1, \"base_backup_interval\" => 3600, \"restore\" => 1 },' >> /tmp/user_data.config\n",
"echo '\"syslog\" => { \"backup\" => 1, \"restore\" => 1, \"restore_period\" => 8 },' >> /tmp/user_data.config\n",
"echo '\"s3_bucket\" => \"",
{ "Fn::If" : [
"CreateS3Bucket",
{ "Ref" : "S3Bucket" },
{ "Ref" : "ExistingS3Bucket" }
]
},"\",' >> /tmp/user_data.config\n",

"echo '\"stack_name\" => \"", { "Ref": "AWS::StackName"},"\",' >> /tmp/user_data.config\n",

"echo '\"elastic_ip\" => \"",
{ "Fn::If" : [
"AllocateElasticIP",
{ "Ref" : "IPAddress" },
{ "Ref" : "ExistingElasticIP" }
]
}, "\",' >> /tmp/user_data.config\n",

"echo '\"trusted_network\" => \"", { "Ref": "TrustedNetwork"},"\"' >> /tmp/user_data.config\n",
"echo '}' >> /tmp/user_data.config\n",

"/usr/local/bin/confd-client.plx -noquote -stdin set \\'ha\\' \\'aws\\' < /tmp/user_data.config >> /tmp/user_data.log\n",

"date +'UserData confd config end %c' >> /tmp/user_data.log\n",

"date +'UserData ha aws start %c' >> /tmp/user_data.log\n",
"/etc/init.d/ha_aws start\n",

"date +'UserData awslogs agent setup start %c' >> /tmp/user_data.log\n",
"/usr/local/bin/awslogs-agent-setup.py -n -r ", { "Ref" : "AWS::Region" }, " -c /etc/cloud/awslogs.conf >> /tmp/user_data.log\n",
"date +'UserData awslogs agent setup end %c' >> /tmp/user_data.log\n",

"/usr/local/bin/confd-client.plx trigger ha_aws\n",

"exit 0\n"
] ] } }
}
},


Thanks,

Ted