Apricate.dev

Crafting a Migration Plan: PostgreSQL to AWS with Terraform

I’d like to share my insights on migrating an on-premises PostgreSQL database to AWS using Terraform. This approach is not just about the technical steps but also about the strategic planning that goes into a successful migration.

Setting the Stage for Migration

Understanding Terraform’s Role

Terraform is our tool of choice for this migration, owing to its prowess in Infrastructure as Code (IaC). It allows us to define and provision the AWS environment needed for our PostgreSQL database with precision and repeatability.

Prerequisites

  • Ensure Terraform is installed and configured.
  • Secure AWS credentials for Terraform.

The Migration Blueprint

1. Infrastructure Definition

We start by scripting our infrastructure requirements in Terraform’s HCL language. This includes:

  • AWS RDS Instance: Our target PostgreSQL instance in RDS.
  • Networking Setup: VPC, subnets, and security groups.
  • AWS DMS Resources: The DMS instance, endpoints, and migration tasks.
# AWS RDS Instance for PostgreSQL
resource "aws_db_instance" "postgres" {
  allocated_storage    = 20
  storage_type         = "gp2"
  engine               = "postgres"
  engine_version       = "12.4"
  instance_class       = "db.m4.large"
  name                 = "mydb"
  username             = "myuser"
  password             = "mypassword"
  parameter_group_name = "default.postgres12"
  skip_final_snapshot  = true
}

# AWS DMS Replication Instance
resource "aws_dms_replication_instance" "dms_replication_instance" {
  allocated_storage            = 20
  replication_instance_class   = "dms.t2.micro"
  replication_instance_id      = "my-dms-replication-instance"
  replication_subnet_group_id  = aws_dms_replication_subnet_group.dms_replication_subnet_group.id
  vpc_security_group_ids       = [aws_security_group.dms_sg.id]
}

# DMS Replication Subnet Group
resource "aws_dms_replication_subnet_group" "dms_replication_subnet_group" {
  replication_subnet_group_id          = "my-dms-subnet-group"
  replication_subnet_group_description = "My DMS Replication Subnet Group"
  subnet_ids                           = [aws_subnet.subnet1.id, aws_subnet.subnet2.id]
}

# Security Group for DMS
resource "aws_security_group" "dms_sg" {
  name        = "dms_sg"
  description = "Security Group for DMS"
  vpc_id      = aws_vpc.main.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# DMS Source Endpoint (On-Premises PostgreSQL)
resource "aws_dms_endpoint" "source_endpoint" {
  endpoint_id                  = "source-endpoint"
  endpoint_type                = "source"
  engine_name                  = "postgres"
  username                     = "source_db_username"
  password                     = "source_db_password"
  server_name                  = "onpremises-db-server-address"
  port                         = 5432
  database_name                = "source_db_name"
  ssl_mode                     = "none"
  extra_connection_attributes  = "key=value;"
}

# DMS Target Endpoint (AWS RDS PostgreSQL)
resource "aws_dms_endpoint" "target_endpoint" {
  endpoint_id                  = "target-endpoint"
  endpoint_type                = "target"
  engine_name                  = "postgres"
  username                     = "myuser"
  password                     = "mypassword"
  server_name                  = aws_db_instance.postgres.address
  port                         = aws_db_instance.postgres.port
  database_name                = "mydb"
  ssl_mode                     = "require"
}

# DMS Replication Task
resource "aws_dms_replication_task" "dms_replication_task" {
  replication_task_id          = "my-dms-task"
  source_endpoint_arn          = aws_dms_endpoint.source_endpoint.arn
  target_endpoint_arn          = aws_dms_endpoint.target_endpoint.arn
  replication_instance_arn     = aws_dms_replication_instance.dms_replication_instance.arn
  migration_type               = "full-load"
  table_mappings               = "{\"rules\":[{\"rule-type\":\"selection\",\"rule-id\":\"1\",\"rule-name\":\"1\",\"object-locator\":{\"schema-name\":\"%\",\"table-name\":\"%\"},\"rule-action\":\"include\"}]}"
}

# Output RDS Instance Address
output "rds_instance_address" {
  value = aws_db_instance.postgres.address
}

# Output RDS Instance Endpoint
output "rds_instance_endpoint" {
  value = aws_db_instance.postgres.endpoint
}

Notes:

  1. Security: This script doesn’t include detailed security configurations. You should configure security groups and IAM roles/policies according to your security standards.
  2. Network Configuration: The script assumes existing VPC, subnets, etc. You should adjust these according to your AWS network setup.
  3. Credentials: Never hardcode sensitive information like usernames and passwords. Use a secure method like AWS Secrets Manager or environment variables.
  4. Customization: Adjust database sizes, instance

2. Initialization and Planning

Run terraform init to prepare your Terraform environment. Follow this with terraform plan to review the actions Terraform will perform.

3. Executing the Plan

Apply the configuration using terraform apply. This step will bring up our necessary AWS infrastructure.

4. The Migration Process

With the infrastructure in place, we manually initiate the data migration using AWS DMS. This step is crucial and requires a meticulous approach to ensure data integrity.

5. Post-Migration Strategies

After migration, we’ll perform tasks like data validation, application redirection, and performance tuning. Terraform can assist in setting up additional resources for monitoring and management.

6. Ongoing Infrastructure Management

Use Terraform for any future updates or changes in the AWS environment. Keep these configurations in a version control system for better management and collaboration.

Key Considerations

  • Complex Configurations: Some aspects may require manual intervention, especially in complex database setups.
  • Learning Curve: If you’re new to Terraform, allocate time for learning and experimentation.
  • State Management: Handle Terraform’s state file with care, particularly in team settings.

Conclusion

Migrating to AWS using Terraform presents a structured and reliable approach. It’s a journey that requires careful planning, execution, and post-migration management. By following this plan, we can ensure a smooth transition to AWS, setting the stage for a more efficient, scalable cloud environment.

Bohdan