The Idle EC2 Reaper: Automating Non-Prod AWS Shutdowns

TL;DR

Your non-production AWS environments run around the clock, but your team uses them for roughly a third of the week. The rest is paid-for idle compute.

The Stack: Terraform (infrastructure as code), AWS Lambda (serverless execution), EventBridge (cron scheduling), Python (Lambda runtime).
The Verdict: A single afternoon of engineering work removes about two thirds of the compute hours your dev/staging environments burn, by shutting them down outside business hours. The ROI starts the first night.

Rather not deploy it yourself? Idle non-prod is one of the first things I check on the free 15-minute AWS bill review. If it's worth doing on your account, I'll tell you on the call.

The $3,000 Night Shift

Open your AWS Cost Explorer right now. Filter by environment tag — staging, dev, qa, whatever your team uses. Now look at the 24-hour usage graph.

You will see a flat line. No dips at night. No gaps on weekends. Your staging environment runs exactly the same at 3 AM on a Sunday as it does at 10 AM on a Monday.

Your engineers work roughly 8 hours a day, 5 days a week. That is 40 out of 168 hours per week. Your non-production infrastructure is idle for 76% of every week.

If your non-prod compute bill is $5,000/month, you are burning $3,800 every month on servers that are doing absolutely nothing.

The Reaper is a Lambda function triggered by EventBridge schedules. The code below is the same code as the aws-ec2-reaper repository on GitHub, so you can git clone it and terraform apply it as is. Want a number first? The AWS cost waste calculator will estimate what idle compute is costing you today.

With the default schedules (UTC, all overridable):

Weekdays 19:00: stops tagged EC2 instances, stops tagged RDS instances and scales tagged ECS services to 0.
Weekdays 08:00: starts everything back up before engineers arrive.
Friday 19:00: an extra stop so nothing is left running over the weekend.

Nothing runs Saturday and Sunday; Monday's 08:00 start brings everything back.

The Lambda Function

lambda/reaper.py

"""AWS non-prod "reaper" Lambda.
 
Stops EC2 / RDS / ECS resources tagged for non-production outside working hours
and starts them back up in the morning. Triggered by EventBridge with an
``{"action": "stop"}`` or ``{"action": "start"}`` payload.
 
Resources are only touched when they carry BOTH tags:
    Environment = dev | staging | qa
    Reaper      = enabled
"""
 
import logging
 
import boto3
 
logger = logging.getLogger()
logger.setLevel(logging.INFO)
 
ec2 = boto3.client("ec2")
rds = boto3.client("rds")
ecs = boto3.client("ecs")
 
NON_PROD_ENVS = ("dev", "staging", "qa")
 
 
def get_tagged_instances(action: str) -> list[str]:
    """Find EC2 instances tagged for the Reaper in the right state."""
    filters = [
        {"Name": "tag:Environment", "Values": list(NON_PROD_ENVS)},
        {"Name": "tag:Reaper", "Values": ["enabled"]},
    ]
 
    if action == "stop":
        filters.append({"Name": "instance-state-name", "Values": ["running"]})
    elif action == "start":
        filters.append({"Name": "instance-state-name", "Values": ["stopped"]})
 
    response = ec2.describe_instances(Filters=filters)
    instance_ids = []
    for reservation in response["Reservations"]:
        for instance in reservation["Instances"]:
            instance_ids.append(instance["InstanceId"])
 
    return instance_ids
 
 
def _is_reaper_rds(db: dict) -> bool:
    tags = rds.list_tags_for_resource(ResourceName=db["DBInstanceArn"])["TagList"]
    tag_map = {t["Key"]: t["Value"] for t in tags}
    return (
        tag_map.get("Environment") in NON_PROD_ENVS
        and tag_map.get("Reaper") == "enabled"
    )
 
 
def stop_rds_instances() -> None:
    """Stop all non-prod RDS instances that are currently available."""
    for db in rds.describe_db_instances()["DBInstances"]:
        if db["DBInstanceStatus"] == "available" and _is_reaper_rds(db):
            rds.stop_db_instance(DBInstanceIdentifier=db["DBInstanceIdentifier"])
            logger.info("Stopped RDS: %s", db["DBInstanceIdentifier"])
 
 
def start_rds_instances() -> None:
    """Start all non-prod RDS instances that are currently stopped."""
    for db in rds.describe_db_instances()["DBInstances"]:
        if db["DBInstanceStatus"] == "stopped" and _is_reaper_rds(db):
            rds.start_db_instance(DBInstanceIdentifier=db["DBInstanceIdentifier"])
            logger.info("Started RDS: %s", db["DBInstanceIdentifier"])
 
 
def scale_ecs_services(desired_count: int) -> None:
    """Scale non-prod ECS services to the desired task count."""
    for cluster_arn in ecs.list_clusters()["clusterArns"]:
        service_arns = ecs.list_services(cluster=cluster_arn)["serviceArns"]
        if not service_arns:
            continue
 
        described = ecs.describe_services(
            cluster=cluster_arn,
            services=service_arns,
            include=["TAGS"],
        )["services"]
 
        for service in described:
            tags = {t["key"]: t["value"] for t in service.get("tags", [])}
            if (
                tags.get("Environment") in NON_PROD_ENVS
                and tags.get("Reaper") == "enabled"
            ):
                ecs.update_service(
                    cluster=cluster_arn,
                    service=service["serviceArn"],
                    desiredCount=desired_count,
                )
                logger.info(
                    "Scaled ECS %s to %s", service["serviceName"], desired_count
                )
 
 
def handler(event, context):
    action = event.get("action", "stop")
 
    if action == "stop":
        instance_ids = get_tagged_instances("stop")
        if instance_ids:
            ec2.stop_instances(InstanceIds=instance_ids)
            logger.info("Stopped EC2 instances: %s", instance_ids)
 
        stop_rds_instances()
        scale_ecs_services(desired_count=0)
 
        return {
            "action": "stop",
            "ec2_stopped": len(instance_ids),
            "message": "Non-prod environments shut down for the night.",
        }
 
    if action == "start":
        instance_ids = get_tagged_instances("start")
        if instance_ids:
            ec2.start_instances(InstanceIds=instance_ids)
            logger.info("Started EC2 instances: %s", instance_ids)
 
        start_rds_instances()
        scale_ecs_services(desired_count=1)
 
        return {
            "action": "start",
            "ec2_started": len(instance_ids),
            "message": "Non-prod environments are waking up.",
        }
 
    raise ValueError(f"Unknown action: {action!r} (expected 'stop' or 'start')")

Two details that are easy to get wrong if you write this from scratch:

ECS tags are not returned by default. describe_services only includes tags when you pass include=["TAGS"]. Without it, the tag check never matches and nothing is ever scaled down.
Tag filters differ per service. EC2 supports server-side tag filters in describe_instances. RDS doesn't, so the function lists instances and checks tags with list_tags_for_resource.

The Terraform Module

Deploy the Reaper as infrastructure-as-code so it is version controlled, reviewable, and reproducible. main.tf packages the Lambda and gives it a least-privilege role. EC2 start/stop is restricted to instances tagged Reaper = enabled. ec2:DescribeInstances sits in its own statement because Describe calls don't support resource-tag conditions.

main.tf

terraform {
  required_version = ">= 1.3"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
    archive = {
      source  = "hashicorp/archive"
      version = ">= 2.0"
    }
  }
}
 
# Package the Lambda source into a zip at apply time.
data "archive_file" "reaper_zip" {
  type        = "zip"
  source_file = "${path.module}/lambda/reaper.py"
  output_path = "${path.module}/build/reaper.zip"
}
 
resource "aws_lambda_function" "reaper" {
  function_name = var.function_name
  runtime       = "python3.12"
  handler       = "reaper.handler"
  timeout       = 300
  memory_size   = 256
 
  filename         = data.archive_file.reaper_zip.output_path
  source_code_hash = data.archive_file.reaper_zip.output_base64sha256
 
  role = aws_iam_role.reaper_role.arn
}
 
# ---------------------------------------------------------------------------
# IAM: least-privilege role for the reaper
# ---------------------------------------------------------------------------
resource "aws_iam_role" "reaper_role" {
  name = "${var.function_name}-role"
 
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}
 
resource "aws_iam_role_policy" "reaper_policy" {
  name = "${var.function_name}-policy"
  role = aws_iam_role.reaper_role.id
 
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ec2:DescribeInstances"]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["ec2:StartInstances", "ec2:StopInstances"]
        Resource = "*"
        Condition = {
          StringEquals = { "ec2:ResourceTag/Reaper" = "enabled" }
        }
      },
      {
        Effect = "Allow"
        Action = [
          "rds:DescribeDBInstances",
          "rds:ListTagsForResource",
          "rds:StartDBInstance",
          "rds:StopDBInstance",
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ecs:ListClusters",
          "ecs:ListServices",
          "ecs:DescribeServices",
          "ecs:UpdateService",
        ]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
        Resource = "arn:aws:logs:*:*:*"
      },
    ]
  })
}

schedules.tf wires the three EventBridge rules to the function. The aws_lambda_permission resource is the one people forget: without it, EventBridge has no right to invoke the Lambda and the rules fire into nothing.

schedules.tf

# EventBridge schedules that drive the reaper. Times are UTC — adjust the cron
# expressions to your team's working hours (see README).
 
locals {
  schedules = {
    # Stop every weekday evening.
    stop_weekday = {
      cron   = var.stop_weekday_cron
      action = "stop"
    }
    # Start every weekday morning (also covers Monday).
    start_weekday = {
      cron   = var.start_weekday_cron
      action = "start"
    }
    # Belt-and-braces: ensure everything is down Friday night for the weekend.
    stop_weekend = {
      cron   = var.stop_weekend_cron
      action = "stop"
    }
  }
}
 
resource "aws_cloudwatch_event_rule" "schedule" {
  for_each            = local.schedules
  name                = "${var.function_name}-${each.key}"
  schedule_expression = each.value.cron
}
 
resource "aws_cloudwatch_event_target" "schedule" {
  for_each = local.schedules
  rule     = aws_cloudwatch_event_rule.schedule[each.key].name
  arn      = aws_lambda_function.reaper.arn
  input    = jsonencode({ action = each.value.action })
}
 
# Allow EventBridge to invoke the Lambda for each rule.
resource "aws_lambda_permission" "allow_eventbridge" {
  for_each      = local.schedules
  statement_id  = "AllowExecutionFromEventBridge-${each.key}"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.reaper.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.schedule[each.key].arn
}

The cron expressions live in variables.tf (stop_weekday_cron, start_weekday_cron, stop_weekend_cron) so you can shift them to your team's time zone without touching the module.

Tagging Your Resources

The Reaper uses two tags to decide what to manage:

Tag Key	Tag Value	Purpose
`Environment`	`dev`, `staging`, or `qa`	Identifies non-production resources
`Reaper`	`enabled`	Opt-in flag for automatic shutdown

Any resource without both tags is completely ignored. This gives teams fine-grained control — if a specific staging database must stay running overnight for a nightly integration test, just remove the Reaper: enabled tag.

The Operational Reality

RDS cold-start delay. Starting a stopped RDS instance takes 5 to 10 minutes. If your engineers arrive at 8 AM sharp and immediately hit the database, they will see connection errors. Set the "start" cron 15 minutes earlier than the earliest engineer's workday.
ECS task definition drift. When you scale an ECS service to 0 and back to 1, it uses the current task definition. If someone deployed a broken version at 6:55 PM, the Reaper will dutifully bring up the broken version at 8 AM. Always verify your latest deployment is healthy before end-of-day.
Spot instances. EC2 can only stop Spot instances launched with a persistent request; one-time Spot instances cannot be stopped, and a StopInstances call that includes one fails. Don't put the Reaper tag on one-time Spot instances.
RDS restarts itself after 7 days. AWS automatically starts a stopped RDS instance after seven days. With a weekday schedule that never matters, but if you extend the Reaper to stop databases for longer (holidays), plan for it.
Pagination. The function makes single list_services / describe_db_instances calls. list_services returns 10 services per page and describe_services accepts at most 10 at a time, so if a cluster has more than 10 services, switch to boto3 paginators and batch the describe calls.
Multi-timezone teams. If your engineers span US, Europe, and Asia, set the "start" time to the earliest timezone and the "stop" time to the latest. Or deploy separate Reaper instances per region.

The Math

With the default schedule, tagged resources run 08:00–19:00 on weekdays: 11 hours × 5 days = 55 hours a week instead of 168. That removes about two thirds of their compute hours.

A worked example with 10 m6i.large dev/staging instances at the us-east-1 On-Demand rate of $0.096/hour (check current pricing for your region):

Metric	Before Reaper	After Reaper
Running hours per week	168	55
Share of the week running	100%	~33%
Monthly EC2 compute (10 × m6i.large)	~$701	~$229
Monthly saving	—	~$471

Scale that to your own non-prod footprint: the saving is roughly 67% of the compute cost of whatever you tag. Two things keep billing while resources are stopped: EBS volumes and RDS storage, so the real saving is a bit lower than the compute figure. The Lambda itself runs a few times a day and costs effectively nothing.

The Reaper pays for itself in the first night it runs. Idle non-prod is one of the seven fixes in my AWS cost optimization playbook; if you want someone to find the others on your bill, the $250 fixed-price AWS Cost Audit covers all seven.

Your dev and staging environments are billing you while your engineers sleep. Idle non-prod is usually one leak among several; the free review finds the others.