Create intelligent CloudFormation templates with Custom Resources

When I wrote my first CloudFormation template to facilitate Ethereum mining on AWS it was a relatively simple solution used only by me. I could handle the little inefficiencies here and there. But then I made it public and people started using it. And they ran into issues.

“The template doesn’t work in region x-y-z!” Oh…
“The template doesn’t work with instance type abc.xlarge!” Oh…
“I want to use both spot and on-demand!” Oh…
“Why is it starting g5.xlarge when g4ad.2xlarge is more efficient?” Oh…

To please the crowd I had to improve the template and make it more intelligent.

Dashboard with charts on a screen — Photo by Stephen Dawson on Unsplash

AWS Ethereum Miner challenges

The AWS Ethereum Miner is an easy to use CloudFormation template that spins up AWS GPU instances to mine the Ethereum crypto coins. On the surface it’s nothing too complicated. However, the devil is in the detail as usual:

Different regions support different EC2 instances. For example, the best performing G4AD instance type is available in us-east-2 but the 2nd best performing G5G is not. In ap-southeast-1 it’s the opposite. Yet I still want the template to work in both these regions and pick the best available instance type.
Spot prices fluctuate. Right now in us-west-2 the g4ad.xlarge is more expensive than the g4ad.2xlarge, yes, really. Therefore it is more economical to start the 2xlarge instead of xlarge. In us-east-1 it’s the opposite.
Different instance types deliver vastly different ethash hashrates. Therefore instead of trying to figure out the number of instances to start it’s better to specify the total hashrate. Hey AWS, I don’t care if you give me 4x p3.2xlarge or 1x p3.16xlarge as both ways I get the same hashrate for the same price.

All these challenges could in theory be addressed by introducing a large number of stack parameters, requiring the user to spend a significant amount of time on research and calculations before spinning up the stack.

Or instead, we can use CloudFormation Custom Resources to do it all for us. Custom Resources add executable logic to your templates. They are implemented as Lambda functions that can process the given inputs to calculate and make available the required outputs (aka attributes). All that is in the form of a CloudFormation resource. Let’s see how that works in practice.

We want to spend the least amount of dollars to achieve the maximum performance.

ASG Weighted Capacity

One thing that I learned about Auto Scaling Groups while developing the miner solution was the magic of the Weighted Capacity.

Weighted Capacity essentially tells AWS that for our purpose the instance type A is worth X units of whatever performance, while instance B is worth Y units. AWS ASG will then try to launch the right number of instances to fulfil the required capacity of Z units.

Now what are these units? It turns out that we can use the hashrate of different instance types directly in the template.

InstanceTypesAttributes:
 # WeightedCapacity is the measured instance type Hashrate in MH/s
 - InstanceType: g4ad.xlarge
   WeightedCapacity: "48"
 - InstanceType: g4ad.2xlarge
   WeightedCapacity: "48"
 - InstanceType: g4ad.8xlarge
   WeightedCapacity: "96"
 - InstanceType: g5g.xlarge
   WeightedCapacity: "25"
 - InstanceType: g5.xlarge
   WeightedCapacity: "56"
 - InstanceType: p3.2xlarge
   WeightedCapacity: "93"
 - InstanceType: p3.8xlarge
   WeightedCapacity: "372"
 - InstanceType: p4d.24xlarge
   WeightedCapacity: "999" # It's in fact ~1400 but
                           # the max value allowed here is 999

With this configuration, we can now simply tell the ASG to allocate for example 250 MH/s hashrate capacity and it will pick and mix from the available instance types. No more guessing how many instances and of what type we need. The trouble is that it won’t do the most cost-efficient decisions. Read on, we’ll fix that later.

“Instance Filter” Custom Resource

As I already mentioned the CloudFormation Custom Resource is a method for adding a programmable logic to our CloudFormation templates.

For example, we can create a Custom Resource “MyHelloResource” with a custom attribute “HelloWorld” returning a string “Hello AWS World”. Now whenever we need the string “Hello AWS World” in our template we can get it from our custom resource with !GetAtt MyHelloResource.HelloWorld. Ok, all right, I accept that this is not very useful for most people.

Let’s create a more useful one — a custom resource that takes the above InstanceTypesAttributes list as a Parameter and then does 5 things:

Filters out only the instances wanted by the user, e.g. * or g4dn.* or g5g.xlarge.
Filters out the instances that are not available in the current region.
Fetches the average spot price for the remaining instance types.
Calculates the cost per megahash and orders the list by this value, while putting the more cost effective instance types first.
Makes the filtered and ordered list available as a new InstanceTypesAttributes resource attribute.

This way the template will adjust to the current conditions every time it’s deployed.

The nitty-gritty of creating the Custom Resources

Creating a CFN Custom Resource requires two things — the Custom Resource definition and the Lambda implementing the logic.

The actual Custom Resource definition can look something like this:

InstanceFilter:
    Type: Custom::InstanceFilter
    Properties:
      ServiceToken: !GetAtt InstanceFilterLambda.Arn
      InstanceTypesWanted: !Ref InstanceTypesWanted
      InstanceTypesAttributes:
      - InstanceType: g4ad.xlarge
        ...

Next, we will need a Lambda function that can handle the Create, Update and Delete lifecycle events of the template:

InstanceFilterLambda:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.9
      Role: !GetAtt LambdaExecutionRole.Arn
      Code: #!/usr/bin/env python ...

The Lambda can be inline if it’s short enough or specified as an external resource in S3, just like any other Lambda in CFN. I won’t go into all the details of how to write Lambda functions, that’s beyond the scope of this article. If you’re interested in the implementation details feel free to check out the instance-filter.py file from the aws-ethereum-miner GitHub repo.

The important points are these:

The custom resource can have parameters, in our case, it’s the list of instances, their measured hashrate and some other details
When the custom resource is created it calls the Lambda function and sends the parameters to it for processing, as a JSON structure.
When the processing is done the Lambda send the results back to the custom resource, again in a JSON format.
The results can then be accessed as the resources Attributes using the Fn::GetAtt function.

To have a peek at how the Lambda calculates the order of instances we can check the CloudWatch Logs where the Lambda emits the debugging information.

Instances sorted: [
{
    "WeightedCapacity": "48",
    "InstanceType": "g4ad.2xlarge",
    "_spot": 0.21756666666666669,
    "_efficiency": 220.62203156120728
}, 
{
    "WeightedCapacity": "25",
    "InstanceType": "g5g.xlarge",
    "_spot": 0.126,
    "_efficiency": 198.4126984126984
}, 
{
    "WeightedCapacity": "56",
    "InstanceType": "g5.xlarge",
    "_spot": 0.3018,
    "_efficiency": 185.5533465871438
}, 
...]

The higher the “_efficiency” the higher the calculated performance (hashrate) per dollar (spot price). And that’s what we want — spend the least amount of dollars to achieve the maximum performance.

Light trails of spinning steel wool sparks — Photo by Todd Quackenbush on Unsplash

The intelligent Auto Scaling Group

Now that we’ve got the logic implemented what do we do with it? Of course we’ll use it to create the Auto Scaling Group. Because we want the ASG to honour the order of our instances we need to use a Mixed instances policy with capacity-optimized-prioritized Spot allocation strategy. AWS will try quite hard to allocate our most preferred instance type before going down the list to the less preferred and less cost-efficient ones. There are other allocation policies as well, for example, one that allocates as many different instance types as possible to minimise the risk of interruptions, but that’s not what we want.

Asg:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MixedInstancesPolicy:
        InstancesDistribution:
          # InstanceFilter returns the instances
          # in an order from the most efficient ones
          # hence we use the 'prioritized' allocation strategies.
          SpotAllocationStrategy: capacity-optimized-prioritized
        LaunchTemplate:
          LaunchTemplateSpecification: ...
          Overrides: !GetAtt InstanceFilter.InstanceTypeAttributes
          #          ^^^^^^^ Calling our custom resource

What the ASG sees is the ordered list of instance types:

Overrides:
      - InstanceType: g4ad.2xlarge
        WeightedCapacity: "48"
        # spot price: $0.218/hr
      - InstanceType: g5g.xlarge
        WeightedCapacity: "25"
        # spot price: $0.126/hr
      - InstanceType: g5.xlarge
        WeightedCapacity: "56"
        # spot price: $0.302/hr

Unlike a typical ASG this one adjusts to the current conditions — the spot prices, the regional instance availability, etc. However only at the time of deployment! It’s a good idea to re-deploy the ASG once a week or so to make sure that it’s up to date with the current spot prices.

The maximum spot price

In the CloudWatch Logs we have seen the different spot prices: g4ad.2xlarge costs $0.218/hour, g5g.xlarge costs $0.126/hour while g5.xlarge costs $0.302/hour. Now how do we set the maximum spot price when using different instance types? We may be willing to pay let’s say up to $0.310/hour for g5.xlarge but no chance we would be ready to pay that for the much slower g5g.xlarge for example.

It turns out that then using the Weighted Capacity the spot price can be specified per capacity unit. That is in our case a price that we are willing to pay per 1 MH/s. And that makes sense — I don’t care if I run 1 instance that can do 50 MH/s or 2 instances that can do 25 MH/s as long as the price per MH/s is the same.

Having said that we can now divide the Spot Price by the Hashrate to get the price per MH/s (megahash per second) and that’s our cost per capacity or cost per performance. We’ll quickly figure out that g4ad.2xlarge costs $0.0045 per MH/s, g5g.xlarge costs $0.0050 per MH/s and g5.xlarge costs $0.0054 per MH/s — hey they are close!

If we’re happy to spend for example up to $0.0055 per MH/s we can let the ASG decide how to fulfil the capacity from these three instance types. It will still try the most efficient instances first, but if it fails for example due to a temporary capacity shortage it will try to allocate the little less efficient ones up to our spot price limit. Some people prefer to keep the ETH mining running even at a lower efficiency than stopping it due to capacity issues.

Here is the final bit that we have to add to the template:

Asg:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MixedInstancesPolicy:
        InstancesDistribution:
          SpotMaxPrice: !Ref MaxPricePerMHs    # <-- here

The actual maximum Spot price that we are willing to pay will be calculated for each instance type based on its performance — how convenient!

And with that we are pretty much done!

Our intelligent CloudFormation template

In this post we have seen an interesting use-case for CloudFormation Custom Resources that allowed us to create a very flexible, self-configuring CFN template. You can check out the complete source code on my AWS Ethereum Miner GitHub repository and if that’s your thing you can use it for mining some ETH. Just be aware that the profitability of such an operation is questionable at best — consult my previous two articles for details before you jump onto it.

Mining Bitcoin and other crypto on AWS
The Bitcoin mania has struck again. Take part in the frenzy and start mining crypto on AWS using a complete working…

New AWS instance that makes ETH mining profitable
Mining Ethereum on AWS so far made no sense. Until now! One new AWS instance is the game changer and mining on AWS is…

Take care and happy mining!