AWS.NZ

Crypto mining on AWS and GCP after “The Merge”

Tue, 06 Sep 2022 13:36:41 +1200

Yes, you can still mine crypto in the public clouds even after Ethereum switch to Proof of Stake!

Ethereum mining and “The Merge”

For a long time Ethereum (ETH) used to be the favourite (= most profitable) crypto coin to mine on GPUs. People were building “mining rigs” — specialised computers equipped with lots and lots of NVidia or AMD Radeon GPUs — and operated them 24/7 in order to earn new ETH coins. The cost of setting up these rigs was massive, many thousands or even tens of thousands of dollars. And this upfront cost was obviously a big barrier for many wanna-be miners.

The new AWS instance that makes ETH mining profitable

Sun, 02 Jan 2022 12:38:00 +1300

Mining Ethereum on AWS until now wasn’t profitable. This new AWS instance is the game changer that no only breaks even but can finally make you money!

Ever since I wrote my first article about Ethereum and Bitcoin mining on AWS more than a year ago I was bombarded over and over again with the same sort of questions :

Is it profitable now that the ETH price has increased?
… that this new AWS instance type was released?
… that this new coin is the hype of the day?

So far the answer was always a NO. The profitability has gradually improved from some $20 mined in ETH for every $100 spent on AWS services a year ago, to $60 or $65 in ETH for the same $100 AWS spent a month ago. You would still be losing money though.

CloudFormation Service Roles

Sat, 10 Aug 2019 20:45:33 +1200

Sometimes we want to give users the ability to create pretty much anything with CloudFormation but at the same time prevent them from doing the same through the console or aws-cli. Perhaps it’s a company policy that everything must be managed using CloudFormation. Or on the other hand you may have Admin privileges but want to CloudFormation can do on your behalf.

Solution?

CloudFormation Service Role

CFN Service Role can achieve exactly that – users can have a separate set of credentials for work outside of CloudFormation and a completely different set of credentials for creating, updating and deleting CloudFormation stacks.

ssm-tunnel

Sun, 21 Jul 2019 23:22:51 +1200

EC2 instances, RDS database and many other resources reside in VPCs. To access them we need a network connectivity – either through a VPN, Direct Connect, or through a Jump / Bastion Host. That’s stating the obvious.

Sometimes, however, it’s inconvenient or difficult to achieve a direct connectivity ad-hoc – maybe you are in a location where your VPN is filtered, or from where the Bastion host Security Group won’t allow access but you still need access to the RDS for example.

SSM Sessions the easy way

Mon, 27 May 2019 08:05:08 +1200

In the previous post – Using SSM Session Manager for interactive instance access – I showed you how to access EC2 instances through AWS Systems Manager (SSM) Sessions without having to open Security Groups or firewall ports, maintain SSH keys, VPNs, Jump Hosts, and so on.

The native way for starting a SSM Session with aws ssm start-session requires the knowledge of the exact Instance ID, e.g. i-01234567890abcdef. That’s not very user friendly unfortunately. I know what my “pet” instances Host Names or Instance Names are. And I can often see the IP addresses of a misbehaving auto-scaling instances in the logs. However I don’t always know the corresponding Instance ID.

Using SSM Session Manager for interactive instance access

Sat, 25 May 2019 00:41:17 +1200

You may argue that interactive login to EC2 instances should never be needed. Everything is dynamic, automated, self healing, centrally logged, and so on and there is no place for human interaction, right? But lets be honest – the world isn’t perfect and we all sometimes need to jump into bash to do stuff for one reason or another.

Traditionally our EC2 instances would have a SSH KeyPair assigned and a Security Group with SSH port open. If they were on private IPs only we would also need a jump host or VPN to access them. That works but it’s got its own problems – for example rotation of the SSH keys, keeping the Security Group up to date, etc.

CloudFormation package & deploy

Wed, 20 Feb 2019 20:51:08 +1300

Easily deploy complex CloudFormation templates with external resources such as Lambdas or Nested Stacks.

Many CloudFormation templates are completely standalone – one single YAML or JSON file and that’s it. Easy to deploy. However in some cases CFN templates refer to other files, or artifacts. For example Lambda source or ZIP file, nested CloudFormation Template file, or an API definition for API Gateway may be such “artifacts”. These files have to be available in S3 before we can deploy the main CloudFormation template.

IP-Ranges Updater

Fri, 28 Dec 2018 17:16:18 +1300

Back in March I wrote about a command line script filter-ip-ranges that can parse the Amazon-published ip-ranges.json file and look up address ranges by region, service or IP address. That’s handy for the occasional manual use or for creating or updating some whitelists or Security Groups through aws-cli.

But running an aws-cli script from cron on some EC2 instance is so old school!

IP-Ranges-Updater Lambda

Much better way is to have this functionality in a Lambda function and let it update your Security Groups or Route Tables automatically.

New C5 / M5 / T3 instance disk names

Tue, 02 Oct 2018 21:07:44 +1300

When we started using the new C5, M5 or T3-class instances (e.g. t3.micro, m5.large, m5d.large, c5.xlarge, etc) we noticed that the disk device names are no longer the classic /dev/sda1 or /dev/xvda that we were used to for years.

TL;DR

The new root disk name is /dev/nvme0n1 with the partition device names being /dev/nvme0n1p1, /dev/nvme0n1p2, etc.

… just in case you’re lazy to read the whole post 🙂

Identifying the disks

Now that the disk names in the OS are no longer the names shown in the web console how do we know which disk is which? That’s even more pressing issue if you launch an M5D instance (e.g. m5d.large) with instance ephemeral disk.

CloudFormation Exports

Mon, 17 Sep 2018 22:26:01 +1200

If you are a CloudFormation user you have probably come across a situation where an ID of a resource created in one stack (e.g. Subnet ID from VPC-Stack) is needed in another, independent stack, for example to create an EC2 instance in EC2-Stack.

The traditional approach is to pass Outputs from one template to Parameters in another template. However there are a few problems…

The first problem with this method is that is that you somehow have to collect the Outputs from one stack and pass them to the Parameters in another. This can be done either manually or through some scripting like AWS-CLI or Ansible, but it’s an extra, mundane work.
The second problem is that many stacks need the same set of parameters – VPC ID, Subnet IDs, perhaps some Security Group IDs, Route53 hosted zone name and ID, etc. The Parameters list can get very long and not very interesting.

There is a better option – CloudFormation Stack Exports

CloudWatch Log Mailer

Sat, 30 Jun 2018 21:34:06 +1200

Feeding logs from various AWS services to CloudWatch Logs is a common pattern. From Lambda logs, through ECS Task logs, to Linux /var/log/* messages forwarded by awslogs daemon, they can end up in CloudWatch. How can we monitor them – and get alerted – when something of interest gets logged?

People often use Elastic Search, Splunk, Graylog or similar heavy weight solutions. But do we really need that? Very often all we want is to know when a certain keyword, e.g. “Error”, appears in the messages. Luckily that’s very easy to do.

filter-ip-ranges

Sat, 31 Mar 2018 19:45:34 +1300

Sometimes it’s handy to know what IP address ranges are used in different AWS Regions. You may want to whitelist access to your region AWS endpoints, or from global CloudFront location, or set up some special routing rules, or look up which region a certain IP belongs to.

ip-ranges.json & filter-ip-ranges

AWS publishes ip-ranges.json file with all that info. But it’s a huge JSON and not really meant for human consumption. So here I wrote a small parser: filter-ip-ranges

Cross-account access with aws-cli

Wed, 10 Jan 2018 13:34:27 +1300

In the previous post I explained how to Use Cross-account access through AWS Console. Today I’ll show you how to do the same in the command line using aws-cli.

We’ve got Access and Secret keys for the Login account and want to use aws-cli to create and manage resources in the Dev account above.

First step is to configure aws-cli with credentials for the aws-nz-login account.

Using Cross-account access (AWS Console)

Sun, 10 Dec 2017 10:45:25 +1300

Short post to demonstrate how to use AWS Cross-account access from AWS Console. To re-iterate this is what we what we have set up in our previous post:

First step is to login to the aws-nz-login account as an non-privileged IAM user, e.g. as michael.ludvig in my case.

Select Switch Role

In the top-right menu under your IAM username select Switch Role link.

Dev account Role details

Fill in the Dev account name and Role

Cross-account access

Wed, 06 Dec 2017 14:50:01 +1300

In bigger organisations it is common to have one central AWS account with IAM User accounts and a whole lot of independent per-project or per-team accounts that are only through cross-account access from this central account.

The benefits is obvious – company has a single place where they manage user accounts, credentials, passwords, permissions, etc. It is a very similar concept to switching from local Linux accounts to a centralised LDAP or AD authentication.

Linux user authentication with SSSD / LDAP

Tue, 17 Oct 2017 19:42:16 +1300

Current Linux distributions can seamlessly work as members of Active Directory domains which gives them access to the AD authentication system. However it requires the Linux hosts to “join” the AD domain, for which one has to posses some special AD privileges. In many cases this is not viable and we may only want a simple user authentication without any write privileges to the Active Directory. Good old LDAP provides such an option.

Let’s Encrypt certbot-auto problems on Amazon Linux

Sun, 08 Oct 2017 14:37:03 +1300

Let’s Encrypt certbot-auto support for Amazon Linux is still marked as experimental and as such we occasionally encounter unexpected problems. The most recent one was introduced with certbot-auto 0.19.0. During install or upgrade from previous versions you may get an error like this:


Upgrading certbot-auto 0.18.2 to 0.19.0...
Replacing certbot-auto...
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Traceback (most recent call last):
 File "/opt/eff.org/certbot/venv/bin/letsencrypt", line 7, in

It took me quite a while to figure out that it’s got nothing to do with Zope.

Using AWS Storage Gateway Virtual Tape Library in Linux

Wed, 20 Sep 2017 17:31:49 +1200

If you are of around my age or younger you probably didn’t have much exposure to tape backup technologies. Tapes are sooo 90’s right?! I definitely didn’t expect that as an AWS Consultant I will have to learn about tapes. But I did! One of our customers wanted to use AWS Storage Gateway (SGW) in the Virtual Tape Library (VTL) mode and use it for backups with Veeam Backup suite. Veeam seems to “just work” with SGW VTL but I like to understand how things work under the hood so I decided to backup my Linux test system to VTL using just the low level Linux tools. That’s the best way to learn.

Monitoring temperature with Raspberry Pi and AWS IoT

Tue, 25 Jul 2017 23:48:27 +1200

Working as a Senior AWS Consultant for one of the New Zealand’s leading AWS consulting providers means that I’m expected to have a hands on experience with as many AWS services as possible. In this series I will walk you through my first project that makes use of AWS IoT service.

Background

In one of our offices we’ve got a small server room with a handful of old servers – mostly sandpit stuff, nothing mission critical. The rack is cooled by an old unreliable Air Conditioning unit that sometimes stops cooling and sometimes even leaks water. Not good but not painful enough to warrant an installation of a new commercial grade cooling.

Joining a Linux EC2 instance to Active Directory realm

Tue, 06 Jun 2017 15:34:38 +1200

This post is not too AWS-specific, in fact the steps below should work not only on Amazon Linux but also on RedHat Linux, CentOS and Oracle Linux and posibly on Debian and Ubuntu based distros as well.

There is a number of prerequisities for a successful completion of this task.

At least the following records should be resolvable:

S3 Pre-signed URL example

Wed, 25 Jan 2017 19:13:17 +1300

S3 Pre-signed URLs can be used to provide a temporary 3rd party access to private objects in S3 buckets. For example non-public files on a file sharing site can only be made available to the approved users with one-off URLs that expire after 10 minutes. Here we offer a simple demo for testing the concept.

The demo consists of a number of parts:

S3 bucket with a Private access policy, i.e. non-public

EC2 instance with IAM Role attached

IAM Policy that gives the EC2 instance access to the S3 Bucket through the IAM Role

Some files, images, etc uploaded to the S3 bucket

Simple PHP Script running on the EC2 instance that lists the objects (files) in the bucket and generates signed URLs

Preparation

Spin up Amazon Linux EC2 instance

Re-using EC2 Key Pair in multiple regions

Mon, 04 Jul 2016 17:33:14 +1200

One of the parameters required for launching an EC2 instance is a Key Pair which is effectively an SSH Key used for interactive logging into the default user account – on Amazon Linux it’s the ec-user account – or for decrypting the Windows Administrator’s password.

It is easy to create a new Key Pair / SSH Key as part of the EC2 launch process however as soon as you start using more regions and more accounts you will quickly end up with heaps of stored SSH Keys and unless you are diligent with their naming both on the filesystem an in AWS you’ll end up with a mess. Like I did.

Using LastPass with AWS Console

Sun, 17 Jan 2016 01:02:56 +1300

As an Amazon Web Services professional working on projects for multiple customers I need an easy way to login to their many AWS accounts through the web console.

Since I use LastPass for storing my website passwords it’s only logical to use it for AWS logins as well. The problem is that there are 3 login fields – Account, User Name and Password – instead of the usual 2. LastPass has support for filling the additional field however it may not be entirely obvious. Follow these steps…

EC2 Instance Roles

Fri, 27 Nov 2015 13:15:39 +1300

In the previous post about Access & Secret Keys I emphasized that those keys – static keys – should never be used in production and in fact they should never leave your laptop. But what if you need to call the AWS services from scripts or programs running on an EC2 instance?

In a typical scenario such a script is called either by a scheduler (cron) or by another program like CI/CD tool (Jenkins) to automatically perform a task that needs certain AWS privileges. Say it has to upload a newly build package to an S3 bucket at the final stage of Jenkins run or it shut down all non-production EC2 instances in the evening and restart them in the morning.
The answer is – assign IAM Role to the EC2 Instance

Michael Ludvig

Mon, 23 Nov 2015 01:17:26 +1300

Who am I?

Michael Ludvig — Experienced IT Professional focused on Architecting, Designing and Implementation of Amazon Web Services solutions for customers of various sized and various AWS adoption levels.

My background in all things Linux with over 15 years of professional experience from Linux Kernel development through System, Security and Network Administration to a more high level role of Design and Delivery of new projects makes a perfect foundation for the varied and complex skill set required for successful Amazon Web Services adoption.

Access & Secret Keys

Wed, 22 Jul 2015 00:14:16 +1200

When working with AWS API the clients (programs, scripts, etc) must have a way to prove who they are and what level of access to Amazon services they should have.

In a typical scenario an AWS user runs aws cli (or a script using aws cli) to interact with Amazon. For example to create a VPC and Subnets or to backup some files to S3 bucket. To perform any non-anonymous operation like these the script will need a set of AWS Credentials – also known as Access Key and Secret Key.