Discover ecs-session — the new easy to use tool for interactive access to ECS tasks
Running apps in Docker containers comes with some challenges. Most notably we are not supposed to “login” to the containers and should only observe them from the outside. However let’s be honest — sometimes it may be handy to login to a running container and “do something”. For example we may want to check the network connectivity from the container point of view. Or to test access to the AWS services with the container’s IAM role credentials. Or to check some logs that we forgot to forward to CloudWatch. And so on. I’m not saying that we should be doing it on daily basis, but sometimes it’s required…
Everyone is probably familiar with “docker exec” and we can use this very method when the ECS tasks run on an EC2 instance. We can SSH or SSM into the instance, figure out the running container name and then “docker exec” into it. We have all done that in the past, didn’t we?
But what if the container runs on AWS Fargate when there is no EC2 to login to? Or when we don’t have access to the underlying instance and in turn nowhere to run the “docker exec” command from. Now what?
ECS Exec — the “docker exec” for ECS
In March 2021 AWS announced the general availability of ECS Exec, a feature that lets us login to the running containers, both on Fargate and on EC2. It’s not a “docker exec” as such as it uses a different mechanism that’s based on the AWS SSM protocol, but it serves the same purpose — executing commands or opening an interactive shell in the running containers. Unfortunately the “official” way using AWS CLI is horribly user unfriendly. We have to know the ECS cluster name, the task ID and the container name at the very least. Building the command line then typically involves a number of lookups and a tedious copy and pasting of long ID strings.
Yields the cluster names, e.g. ecs-demo-EcsCluster-mWkD7Y4zt5W5
~ $ aws ecs list-clustersYields the task IDs, e.g. 11f37244484b486e9e0eda9b94cc9f74
~ $ aws ecs list-tasks --cluster ecs-demo-EcsCluster-mWkD7Y4zt5W5Yields the container names in the task, e.g. nginx
~ $ aws ecs describe-tasks \ --cluster ecs-demo-EcsCluster-mWkD7Y4zt5W5 \ --task 11f37244484b486e9e0eda9b94cc9f74Now finally build the “aws ecs execute-command”…
~ $ aws ecs execute-command \ --cluster ecs-demo-EcsCluster-mWkD7Y4zt5W5 \ --task 11f37244484b486e9e0eda9b94cc9f74 \ --container nginx \ --command "/bin/bash" \ --interactive
That’s indeed quite convoluted and I usually had to consult the documentation every time I wanted to exec into a container.
So I decided to create a more convenient tool: ecs-session, now part of the popular AWS SSM Tools suite.
Photo by Michael Dziedzic on Unsplash
ecs-session — ECS Exec the easy way
For example — if we’ve got only a single apache container then we don’t really need to know the Cluster name or the Task ID, why should we? The client tool can figure it out for us just from the container name. With ecs-session the usage can therefore be as simple as this:
~ $ ecs-session apache
Of course we typically have more tasks and more containers and that’s where the --list parameter comes handy.
~ $ ecs-session --list
Cluster Group Task Cont. IP
# Apache service - 1 task with a single container
apache-Cluster-cxYDZkf service:apache-Service-CUC9JM 6164a56....211c5 apache 172.31.35.190
# Nginx service - 3 tasks each with 1 container
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj a3875d0e...7d488 nginx 172.31.107.214
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj e0d6b610...52371 nginx 172.31.73.13
nginx-Cluster-8SqXVHC service:nginx-Service-gL6Cnj fb4099e3...aba46 nginx 172.31.94.23
# App Server service - 2 tasks x 2 containers
app-Cluster-mWkD7Y4 service:app-Service-09C2Vy 11f3..9b94cc9f74 appsrv 172.31.21.13
app-Cluster-mWkD7Y4 service:app-Service-09C2Vy 11f3..9b94cc9f74 logger 172.31.21.13
app-Cluster-mWkD7Y4 service:app-Service-09C2Vy 4b5f..bc30edf99e appsrv 172.31.39.74
app-Cluster-mWkD7Y4 service:app-Service-09C2Vy 4b5f..bc30edf99e logger 172.31.39.74
ecs-session --list
Now we can use any of these identifiers — Task ID, Container name, or IP address — to select the desired container and login to it. In the case of the Apache service it’s easy as there’s only one.
In case of the Nginx service above we can use some other unique identifier — the Task ID or the Task IP address, because the “nginx” container name is not unique. We can also specify the command to run, the profile from ~/.aws/credentials or the region.
~ $ ecs-session --profile demo --command "/bin/bash -i" 172.31.73.13
Task ARN: arn:aws:ecs:ap-southeast-2:123456789012:task/nginx-Cluster-8SqXVHC/e0d6b610...52371
Container Name: nginx
The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.
Starting session with SessionId: ecs-execute-command-0abcdef20
root@ip-172-31-73-13:/usr/lib/nginx#
Here you can see that we have used the IP (172.31.73.13) instead of the container name (nginx). This works because each task (= each IP address) consists only of a single container and ecs-session can figure out all the required details just from the IP address. We could just as well have used the Task ID.
However in the case of the App Server in the example above it’s more complicated. Here we have to use two identifiers together because neither one of the Container name, the Task ID or the Task IP does on its own uniquely identify a container. However an IP plus a Container name will do the trick.
~ $ ecs-session appsrv 172.31.21.13
Task ARN: arn:aws:ecs:ap-southeast-2:123456789012:task/app-Cluster-mWkD7Y4/11f3..9b94cc9f74
Container Name: appsrv
...
tomcat@ip-172-31-21-13 /app $
What if ecs-session is unable to figure out which container we want to login to? Well it will tell us and give us a list of all the matching ones that we can choose from:
~ $ ecs-session nginx
WARNING: Found 3 instances for: nginx
WARNING: Use Container IP or Task ID to connect to a specific one
nginx-Cluster-8SqXVHC [...] a3875d0e...7d488 nginx 172.31.107.214
nginx-Cluster-8SqXVHC [...] e0d6b610...52371 nginx 172.31.73.13 nginx-Cluster-8SqXVHC [...] fb4099e3...aba46 nginx 172.31.94.23
This makes logging into ECS Fargate and EC2 containers very simple. In fact too simple some people may say.
As always please remember — just because you can do it doesn’t mean that you should do it. Or at least not too often :)
Configuring Tasks for ECS Exec
Now that we know how to connect to the containers we should also briefly touch on how to configure your ECS deployments to support ECS Sessions. These are the 4 requirements…
- The task IAM role must permit SSM Sessions. That is typically done by attaching AmazonSSMManagedInstanceCore managed policy.
- The task or service must have Execute Command setting enabled. Typically it’s a setting in your CloudFormation or Terraform template for the ECS Service.
- The tasks must have outbound access to the SSM service over port 443. This typically means opening egress access on port 443 in the Security Group. The outbound access can be either direct via a Public IP and IGW, or from a private subnet through a NAT gateway. It can also be achieved with VPC endpoints for SSM, but that’s beyond the scope of this post.
- The tasks must have Platform Version 1.4 or later and can run either on Fargate or on EC2, it doesn’t matter, both launch types work.
If any one of these conditions is not met then the task will not show up in the ecs-session --list and you won’t be able to open a shell session to it.
Check out my other article for both CloudFormation and Terraform examples of the required configuration.
Setting up Fargate for ECS Exec
Examples for both CloudFormation and Terraform.
The complete templates are also available in the project GitHub.
Installing AWS SSM Tools
The AWS SSM Tools package is available through PyPi and can be installed with pip in Python 3.6 or newer.
~ $ sudo pip3 install aws-ssm-tools
~ $ ecs-session --version
ssm-tools/1.4.2
Alternatively (and perhaps preferably) you can use pipx for a standalone installation that won’t pull any dependencies into your shared Python environment.
~ $ pip3 install --user pipx
~ $ pipx install aws-ssm-tools
~ $ ecs-session --version
Now that you’ve got aws-ssm-tools package installed don’t forget to check out the other scripts that came with it:
- ssm-session — for easy SSH-less logging into EC2 instances
- ssm-tunnel — for creating TCP tunnels over SSM
- ssm-ssh — for running SSH over SSM, which in turn allows for example RSYNC copying to and from EC2 instances without a direct SSH access.
In the end…
Let me know of any issues with any of these scripts or create pull requests for enhancements in the AWS SSM Tools GitHub project.
I hope that you will find ecs-session and all the other SSM tools useful. Let me know in the comments.
Now it’s time to check out some of my other posts!
New AWS instance that makes ETH mining profitable
Mining Ethereum on AWS so far made no sense. Until now! One new AWS instance is the game changer and mining on AWS is…
Also consider subscribing to medium using my referral link…