Using SSM Session Manager for interactive instance access

You may argue that interactive login to EC2 instances should never be needed. Everything is dynamic, automated, self healing, centrally logged, and so on and there is no place for human interaction, right? But lets be honest – the world isn’t perfect and we all sometimes need to jump into bash to do stuff for one reason or another.

Traditionally our EC2 instances would have a SSH KeyPair assigned and a Security Group with SSH port open. If they were on private IPs only we would also need a jump host or VPN to access them. That works but it’s got its own problems – for example rotation of the SSH keys, keeping the Security Group up to date, etc.

AWS Systems Manager offers a better solution – the SSM Session Manager.

Session Manager enables ad-hoc shell access for any authorised IAM User completely outside of your Network / VPC / Security Group infrastructure. In fact your instance doesn’t even need to have sshd running! All it needs is a running amazon-ssm-agent, which comes preinstalled in all recent AMIs, and a correct EC2 IAM Policy for connecting to the SSM service.

Any recent Amazon Linux, Amazon Linux 2 as well as the official Ubuntu and RedHat AMIs come with SSM Agent ready to use. If it is not installed on your instance check out Installing and Configuring SSM Agent on Amazon EC2 Linux Instances.

The next requirement is AmazonEC2RoleforSSM managed policy attached to your instance role. In my CloudFormation templates I use this role definition:

InstanceRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
         Version: '2012-10-17'
         Statement:
         - Action: sts:AssumeRole
           Effect: Allow
           Principal:
             Service: ec2.amazonaws.com
       Path: /
       ManagedPolicyArns:
       - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM

Provided the instance can connect from its subnet to ssm.{region}.amazonaws.com – either directly, through NAT or through a Proxy – you are now all set on the instance side.

One way to open a new shell session is from the AWS Console. Simply go to AWS Systems Manager service dashboard, under Actions open the Session Manager, select your instance and click Start session. It can’t be any easier!

Unfortunately I’ve got two problems with this setup:

1. You can’t work on instances in more than one AWS account at the same time. That’s the nature of AWS Console unfortunately.
2. I keep closing the browser window!
   Whenever I press Ctrl+W to delete the previous word in the shell or in vim my Chrome intercepts it and closes the open tab. Arrrgghhh!!

I find it much better to open the shell session through aws cli. There I can keep pressing Ctrl+W as much as I want 🙂

Make sure you’ve got the session-manager-plugin installed. Otherwise you will get a message prompting you to do so.

~ $ aws ssm start-session --target i-0123abcd1234abcd
SessionManagerPlugin is not found. Please refer to SessionManager
Documentation here:
http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-found

With the plugin installed the usage is really simple:

~ $ aws ssm start-session --target i-0123abcd1234abcd
Starting session with SessionId: michael.ludvig-006c1ff2b131a2531

sh-4.2$ bash
ssm-user@test1 /usr/bin $ id
uid=1001(ssm-user) gid=1001(ssm-user) groups=1001(ssm-user)

ssm-user@test1 /usr/bin $ sudo su -
Last login: Thu May 23 17:14:29 NZST 2019 on pts/0

root@test1 ~ # id
 uid=0(root) gid=0(root) groups=0(root)

UPDATE: I wrote a script to start SSM Sessions by instance name, IP, etc. Check it out here: ssm-session – SSM Sessions the easy way

SSM Agent has automatically created this ssm-user for us and gave it sudo privileges. You can check what has been done in /var/log/amazon/ssm/amazon-ssm-agent.log:

2019-05-24 22:05:58 INFO [MessageGatewayService] Successfully created ssm-user
2019-05-24 22:05:58 INFO [MessageGatewayService] Successfully created file /etc/sudoers.d/ssm-agent-users
2019-05-24 22:05:58 INFO [MessageGatewayService] Successfully changed mode of /etc/sudoers.d/ssm-agent-users to 288

One little unsettling thing is that this session doesn’t appear in who, w or last outputs…

root@test1 ~ # w
  22:20:09 up  3:53,  0 users,  load average: 0.00, 0.01, 0.05
 USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT

root@test1 ~ # who

root@test1 ~ # last
 reboot   system boot  3.10.0-957.1.3.e Thu May 23 13:26
 wtmp begins Thu May 23 13:26:23 2019

root@test1 ~ # 

It would be nice to have a record of these logins in the standard places – if you know how to do that let me know in the comments.

Quite surprisingly SSM Sessions also works with Windows EC2 instances. However don’t expect a full graphical interface, you will only get a PowerShell command prompt…

~ $ aws start-session --target i-1234abcd1234abcd
 Starting session with SessionId: botocore-session-1558574990-abcd1234abcd1234

 Windows PowerShell
 Copyright (C) Microsoft Corporation. All rights reserved.
 
 PS C:\Windows\system32>
 PS C:\Windows\system32> exit

Exiting session with sessionId: botocore-session-1558574990-abcd1234abcd1234.

As I mentioned above the Session Manager doesn’t need inbound access to the instances. Instead all the traffic is relayed through the SSM service.

1. When the instance boots up it connects to AWS SSM service endpoint and awaits the commands.
   
2. When we run aws ssm start-session also connects to the AWS SSM service endpoint (very likely a different node though).
   
3. AWS-CLI then executes the session-manager-plugin that we installed above which then negotiates a secure WebSocket channel with SSM. Something similar then happens on the instance where a ssm-session-worker process is in charge of its side of the WebSockets stream.
   
4. All the keystrokes and all the screen outputs is passed from the instance through this channel between the instance and us. Note that at no time there is a direct connection between us and the instance. Everything is passed through the AWS SSM service.

I found the SSM Session Manager to be a very good option for the occasional ad-hoc access to our instances. No need to keep track of SSH keys, opening Security Group ports, etc. Give it a try!