There are many ways to securely transfer data around today. One approach that has been around for a long time involves using the Secure File Transfer Protocol (SFTP). SFTP is a network protocol that provides file access, file transfer, and file management over a reliable data stream. The standard approach for SFTP is using TCP port 22.

One of the most popular places to store files and other objects today is in cloud storage services like S3 on AWS. S3 provides highly available and highly durable storage of data. S3 buckets are almost infinitely scalable in how much data they can hold and have lots of tooling around them and a huge number of configuration options.

This article and associated Github repo provide a low touch implementation of a SFTP Server that is backed by an S3 bucket in your AWS account. The SFTP server will have a set of static IP addresses for users to reach it and will optionally restrict incoming connections to IP ranges you specify. The setup is done using Terraform Infrastructure as Code (IaC) and can be up and running in minutes allowing users to upload and download files from one of your AWS S3 buckets using any standard SFTP client.

AWS Transfer Family

AWS supports a fully managed service call the AWS Transfer Family which works with S3 as well as Amazon EFS to provide a way to ingest data into AWS via standard protocols like SFTP and others. Files uploaded via AWS Transfer Family can be put directly into S3 buckets or EFS filesystems. Users can be easily setup in AWS Transfer Family servers and you have a number of options for authentication including using SH Keys, Username/Password, integration with Microsoft Active Directory, and more.

AWS Transfer Family servers are highly available and can scale to very large throughput ranges (multiple Gbps when using parallel transfers). Cloudwatch logging can track all activity on your servers and you can use IAM permissions to define what users are able to do.

AWS Transfer Family Servers can be setup as public facing and can have custom domain names associated with them. This will make it easy for clients to use hostnames to interact with the server. One drawback of this approach is that the IP addresses the servers run on may change over time. The DNS lookup of the hostnames will be able to resolve the IP addresses though.

The SFTP servers with this service can also be setup inside a Virtual Private Cloud (VPC) where static public IP addresses can be setup to reach them. In this case, the servers can still be reachable from the Internet (if desired), but you can also setup firewall rules and limit access using various mechanisms.

Transferring Data in A Business Environment

Security is at the top of mind for most businesses today and when enabling transfer of files outside of your company it is typically required to setup firewall rules. Most companies will block outgoing SSH or file transfer traffic. This is to ensure and corporate data cannot easily be transferred out.

When there is a need to setup a firewall rule to allow such traffic, typically static IP addresses need to be involved. If a long running SFTP server needs to be accessed from inside the company network then a special rule can be created based on the fixed IP addresses the external server will have.

Another key requirement in this area revolved around the organization setting up an SFTP server for receiving files. If you setup a server that is internet facing with no restrictions on it then you will surely be bombarded with random incursion attempts from all over the world. If you look closely at the logs for servers you run on the internet you will see what I mean. Since SFTP runs over the same port, 22, as Secure SHell (SSH) it will be a prime candidate for incursion attempts.

One great way to handle this is by using the VPC option for setting up your SFTP server based on AWS Transfer Family. When running in a VPC you can setup firewall rules (using Security Groups) which limit what traffic is allowed in using rules like IP ranges. When these are setup and incoming requests that do not match will get instantly dropped.

My Solution for SFTP access

When building solutions on AWS I like to take advantage of managed offerings whenever possible - where the responsibility to setup and maintain resources is with AWS and I can focus on the business logic and specific problems i am trying to solve. Offloading this responsibility typically comes at a cost in terms of actual $$$’s for the service and potential loss of flexibility. In any kind of solution you build in the cloud (or other environments) there are always trade-offs and you have to try and come up with a plan using the tools available that meets your business parameters.

AWS Transfer Family is not free and there is no free tier. It does cost $0.30 US per hour while your SFTP server is running and there is an additional cost of around $0.04 per GB uploaded or downloaded (depending on the AWS region you use). This likely sounds expensive to many people and you can certainly write your own code to setup your own SFTP servers for much less money. You will be required to build and maintain this and you need to determine what is best for your use case. I like the super convenience of using AWS Transfer Family and the cost is worth it in many cases for me.

Terraform Configuration

The solution i have built is based on Terraform. There are a number of files in the repo that contain all the information required by Terraform to setup an SFTP server in your AWS account which uses an S3 bucket for file storage. If you’re going to use the files in my Github repo and set this up yourself than you will want to update the values in the configuration.tf file in the terraform directory.

variable "aws_region" {
  description = "AWS region"
  default     = "us-east-1"
}

variable "az_list" {
  description = "AWS availability zones"
  default     = ["us-east-1a", "us-east-1b"]
}

variable "storage_bucket_name_prefix" {
  description = "Prefix for the bucket name to store the files in"
  default     = "sftp-s3-bucket-"
}

variable "transer_familiy_default_user" {
  description = "The Username for the SFTP server"
  default     = "sftpuser"
}

variable "allowed_incoming_cidr_list" {
  description = "List of CIDR blocks to allow incoming traffic from for the SFTP server"
  default     = ["0.0.0.0/0"]
}

The storage_bucket_name_prefix value is the prefix for the S3 bucket name in your account which will be associated with the SFTP server. A random string will be generated by terraform and appended to this prefix to ensure your bucket name is globally unique. The transer_familiy_default_user is the sftp username to login to the server.

The allowed_incoming_cidr_list is the list of IP address ranges (or CIDR blocks) that are allowed to connect to the SFTP server. In the repo this is set to "0.0.0.0/0" by default but i highly recommend setting this to a single IP for security if you know where files will be uploaded/downloaded from. To set it to your own IP you can use a tool like https://whatismyipaddress.com/ to find your internet IPv4 address and then put that IP address in the allowed_incoming_cidr_list variable as the value with a “/32” suffix (i.e. “1.2.3.4/32” if your address is 1.2.3.4) instead of “0.0.0.0/0”. If you do this then nobody else in the world can connect to the server.

Once you have made the appropriate changes in the configuration.tf file you can run the following commands to install the sftp server in your AWS account. (Of course you need to be setup with credentials in your current terminal shell for whatever AWS account you want to set this up in.)

terraform init
terraform apply

When these commands have been run and you have said “yes” to perform the apply, terraform will create the resources in your AWS account to setup the SFTP server including creating the S3 bucket, setting up the VPC and resources needed for it, create the AWS Lambda function for authentication, create a random password for your user and store it in AWS Secrets Manager, and create the actual AWS Transfer Family SFTP Server.

Once it’s done it will output the following information.

The S3 bucket name that will be used for the SFTP server files is listed. You can go and look in the AWS console in your account and you will see this. The SFTP password for the user of the SFTP server will be displayed (NOTE the username is what is define in transer_familiy_default_user in the terraform configuration.tf file. The list of static internet routable IPv4 addresses is also listed here. You should be able to now connect to any of these IP addresses using any standard SFTP client using the username and password now.

How the authentications works

The authentication is for this solution is setup using an AWS Lambda function that compares the user entered password in an SFTP client with the randomly generated 64 character password that is stored in AWS Secrets Manager.

If you look in the AWS console under the AWS Transfer Family Page you will see something like below.

The AWS Lambda function that is used for authentication will be shown on this page and you can jump to see the details of it. Here is what the AWS Lambda function code looks like.

import os
import json
import boto3
import traceback
from aws_lambda_powertools import Logger, Tracer, Metrics

logger = Logger(service="PrivateSFTP")
tracer = Tracer(service="PrivateSFTP")
metrics = Metrics(namespace="PrivateSFTP", service="PrivateSFTP")

secrets_region = os.environ["SecretsManagerRegion"]
secrets_client = boto3.session.Session().client(service_name="secretsmanager", region_name=secrets_region)

@tracer.capture_lambda_handler
@logger.inject_lambda_context(log_event=True)
@metrics.log_metrics(capture_cold_start_metric=True)
def handler(event, context):
    try:
        server_id = event["serverId"]
        user_name = event["username"]
        protocol  = event["protocol"]
        source_ip = event["sourceIp"] 
        password  = event.get("password", "")

        logger.info(f"server ID: {server_id}, username: {user_name}, protocol: {protocol}, source IP: {source_ip}")

        secret_id = f"aws/transfer/{server_id}/{user_name}"
        expected_password_secret = secrets_client.get_secret_value(SecretId=secret_id).get("SecretString", None)

        if expected_password_secret is not None:
            expected_password_secret_dict = json.loads(expected_password_secret)

            expected_password = expected_password_secret_dict.get("password", None)
            if password == expected_password:
                logger.info(f"Password for user: {user_name} matches expected password")
                response = {
                    "Role": expected_password_secret_dict.get("role", None),
                    "HomeDirectory": expected_password_secret_dict.get("home_dir", None)
                }
                logger.info(f"Response: {response}")
                return response
            else:
                logger.error(f"Password for user: {user_name} does not match expected password")
                return {}
        else:
            logger.error(f"No secret found for user: {user_name}")
            return {}

    except Exception as e:
        traceback.print_exc()
        logger.info(f"traceback={traceback.format_exc()}")   
        return {}

The Lambda code is using the AWS Powertools for Lambda library which I highly recommend everyone uses for the code. The AWS Transfer Family integration with AWS Lambda passes in key information like the username, password, server id, IP address, etc and the code above tries to find a secret in AWS Secrets Manager matching the server id and username. If it does find this then it compares the entered password with the expected value from the secret. If they match it returns the IAM Role containing the permissions the user has for interacting with AWS resources and the home directory they have (which maps to the root of the S3 bucket associated with the server.

Potential Enhancements

The default configuration in the Github repo only creates one user. There may be a need to have more than 1 user so changes could be made to allow a list of users.

The authentication could be done using SSH keys instead of username/password authentication. This could be considered more secure.

I have used AWS Secrets Manager for storing the password. Typically i use the SSM Param Store for holding key/value pairs. Secrets Manager is used here because it has support for setting up secrets rotation via AWS Lambda functions (see here for more details). It would likely be best to rotate the password regularly if the SFTP server is running over a long period of time. The updated password would have to be shared with the clients.

Please let me know if you can suggest other improvements.

CLEANUP (IMPORTANT!!)

As mentioned near the top, AWS Transfer Family does not have a free tier!! Setting up a server using this will cost you real $$$s. It has pricing of $0.30 USD per hour (depending on which AWS region you use). There is also a charge based on GB’s uploaded and downloaded.

If you are going to setup a server using the repo i provide, please MAKE SURE TO DELETE the server if you are no longer using it. Running terraform destroy can take care of this or you can delete the server in the AWS console.

Try the setup in your AWS account

You can clone the Github Repo and try this out in your own AWS account. The README.md file mentions any changes you need to make for it to work in your AWS account.

Please let me know if you have any suggestions or problems trying out this example project.

For more articles from me please visit my blog at Darryl's World of Cloud or find me on Bluesky, X, LinkedIn, Medium, Dev.to, or the AWS Community.

For tons of great serverless content and discussions please join the Believe In Serverless community we have put together at this link: Believe In Serverless Community