Pro2col’s Senior Technical Consultant Sam Fry demonstrates how to set up an S3 resource in AWS and configure it as a project resource in GoAnywhere.
Today’s businesses operate in a truly cloud-centric world. The adoption of cloud-based services is on the increase, as enterprises identify the significant benefits and cost savings. The ‘de facto’ infrastructure set-up for enterprises is shifting from pure on-prem to hybrid, and it is now a case of how enterprises use the cloud and not when.
Application and software developers are adopting connectors and open APIs to allow their software to integrate with a wide range of cloud platforms and services. GoAnywhere MFT from Fortra is an enterprise-level Managed File Transfer application that has been ahead of the game for a while. As well as cloud connectors, it also integrates with storage platforms like Amazon’s Simple Storage Service (Amazon S3) buckets.
Amazon S3 provides businesses of all sizes with an affordable and scalable cloud storage solution. GoAnywhere allows an Amazon S3 bucket to be configured as a resource and then specified as file repositories from within Domains or as a Web User virtual folder. Using the intuitive workflow builder, you can use the Amazon S3 component to build projects that retrieve or modify object metadata using the S3 task, or upload/download/manage documents by using qualified file paths.
GoAnywhere supports Amazon’s server-side SSE-S3 method for encrypting data at rest by using the AES 256-bit encryption standard. Advanced Encryption Standard (AES) – also known by its original name ‘Rijndael’ – has been adopted by the US government and is now used worldwide. When AES-256 is selected from within GoAnywhere, all files and data uploaded to the Amazon S3 resource will automatically be encrypted on the Amazon platform. Files that are already encrypted in the Amazon S3 bucket will be automatically decrypted when downloaded by GoAnywhere.
In addition, in GoAnywhere you can monitor Amazon S3 buckets for changes or modifications to data. Monitoring can help with automation tasks, such as email alerts when a file has been uploaded to a bucket, changed, or even deleted.
GoAnywhere MFT also supports Azure blob storage with the same level of functions and features in the Amazon S3 bucket resource.
To set up an S3 bucket for use within GoAnywhere:
- Login to your AWS console account and select Services > Storage > S3.
- In the Buckets window, select the Create Bucket button.
- Give the bucket a name (lowercase characters are only allowed in a bucket name).
- Select the AWS Region for the bucket to reside in.
- In this example, we are configuring a simple bucket access with basic access granted. Leave ACLs disabled (recommended).
It is possible to enable Access Control Lists relating to other AWS accounts, but this example does not use ACLs.
6. In the section ‘Block Public Access settings for this bucket’ we are blocking all public access as GoAnywhere will use an access key to connect.
7. Leave Bucket Versioning disabled. Versioning creates multiple variants of objects in the bucket, but this example does not use versioning.
8. Scroll down to the Default Encryption section.
In this example we are selecting the option Server-side encryption with Amazon S3 managed Key (SSE-S3).
9. Select the Create Bucket button at the bottom of the screen. Your new bucket has been created and is displayed in a list on screen.
10. Go back to the Services menu and type Users into the search bar. Navigate to Features > Users/
11. In the Users page select the Add Users button.
12. Select or enter the email address for the relevant user account. We are creating a new user in this example. Select Next.
13. On the Set Permissions page, select the Attach Policies Directly option. In the search box type in S3 to see the various S3 permission level policies available. A new policy can also be created.
14. Select a policy to grant the policy access to your required user.
15. After selecting the relevant policy, select Next. Then select the Create User button.
16. Double-click the user in the user list and navigate to the Security credentials tab.
17. Scroll down to the Access Keys section, and select the Create Access Key button.
18. Select the option Application running outside AWS and read the best practice guidance that is then displayed on the screen when using this option. Select Next.
Optionally, set a description tag for the access key you are about to create.
19. Select the Create Access Key button, the Access key, and the Secret access key are then displayed on the screen and can either be copied or saved into a .csv file.
The secret key is hidden and must be saved before leaving this page or the secret key will be lost.
20. Once both are copied, select the Done button. Navigate back to your new Bucket through the top menu Services > Storage >S3.
21. Click on the bucket to open the configuration tabs, and select the Access Points tab.
22. Select the Create access point and enter a name for the access point for this bucket (lowercase characters only including numbers).
23. Set the Network origin to the Internet. Ensure that all public access is Blocked as we did when creating the bucket.
24. Scroll down to the end of the page and select the Create access point button. This completes a basic setup for a new AWS S3 bucket. The bucket is now ready to be set up as a Resource in GoAnywhere.
25. In the GoAnywhere Admin Console, navigate to Resources, and select Amazon A3 Buckets. Select the Add Amazon S3 bucket button at the top of the screen. Enter a Name for the resource (this does not have to be the same as the bucket name).
26. Select the Authentication Type (we have used basic to facilitate initial connectivity). Enter the Access Key you copied from the Access Key creation stage in AWS.
Enter the Secret Access Key you copied from the Access Key creation stage in AWS. Enter the Bucket Name as created, and displayed in AWS
Select the Region (although it is also possible that GoAnywhere will recognise the bucket without setting this to be the same region as the bucket location).
We now have a functioning AWS S3 Resource in GoAnywhere which can be used in Projects and project Monitors, as a Webdocs location, or as a Web User Virtual Folder.
Examples of the S3 bucket in use:
- Using the S3 Upload (or Download) commands within a project. As with many of the GoAnywhere connectors, the upload and download commands take a single file so must be used within a loop to process multiple files.
- Using the S3 bucket as the source of a Create File List command in a Project.
- Using the S3 bucket in a Copy statement to copy to another resource location.
- Using the S3 Bucket as the source directory for a monitor.
When run in conjunction with a monitor, the monitor file set variable from the S3 source is passed into the project just like any other type of monitor directory file set.
In the project below, the S3 monitor file set variable is passed through to the project via the variable ${files}
