Storage Gateway

Introduction to AWS Storage Gateway

An on-premises gateway to infinite cloud storage is offered by the hybrid cloud storage solution known as AWS Storage Gateway.

By offering a range of industry-standard storage protocols including iSCSI, SMB, and NFS, Storage Gateway enables you to use AWS storage without having to modify your existing applications. By caching frequently requested data locally while securely and durably storing data on Amazon cloud storage services, it offers low-latency performance.

By transferring just updated data and compressing data, Storage Gateway increases data delivery to AWS.

Storage Gateway is compatible with four major hybrid cloud use cases: (1) Moving backups and archives to the cloud (2) Reducing on-premises storage with cloud-backed file shares (3) Providing on-premises applications with low-latency access to data stored in AWS (4) Data lake access for pre and post-processing workflows.

Types of Storage Gateways

Storage Gateway offers three types of storage interfaces namely file, volume, and tape.

File Gateway:

The Amazon S3 File Gateway allows you to save and retrieve objects from AWS S3 using file protocols like NFS and SMB. Direct access to the objects produced by the S3 File Gateway is possible.

FSx File Gateway is a new File Gateway type that allows low latency and efficient access from your on-premises facility to in-cloud FSx for Windows File Server file shares.

Volume Gateway:

A Volume Gateway delivers cloud-backed storage volumes that you may mount as Internet Small Computer System Interface (iSCSI) devices from your on-premises application servers.

A Volume Gateway can be deployed on-premises as a VM appliance operating on VMware ESXi, KVM, or Microsoft Hyper-V hypervisors, as a hardware appliance, or in AWS as an Amazon EC2 instance.

Tape Gateway:

Tape Gateway is a cloud-based virtual tape storage solution.

With a Tape Gateway, you may archive backup data in S3 Glacier Flexible Retrieval (or) S3 Glacier Deep Archive is a cost-effective and long-lasting manner.

It offers a virtual tape infrastructure that is scalable with your company's demands and removes the operational complexity of providing, scaling, and maintaining a physical tape infrastructure.

File Gateway

Amazon S3 File Gateway

It combines a service and a virtual software appliance to provide a file interface into AWS S3. You may use industry-standard file protocols like Network File System (NFS) and Server Message Block (SMB) to store and access data in Amazon S3.

The software appliance, or gateway, is installed as a virtual machine (VM) in your on-premises environment and runs on a hypervisor such as VMware ESXi, Microsoft Hyper-V, or Linux Kernel-based Virtual Machine (KVM).

The gateway grants access to S3 items in the form of files or file share mount points. You can perform the following with an S3 File Gateway:

• You may save and retrieve files immediately using the NFS version 3 or 4.1 protocol. • You may use the SMB file system protocol versions 2 and 3 to store and retrieve files directly. • You may easily access your data in AWS S3 from any AWS Cloud application • S3 data may be managed using lifecycle controls, cross-region replication, and versioning. Consider an S3 File Gateway to be a file system mount on Amazon S3.

An S3 File Gateway simplifies Amazon S3 file storage, links to existing applications using industry-standard file system protocols, and offers a cost-effective alternative to on-premises storage.

It enables low-latency data access via transparent local caching. An S3 File Gateway controls data transmission to and from Amazon Web Services (AWS), buffers applications from network congestion, optimizes and streams data in parallel, and monitors bandwidth use. S3 File Gateway integrates with AWS services, such as the ones listed below.

• AWS IAM is used to manage common access
• AWS Key Management Service encryption (AWS KMS)
• Amazon CloudWatch monitoring (CloudWatch)
• AWS CloudTrail auditing (CloudTrail)
• AWS Management Console and Command Line Interface operations (AWS CLI)
• Billing and expense control

FSx File Gateway:

FSx File Gateway is a new File Gateway type that allows low latency and efficient access from your on-premises facility to in-cloud FSx for Windows File Server file shares.

If you need on-premises file storage due to latency or bandwidth constraints, you may use FSx File Gateway to gain smooth access to fully managed, highly dependable, and practically infinite Windows file shares offered by FSx for Windows File Server in the AWS Cloud.

FSx File Gateway offers the following advantages:

• Assists in the elimination of on-premises file servers by consolidating all data in AWS to take advantage of the size and economics of cloud storage.

• Apps that must remain on premises may now enjoy the same low latency and excellent performance as those in AWS without stressing your networks or affecting the latencies experienced by your most demanding applications.

To utilize Amazon FSx File Gateway, you must have at least one Amazon FSx for the Windows File Server file system. You must also have access to FSx for Windows File Server on-premises, either through a VPN or an AWS Direct Connect connection.

In an on-premises environment, you download and install the FSx File Gateway VMware virtual appliance or an AWS Storage Gateway Hardware Appliance. After you've deployed your appliance, you can use the Storage Gateway interface or the Storage Gateway API to enable the FSx File Gateway. You may also use an Amazon Elastic Compute Cloud (Amazon EC2) image to establish an FSx File Gateway.

Use the Storage Gateway panel to join the Amazon FSx File Gateway to your Microsoft Active Directory domain when it has been enabled, and can access FSx for Windows File Server. After the gateway has successfully joined a domain, you may attach it to an existing FSx for Windows File Server using the Storage Gateway interface.

The diagram below illustrates the implementation of file storage for Storage Gateway.

Volume Gateway (iSCSI)

A Volume Gateway delivers cloud-backed storage volumes that you may mount as Internet Small Computer System Interface (iSCSI) devices from your on-premises application servers.

A Volume Gateway can be deployed on-premises as a VM appliance operating on VMware ESXi, KVM, or Microsoft Hyper-V hypervisors, as a hardware appliance, or in AWS as an Amazon EC2 instance.

The gateway supports the following volume configurations:

• Gateway Cached Volumes
• Gateway Stored Volumes

The Architecture of Gateway Cached Volumes

You may utilize Amazon S3 as your primary data storage while keeping frequently requested data locally in your Storage Gateway by employing cached volumes.

Cached volumes reduce the requirement to expand your on-premises storage infrastructure while still delivering low-latency access to frequently requested data to your applications.

Storage volumes of up to 32 TiB in size may be created and attached to iSCSI devices from your on-premises application servers. Your gateway saves data that you write to these volumes in Amazon S3 and keeps recently read data in the cache, and uploads buffer storage of your on-premises Storage Gateway.

The size of cached volumes might range from 1 GiB to 32 TiB and must be rounded to the closest GiB. Each cached volume gateway may accommodate up to 32 volumes for a total storage volume of 1,024 TiB. (1 PiB).

Storage Gateway's cached volumes solution saves all of your on-premises application data in an Amazon S3 storage volume. The graphic below depicts an overview of the stored volumes deployment.

The Architecture of Gateway Stored Volumes

You may use stored volumes to keep your primary data locally while asynchronously backing it up to AWS. Stored volumes enable low-latency access to your on-premises applications' complete datasets.

At the same time, they offer long-term, offsite backups. Storage volumes may be created and mounted as iSCSI devices from your on-premises application servers.

Data written to saved volumes are kept on your on-premises storage hardware. This information is stored asynchronously to Amazon S3 as Amazon Elastic Block Store (Amazon EBS) snapshots.

The size of stored volumes can range from 1 GiB to 16 TiB and must be rounded to the closest GiB. Each gateway set for stored volumes may accommodate up to 32 volumes and 512 TiB of total volume storage (0.5 PiB).

You keep your volume storage on-premises in your data center with stored volumes. That is, all of your application data is stored on your on-premises storage infrastructure. The gateway then uploads data to the Amazon Web Services Cloud for cost-effective backup and speedy disaster recovery, employing features that assist protect data security.

Because you require low-latency access to all of your data and backups in AWS, this solution is appropriate if you wish to retain data locally on-premises.

The following diagram depicts the deployment of stored volumes.

Tape Gateway (VTL)

Tape Gateway is a cloud-based virtual tape storage solution.

With a Tape Gateway, you may archive backup data in S3 Glacier Flexible Retrieval (or) S3 Glacier Deep Archive is a cost-effective and long-lasting manner.

It offers a virtual tape infrastructure that is scalable with your company's demands and removes the operational complexity of providing, scaling, and maintaining a physical tape infrastructure.

Storage Gateway may be deployed on-premises as a VM appliance operating on VMware ESXi, KVM, or Microsoft Hyper-V hypervisors, as a hardware appliance, or as an Amazon EC2 instance on AWS.

To provision iSCSI storage volumes in AWS, you deploy your gateway on an EC2 instance. Gateways hosted on EC2 instances can be used for disaster recovery, data mirroring, and storage for Amazon EC2-based applications.

Tape Gateway provides a long-lasting, low-cost method for archiving data in the Amazon Web Services Cloud.

You leverage your current tape-based backup architecture with its virtual tape library (VTL) interface to store data on virtual tape cartridges created on your Tape Gateway.

These are offered as iSCSI devices to your current client backup programs. Tape cartridges are added as needed to archive your data.

The figure below depicts the deployment of Tape Gateway. The following Tape Gateway components are identified in the diagram:

• Virtual tape - A virtual tape functions similarly to an actual tape cartridge. Virtual tape data, on the other hand, is kept on the Amazon Web Services Cloud.

• Virtual Tape Library (VTL): It is similar to a physical tape library that is provided on-premises and equipped with robotic arms and tape drives. Your VTL contains a collection of virtual tapes. Each Tape Gateway includes one VTL.

• Tape drive - A VTL tape drive functions similarly to a physical tape drive, allowing I/O and search operations on a tape. Each VTL includes a set of 10 tape drives that are accessible to your backup program.

• Media changer - A VTL media changer is similar to a robot that moves tapes around in the storage slots and tape drives of a physical tape library. Each VTL has one media changer, which is accessible as an iSCSI device.

• Archive - The archive is comparable to a facility that stores tapes offsite. Tapes from the VTL on your gateway can be archived. Tapes from the archive can be retrieved and returned to your gateway's VTL if necessary.

• Tape archiving - Your gateway moves a tape to the archive for long-term storage when your backup program ejects a tape. The archive's tapes are kept (VTS) on a virtual tape shelf.

• Retrieving tapes - Tape archives cannot be read directly. You must first retrieve an archived tape to your Tape Gateway using the Storage Gateway interface or the Storage Gateway API.