Amazon S3 Lifecycle Management

Amazon S3 storage, a leading cloud storage solution, is widely adopted by businesses worldwide for critical data storage. Effective management of this data, guided by AWS S3 Lifecycle Policy, is essential for cost efficiency. The policy involves rules for automating data management in S3, ensuring data is stored appropriately over time. It includes transitioning outdated data to cheaper storage or deleting it after a certain period. This lifecycle management ensures cost-effective storage throughout the data's lifecycle in AWS S3.

Why is AWS S3 Lifecycle Policy Management Needed?

Configure the Amazon S3 Existence of your objects to ensure that they are stored cost-effectively throughout their lifecycle. An S3 Lifecycle configuration is a set of rules that describe the activities that Amazon S3 does on a set of objects. Actions are classified into two types:

Transition Actions These actions determine when objects change storage classes. For example, you may select to move items to the S3 Standard-IA storage class 30 days after they are created or archive them to the S3 Glacier Flexible Retrieval storage class one year after they are created. AWS S3 Lifecycle Policy transition requests are not without cost. 

Expiration Actions When objects expire, these actions determine when they do so. Amazon S3 deletes expired items for you. The cost of lifecycle expiration is determined by when you choose to expire items.

• If there is a time lag between when an object becomes eligible for a lifecycle action and when Amazon S3 transfers or expires your object, billing adjustments are performed as soon as the item becomes eligible. For example, if an item is set to expire and Amazon S3 does not instantly expire it, you will not be charged for storage after the expiration date.
• The only exception to this is if you have an AWS S3 Lifecycle Policy rule that directs the storage to the S3 Intelligent-Tiering storage class. Billing adjustments do not occur in this situation until the item has converted to S3 Intelligent Tiering.
• Set up your objects in AWS S3 Lifecycle Policy to ensure that they are stored affordably throughout the duration of their existence. A set of guidelines known as an AWS S3 Lifecycle Policy configuration outlines the operations carried out by Amazon S3 on a collection of items. Two categories of acts exist:
• Shift operations are used to decide which items migrate to which storage type. For instance, you may choose to move objects to the S3 Standard-IA storage class after 30 days (or) archive them to the S3 Glacier Flexible Retrieval storage class after a year.
• AWS S3 Lifecycle Policy transition requests are not free. Expiration actions provide the time frame for an item's obsolescence. You can have expired items removed via Amazon S3. The cost of the lifecycle depends on when you decide to discard objects.
• Billing adjustments are performed as soon as an object becomes eligible for an AWS S3 Lifecycle Policy action, even if there is a delay between that point and when your object transfers or expires.
• You won't be charged for storage after an item's expiration date if Amazon S3 does not instantly delete it. The sole exception is if an AWS S3 Lifecycle Policy rule directs the storage to the S3 Intelligent-Tiering storage class. Billing adjustments won't take place in this situation until the item has been switched over to S3 Intelligent Tiering.

Managing S3 Object Lifecycle

Create setup guidelines for the AWS S3 Lifecycle Policy for objects with a clear lifecycle. An example would be as follows:

• Your application could need logs for a week or a month if you regularly upload them to a bucket. You then have the option to remove them.
• Some documents are used repeatedly for a brief period of time. After then, they are rarely accessible. At some point, you might not need immediate access to them, but your company or regulations could mandate that you preserve them for a predetermined amount of time. You may then remove them after that.
• Certain types of data may be uploaded to Amazon S3 solely for archival purposes. Among other objects, are digital archive content, financial and medical records, unprocessed genetic sequence data, long-term database backups, and information on regulatory compliance.
• You may instruct Amazon S3 to move objects to cheaper storage classes, archive them, or destroy them using AWS S3 Lifecycle Policy configuration rules.

S3 Lifecycle Policies

AWS S3 Lifecycle Policies

• Using AWS S3 Lifecycle Policy rules inside S3 is a good way to guarantee that your data is appropriately kept (without incurring unnecessary costs) and that it is cleanly deleted once it is no longer required. AWS S3 Lifecycle Policy rules allow you to automatically analyze items in your S3 Buckets and move or delete them from S3.
• This might be done for security, legislative compliance, company policy compliance, or simply good housekeeping. Putting in place the appropriate AWS S3 Lifecycle Policy will help you improve data security. Good lifecycle policies can assist in ensuring that sensitive information is not stored for any longer than is absolutely necessary.
• When required, these policies may change archive data swiftly onto AWS Glacier behind extra security safeguards. Glacier is commonly used as a "cold storage" option for information that must be retained but is seldom accessed, and it is far less expensive than AWS S3.
• AWS S3 Lifecycle Policy is implemented at the Bucket level, with each Bucket containing up to 1000 policies. Using object 'prefixes,' different rules affecting different objects can be shown within the same Bucket. The policies are automatically validated and executed, with no need for a manual start.
• An important note regarding this automation: be aware that AWS S3 Lifecycle Policy may not be active immediately once they are configured since the policy may need to propagate over the AWS S3 Service. It is vital to begin by ensuring that your automation is functional.
• Policies may be defined and implemented through the AWS Console (or) the AWS S3 API. This article will show you how to set up AWS S3 Lifecycle Policy using the AWS interface. Cloud Academy has a good "overview" blog post comparing Amazon S3 with Amazon Glacier.

Use AWS S3 Lifecycle Policy rules to govern your object: An AWS S3 Lifecycle Policy rule explains how Amazon S3 manages objects during their existence and may be used to govern an object's S3 Lifecycle Policy.

Automatic transition to tiered storage: AWS S3 Lifecycle Policy allows you to migrate items to the Standard IA storage class and then to the Glacier storage class.

Expire your objects: You may use the AWS S3 Lifecycle Policy rule to automatically expire your objects.

Setting up a Lifecycle Policy in S3

Using the AWS S3 Console: By utilizing a shared prefix (object names that begin with a similar string) or a tag, you may create AWS S3 Lifecycle Policy rules for all or a subset of items in the bucket. An AWS S3 Lifecycle Policy rule may be used to define actions that are particular to current and non-current object versions. See the following for further information:

To Create an AWS S3 Lifecycle Policy Rule Using AWS Console:

• login to the AWS Management Console and navigate to the link.
• In the Buckets list, select the bucket for which you want to establish an AWS S3 Lifecycle Policy rule.
• Navigate to the Management tab and choose to Create AWS S3 Lifecycle Policy rule.
• Enter a name for your rule in the AWS S3 Lifecycle Policy rule name. (The name should be distinct inside the bucket.)
• Define the AWS S3 Lifecycle Policy rule's scope:
• Select To apply this life cycle rule to all objects with a specified prefix or tag, restrict the scope to those prefixes or tags.
• To limit the scope by Prefix, put the Prefix in Prefix.
• Select Add tag and input the tag key and value to limit the scope by tag.
• Select to execute this AWS S3 Lifecycle Policy rule to all items in the bucket. This rule should apply to all objects in the bucket, and you can pick one of them. I understand that this rule applies to all of the objects in the bucket.
• To filter a rule based on object size, choose to Specify minimum object size, Specify maximum object size, or both.
• When selecting a minimum or maximum object size, the number must be greater than 0 bytes and not less than 5TB. This value can be specified in bytes, KB, MB, or GB.
• If both are specified, the maximum object size must be greater than the minimum object size.
• Select the activities you want your AWS S3 Lifecycle Policy rule to do under AWS S3 Lifecycle Policy rule actions.
• Move current object versions between storage classes
• Move earlier versions of items from one storage class to another.
• Object's current versions will expire.
• Delete older versions of objects permanently.
• Remove expired delete marks as well as incomplete multipart uploads.
• Various alternatives display depending on the activities you select.
• To move current versions of items across storage classes, see Move current versions of objects between storage classes.
• In Storage class transitions, select the storage class to which you want to transition:
  • Standard-IA
  • Intelligent-Tiering
  • One Zone-IA
  • S3 Glacier Flexible Retrieval
  • Glacier Deep Archive
• Enter the number of days after the object's creation in the Days after object creation.
• Transitions can be set up for the most recent version of an item, a previous version, or both. Versioning allows you to keep many copies of a single object together in a single bucket.
• Your objects will still be kept in Amazon S3 even if you use the S3 Glacier Flexible Retrieval or Glacier Deep Archive storage classes. When using the Amazon S3 Glacier service, they are not instantly available.
• For information on transitioning non-current versions of objects between storage classes, see Transitioning non-current versions of objects between storage classes:
• In Storage class transitions, select the storage class to which you want to transition:
  • Standard-IA
  • Intelligent-Tiering
  • One Zone-IA
  • S3 Glacier Flexible Retrieval
  • Glacier Deep Archive
• Enter the number of days after the object becomes non-current in the Days after the object becomes non-current.
• To expire current versions of objects, enter the number of days after object creation in the Number of days after object creation under Expire previous versions of objects.
• In a non-versioned bucket, the expiry action causes Amazon S3 Lifecycle Policy to delete the object permanently.
• To permanently destroy prior versions of objects, specify the number of days in Days after objects become non-current under Permanently delete non-current versions of items. You may choose the number of newer versions to keep by providing a value in the Number of newer versions to keep the field.
• Select Delete expired object delete markers and Delete incomplete multipart uploads from the drop-down menus next to Delete expired delete markers and Delete incomplete multipart uploads, respectively. Enter the number of days you wish to wait before ending and cleaning up any unfinished multipart uploads once the multipart upload has started.
• Select Create rule.
• If the rule is error-free, Amazon S3 activates it, and you can see it under AWS S3 Lifecycle Policy rules on the Management tab.

To Create an AWS S3 Lifecycle Policy Rule Using AWS CLI:

The AWS S3 Lifecycle Policy settings may be managed using the AWS CLI commands listed below:

• put-bucket-AWS S3 Lifecycle Policy-configuration
• get-bucket-AWS S3 Lifecycle Policy-configuration
• delete-bucket-AWS S3 Lifecycle Policy

An XML file serves as the Amazon AWS S3 Lifecycle Policy configuration. However, you are unable to supply the XML when using the AWS CLI. Instead, you must supply the JSON. The XML AWS S3 Lifecycle Policy settings shown below are examples, and the corresponding JSON values may be specified in an AWS CLI command.

Take the sample AWS S3 Lifecycle Policy setup shown below.

Example 1

The put-bucket-AWS S3 Lifecycle Policy configuration can be tested as follows:

In order to verify the configuration

• Create a file and save the JSON AWS S3 Lifecycle Policy configuration (AWS S3 Lifecycle Policy.json).
• To change your bucket's AWS S3 Lifecycle Policy settings, run the following AWS CLI command.

• To confirm, use the get-bucket-AWS S3 Lifecycle Policy configuration AWS CLI command to receive the AWS S3 Lifecycle Policy configuration.

• Use the delete-bucket-AWS S3 Lifecycle Policy AWS CLI command as shown below to remove the AWS S3 Lifecycle Policy settings.

For more information, you can refer to the official AWS docs.

S3 Object AWS S3 Lifecycle Policy Management Rules

Supported AWS S3 Lifecycle Policy Transitions

• Other storage class for S3 Standard storage.
• Any storage class —> storage classes S3 Glacier or S3 Glacier Deep Archive.
• S3 Intelligent-Tiering (or)S3 One Zone-IA -> S3 Standard-IA
• S3 One Zone-IA storage class —> S3 Intelligent-Tiering storage class.
• S3 Glacier Deep Archive storage class is a subclass of the S3 Glacier storage class.

Unsupported AWS S3 Lifecycle Policy Transitions

• Any storage class X S3 Standard
• Any storage class X Reduced Redundancy
• S3 Intelligent-Tiering X S3 Standard-IA
• S3 One Zone-IA X S3 Standard-IA or S3 Intelligent-Tiering

AWS S3 Lifecycle Policy Transitions Constraints

1. STANDARD (128 KB and 30 days)-> S3 Intelligent-Tiering, One Zone-IA, or STANDARD-IA
   • Larger Objects - In order to enjoy the cost savings associated with switching to STANDARD-IA or One Zone-IA, only objects larger than 128 KB may be migrated.
   • Smaller objects (<128KB) - S3 does not migrate objects that are less than 128 KB.
   • Minimum 30 Days - New objects are accessed more often or removed sooner than is appropriate for STANDARD-IA or One Zone-IA. Therefore they must be kept in the present storage class for at least 30 days before moving to one of those classes.
2. Glacier: (90 days) Permanent Deletion OR Glacier Deep Archive: (180 days) Permanent Deletion
   • If the removed items have been archived for three months or more, deleting data that has been stored in Glacier is free.
   • If the item is removed or rewritten within three months of being archived, S3 assesses a prorated early deletion cost.
3. Using object AWS S3 Lifecycle Policy management, objects are archived to Glacier asynchronously, and there may be a delay between the transition date in the AWS S3 Lifecycle Policy configuration rule and the date of the actual transfer. AWS, on the other hand, charges Glacier rates according to the rule's chosen transition date.
4. For a S3 Bucket that Supports Versioning
   • Current versions are affected by the Transition and Expiration acts.
   • The NoncurrentVersionTransition and NoncurrentVersionExpiration actions are applicable to non-current versions, and they function identically to non-versioned objects with the exception that the time period is measured from the moment the objects became non-current.
5. Expiration Guidelines
   • For Non-Versioned Bucket:
     • Object is permanently removed for buckets that are not versioned.
   • For Versioned Bucket:
     • Expiration only affects the current object in a versioned bucket; it has no effect on any of the non-current objects.
     • S3 will insert a non-current version of the preceding item and insert a Delete Marker object with a unique id. When a delete marker is a current object, S3 will not do anything.
     • S3 removes the Deletion Marker if the bucket contains a single item that is the Delete Marker (also known as an expired object delete marker).
   • For Versioned Suspended Bucket:
     • S3 will insert a Delete Marker object with version ID null and replace any object with version ID null for versioned suspended buckets.
6. S3 queues up objects for removal as they approach the end of their useful lives and removes them asynchronously. The time between the expiration date and the day AWS S3 Lifecycle Policy removes an object may differ. An expired object is not charged for the storage period connected with it.
7. There is a fee if an item in STANDARD-IA, GLACIER, or GLACIER DEEP ARCHIVE expires earlier than 30 days, 90 days, or 180 days.

Examples of AWS S3 Lifecycle Policy Configuration

This section contains AWS S3 Lifecycle Policy configuration samples. Each example demonstrates how to provide XML in each of the sample cases.

Topics

Example 1: Choosing a Filter Each S3 Lifecycle Policy rule offers a filter that you can use to identify a subset of items in your bucket that are affected by the S3 Lifecycle Policy rule. The S3 Lifecycle Policy setups below demonstrate how to define a filter.

• The filter defines a key prefix (tax/) in this S3 Lifecycle Policy configuration rule. As a result, the rule applies to items beginning with the prefix tax/, such as tax/doc1.txt and tax/doc2.txt. The rule defines two actions that instruct Amazon S3 to perform the following:
  • After 365 days (one year), move things to the S3 Glacier Flexible Retrieval storage class.
  • Delete items 3,650 days (10 years) after they were created (the Expiration operation).

Example 2: Turning off a AWS S3 Lifecycle Policy Rule A S3 Lifecycle rule can be temporarily disabled. The S3 Lifecycle Policy configuration below defines two rules:

• Rule 1 instructs Amazon S3 to move items with the logs/ prefix to the S3 Glacier Flexible Retrieval storage class as soon as possible after they are created.
• Rule 2 instructs Amazon S3 to move items with the documents/ prefix to the S3 Glacier Flexible Retrieval storage class as soon as possible after they are created.

Rule 1 is activated in the policy, whereas Rule 2 is disabled. Amazon S3 disregards rules that have been deactivated.

Example 3: Tiering Down Storage Classes Over the Lifecycle of an Object

You utilize S3 Lifecycle Policy settings in this example to tier down the storage type of objects over their lifetime. Tiering down might help you save money on storage. The S3 Lifecycle Policy setting that follows defines a rule that applies to objects with the key name prefix logs/. The rule specifies the following acts:

• There are two transition actions:

  • 30 days after creation, move things to the S3 Standard-IA storage class.
  • 90 days after creation, move things to the S3 Glacier Flexible Retrieval storage class.

• One expiry action instructs Amazon S3 to destroy things a year after they are created.

Example 4: Specifying Numerous Rules If you want various S3 Lifecycle Policy actions for distinct items, you may define several rules. The S3 Lifecycle Policy configuration below contains two rules:

• Rule 1 is applicable to objects beginning with the key name prefix classA/. It instructs Amazon S3 to move things to the S3 Glacier Flexible Retrieval storage class one year after they are created and to delete them ten years later.
• The second rule applies to items with the key name prefix classB/. It instructs Amazon S3 to move things to the S3 Standard-IA storage class 90 days after they are created and remove them one year later.

Example 5: Conflicting AWS S3 Lifecycle Policy operations, overlapping filters, and what Amazon S3 does with non-versioned buckets You might create an S3 Lifecycle Policy configuration with overlapping prefixes or operations.

• S3 Lifecycle, in general, optimizes for cost. If two expiry policies overlap, for example, the shorter expiration policy is enforced so that data is not retained for longer than planned. Similarly, if two transition policies overlap, S3 Lifecycle Policy moves your items to the cheaper storage class.
• In all circumstances, S3 Lifecycle Policy attempts to select the least costly option for you. The S3 Intelligent-Tiering storage type is an exception to this general norm. Except for the S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive storage classes, S3 Lifecycle Policy prefers S3 Intelligent-Tiering above any other storage class.

Example 6: Defining a AWS S3 Lifecycle Policy rule for a bucket that supports versioning.

• Assume you have a versioning-enabled bucket, which implies you have a current version and zero or more noncurrent versions for each item. (See Using versioning in S3 buckets for additional information on S3 Versioning.) In this case, you want to save a year's worth of history while deleting noncurrent versions. S3 Lifecycle Policy options allow for the storage of up to 100 versions of any item.
• You wish to shift noncurrent versions to S3 Glacier Flexible Retrieval 30 days after they become noncurrent to decrease storage costs (assuming these noncurrent objects are cold data that doesn't require real-time access). Furthermore, you anticipate that the frequency of access to the present versions will decrease 90 days after their formation, so you may decide to relocate these.

Example 7: Delete Expired Object Markers Each object in a versioning-enabled bucket has one current version and zero or more noncurrent versions. When you remove an item, keep the following in mind:

• When you remove an item without specifying a version ID, Amazon S3 inserts a delete marker instead of removing it. The current object version is no longer current, and the delete marker is the current version.
• If you include a version ID in your delete request, Amazon S3 permanently deletes the object version (a delete marker is not created).
• An expired object delete marker is a delete marker that has zero noncurrent versions.

Example 8: Configuring the AWS S3 Lifecycle Policy to stop multipart uploads

• To upload huge items in sections, utilize the Amazon S3 multipart upload REST API methods.
• You can instruct Amazon S3 to cease unfinished multipart uploads (identified by the key name prefix supplied in the rule) if they are not completed within a defined number of days of beginning using S3 Lifecycle Policy settings.
• When Amazon S3 cancels a multipart upload, it deletes all of the portions linked with the cancelation. This technique helps you limit your storage expenses by preventing incomplete multipart uploads with pieces saved in Amazon S3.

Example 9: Size-based criteria for AWS S3 Lifecycle Policy configuration

• You may write rules to transition items only depending on their size. You can give a minimum (ObjectSizeGreaterThan) or maximum (ObjectSizeLessThan) size, or a range of object sizes in bytes.
• When employing several filters, such as a prefix and a size rule, the filters must be wrapped in an And> element.

For more information, you can click on the link

Implementation of AWS AWS S3 Lifecycle Policy

Follow the steps below to establish S3 life cycle management:

Step 1: Log in to AWS and navigate to services, then S3.

Step 2: Make a bucket since it is applied to the bucket rather than a single item or complete storage.

• Give the bucket a name (try to give it a unique name or it will fail later), and uncheck the Block all public access option.
• Then select the "I acknowledge..." checkbox and press CREATE BUCKET.

Step 3: Transfer data to the bucket.

• Edit the following permissions, then click next and upload.

Step 4: Return to your bucket and select "Management."

• Select "Get Started."
• Give the life cycle a name (it does not have to be unique) and edit the parameters.
• Include transitions (i.e., transferring data from standard storage to ones that cost less after a span of when data is no more useful. )
• Set the expiration date (the number of days after which data must be deleted from storage) and Save.

Step 5: Finally, an S3 bucket with LIFE CYCLE MANAGEMENT is created, which will apply to all future data uploads.

How to Set Up an AWS S3 Lifecycle Policy to Delete Objects?

• S3 Objection expiry of AWS S3 Lifecycle Policy settings may be established with a variety of tools, including the AWS CLI tool, the AWS SDK, the Amazon S3 interface, and RESTful API calls.
• The screenshot below demonstrates how to use the Amazon S3 console UI (accessible through the "Management" tab inside the S3 bucket) to define an AWS S3 Lifecycle Policy rule to expire the current version of S3 objects.
• For the sake of this example, expiration has been specified to occur 90 days after the object was created for the whole bucket.
• The same AWS dashboard can be used to simply establish the AWS S3 Lifecycle Policy rule for deleting older versions of S3 items, as seen below.