Glue

What is Amazon glue?

AWS Glue is a serverless computing platform and ETL (extract, transform, and load) service that simplifies data discovery, preparation, and aggregation for data analysis, machine learning, and application development. Glue provides both visual and code-based solutions to make data integration easier.

The AWS Glue Data Catalog is an ETL engine that automatically generates Python or Scala code and a customizable scheduler that manages dependency resolutions, job monitoring, and restart of Amazon Glue.

Users may rapidly search and retrieve data using the Glue Data Catalog. The Glue service also allows the customization, coordination, and monitoring of complex data streams.

AWS Glue use cases

• Users may use AWS Glue to conduct serverless queries across the Amazon S3 data lake. Amazon Glue can assist users in getting started quickly by making all your data available for analysis at a single interface without relocating it.

• AWS Glue can help you understand your data assets. The Data Catalog makes it simple to locate various AWS data sets. You can also use this Data Catalog to store your data across many AWS services while preserving a consistent representation of your data.

• Glue is handy when creating event-driven ETL (extract, transform, and load) procedures. By calling your Glue ETL(extract, transform, and load) tasks from an AWS Lambda service, you can perform your ETL processes as soon as new data is available in Amazon S3.

• AWS Glue can organize, clean, verify, and format data before storing it in a data warehouse or data lake.

Features of AWS Glue

• Drag & Drop Interface: A drag-and-drop job editor helps to define ETL services, and AWS Glue will quickly develop the code to extract, convert, and upload the data.

• Automatic Schema Discovery: You can use the Glue service to build crawlers that connect to different data sources. It efficiently organizes the data, extracts schema-related information, and puts it in the data catalog. ETL tasks can then use this data to monitor ETL processes.

• Job Scheduling: Glue can be used regularly, on-demand, or in response to an event. Users can use the scheduler to create complex ETL pipelines by specifying task dependencies.

• Built-in Machine Learning: AWS Glue provides "FindMatches" that find and deduplicates records that are imperfect copies of one another.

• Developer Endpoints: AWS Glue provides developer endpoints to alter, debug, and test the created code.

• Glue DataBrew: It is a data preparation solution for users such as data analysts and data scientists to help them clean and normalize data utilizing the active and visible interface of Glue DataBrew.

Components of AWS Glue

Before we can comprehend the architecture of AWS Glue, we must first grasp a few components. AWS Glue relies on the various components to develop and maintain your ETL operations. The important components of AWS Glue are:

• Data Catalog: Metadata repository that offers table, job, and control information to keep your Glue environment working.

• Classifier: The schema of data stored for relational database management systems (RDBMS) and file formats such as CSV, JSON, and XML is defined by the classifier.

• Connection: It has the properties required to connect to a specific storage of data.

• Crawler: A crawler is used to inventory the data in your data stores, but it also has the ability to add metadata tables to Data Catalog.

• Database: A database is a set of interconnected Data Catalog table definitions.

• Data Storage Facility: It is a site that allows the users to save and retrieve data for an extended period of time. Examples are relational databases and AWS S3  .

• Source of Information: A collection of data that is utilized as intake to a process or transformation.

• Data Target: Storage area where the updated data is written by the job.

• Transform: Transformation refers to the reasoning in the code that is used to modify the format of your data.

• Endpoint of Development: Using the development endpoint environment, you may create and test AWS Glue ETL scripts.

• Dynamic Framework: A DynamicFrame is similar to a DataFrame, except each element is self-descriptive. As a result, a schema is not originally required. Dynamic Frame also features extensive data cleaning and ETL capabilities.

• AWS Glue Job: It is a form of business logic needed for ETL work. A transformation script, data sources, and data targets comprise a task.

• Trigger: The ETL procedure is started by the trigger. Triggers can be set to occur at a specific time or in reaction to an event.

• Notebook Server: It is a web-based environment in which you may run PySpark commands. A notebook on a development endpoint enables active ETL script creation and testing.

• Script: A script is a piece of code that retrieves data from sources, alters it, and loads it into destinations. PySpark or Scala scripts are created using AWS Glue. Amazon Glue offers notebooks, and Apache Zeppel is available on notebook servers.

• Table: A table is a metadata representation of the data in storage. A table contains information about a basic dataset such as column names, data type definitions, partition information, and other metadata.

Architecture of AWS Glue

• Decide which datasets you will utilize.

• If you use a data storage source, you must establish a crawler to feed metadata table definitions to the AWS Glue Data Catalog.

• When you target your crawler at a data store, the crawler inserts the Data Catalog with metadata.

• If you utilize streaming sources, you should create Data Catalog tables and data stream characteristics manually.

• After categorizing the Data Catalog, the data is immediately ready for ETL (extract, transform, and load) ,search, and  query.

• AWS Glue then creates a script to transform the data. You may also provide the script using console or API.

• Once the script is created, you may run it on request or plan it to run when a certain event happens.

• When the task is launched, the script will extract data from the data source, convert it, and load it into the data destination. As a consequence, the AWS Glue ETL procedure is successful.

Benefits and drawbacks of using Glue

Benefits of using AWS Glue are:

• Glue is a serverless data integration solution that eliminates infrastructure creation and management requirements.

• It provides simple tools for creating and tracking work activities triggered by schedules, events, or on-demand.

• It is a low-cost solution. Only the resources utilized during the task execution process are charged.

• Glue will develop ETL pipeline code in Scala or Python based on your data sources and destinations.

• Multiple organizations within the corporation may utilize AWS Glue to collaborate on various data integration initiatives. This decreases the time needed to analyze the data.

Drawbacks of using AWS Glue are:

• AWS Glue has integration constraints. Glue only supports ETL from JDBC (Java Database Connectivity) and S3 (CSV) data sources.

• Glue cannot assist you in loading data from other cloud services, such as File Storage Base.

• Glue is not used to regulate individual table jobs. The ETL technique is only utilized to process the entire database.

• AWS Glue only supports two programming languages: Python and Scala, for updating ETL scripts.

• AWS Glue only supports a few data sources, such as S3. As a consequence, there is no method to sync with the data source progressively. It means that real-time data for complex operations will be unavailable.

AWS Glue Pricing

The initial price for Amazon Glue is $0.44. There are four unique plans to choose from here:

1. ETL tasks and development endpoints are available for $0.44.

2. Crawlers and DataBrew interactive sessions are offered for $0.44.

3. DataBrew jobs start at $0.48 per hour.

4. The Data Catalog's monthly storage and requests cost $1.00.

• AWS does not offer a free plan for the Glue service. Each hour will cost roughly $0.44 per DPU.
• So you'd have to pay $21 every day on average. However, pricing may differ by region.