AWS EMR

What is Amazon EMR, and What is Its Purpose?

Amazon EMR (also known as Amazon Elastic MapReduce) is a managed cluster platform that enables big data frameworks such as Apache Hadoop and Apache Spark to process and analyze huge amounts of data on AWS. Users can process data for analytics and business intelligence tasks using these frameworks and related open-source projects. Additionally, Amazon EMR enables you to transform and transmit huge massive volumes of data in and out of AWS data storage and databases, including Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.

Features of AWS EMR

• Simple to use

  Building and running large data environments and apps are simpler with Amazon EMR. Easy provisioning, controlled scaling, cluster reconfiguration, and EMR Studio for collaborative development are other EMR features.

• Elasticity

  Users can quickly add (or) remove capacity, either automatically (or) manually, according to their needs. Two primary ways to increase or decrease capacity are to deploy many clusters (or) Resize a running cluster.

• Low price

  The purpose of Amazon EMR is to lower the cost of processing massive volumes of data. Low per-second price, Amazon EC2 Spot integration, Reserved Instance integration, and Amazon S3 integration are a few of the elements contributing to its low cost.

• Flexible Data Stores

  Amazon EMR supports a variety of data storage, including Amazon S3, Hadoop Distributed File System (HDFS), and Amazon DynamoDB.

• Big Data Applications

  Apache Spark, Apache Hive, Presto, and Apache HBase are among the sophisticated and well-tested Hadoop technologies supported by Amazon EMR. Data scientists utilize EMR to run deep learning and machine learning technologies like TensorFlow and Apache MXNet using bootstrap operations.

• Controlling Data Access

  While calling other AWS services, Amazon EMR application processes use the EC2 instance profile by default. Amazon EMR provides three ways for managing user access to Amazon S3 data in multi-tenant clusters. These include integration with AWS Lake Formation, Native integration with Apache Ranger, and Amazon EMR User Role Mapper.

• EMR Studio

  It is an integrated development environment(IDE) that allows you to create, display, and debug data engineering and data science applications in R, Python, Scala, and PySpark. EMR Studio includes fully managed Jupyter Notebooks and debugging tools such as Spark UI and YARN Timeline Services.

Components of AWS EMR

The cluster is the primary component of AWS EMR. A cluster is an Amazon Elastic Compute Cloud (Amazon EC2) instance collection.

Each instance in the cluster is known as a node. The function of each node inside the cluster is referred to as the node type.

Amazon EMR has the following node types:

1. Master Node: It runs the cluster by executing software components that coordinate data distribution and tasks to other nodes for processing. The master node keeps track of tasks and oversees the cluster's health. Every cluster has a master node, and a single-node cluster can be formed using only the master node.

2. Core node: A node contains software components that conduct jobs and store data in your cluster's Hadoop Distributed File System (HDFS). A minimum of one core node exists in multi-node clusters.

3. Task node: A node that only performs tasks and does not store data in HDFS. Task nodes are not necessary.

How Does AWS EMR Work/ AWS EMR Architecture?

The Amazon EMR service architecture is made up of various levels, each of which offers a cluster with different capabilities and functions. Mainly four layers are:

Storage:

The storage layer contains the many file systems utilized by the cluster. Different types of storage options available are:

• Hadoop Distributed File System (HDFS) It is a scalable, distributed file system for Hadoop. HDFS distributes data across cluster instances, keeping several copies of data on separate instances to guarantee no data loss if a single instance fails.

• EMR File System (EMRFS) Amazon EMR enhances Hadoop by using the EMR File System (EMRFS) to offer the ability to access data stored in Amazon S3 directly. Your cluster's file system can be either HDFS or AWS S3.

• Local file system A locally attached disk is referred to as the local file system. In the Hadoop cluster, each node is built from an Amazon EC2 instance that includes a predefined block of disk storage known as an instance store.

Management of Cluster Resources

This layer manages cluster resources and schedules the jobs for data processing.

Amazon EMR, by default, employs YARN (Yet Another Resource Negotiator), a component introduced in Apache Hadoop 2.0 that allows for the central management of cluster resources for different data-processing frameworks.

However, other frameworks and applications available in Amazon EMR do not employ YARN as resource management. Amazon EMR also includes an agent on each node that manages YARN components, monitors cluster health, and communicates with Amazon EMR.

Data processing frameworks

It is mainly used to handle and analyze data. Many frameworks are available that use YARN or have resource management.

Several frameworks are available for processing demands, such as batch, interactive, in-memory, streaming, etc. This influences the languages and interfaces accessible from the application layer. Hadoop MapReduce and Spark are the primary processing frameworks available for AWS EMR.

Programs and Applications

Many applications Hive, Pig, and Spark Streaming libraries are supported by Amazon EMR to provide capabilities such as processing workloads, leveraging machine learning algorithms, creating stream processing applications, and building data warehouses. To communicate with the applications in Amazon EMR, you use a variety of libraries and languages. For example, you may utilize MapReduce with Java, Hive, Pig, Spark Streaming, Spark SQL, MLlib, and GraphX.

What are the Benefits of Amazon Elastic MapReduce?

• Cost savings

  The instance type determines Amazon EMR cost and quantity of Amazon EC2 instances deployed and the region in which your cluster is launched. On-demand pricing is inexpensive, but you can save even more money by purchasing Reserved Instances or Spot Instances.

• AWS integration

  Amazon EMR works with other AWS services to offer your cluster networking, storage, security, and other functionalities. Several examples of this integration are provided in the following list:

  • Amazon EC2 for the instances that compose the cluster's nodes.
  • Amazon EC2 for the instances that compose the cluster's nodes.
  • Amazon S3 for storing input and output data.
  • Amazon CloudWatch for monitoring cluster performance and configuring alerts.
  • Amazon S3 for storing input and output data.
  • AWS Identity and Access Management (IAM) for setting permissions.

• Deployment

  AWS EMR cluster comprises EC2 instances that handle the tasks you assign to it. AWS EMR configures the instances with the programs you specify when you begin your clusters, such as Apache Hadoop or Spark . Select the instance size and type (batch processing, low-latency queries, streaming data, or big data storage) that best meet the processing requirements of your cluster.

• Scalability and flexibility

  Amazon EMR allows you to scale your cluster up or down as your computing requirements vary. When peak workloads drop, you may adjust your cluster to add instances and delete instances to reduce expenses.

  Amazon EMR allows flexibility to use several file systems for input, output, and intermediate data. For example, you may use the Hadoop Distributed File System (HDFS), which operates on your cluster's master and core nodes, to process data that will not be stored beyond the lifespan of your cluster.

• Reliability

  Amazon EMR monitors cluster nodes and automatically terminates and replaces instances in the event of a failure. Amazon EMR gives configuration choices for determining if your cluster should be terminated automatically or manually.

• Security

  Amazon EMR works with other AWS services like IAM and Amazon VPC and features like Amazon EC2 key pairs to help you protect your clusters and data.

• Monitoring

  You can use the Amazon EMR administration interfaces and log files to debug cluster issues such as failures or errors. Amazon EMR allows you to archive log files on Amazon S3, allowing you to store logs and address issues even after your cluster has terminated.

• Management interfaces

  There are many ways a user can interact with Amazon EMR:

  • Console
  • AWS Command Line Interface (AWS CLI)
  • Software Development Kit (SDK)
  • Web Service API

Difference between AWS EMR and EC2