Introduction to Azure Durable Functions

Prerequisites

Before diving into Azure Durable Functions, it's essential to have a solid understanding of the following concepts and technologies:

• Azure Functions:
  Familiarize yourself with the basics of Azure Functions, which are the serverless compute services that Azure Durable Functions build upon.
• Serverless Computing:
  Understand the fundamental principles of serverless computing, including its benefits, event-driven nature, and cost model.
• Programming Language:
  Choose a programming language you are comfortable with for writing Azure Functions. Common choices include C#, JavaScript/TypeScript, Python, and Java.
• Networking Basics:
  Networking fundamentals are essential when working with cloud services and distributed systems, including Durable Functions.
• Version Control:
  Proficiency in version control systems like Git is vital for collaborative development and managing codebase changes.
• Distributed Systems:
  Durable Functions are part of a distributed system, and understanding distributed systems principles is valuable.

About Azure Functions

Azure Functions is a serverless computing service that allows you to build and deploy event-driven, serverless applications without managing infrastructure. It supports various programming languages, uses event triggers to execute functions, and provides seamless integration with Azure services through bindings. Functions are stateless by default but can be made stateful using Durable Functions, and the service automatically scales to handle incoming events.

Azure Function Name: ImageThumbnailGenerator

Function Description:

This Azure Function processes images uploaded to a cloud storage container and generates thumbnails for each image. It's triggered by new images being added to the storage container.

How It Works?

• Trigger:
  The Azure Function is triggered by a storage event. When an image is uploaded to a specified container in Azure Blob Storage, it automatically invokes the ImageThumbnailGenerator function.
• Processing:
  The function retrieves the newly uploaded image from the storage container.
• Thumbnail Generation:
  It then processes the image, creating a smaller thumbnail version using an image processing library or tool.
• Storage:
  The generated thumbnail is saved back to the same or another container in Azure Blob Storage.

Azure Function Name: SendEmailOnNewContact

Function Description:

This Azure Function sends an email notification whenever a new contact is added to a customer relationship management (CRM) system.

How It Works?

• Trigger:
  The Azure Function is triggered by an event in the CRM system, such as a new contact record being added to a database or an API request.
• Data Retrieval:
  The function retrieves information about the new contact from the CRM system, such as the contact's name, email address, and any relevant details.
• Email Composition:
  It composes an email message with a customized greeting and information about the new contact.
• Email Sending:
  The function uses an email service like SendGrid or SMTP to send the email notification to a specified recipient or group of recipients.

These two examples demonstrate how Azure Functions can be used to respond to events, process data, and trigger specific actions. Azure Functions are highly versatile and can be employed in various scenarios, from data processing and integration to automation and real-time event handling.

About Serverless

Serverless computing, often referred to as Function as a Service (FaaS), is a cloud computing model that allows developers to run code in response to events without the need to manage or provision servers. In a serverless architecture, functions or microservices are executed on-demand, automatically scaling to handle workloads, and users are billed based on actual execution time. This approach enables cost-effective, event-driven, and highly scalable application development while abstracting infrastructure management tasks.

Why Durable Functions?

Durable Functions in Azure simplify the management of complex workflows by allowing developers to define them in code, making it easy to orchestrate the sequence and order of execution for multiple functions, including handling parallel execution and errors throughout the workflow. This simplification and centralized control come from the Orchestrator Function, enhancing workflow visibility and maintainability.

Durable Functions offer several advantages that make them a compelling choice for building serverless applications and orchestrating workflows. Here are some key reasons why you might choose Durable Functions:

• Error Handling and Retry Logic:
  Durable Functions offer built-in error handling and retry mechanisms. Orchestrator functions can gracefully handle exceptions and implement custom retry policies, making your workflows more robust and reliable.
• Timers and Scheduling:
  You can use durable timers to introduce time-based delays and schedule future actions within your workflows. This is useful for scenarios like sending reminders, expiring tokens, or managing time-sensitive tasks.
• Cross-Language Support:
  While Durable Functions are initially developed for .NET, they offer support for multiple languages. You can implement activity functions in the language of your choice, allowing flexibility and accommodating developers with different language preferences.
• Scalability:
  Durable Functions can seamlessly scale to handle increased workloads. As the number of incoming requests or events grows, Azure Functions automatically provision additional resources to keep your workflows responsive.

Benefits of Durable Functions

Azure Durable Functions offer several benefits that make them a valuable tool for building scalable and efficient serverless workflows and applications. Some of the key benefits of Durable Functions include:

• Orchestration:
  Coordinate Azure Functions for structured long-running workflows, simplifying complex processes with orderly execution.
• Human Interaction:
  Functions can pause and wait for user input or external events without incurring additional costs during waiting periods.
• Error Handling:
  Built-in error handling and automatic retries simplify the recovery from errors in workflows.
• Monitoring: Real-time tracking and logging provide insight into the progress and status of workflows for effective troubleshooting.
• Rapid Development:
  Declarative orchestration simplifies complex workflow creation, reducing development time and effort.
• Auto-scalability:
  Automatically scale to handle increased workloads efficiently, distributing tasks across multiple instances for optimal resource utilization.
• Integration:
  Seamlessly integrate with other Azure services like Logic Apps, Event Grid, and Service Bus for comprehensive, event-driven workflows.

Concepts of Durable Function

In Durable Functions, key concepts revolve around Orchestrator Functions and Activity Functions. Here's a brief overview:

Orchestrator Functions:

Orchestrator Functions are a fundamental component of Durable Functions. Their sole purpose is to define workflows. They don't perform any actions like making API calls or interacting with databases. Instead, they delegate the actual steps in the workflow to Activity Functions. When a workflow is initiated, the Orchestrator Function is triggered, and it starts the first activity. Then, it goes to sleep. Once the first activity is complete, the Orchestrator Function wakes up and continues the workflow, calling the next activity. Orchestrator Functions use an OrchestrationClient binding to start or interact with orchestrations.

Example:
Imagine you're building an e-commerce platform. Your orchestrator function could be responsible for managing the order fulfillment process. It would start by checking the inventory, then contacting the shipping service, and finally sending an order confirmation to the customer. If any of these steps fail, the orchestrator can retry or handle errors as needed.

Activity Functions:

Activity Functions are regular Azure Functions that actively participate in a workflow. They can receive input data from the Orchestrator Function and return data to it. These functions are responsible for executing specific tasks or actions within the workflow. Activities can be called sequentially or in parallel, depending on the workflow requirements.

Example:
In the e-commerce scenario, activity functions could include checking inventory levels, contacting the shipping service to schedule a delivery, and sending order confirmations. Each of these operations can be implemented as individual activity functions, making the workflow modular and easier to manage.

Azure Durable Functions are built on several other key concepts that enable you to create stateful and long-running workflows within the Azure serverless environment. Here are the fundamental concepts of Durable Functions:

• State Management:
  Durable Functions automatically manage the state of orchestrator functions and activity functions, including checkpoints, retries, and durable timers. This built-in state management simplifies the development of long-running processes and reduces the complexity of handling the state externally.
• Fan-Out/Fan-In:
  Durable Functions support the fan-out/fan-in pattern, allowing orchestrator functions to create multiple parallel instances of activity functions and wait for all instances to complete before proceeding. This pattern is useful for tasks like parallel data processing.
• Chaining:
  You can chain orchestrator functions together to create sequential workflows. One orchestrator function can trigger another when it completes, enabling the creation of complex multi-step processes.
• Human Interaction:
  Durable Functions support human interaction through external events. Orchestrator functions can pause and wait for external input or approval before continuing the workflow.
• Durable Timers:
  You can use durable timers to introduce time-based delays and timeouts within the workflow, allowing you to build processes that involve time-dependent actions.
• Error Handling:
  Durable Functions provide built-in error-handling capabilities, including retries, timeouts, and the ability to catch and handle exceptions at different levels of the workflow.
• Idempotency:
  Activity functions are designed to be idempotent, meaning they can be safely executed multiple times without causing unexpected side effects. This is essential for ensuring reliability in long-running workflows.
• Concurrency Control:
  Durable Functions can manage the concurrency of orchestrator and activity functions to prevent overloading resources, allowing you to control the number of parallel executions.
• Event-Driven:
  Durable Functions can respond to external events such as Azure Queue messages, HTTP requests, or other triggers. Orchestrator functions can be started in response to these events, making Durable Functions suitable for event-driven scenarios.

Patterns of Durable Function

Durable Functions offer various patterns to address different workflow scenarios. Here are the key patterns:

Function Chaining

In the function chaining pattern, multiple functions are executed sequentially, with each function triggered by the completion of the previous one. This is useful for creating a linear workflow, where the output of one function serves as the input for the next. It simplifies complex, multi-step processes.

• Sequential Execution:
  Function chaining allows you to execute a sequence of functions in a defined order, ensuring that one function's output becomes the input for the next. This simplifies complex workflows by breaking them into smaller, manageable steps.
• State Preservation:
  Azure Durable Functions automatically preserve the state of your workflow between function calls. If a function execution is interrupted, it can continue from where it left off, maintaining the context.
• Common Use Cases:
  Function chaining is suitable for use cases such as data transformation, data validation, notification generation, and multi-step business processes.

Fan-Out/ Fan-In

The fan-out/fan-in pattern is employed when a single orchestration function needs to trigger multiple activity functions in parallel. These activity functions run concurrently and independently, typically processing data from a collection or distributing tasks. Once all activity functions have been completed, the orchestration function collects and combines their results in a fan-in fashion.

• Parallel Processing:
  The fan-out/fan-in pattern is ideal for parallel processing tasks. It enables you to initiate multiple instances of a function concurrently. This is particularly valuable for tasks that can be distributed across multiple resources, enhancing performance and throughput.
• Aggregation:
  After parallel execution, you can aggregate the results from multiple function instances into a single result set or make further decisions based on the aggregated data.
• Use Cases:
  Examples include data partitioning, batch processing, and scenarios where multiple sources of input data need to be processed in parallel.

Async HTTP APIs

Durable Functions can be used to orchestrate asynchronous HTTP APIs. This pattern is valuable when waiting for responses from external services that might take an unpredictable amount of time. The orchestration function can call an HTTP API, go to sleep, and then be awakened upon receiving the response.

• Non-Blocking Execution:
  Durable Functions support asynchronous HTTP APIs, allowing clients to initiate long-running operations without blocking the client. The client initiates the operation, receives a status URL, and can poll or wait for completion.
• Long-Running Tasks:
  This pattern is useful for scenarios where tasks may take a significant amount of time to complete, such as data import, complex calculations, or generating reports.
• Client Notifications:
  Clients can be notified upon task completion or retrieve the final results when ready, providing a responsive and user-friendly experience.

Monitoring

The monitoring pattern is applied for tracking and managing long-running workflows. Orchestrator functions can periodically check the status of running activities or child orchestrations. It's useful for implementing timeout handling, status updates, or automated retries in workflows.

• Telemetry and Observability:
  Azure Durable Functions can be integrated with Azure Application Insights, enabling you to collect telemetry data and monitor the execution of orchestrator and activity functions.
• Logging and Debugging:
  Logging is crucial for troubleshooting and debugging workflows. You can log information and track the progress of your functions.
• Performance Optimization:
  Monitoring helps identify performance bottlenecks, inefficient processes, and areas for optimization, ensuring your workflows run efficiently.

Human Interaction

Durable Functions can also support human interaction in workflows. For example, they can trigger an approval process or send notifications to users. Orchestrator functions can wait for a human's response before proceeding, making it possible to include user decision points in automated processes.

• Pause and Resume:
  Durable orchestrators can pause and wait for external input, such as human approvals or reviews. When human interaction is required, the workflow pauses until the necessary input is provided.
• Approval Workflows:
  This pattern is commonly used in approval workflows, where a task requires validation or approval from a human stakeholder before the workflow can proceed.
• Task Assignment:
  It can also be used for task assignment and delegation, ensuring the right individuals or groups are involved in the workflow process.

What is Event Sourcing?

Event sourcing is a data storage and management approach where, instead of storing the current state of an application, you store a sequence of immutable events that led to that state. It keeps a historical record of all actions that have occurred to reach the present state. This history is stored in a structured and ordered manner, making it easy to understand how the application's state evolved.

What are Task Hubs?

Durable Functions use the concept of "task hubs" to refer to the storage associated with a specific set of orchestrations and activities. These task hubs are essentially isolated containers for storing the state of workflows. You can have multiple task hubs associated with different sets of functions.

Single Task Hub:
In most scenarios, having a single task hub in a storage account is sufficient for managing the state of Durable Functions. This simplifies the setup and management of the system. However, it is also possible to have multiple task hubs within the same storage account if needed, enabling more granular control over different sets of functions.

Ways to Develop Durable Functions

When developing Durable Functions, there are three main ways to get started:

• Azure Portal:
  You can create functions directly in the Azure portal. This approach is useful for trying out Azure Functions without the need to install any development tools on your local machine. It's excellent for experimentation and learning about Azure Functions, but it's not typically recommended for building production-ready code.
• Command Line:
  You can develop Durable Functions from the command line on any platform. Using your favorite text editor, such as Visual Studio Code, in combination with the cross-platform Azure Functions command line interface, you can build functions locally. You can find the necessary tooling for this approach at the following link.
• Visual Studio:
  Visual Studio 2017 provides an Azure Functions extension that simplifies the development process. It offers project and function templates and provides features like IntelliSense and debugging support. This is a robust and powerful development environment for building Durable Functions.

Set Up Durable Functions in VS2022

To set up Durable Functions in Visual Studio 2022, follow these steps:

• Install Visual Studio:
  If you don't already have Visual Studio 2022 installed, you can download the free Visual Studio Community edition from https://visualstudio.microsoft.com/vs/community/.

  

• Select Azure Development Workload:
  During the installation process, make sure to select the "Azure development" workload. This ensures that you have the necessary tools and extensions for Azure development.

  

• Add Azure Development Workload:
  If you've already installed Visual Studio without the Azure development workload, don't worry. You can modify your installation later. Open the Visual Studio Installer, select "Modify", and then add the "Azure development" workload.

  

• Install Azure Development Extensions:
  With the Azure development workload, you'll receive several Azure development extensions and the Azure SDK. Two essential components you'll get are the following:

  • Azure Functions and WebJob Tools Visual Studio Extension:
    This extension allows you to create new Azure Functions apps from the "File-New Project" menu.

    

• Azure Storage Emulator:
  This is necessary for testing Durable Functions locally. Durable Functions use Azure storage tables and queues to implement task hubs for storing orchestration state and control messages. The Storage Emulator enables you to run and test Durable Functions against an emulated storage account.

• Creating a New Azure Function App:
  With Visual Studio set up and the necessary extensions in place, you can create a new Azure Function app. This provides the foundation for developing Durable Functions in your preferred development environment.

Creating a First Project

In this demo, you're using Visual Studio 2022 to create a new Azure Function app and enable Durable Functions by adding the Durable Functions NuGet package to the project. Here's an overview of the steps:

• Setting Up Visual Studio:
  Ensure you have Visual Studio 2022 installed with the Azure development workload. This workload is necessary for Azure Functions development.

• Creating a New Azure Function App:
  Go to "File" and select "New Project". Choose the "Azure Functions" template. Provide a name for your project and proceed. Visual Studio will prompt you to choose the version of the Azure Functions Runtime you want to use.

  

• Configuring the Project:
  You can choose to create an empty function app or one with a starter function. In this demo, select the option with an HTTP function, which will be useful for testing.

  

• Setting Up Local Development:
  Visual Studio allows you to configure settings for your function app. This includes specifying the storage account to use. For local development and testing, it's set to the Storage Emulator. Keep in mind that in a production environment, you'll configure it to use a real storage account.

  

• Project Structure:
  Once the project setup is complete, you'll see a basic project structure. It includes a local.settings.json file with connection strings pointing to the Storage Emulator for local development. Notably, these connection strings are not stored in source control. When deploying your function app to Azure, the connection strings will be obtained from your function app's app settings. You'll also find an example function named Function1.cs, which responds with a hello message when invoked.

How to Enable Durable Functions?

To enable Durable Functions in your project, follow these steps:

• Open Your Project in Visual Studio:
  Ensure that you have your Azure Function app project open in Visual Studio.

  

• Access NuGet Package Manager:
  In the Solution Explorer, locate the "Dependencies" node. Right-click on it, and then select "Manage NuGet Packages" from the context menu.

  

• Search for Durable Functions Package:
  In the NuGet Package Manager, enable the "Include prerelease" option, as Durable Functions may have newer releases. Then, use the search box to look for "Durable Functions".

  

• Select the Durable Functions Package:
  You will see the "Microsoft Azure WebJobs Extensions Durable Task" NuGet package listed. This package is what you need to enable Durable Functions. Select it for installation, and you can choose the specific version you want to install (e.g., version 1.6.2).

  

• Install the Package:
  Click the "Install" button to add the Durable Functions extension to your project.

• Check the csproj File:
  After the installation is complete, you can inspect the changes made by right-clicking on your project in the Solution Explorer. Select "Edit .csproj" to view the project file.

• Verify Package Reference:
  Inside the .csproj file, you'll notice that a new package reference to the DurableTask extension has been added. This indicates that the Durable Functions extension is now included in your project.

With Durable Functions enabled, you can start building robust and scalable serverless workflows and orchestrations within your Azure Function app, making it easier to handle complex business processes and integrate with various Azure services.

Test the Function App

To test your Azure Function App and ensure everything is working correctly, follow these steps:

• Run the Project:
  In Visual Studio, start debugging by pressing the "Start Debugging" button (usually F5). This action will launch the local Azure Functions host to run your project.

• Note the Function URI:
  After starting the debugging session, you will see the URI of your function in the debug output. For example, it might be something like http://localhost:7140/api/Function1. Make note of this URI.

  

• Test the Function:
  Open a web browser or use a tool like Postman to test the function. Simply paste the function's URI into the browser's address bar.

  

  • The desired message will be printed on the screen.

    

• Query Parameters:
  By default, your function might require specific query parameters to work correctly. In this example, the query parameter "name" is expected. Add the required parameters to the URL, like http://localhost:7140/api/Function1?name=scalerLearner.

• Execute the Function:
  Access the updated URL, and you should see the expected result. In the example provided, visiting http://localhost:7140/api/Function1?name=scalerLearner would display "Hello scalerLearner....".