Building Resilient .NET Microservices on Azure

Microservices are an appealing method for developing applications today. Developers separate a single large application into smaller services. Each smaller service encapsulates specific functionality and can be developed, deployed, and scaled independently.

Microservices trade the monolithic application flexibility for new problems. Now, multiple services must be reached to fulfil a single user request. Should any of the services be unavailable, the responsiveness of the entire application will suffer.

An example of this occurred during a recent project I worked on. One of the services relied a lot on third-party APIs and message processing. During high-volume usage, temporary service disruptions on the network caused multiple message retries, slow network responsiveness, and eventually caused problems for the remaining services. Such issues are endemic to distributed systems and are the reason why such systems must be designed with resilience.

When dealing with such problems, the Circuit Breaker is one of the most successful patterns and is the main focus of this article. The article covers how Circuit Breaker can be implemented in .NET using the Polly library. It also focuses on how Azure Service Bus can be used to build a reliable microservices communication layer.

Understanding Failure in Microservices

Revisiting a previously examined concept will help with understanding monolithic and microservices.

Let’s simplify a microservices structure as an e-commerce system composed of:

• Order Service
• Payment Service
• Inventory Service
• Notification Service

When the Payment Service goes down, it can create quite a problem.

If a customer places an order, the Order Service follows, and the system automatically retries the Payment Service. After so many failed attempts, the system resources will succumb to failed connections even more. The system will think it is too busy and be in a failed state, but in fact, it is resources going to wasted connection attempts.

This is called a “cascading failure”. There are many things that may create cascading failures. Some are:

• Temporary network interruptions
• Database connectivity issues
• Service timeouts
• API rate limiting
• Cloud infrastructure outages

It is certainly more intelligent to have a smarter strategy to deal with manual connection attempts rather than a broken system that attempts to connect automatically.

What is the Circuit Breaker Pattern?

The Circuit Breaker Pattern is based on the example of a circuit breaker used in electrical systems in a home.

When a fault in the system occurs, the circuit breaker trips and cuts the flow of current in the electrical system, preventing further damage.

In the example of a software system, a Circuit Breaker can temporarily prevent the software client from making additional requests to an already failing service. The client, instead of making requests to the service that will most likely continue to fail, will allow the failing service to recover.

The Circuit Breaker operates in three states:

Closed State

Everything is functioning normally.

Requests can be sent to the service.

Open State

The Circuit Breaker has reached a threshold of failures.

The circuit opens and proactively denies any further requests sent to the service.

Half-Open State

After a defined period of time, the service has a limited number of requests sent to it.

The Closed State is reached if the sent requests to the service are successful.

The Open State is returned if the sent requests to the service are not successful.

This pattern is, therefore, protective to both the client and the service that is failing.

Why Polly?

Polly is an extremely popular resilience framework in the .NET space.

Polly supports the built-in functionality of the following features:

• Retry
• Circuit Breaker
• Timeout
• Fallback
• Bulkhead Isolation

Being able to rely on Polly to handle resilience means developers do not have to write custom resilience logic. Rather, developers have the option to configure a policy that is simple and easy to test.

Payment Service Minimal API

Payment Service has one successful payment endpoint and one gateway failure simulation endpoint.

Setting Up Polly in a .NET Application

Install the packages used by the OrderService demo:

dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly
dotnet add package Azure.Messaging.ServiceBus

Configure the circuit breaker and HttpClientFactory in OrderService/Program.cs:

Here is a sample output:

• The circuit will open when there are 3 failed attempts in a row.
• For 30 seconds, all new requests will be denied.
• Then Polly will enter the Half-Open state.

The Http Client Circuit Breaker Demo

Within the example, the OrderService accesses the PaymentService through the specified HttpClient. The OrderService has two available endpoints: the healthy /order endpoint and the failure demonstration /order-fail.

Instead of making unhealthy service calls repeatedly, Polly simply stops the requests once the failure threshold has been hit.

Where Azure Service Bus Fits In

Service-to-service communication isn’t always the optimal choice.

Take, for example, the Notification Service going temporarily offline.

Would the Order Service be expected to fail, therefore, because the notifications stop sending?

Not ideally.

This is where Azure Service Bus is handy.

With Azure Service Bus, services can communicate through queues and topics, but do so asynchronously. There won’t be an immediate call to the Notification Service.

Instead, what would actually happen is:

1. The Order Service would send out a message.
2. The Azure Service Bus would hold onto the message.
3. The Notification Service would call and pick up that message when it is ready.

This model inherently lowers the level of service dependencies and ultimately increases reliability.

Sending Messages to Azure Service Bus

The Service Bus package is already installed in OrderService:

dotnet add package Azure.Messaging.ServiceBus

The /send-order endpoint adds a message to the orders queue with a timestamp:

Messages remain in the queues until received, even if the receiving service is temporarily unavailable.

Combining Polly and Azure Service Bus

The true power of Polly and Azure Service Bus is shown in the following example:

Consider this workflow:

1. The customer places an order.
2. The Order Service calls the Payment Service.
3. The call to the Payment Service is protected by Polly.
4. The order is successfully sent to Azure Service Bus.
5. Both the Inventory and Notification services read messages in parallel.

This leads to:

• Low service coupling
• High fault tolerance
• High scalability
• Prevention of cascading failures
• Improved user experience

Real Life Example

A retail service uses a third-party payment service, which is frequently unavailable.

No resilience strategies in place lead to:

• Many services were requesting the payment service needlessly.
• The payment services’ response time increased.
• The service’s thread pool was completely utilised.
• Other services became sluggish.

With Polly Circuit Breakers in place:

• Faulty services were quickly isolated.
• Useless requests were stopped.
• Other services had the resources to function normally.

With the addition of the Azure Service Bus:

• Orders were safely placed in the queue.
• Services downstream were unaffected.
• Temporary outages of the payment service did not affect the entire system.

The system became more predictable and less volatile.

Additional Best Practices

When we use Circuit Breakers in production, we should think about these things:

Monitor the state of the Circuit Breakers

We should write down when a Circuit Breaker opens or closes.

This helps us find problems that keep happening before they become issues.

We can use this information to fix things before they get worse.

Use Retry and Circuit Breaker together carefully

Retry is helpful when things fail for a time.

If we retry too many times, it can make things worse.

A common way to do this is:

Retry -> Circuit Breaker -> Timeout

Check the health of our services

We should make it easy for our services to tell us if they are working or not.

This makes it easy to check if our services are available using Azure tools.

 Set limits based on traffic

We should not just pick numbers out of thin air.

We should look at how our servicers are used and set the limits for when they fail based on that.

This way, the Circuit Breakers will work better with our services. Help us find problems.

Conclusion

Building microservices is not that hard. Building microservices that keep working when something goes wrong is a different story.
The Circuit Breaker pattern is a way to stop services that are failing from affecting the rest of the system. With Polly, it is easy to use this pattern in.NET. It is also easy to maintain.

When you use the Circuit Breaker pattern with Azure Service Bus, you can build systems that can handle problems and get bigger at the time. By letting problems spread across the entire system, microservices can find problems and fix them without affecting everything else.

As applications get bigger and have dependencies, microservices should be able to handle problems from the start. It is not something you can add later; it should be a part of the architecture, from the beginning.

FAQs