Spring Cloud Hystrix example

Goal of this example

This example shows how to implement the following microservice patterns and techniques using Hystrix:

• Circuit breaker
• Request collapser
• Fallback
• Monitoring / dashboard
• Caching

Technology Used

• Spring boot 1.3.5.RELEASE
• Eureka
• Hystrix

This article is part of a Spring Cloud / Netflix OSS series

• Hystrix tutorial
• Eureka, Ribbon, Feign tutorial
• Microservice ecosystem with Docker, Spring Cloud and Spring Boot
• Zuul tutorial

Download example project from GitHub, working code tells more than hundred blog posts.

Avoiding cascading failures with circuit breaker

Cascading failure can happen when multiple services are calling each other. For example if we have the below service topology

If service 1 fails then all other services will fail too. Additional risk is that Service 2 and 4 will keep sending requests to service 1, so it will be completely overloaded and will never be able to recover. The circuit breaker pattern addresses this problem. Just like an electrical switch, when closed electrons (requests) can flow through it, when open the flow is stopped. Hystrix can wrap methods with circuit breaker that uses the following logic :

1. By default the circuit is closed and the original method is executed.
2. The original method throws an exception, the fallback is executed.
3. Error rate hits the threshold, the circuit opens.
4. Until the circuit is open the original method is not executed anymore, only the fallback.
5. After a predefined amount of time the circuit is closed, and the flow starts from the beginning.

A typical scenario is when a REST client is wrapped with circuit breaker. The client makes a call to the backend, after subsequent exceptions the circuit opens and the fallback method is called that is serving a static content or an error message. Circuit breaker can be used in any place where there is a chance that operation will fail, but most often this is used in clients of external systems. The actual implementation is quite straightforward :

    @HystrixCommand(fallbackMethod = "fallbackGetCustomer")
    public MessageWrapper<Customer> getCustomer(int id) {

        throw new RuntimeException("Simulating downstream system failure");
    }

    public MessageWrapper<Customer> fallbackGetCustomer(int id, Throwable t) {
        return new MessageWrapper<>(null, "Fallback method handled exception for id " + id + ". The original exception was " + t.toString());
    }

Calling the above code will initially fail because of new RuntimeException(“Simulating downstream system failure”), but after a few calls it will fail because the circuit is open. By convention the fallback method has the same signature as the method it belongs to. Additionally it can take a throwable parameter so it can process the exception occurred in the original method. There are two disadvantages of this approach

• The fallbackMethodName is a string parameter, therefore it can’t be checked at compile time that the fallback is really present.
• If the signature of the original and the fallback method diverges then the fallback will not be called.

To address these issues either appropriate unit tests should be written, or the underlying non annotation based Hystrix api should be used. Circuit breaker is somewhat similar to a try-catch block, but has two significant differences

• The body of the try catch block is always executed, the circuit breaker can decide to call only the fallback method
• Circuits can be monitored

Monitoring circuits with Hystrix dashboard and turbine

The Hystrix dashboard application can display detailed information about circuits. The dashboard application is yet another spring boot application. All we need to do is to add the @EnableHystrixDashboard to our spring boot main class

@SpringBootApplication
@EnableHystrixDashboard
public class HystrixDashboardApplication {

    public static void main(String[] args) {
        SpringApplication.run(HystrixDashboardApplication.class, args);
    }
}

The dashboard will ask for the URL a Hystrix stream. Every hystrix enabled application produces a stream where the status of all circuits are constantly written. The application stream contains circuit information of the given application. The URL is

http://application-host/hystrix.stream

Monitoring an individual application in an single host is not really useful. what we really need is circuit information of all applications running in all hosts. Turbine addresses this problem by aggregating multiple application level streams into a single one. Turbine is yet another spring-boot application

@SpringBootApplication
@EnableTurbinepublic class TurbineApplication {

We will need to tell Turbine which applications should it process. The best way to do it is to look up services from Eureka. The advantage of this is that Turbine will notice if a new service instance starts up. The below example explicitly limits which services should Turbine look up from Eureka

turbine:
  clusterNameExpression: new String("default")
  appConfig: HYSTRIX-EXAMPLE-SERVICE,HYSTRIX-EXAMPLE-SERVICE2,CUSTOMER-CLIENT-SERVICE
  combineHostPort: true

Once turbine is running it will procude it’s stream at

http://turine-host/turbine.stream

The stream can be monitored from the dashboard at

http://dashboard-host/hystrix/monitor?stream=http://turbine-host/turbine.stream

And it will show data like this :

The meaning of the diagrams is explained below (source)

Request collapser

One of the disadvantages of microservice architecture is the performance penalty caused by high number of cross service HTTP calls. Advantage of batching the calls are :

• The communication overhead of multiple calls are reduced into a single call
• The backend may be able to more efficiently serve a single request with more parameters then multiple requests
• If there are multiple reuquests with the same parameter then only one of the requests have to be executed.

The implementation is quite straightforward again. First we define a method

• returning a Future
• Annotated with @HystrixController

The method body will never be actually executed, it can be left empty.

    @HystrixCollapser(scope = com.netflix.hystrix.HystrixCollapser.Scope.GLOBAL, batchMethod = "getCustomerByIds")
    public Future<MessageWrapper> getCustomerById(Integer id) {
        throw new RuntimeException("This method body should not be executed");
    }

Then we define the collapser method

• Taking a list of the same type as the original method
• Returning a list of the same type as the original method
• Annotated with HystrixCommand

    @HystrixCommand
    public List<MessageWrapper> getCustomerByIds(List<Integer> ids) {

This approach can be used not only to to collapse multiple HTTP calls, but for SQL queries or anything else.

Tags: Eureka, Hystrix, Spring boot