Goal of this example
This example shows how to implement the following microservice patterns and techniques using Hystrix:
- Circuit breaker
- Request collapser
- Fallback
- Monitoring / dashboard
- Caching
Technology Used
- Spring boot 1.3.5.RELEASE
- Eureka
- Hystrix
This article is part of a Spring Cloud / Netflix OSS series
- Hystrix tutorial
- Eureka, Ribbon, Feign tutorial
- Microservice ecosystem with Docker, Spring Cloud and Spring Boot
- Zuul tutorial
Avoiding cascading failures with circuit breaker
Cascading failure can happen when multiple services are calling each other. For example if we have the below service topology
If service 1 fails then all other services will fail too. Additional risk is that Service 2 and 4 will keep sending requests to service 1, so it will be completely overloaded and will never be able to recover. The circuit breaker pattern addresses this problem. Just like an electrical switch, when closed electrons (requests) can flow through it, when open the flow is stopped. Hystrix can wrap methods with circuit breaker that uses the following logic :
- By default the circuit is closed and the original method is executed.
- The original method throws an exception, the fallback is executed.
- Error rate hits the threshold, the circuit opens.
- Until the circuit is open the original method is not executed anymore, only the fallback.
- After a predefined amount of time the circuit is closed, and the flow starts from the beginning.
A typical scenario is when a REST client is wrapped with circuit breaker. The client makes a call to the backend, after subsequent exceptions the circuit opens and the fallback method is called that is serving a static content or an error message. Circuit breaker can be used in any place where there is a chance that operation will fail, but most often this is used in clients of external systems. The actual implementation is quite straightforward :
@HystrixCommand(fallbackMethod = "fallbackGetCustomer") public MessageWrapper<Customer> getCustomer(int id) { throw new RuntimeException("Simulating downstream system failure"); } public MessageWrapper<Customer> fallbackGetCustomer(int id, Throwable t) { return new MessageWrapper<>(null, "Fallback method handled exception for id " + id + ". The original exception was " + t.toString()); }
Calling the above code will initially fail because of new RuntimeException(“Simulating downstream system failure”), but after a few calls it will fail because the circuit is open. By convention the fallback method has the same signature as the method it belongs to. Additionally it can take a throwable parameter so it can process the exception occurred in the original method. There are two disadvantages of this approach
- The fallbackMethodName is a string parameter, therefore it can’t be checked at compile time that the fallback is really present.
- If the signature of the original and the fallback method diverges then the fallback will not be called.
To address these issues either appropriate unit tests should be written, or the underlying non annotation based Hystrix api should be used. Circuit breaker is somewhat similar to a try-catch block, but has two significant differences
- The body of the try catch block is always executed, the circuit breaker can decide to call only the fallback method
- Circuits can be monitored
Monitoring circuits with Hystrix dashboard and turbine
The Hystrix dashboard application can display detailed information about circuits. The dashboard application is yet another spring boot application. All we need to do is to add the @EnableHystrixDashboard to our spring boot main class
@SpringBootApplication @EnableHystrixDashboard public class HystrixDashboardApplication { public static void main(String[] args) { SpringApplication.run(HystrixDashboardApplication.class, args); } }
The dashboard will ask for the URL a Hystrix stream. Every hystrix enabled application produces a stream where the status of all circuits are constantly written. The application stream contains circuit information of the given application. The URL is
http://application-host/hystrix.stream
Monitoring an individual application in an single host is not really useful. what we really need is circuit information of all applications running in all hosts. Turbine addresses this problem by aggregating multiple application level streams into a single one. Turbine is yet another spring-boot application
@SpringBootApplication @EnableTurbinepublic class TurbineApplication {
We will need to tell Turbine which applications should it process. The best way to do it is to look up services from Eureka. The advantage of this is that Turbine will notice if a new service instance starts up. The below example explicitly limits which services should Turbine look up from Eureka
turbine: clusterNameExpression: new String("default") appConfig: HYSTRIX-EXAMPLE-SERVICE,HYSTRIX-EXAMPLE-SERVICE2,CUSTOMER-CLIENT-SERVICE combineHostPort: true
Once turbine is running it will procude it’s stream at
http://turine-host/turbine.stream
The stream can be monitored from the dashboard at
http://dashboard-host/hystrix/monitor?stream=http://turbine-host/turbine.stream
And it will show data like this :
The meaning of the diagrams is explained below (source)
Request collapser
One of the disadvantages of microservice architecture is the performance penalty caused by high number of cross service HTTP calls. Advantage of batching the calls are :
- The communication overhead of multiple calls are reduced into a single call
- The backend may be able to more efficiently serve a single request with more parameters then multiple requests
- If there are multiple reuquests with the same parameter then only one of the requests have to be executed.
The implementation is quite straightforward again. First we define a method
- returning a Future
- Annotated with @HystrixController
The method body will never be actually executed, it can be left empty.
@HystrixCollapser(scope = com.netflix.hystrix.HystrixCollapser.Scope.GLOBAL, batchMethod = "getCustomerByIds") public Future<MessageWrapper> getCustomerById(Integer id) { throw new RuntimeException("This method body should not be executed"); }
Then we define the collapser method
- Taking a list of the same type as the original method
- Returning a list of the same type as the original method
- Annotated with HystrixCommand
@HystrixCommand public List<MessageWrapper> getCustomerByIds(List<Integer> ids) {
This approach can be used not only to to collapse multiple HTTP calls, but for SQL queries or anything else.
Good article! Thanks!
How to run it? I did an mvn install then mvn exec:java
mvn sping-boot:run
Good Article !!!!! Thank you.
I have performed the same step, i application is working but hystrix dashboard not updating the data continuously?
Hi, the hystrix dashboard is updating real time, check the example application on the github link.
Pingback: Spring Cloud Zuul example | Example Driven (Self) Development
Pingback: Spring Cloud, Eureka, Ribbon, Feign example | Example Driven (Self) Development
Pingback: Microservice ecosystem with Docker, Spring Cloud and Spring Boot | Example Driven (Self) Development
Do you have example of setting threshold to open circuit and close circuit after certain number of fallback calls
I don’t have an example, but with a little reading and debugging you can find it out for yourself. I suggest reading these
https://github.com/Netflix/Hystrix/wiki/Configuration#intro
https://stackoverflow.com/questions/31211685/configuring-hystrix-command-properties-using-application-yaml-in-spring-boot-app
and debugging these
HystrixCommandProperties
AbstractCommand
The general approach is when you get an exception, e.g: when circuit is open, then you print out the stacktrace and follow it. The netflix code is quite readable you should find what you are looking for
Hi,
I have a doubt. Is it possible to balance the load among the different instances of a microservice. Say for example, can we limit the number of requests to be processed by an instance should be 50 and the rest has to be processed by another instance.
Do we require any configuration changes related to ribbon. Please help.
It is certainly possible, the ribbon client allows has some customization options, see the official documentation here http://cloud.spring.io/spring-cloud-static/Dalston.SR1/#spring-cloud-ribbon if that is not enough you can write your own ribbon rule.
Good Article, very helpful !!!!! Thank you.