Hystrix and Database Transactions

In my previous post I referenced the NetFlix HystrixCommand and how it implements the Circuit Breaker  pattern.  Let take a quick look at how it works.

Here is the example from the previous post.

public class CommandHelloWorld extends HystrixCommand<String> {

    private final String name;

    public CommandHelloWorld(String name) {
       super(
         HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
        this.name = name;
    }

    @Override
    protected String run() {
        return "Hello " + name + "!";
    }
}

Then to execute the command

String s = new CommandHelloWorld("Bob").execute();

When the execute() method is called the HystrixCommand calls a ThreadPoolExecutor to run a FutureTask.  The ThreadPoolExecutor then blocks on the FutureTask by calling get(). When the thread finishes running the task the ThreadPoolExecutor unblocks from the get() method and returns the value of the get() method as the results of the execute() command.

The HystrixCommand class handles all the details of timeouts, exceptions, recording timings, most importantly all the details around being a good circuit breaker.  All you need to is call the execute method on your class. This is a high level diagram of the interactions

circuit_breaker_1_seq

Note that the FutureTask calls the run() method on the HytrixCommand. That means that the guts of the HystrixCommand run in a different thread than that of the caller of the execute() method.

At this point I am assuming that you know how aspects work and how they work in Spring.  If not read here.

Now lets change this to use the @CircuitBreaker annotation. I have an example that uses the @CircuitBreaker pattern at

https://github.com/marblesit-blog/circuit-breaker-example.git

The application is a REST app using an HSQLDB in memory database and Hibernate mappings.  It also uses Spring Transaction annotations and Spring Data for its DAO repository.

When using an annotations and AspectJ Spring will not inject the object that you expect but instead will inject a proxy of the class (or interface).

Here is the REST controller from the example code:

@RestController
public class ExampleRestController {

  @Autowired
  private UserService userService;

  @RequestMapping(path="/users", method=RequestMethod.GET)
  public @ResponseBody Collection<User> getUsers() {
    return userService.getAllUsers();
  }
}

Below is the UserService that the writer of the REST controller is expecting to be injected into his(her) controller.

@Service
@Transactional
public class UserServiceImpl implements UserService {

  @Autowired
  private UserRepository repo;

  @Override
  @CircuitBreaker
  public Collection<User> getAllUsers() {
    return
      StreamSupport.stream(repo.findAll().spliterator(), false).
        map(userEntity -> map(userEntity)).
        collect(Collectors.toList());
  }

  private User map(UserEntity userEntity) {
    Set<String> emails =
      userEntity.getEmailAddresses().stream().
      map(ema -> ema.getEmail()).
      collect(Collectors.toSet());

    return new User(userEntity.getId(), userEntity.getName(), emails);
  }
}

But when the controller calls getAllUsers() on the proxy, the proxy will call the Spring Transaction to start a transaction and then call the @CircuitBreaker.

The @CircuitBreaker will build the HystrixCommand object and will provide a run() method that will call the method that you have annotated.

When the @CircuitBreaker is done running, the Spring Tranaction will commit the transaction and close it out.

So when our REST controller calls the service

@RequestMapping(path="/users", method=RequestMethod.GET)
public @ResponseBody Collection<User> getUsers() {
  return userService.getAllUsers();
}

the following will happen (from a very high level):

circuit_breaker_2_seq

If you run the example you will notice that you will get an exception.

Capture

Ahhhh…  The infamous LazyInitializationException. That means that after the users are fetched from the database the transaction is committed and the database session is closed.

But the service call is annotated with @Transactional and so the transaction should start when the service method is called and committed and closed when the service method exits.  Why doesn’t it work?

The secret is the ThreadExecutor.  The HytrixCommand runs in a separate thread.  But Spring stores the transaction in thread local storage so that when the transaction (and the database connection) is needed it can be found without having to pass it to every method call as a parameter.

So what happens is that the transaction gets created and stored in thread local storage.  When the service attempts at access the transaction in thread local storage it’s not there because the transaction was created in a DIFFERENT thread.  Since there is not a transaction one is created inside of the DAO, Hibernate executes the query, extracts the results set, and closes the transaction.  Then we attempt to access the users emails which are in another table, Hibernate needs to make another query, but the database session is closed and voila, LazyInitializationException.

The key then is to not use a separate thread but to execute the HystixCommand in the same thread as the caller of the original method. Fortunately the @CircuitBreaker annotation does support an option of the HystrixCommand to not use threads but to use java semaphores instead. To use add the useThreads option.

@Override
@CircuitBreaker(useThreads=false)
public Collection<User> getAllUsers() {
  return
    StreamSupport.stream(repo.findAll().spliterator(), false).
      map(userEntity -> map(userEntity)).collect(Collectors.toList());
}

Do this and the code will work!

Using NetFlix Hystrix Circuit Breaker With Annotations

Recently I was at a customer site building a web service that needed to hit a database. I wanted to use the Circuit Breaker pattern from Mike Nygard’s excellent book Release It! (which is finally available on the Kindle).  I had built my own circuit breakers before but thought someone else must have done it also. So with a bit of googling I found the NetFlix Hystrix project.

Not only was there an implementation of a circuit breaker but it also kept track of performance statistics and there is another project with a dashboard that has a very nice JavaScript UI with real-time updates.

The Hystrix project implements the circuit break by using command objects to wrap your code.  To use the framework you create a subclass of the HystrixCommand object and then override the run() method with your code. For example

public class CommandHelloWorld extends HystrixCommand<String> {

    private final String name;

    public CommandHelloWorld(String name) {
       super(
         HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
        this.name = name;
    }

    @Override
    protected String run() {
        return "Hello " + name + "!";
    }
}

Then to execute the command

String s = new CommandHelloWorld("Bob").execute();

This works great but you do end up writing a lot of boilerplate code.  I had done annotations before and this seemed like the perfect candidate.  I searched but the ones I found were not to my liking.

But I did find someone who had implemented a basic annotation.  I took the basic code and enhanced it with more options.

To use the annotation

    @CircuitBreaker
    public String methodName() throws MyException {
    }

There are options to set the name of the circuit breaker, the timeout value in milliseconds, the fallback code in case of an error, and whether or not to use threads.

This code is open sourced in GitHub and in my next post I will go over how Hytrix can be problematic with database transactions and how to get around it.