Two steps are needed in order to correctly apply back-pressure in an Akka system:
Step 1: Bounded Mailboxes and Push Timeouts
The default mailbox for an actor is an
UnboundedMailbox backed by Java’s
ConcurrentLinkedQueue. As the name indicates, this mailbox grows without bound and will end up crashing the JVM with an
OutOfMemoryError if the consumer significantly slower than the producer. If we want to be able to signal the producer to slow down, the first step is to switch to a
BoundedMailbox backed by Java’s
LinkedBlockingQueue that will block the producer if the mailbox is full. More info about different types of mailboxes can be found here.
Blocking the producer forever is not a good solution because:
Rule #1 of Akka => don't block inside actors. The solution to this problem is provided to us by Akka in the form of a
push timeout for an Actor’s mailbox. A push timeout is exactly what it sounds like: when you try to push a message to an actor’s mailbox, if the mailbox is full, the action will timeout and the message will get routed to the
Configuring an actor to use a bounded mailbox with a 1000 message capacity and a push timeout of 100ms requires the following addition to the
1 2 3 4 5
The actor can then be initialized as follows
Step 2: DeadLetter Watcher
When an actor’s mailbox is full and sent messages start timing out, they get routed to the
DeadLetterActorRef via the Event Stream of the actor system. Akka allows actors to subscribe to event streams and listen in on all, or a filtered subset of, the messages flying around in the actor system. Since the dead letters service also utilizes the event stream infrastructure, we can subscribe to all
DeadLetter messages being published in the stream and signal the producer to slow down.
The following snipped can be used to get an actor subscribed to all the
DeadLetter messages in a system
Tying it all together with an Example
In this example, a fast producer sends messages to a slow consumer. The slow consumer has a bounded mailbox of size 10 and a push timeout of 10ms.
SlowReceiver blocks for 100ms after printing each message it receives. The blocking is only present to ensure that it’s mailbox fills up.
1 2 3 4 5 6 7 8 9 10 11 12 13
FastSender waits for a kickoff message and then sends 15 messages to the
SlowReceiver and a
PoisonPill to itself. After terminating itself, the actor’s
postStop hook schedules a
PoisonPill to be sent to the
SlowReceiver 3 seconds after the
FastSender has been terminated.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Watcher watches for and prints
DeadLetters being sent to the
SlowReceiver. It also
SlowReceiver and terminates the actor system when the
SlowReceiver is killed.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
The App that ties it all together.
1 2 3 4 5 6 7 8 9 10 11
BackPressureTest app gives the following output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
While this example doesn’t actually implement back-pressure, it provides the infrastructure for applying a back-pressure strategy. A possible strategy would be to send
SlowDown message from within the
Watcher for each dead letter received. The
SlowDown case class could be defined as
FastSender receives a
SlowDown message, it could either throttle itself or tell its upstream systems to slow down. The
SlowDown message also encapsulates the relevant
DeadLetter object to allow for retry logic.