Socket.IO with HAProxy on AWS
We all know this feeling… You’ve done all your dev work on local environment, but it’s not working as planned on production. Now you think like you’ve wasted so much time trying to figure out what’s wrong. Here are some highlights from my story to make your life easier.
We deploy our services on Amazon ECS clusters.Everything was working fine, until a new project heavily depending on real-time notifications came up.
We decided to used two load balancers – ELB and HAProxy. These two share some responsibilities, but we used them to serve a different purpose. Our HAProxy is deployed as a Docker container and configuration is generated on the fly using consul-template. The ELB is used mainly for one reason – it’s static, so we easily configure Route 53 (AWS Domain Name Server).
HAProxy + ELB + SocketIO stack
In order to use socket.io we need to set up listeners in a correct way. Websockets use TCP as a transport layer, so we need to configure ELB to transfer all TCP traffic from port 80 to our HAProxy port. The same goes for secure websockets (wss) – all traffic from SSL on port 443 needs to be forwarded to 8005 – a port which our HAProxy container is running on.
ELB listeners configuration
The last step to enable *wss * is to check SSL Protocol setting under Cipher section:
The main problem comes from socket.io performing its handshake in the beginning. This means that the second request must end up with the same server as the first one. There are multiple ways to ensure this outcome, but with ELB + HAProxy stack we are limited in options.
HAProxy gives us an option to select load balancing algorithm (https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-balance), which means that connections will be assigned to a webserver based on IP address. We have to select that one specifically, because it will allow our websocket session to stick.
The last, but not least thing to do is to configure ELB listener to allow proxy on TCP. There is an excellent article on this issue (possibly doubling some knowledge here) written by Philippe Modardhere.
And here is a configuration I came up with in the end:
In order to ensure HAProxy is working as expected, it’s valuable to enable statistics. It will provide us with visual breakdown of open sessions, server information, data flow and many more. I highly encourage to enable them, as it provides us with valuable information for load testing and performance monitoring.
After stats are enabled, we can view the page by visiting
http://[address]:[port]/haproxy?stats and authenticating with Username/Password specified in the config. It’s mainly useful to track and record important metrics as well as monitor the number of open connections, which will be useful for load balancing.
It’s always beneficial to see how our application behaves under a bigger load and test that nothing breaks. There are some notable tools available for load testing like ApacheBench or JMeter, but I was looking for something simpler, especially to load test websockets.
The tool I found and used was https://github.com/M6Web/websocket-bench. With only a simple command I could test 1,000 concurrent connections and verify that nothing breaks:
$ websocket-bench -a 1000 -c 200 http://my-app.io
The main advantages of using websocket-bench over other tools is its simplicity and support for socket.io out of the box. Besides it supports writing your own workflows (called generators), to load test more complex scenarios.
Distributed Load Testing
It’s pretty simple to use it to load balance one machine, but with a source load balancing algorithm it’s impossible to test it on multiple machines, as the traffic will only come from one IP.
What I ended up doing was going a similar route as the project with the coolest name ever – Bees with Machine Guns – spawn a couple of AWS t2.micro instances and start websocket-bench simultaneously on every machine.
Thanks to fabric library I ended up doing it quite painlessly, as it allows to execute commands via SSH on many machines in parallel. The command line to achieve it looks as following:
$ fab -H [LIST OF HOSTS] -u [USER] -P -- websocket-bench -a 1000 -c 200 http://my-app.io
It was quite a journey to ensure that socket.io is working on every part of the stack. Working with ELB definitely wasn’t a most pleasurable experience and many choices came from extensive hours of trial and error. I hope that you’ll find this story useful.