Newer Post

Slack Tips Tuesday: The employee spotlight channel

Older Post

Awesome Backpacks Recommendations from Our X-Teamers

Socket.IO with HAProxy on AWS

Socket.IO with HAProxy on AWS

We all know this feeling… You’ve done all your dev work on local environment, but it’s not working as planned on production. Now you think like you’ve wasted so much time trying to figure out what’s wrong. Here are some highlights from my story to make your life easier.

Our setup

We deploy our services on Amazon ECS clusters.Everything was working fine, until a new project heavily depending on real-time notifications came up.

We decided to used two load balancers – ELB and HAProxy. These two share some responsibilities, but we used them to serve a different purpose. Our HAProxy is deployed as a Docker container and configuration is generated on the fly using consul-template. The ELB is used mainly for one reason – it’s static, so we easily configure Route 53 (AWS Domain Name Server).

stack

HAProxy + ELB + SocketIO stack

ELB Configuration

In order to use socket.io we need to set up listeners in a correct way. Websockets use TCP as a transport layer, so we need to configure ELB to transfer all TCP traffic from port 80 to our HAProxy port. The same goes for secure websockets (wss) – all traffic from SSL on port 443 needs to be forwarded to 8005 – a port which our HAProxy container is running on.

elb

ELB listeners configuration

The last step to enable *wss * is to check SSL Protocol setting under Cipher section:

cipher

Ciphers configuration

HAProxy Config

The main problem comes from socket.io performing its handshake in the beginning. This means that the second request must end up with the same server as the first one. There are multiple ways to ensure this outcome, but with ELB + HAProxy stack we are limited in options.

HAProxy gives us an option to select load balancing algorithm (https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-balance), which means that connections will be assigned to a webserver based on IP address. We have to select that one specifically, because it will allow our websocket session to stick.

The last, but not least thing to do is to configure ELB listener to allow proxy on TCP. There is an excellent article on this issue (possibly doubling some knowledge here) written by Philippe Modardhere.

And here is a configuration I came up with in the end:

HAProxy Stats

In order to ensure HAProxy is working as expected, it’s valuable to enable statistics. It will provide us with visual breakdown of open sessions, server information, data flow and many more. I highly encourage to enable them, as it provides us with valuable information for load testing and performance monitoring.

After stats are enabled, we can view the page by visiting http://[address]:[port]/haproxy?stats and authenticating with Username/Password specified in the config. It’s mainly useful to track and record important metrics as well as monitor the number of open connections, which will be useful for load balancing.

Load Testing

It’s always beneficial to see how our application behaves under a bigger load and test that nothing breaks. There are some notable tools available for load testing like ApacheBench or JMeter, but I was looking for something simpler, especially to load test websockets.

The tool I found and used was https://github.com/M6Web/websocket-bench. With only a simple command I could test 1,000 concurrent connections and verify that nothing breaks:

$ websocket-bench -a 1000 -c 200 http://my-app.io

The main advantages of using websocket-bench over other tools is its simplicity and support for socket.io out of the box. Besides it supports writing your own workflows (called generators), to load test more complex scenarios.

Distributed Load Testing

It’s pretty simple to use it to load balance one machine, but with a source load balancing algorithm it’s impossible to test it on multiple machines, as the traffic will only come from one IP.

What I ended up doing was going a similar route as the project with the coolest name ever – Bees with Machine Guns – spawn a couple of AWS t2.micro instances and start websocket-bench simultaneously on every machine.

Thanks to fabric library I ended up doing it quite painlessly, as it allows to execute commands via SSH on many machines in parallel. The command line to achieve it looks as following:

$ fab -H [LIST OF HOSTS] -u [USER] -P -- websocket-bench -a 1000 -c 200 http://my-app.io

Conclusion

It was quite a journey to ensure that socket.io is working on every part of the stack. Working with ELB definitely wasn’t a most pleasurable experience and many choices came from extensive hours of trial and error. I hope that you’ll find this story useful.

We'll help you unleash.

Join the 20,000 developers who subscribe to our newsletter.

Scale your
Development team

We help you execute projects by providing trusted developers who can join your team and immediately start delivering high-quality code.

Hire Developers