The Super Bowl broke all streaming records. An increasing number of Americans watch the Super Bowl on a live stream from a variety of devices instead of watching the event on their TV. This poses new challenges for the streaming services of broadcasting companies.
When Fox Sports experienced streaming performance issues and higher infrastructure cost, they realized they needed to accommodate for increased traffic on their streaming services. Fox Sports and X-Team strategized together a solution that would provide a better streaming experience, as well as reduce the infrastructure cost of future Super Bowls.
After a performance analysis, Fox decided to migrate some of their Node.js services over to Go. X-Team developers created a supporting library with tools that could be used across the entire Go project, which contained NewRelic, Splunk, and InfluxDB integrations, as well as default middleware and configuration management. The toolkit made it easier and faster for X-Team to develop some of the services that Fox needed for this project.
One such service that X-Team developers created is the Disaster Recovery Service, which identifies streaming channels that are overloaded with traffic and seamlessly switches users over to other streaming providers. Another service is the Decisioning Service, which provides new viewers with the best streaming path and CDN based on their IP address. This is critical for the Super Bowl, as providers and CDNs are often brought down if traffic isn't split appropriately between them.
Finally, there's the ConfigTool Service, which is a distributed configuration manager that acts as a proxy to A/B testing providers as well. This service can be used to configure experiments and variations for any backend service. For example, the ConfigTool Service was used to test whether the Decisioning Service helped reduce the load on individual CDNs (it did!).
Tech Stack Used
The tech stack X-Team uses to solve Fox Sports' streaming issues consisted of the following:
Initial testing at 30,000 requests per second had Fox' Node.js services crash and restart, because of the load on their servers' CPUs. The Go services, however, kept at a constant 12% CPU load and 20MB RAM consumption. This has reduced the cost of Fox' cloud infrastructure and significantly improved the amount of traffic their streaming services can now handle.