Fox Sports, also referred to as Fox Sports Media Group, is the sports programming division of the Fox Broadcasting Company, owned by Fox Corporation, that is responsible for sports broadcasts on the Fox network and its dedicated national sports cable channels.

With X-Team’s help, FOX Sports reduced the cost of their cloud infrastructure and significantly improved the amount of traffic their streaming services could handle.

Project Description

The Super Bowl broke all streaming records. An increasing number of Americans watch the Super Bowl on a live stream from a variety of devices instead of watching the event on their TV. This posed new challenges for the streaming services of broadcasting companies.

When Fox Sports experienced streaming performance issues and higher infrastructure cost, they realized they needed to accommodate for increased traffic on their streaming services. Fox Sports and X-Team came up with a solution that would provide a better streaming experience, as well as reduce the infrastructure cost of future Super Bowls.

Solution

After a performance analysis, Fox decided to migrate some of their Node.js services over to Go. X-Team developers created a supporting library with tools that could be used across the entire Go project, which contained NewRelic, Splunk, and InfluxDB integrations, as well as default middleware and configuration management. The toolkit made it easier and faster for X-Team to develop some of the services that Fox needed for this project.

One such service that X-Team developers created is the Disaster Recovery Service, which identifies streaming channels that are overloaded with traffic and seamlessly switches users over to other streaming providers. Another service is the Decisioning Service, which provides new viewers with the best streaming path and CDN based on their IP address. This is critical for the Super Bowl, as providers and CDNs are often brought down if traffic isn't split appropriately between them.

Finally, there's the ConfigTool Service, which is a distributed configuration manager that acts as a proxy to A/B testing providers as well. This service can be used to configure experiments and variations for any backend service. For example, the ConfigTool Service was used to test whether the Decisioning Service helped reduce the load on individual CDNs (it did!).

Tech Stack Used

The tech stack X-Team used to solve Fox Sports' streaming issues consisted of the following:

  • AWS as the cloud provider
  • Cloudformation as the 'infrastructure as code' handler
  • Jenkins as the CI/CD manager
  • Docker as the container tech of choice
  • ECS as the container orchestration tech
  • Go as the programming language
  • NewRelic as the Application Monitoring tech
  • Splunk as the distributed logging solution
  • InfluxDB as the business metrics aggregator
  • K6 as the load testing tool

Results

Initial testing at 30,000 requests per second had Fox' Node.js services crash and restart, because of the load on their servers' CPUs. The Go services, however, kept at a constant 12% CPU load and 20MB RAM consumption. This has reduced the cost of Fox' cloud infrastructure and significantly improved the amount of traffic their streaming services can now handle.

Read about some of the other companies X-Team has worked together with:

Interested in learning more? Contact us today.