Acquiring and Processing Stock Exchange FIX-based Data

Our client is one of the world’s leading data providers. The company offers a complex cloud-born solution with up-to-date financial data from a variety of sources, including FIX-based stock exchanges.

Considering the users’ needs, our client decided to replace the existing platform with a new one that would reduce latency and increase stock exchange coverage. The new solution was supposed to give our client a clear advantage over other providers.

Software Country’s team has been involved in the development of the entire platform since the beginning. The platform was designed as a type of service-oriented architecture to improve scalability and throughput. To bring forth a high-quality product in a timely fashion, we decided to build the solution on top of the .NET platform using C# language.

In addition to native support of C# applications, the chosen Microsoft Azure Cloud provides an ecosystem for service-oriented architecture with all the necessary resources, such as the following:

• Service Fabric for easier deployment and management;
• Redis as an internal message bus for the best publisher-subscriber performance;
• Cosmos DB as a cheap, fast and scalable storage system;
• Event Hubs as a cheap, performant message streaming platform;
• Application Insights for telemetry and error notifications.

The main challenge of acquiring market data from various sources is that they all support different protocols. Usually, this is a FIX protocol or one of its variations, e.g., FAST (FIX Adapted for Streaming) or SBE (Simple Binary Encoding). To implement data providers in a timely manner with consideration of performance and latency issues, we used high-quality libraries that perform most of the protocol jobs for different venues. These included an OnixS MDP adapter for the Chicago Metal Exchange (CME, which uses SBE) and Rapid Addition Cheetah for the London Metal Exchange (LME, which uses FIX).

An additional application of the FIX protocol was the internal messaging system (called Message Bus, which is built on top of the Redis implementation of the publisher-subscriber pattern). This system transfers a lot of market and trading data over the platform. Since the FIX protocol’s performance limitations are widely known, we have used a much faster SBE protocol.

Special attention was paid to the platform stability, robustness, and specification verification that were achieved through thorough code reviews and unit and specification auto-testing as an addition to high-quality manual testing. Failure discovery and telemetry were supported via application insights with alerting.

For downstream clients, such as web applications or Excel, the system exposes REST API and streaming-ready SignalR endpoints. Although it is possible to provide data in the SBE messaging format for consumers, its representation is obscure, and it is not easy to use. Therefore, internal SBE messages were transformed into plain objects for client usage.

We were also partly responsible for the architecture, implementation, testing, and deployment of the application.

We have developed a best-in-class solution that is able to collect as many representative data points as possible. As a result, our client’s customers can receive the most up-to-date trading data from various sources around the world. The company has big plans for the development of this platform, so we will continue to work together for the foreseeable future.