Study Data Warehouse

Our client is one of the largest contract research organisations (CRO) providing clinical trial services, with annual revenues of over $2 billion and over 11,000 employees in more than 60 countries.

The task was to deliver a Study Data Warehouse to store both clinical and pre-clinical data in a single repository using modern web technology that can provide an extremely fast and highly scalable data integration solution.

Clinical data has a heterogeneous nature: various types of data sources and different formats of the same data. The proposed system would enable our client to offer the only data warehouse in the marketplace that can handle both pre-clinical and clinical data in a single repository and enhance strength as the only end-to-end CRO.

The Study Data Warehouse would provide standardised data to medical information systems; that would make it an industry-leading and differentiating solution and give a significant marketing advantage.

We chose JavaScript as the main development language for both server-side and client-side development. NodeJS was selected as a server-side framework for its asynchronous model of work, which perfectly suited for the simultaneous operation with a large number of data sources.

The development of large applications using JavaScript is extremely demanding in terms of compliance with the best practices of development. Therefore, we monitor code quality using static code analysers and always perform code reviews. Particular attention was paid to unit testing. For unit testing, we chose Mocha framework, since it allows the development of tests for the client and server scripts.

The specifics of the project was working with a vast variety of data sources, from flat files and traditional relational databases to NoSQL storages. We implemented a level of abstraction that allows adding support of new data sources easily with special adapters. Data can be loaded from data sources asynchronously, which is important when handing a large amount of data.

Special attention was paid to memory management and application stability,  that is why we used separate workflows and task queues for complex calculations.

When designing the user interface, our priority was to create an attractive and responsive interface that allows most of the actions to be done on mobile devices as well as on conventional desktop screens. Therefore, we applied the approach “mobile first development”. The AngularJS framework was the most appropriate choice, as it allowed developing a flexible user interface without downloading large libraries. We were loading what we exactly need on the client side.

Our solution enabled processing large amounts of data without using a lot of memory. The asynchronous model of NodeJS increased the number of simultaneously processed requests per unit of time since there was no request processing and maintenance overhead in NodeJS.

The use of one language for the server and client sides, as well as a query language for such NoSQL database as CouchDB, increased the degree of code reuse and improved cross-functionality of team members. It allowed raising the degree of ownership of the project by developers; besides, it reduced the time for development of new features.