$7 Billion Multi-Strategy Hedge Fund was looking to perform some big data research to devise data trends and use the data to drive investment decisions. Because of the large amount of data and perceived daily growth, it was determined to load the data into a solution designed for large datasets. Collectively, it was concluded that Amazon Redshift would be the best option for the particular dataset.

Additionally, the project required backloading 5 years of daily data from a vendor API. Signalbase wrote Python scripts to backload the data based on conversations with the API vendor, intending to minimize the total time, without comprising the vendor’s API backend. In order to minimize the time, the scripts were designed to run in parallel to each other, seperately executing on Digital Ocean machines specifically built and deployed for the task.

Signalbase completed the backloading and then maintained the inclusion of each day’s information into the existing database and pre-aggregated data. The Signalbase team also worked with the firm’s internal development staff to design more efficient queries and incorporate the data into an internal research system.