Modern ELT Data Processing Pipeline and API Platform for Fortune 100 Wealth Management Firm | Sovereign Solutions

Modern ELT Data Processing Pipeline and API Platform for Fortune 100 Wealth Management Firm

Challenge

Our client is a world-leading financial services firm that handles wealth management, insurance, estate planning, financial advisory, retirement savings, and college savings for its private and institutional clients.

Within its wealth management operations, our client is faced with some strategic business challenges that include:

  • Difficulty in aggregating data from various sources such as, financial services

providers, financial advisors, and brokers in near real-time. These diverse data sources utilize different data formats and protocols which can sometimes delay data propagation.

  • A variety of data pipeline issues such as, failure to deliver data at the same time, errors in the data, missing data, resubmitted data, partial data, data dependencies, poor data replication, slow data transformation, time-consuming data testing and validation, and other disruptive data problems.
  • Average throughput of 4-6 hours for data to go through the pipeline, causes unsatisfactory delays and discrepancies in client services and support in the areas of financial reporting, investment valuations, account updates, communication, research, and analytical feeds.
  • Lack of a robust, well-defined API offerings and management platform for sharing data with internal and external consumers.

Solution

Sovereign Solutions was engaged to help design a more agile enterprise architecture that would lead to better accuracy, throughput, and overall better performance. Our client took this significant step in order to automate its business processes and provide world-class investment services to its clients.

Our dedicated team carried out an extensive assessment of the client’s existing legacy architecture in order to identify the client’s pain points. Working with the client’s in-house developers and engineers, Sovereign Solutions defined and highlighted the shortcomings with the client’s current legacy architecture, designed and built a new ultra-high-speed data-processing pipeline, and an automated testing platform with the following redesigning procedures:

  • We onboarded the client on to the Amazon AWS platform, and provided guidance on best practices and reference architecture. Our team set up EC2, S3, and VPC services on AWS for the client, in addition to helping them set up DevOps and CI/CD capabilities on AWS to streamline and optimize development and testing activities.  The AWS platform was used as the foundational infrastructure for all new application and business logic.
  • The Informatica transformations, configurations, and workflows were exported as XML configuration files. The exported XML files were used as inputs to a new Java-based framework that was created for the client. The Java framework parsed the XML files and automatically generated the necessary transformation, configuration, and workflow logic. This necessitated the reuse of the existing transformation, workflow, and configuration information without having to redesign that logic, which would have significantly increased complexity and made it very difficult to test. Hence, the reuse of the same transformation definitions and logic brought about consistent transformation patterns across the organization, from the primary operational data store to the various data marts reporting databases and analytics data stores.
  • Our team built an API Platform that was backed by a Data Lake, which was comprised of a big data repository built using Cassandra with an in-memory cache to boost the performance of the APIs. The API Platform offered request-response REST APIs and real-time publish-subscribe messaging APIs. The Platform offered full API management capabilities, developer portal, throttling, sample code, API gateways, API lifecycle management, and API analytics engine.

Results

Upon the completion and implementation of the new, ELT-based architecture and efficient API Platform, the resulting, ultra-fast data-processing pipeline performance exceeded the client’s expectations as the following landmark achievements were recorded:

  • Data extraction, loading, and transformation that usually took 4-6 hours can now be successfully completed in 5-10 minutes.
  • The time required for testing new changes has now been reduced from 1 to 3 days to 10 to 60 minutes.
  • The overall processing time and throughput went from 4- 6 hours to 5-10 minutes to get the data to the downstream.
  • The error rate was significantly reduced while increasing client satisfaction because updates could be processed quickly and consistently across the organization, which helped ensure that the latest data was available at near real-time.

A better data pipeline model and a faster data processing helped the client improve on its agility and improve customer satisfaction and retention.  The newly deployed cloud capabilities enabled our client to quickly and easily spin up testing environments to perform regression testing on changes to their data processing pipeline and reduce the time to market for rolling out new system features.  It also enabled them to elastically scale their production environments as necessary to support fluctuating transaction volumes for the new API platform and the redesigned data processing pipeline.


Talk to our experts now to learn how Sovereign Solutions can help your organization meet its business objectives.

*This content is being provided for informational and educational purposes, and should not be regarded as a specific recommendation for your situation.