ETL Solution for Financial Services Groups

Overview:

The client sought a business solution to architect, configure and support their Extraction, Transformation and Loading (ETL) processes for a large dataware housing project. we delivered a highly scalable solution.

telco_img

Business Need:

  • Consolidate the ETL processes.
  • Optimize the extraction and data loading process..
  • Customize the ETL product as per requirement..
  • Better error handling and better notification system..

The Challenges

The biggest challenge was to provide scalable architecture for consolidating huge amounts of data.

Our Solution

We provided consulting services to the client data warehousing initiative. We reviewed the data source systems, and data structure needs. Based on the client's requirement, we developed ETL product using various prominent Bigdata tools.


We administered cluster for development, testing as well as deployment and also helped the client to tune the cluster for optimized performance.


Tested the product with different load for stress testing and performance testing. Our staff is also responsible for performance tuning of ETL processes. We also implemented new capabilities like bulk loading the data to optimize the load in ETL processes.


We also inplemented the notification and alerting system for ETL product which enabled better execution of hadoop jobs.

Implementation Process

We followed an iterative phased approach to implement the solution that included the following phases:

  • Business requirements analysis.
  • Architecture design
  • ETL (Extract, Transform and Load).
  • Designed a proper alerting system for errors.

Business Benefits

By using our enhanced bigdata solution, the client got performance and revenue gain by 12% monthly.