Telco Analyzer

Overview:

A prominent telecom company needed assistance in building their customized product for data analytics and recommandation tool using bigdata tools.


The primary goal of the system was to transform the existing system's data to newly developed scalable system for better analytics.

Business Need:

  • Consolidate the data generated from various sources using RabbitMQ.
  • Optimize the extraction and data loading process and stage the data in Hbase.
  • Perform aggregation of data from Hbase to Hive.
  • Perform analytics on top of Hive using Impala.
  • Customize the ETL product as per requirement.

The Challenges

The biggest challenge was to provide scalable architecture for consolidating huge amounts of data.

Our Solution

The high level block diagram of our proposed system is shown below.

telco_img

We provided consulting services to the client data warehousing initiative. We reviewed the data source systems, and data structure needs. Based on the client's requirement, we developed ETL product using various prominent Bigdata tools.


We administered cluster for development, testing as well as deployment and also helped the client to tune the cluster for optimized performance.


Tested the product with different load for stress testing and performance testing. Our staff is also responsible for performance tuning of ETL processes. We also implemented new capabilities like bulk loading the data to optimize the load in ETL processes.


Implementation Process

We followed an iterative phased approach to implement the solution that included the following phases:

  • Business requirements analysis.
  • Architecture design
  • Developed the ETL (Extract, Transform and Load) Suit.