To power real-time decision making on large data sets, enterprises need an expert team, high-performing hardware systems, and a scalable ETL solution that can accelerate development and deployment of ETL frameworks, while swiftly accommodating changing business needs.

Next-generation ETL tools allow enterprises to eff­ectively design and create an environment to mine and analyze data for making informed decisions. They isolate data from transactional systems, which ensures business-as-usual while data is analyzed in an optimized environment. These frameworks also help users solve business problems without spending cycles perfecting boilerplate code.

Business needs

A communication analytics solutions provider wanted to modernize their existing data applications and was looking for an easy-to-use and scalable solution that could process over 1.5 billion user interactions generated per day from multiple real-time feeds.

Solution

Gathr enabled the client to implement applications that run on a scalable Spark compute engine as structured streaming data pipelines while providing self-service and analytics capabilities for large-scale data processing.
The ETL solution used Gathr’s vast library of components for data acquisition, processing, enrichment, and storage. The entire data flow was created and orchestrated using a low-code methodology.

Results

Gathr enabled end-to-end data ingestion, enrichment, machine learning, action triggers, and visualization to modernize hand-written data applications to Spark structured streaming in weeks. This, in turn, helped the customer realize several strategic benefits:

• Replaced roughly ~1 million lines of code in ~3 weeks using Gathr frameworks
• Achieved a high throughput of 100000+ transactions/second, enabling processing of 1.5 billion records per day
• Reduced the overall release cycle from 8 months to 8 weeks