Business activity monitoring (BAM) provides end-to-end visibility of business processes, enabling enterprises to make better-informed decisions and quickly address problem areas. A BAM framework is used for continuous event correlation, real-time alerts, and monitoring KPI statistics for business activities in real-time.

Business needs

A US-based Fortune 500 mortgage lender wanted a BAM solution that could apply data reconciliation checks, aggregate data anomalies for reporting, and perform rule-based data quality checks in real-time. They were looking for a solution that would:

• Power real-time ingestion, event processing, and KPI computing
• Detect data issues with continuous matching and monitoring
• Ensure high performance, alerts, and customization

Solution

Gathr enabled the client to implement an end-to-end continuous monitoring solution leveraging out-of-the-box drag-and-drop operators and a custom development framework. Using a data-first, code-free approach, Gathr allowed users to effortlessly design and manage complex data flows on a visual canvas.
Highlights:

• Deployed and managed data pipelines on Apache Spark-based infrastructure
• Created streaming and batch ETL pipelines for real-time alerting and historical data processing to generate SLA reports
• Enabled real-time data summary generation at different levels (including historical data) and provided an aggregated view of these
• Provided 200+ operators for data wrangling and transformation with support for custom extensions and late data arrival
• Enabled rapid, efficient prototyping, and operationalization of ETL frameworks
• Delivered an SLA monitoring solution in 3 modules based on different types of pipelines:
  • Data ingestion, quality management, and landing
  • Data correlation and SLA monitoring
  • End of day and month views of ETL processing data

Results

• Enabled ingestion and processing of 1 TB data on a daily basis
• 10x faster data processing compared to the existing system
• Efficient near real-time KPI tracking with code-free business rule updates at runtime
• Improved visibility through correlation of real-time and historical data
• Better decision-making with instant SLA status checks