What's brewing

Month: April 2023

Move away from batch ETL with next-gen Change Data Capture

Posted on by Saurabh Dutta

As data volumes continue to grow, enterprises are constantly looking for ways to reduce processing time and expedite insight extraction. However, with traditional databases and batch ETL processes, performing analytical queries on huge volumes of data is time-consuming, complex, and cost intensive.

Change Data Capture helps expedite data analysis and integration by providing the flexibility to process only the new data through a smart data architecture, thus incrementally computing updates without modifying or slowing down source systems. This enables enterprises to reduce overheads, improve hardware lifetime, and ensure timely data processing without facing the limitations of batch processing.

That probably sounds like an oversimplified solution to a very complex problem. So, let’s break it down.

Data extraction is integral to data warehousing and data is often extracted on a scheduled basis in bulk and transported to the data warehouse. Typically, all the data in the data warehouse is refreshed with data extracted from the source system. However, an entire refresh involves the extraction and transportation of huge volumes of data and is very expensive in both resources and time. This is where Change Data Capture plays an important role by replicating data in a big data lake with incremental computing updates without modifying or slowing down source systems.

This makes it easy to structure and organize change data from enterprise databases to provide instant insights. Gathr allows you to ingest, blend, and process high velocity big data streams as they arrive, run machine learning models, and train and refresh models in real-time or in batch mode. There are two parts to a Change Data Capture solution with Gathr :

ETL vs ELT: Which data integration practice is right for you

Posted on by Saurabh Dutta

ETL and ELT are common data integration practices that are extensively used in data science and business intelligence. While leading enterprises use both approaches, you might be wondering what is the difference between ETL and ELT?

Find out which data integration practice is right for you in this head-to-head ETL vs ELT comparison.

These 3 Trends will shape the data and analytics landscape in 2022

Posted on by PC Kiran Kumar

In 2021, we witnessed an explosion in cloud adoption as enterprises continued to focus on digital transformation in response to the global pandemic. Many companies tested the waters by moving data, analytics, and business use cases to the cloud for the first time. Both cloud adoption and migration will continue to gain momentum in 2022 across industries, with DataOps, 5G, and edge analytics playing key roles in the digital transformation journey. Here’s what we can look forward to in 2022.

Self-service ingestion: The key to creating a unified, scalable, cloud data lake

Posted on by PC Kiran Kumar

Enterprises are increasingly leveraging cloud-based data lakes to run large-scale analytics workloads and tap data-driven insights for better decision making. Cloud-based data lakes offer unmatched elasticity and scalability, enabling businesses to save costs and improve time-to-market.

The first step in creating a data lake on a cloud platform is ingestion, yet this is often given low priority when an enterprise enhances its technology. It’s only when the number of data feeds from multiple sources starts increasing exponentially that IT teams hit the panic button as they realize they are unable to maintain and manage the input.

Self-service ingestion can help enterprises overcome these challenges and unlock the full potential of their data lakes on the cloud. Here are a few of the benefits.

Bloor research 2018: Gathr a key challenger in streaming analytics

Posted on by Daniel Howard

“Gathr provides an excellent solution for streaming analytics that combines the strengths of open source with the reliability, manageability, and support of an enterprise solution” – reads the Report.

Gathr is going places. And we can’t be more excited about it.

Bloor Research recognized Gathr as one of the key challengers to the streaming analytics platforms available in the market. The report evaluated the top streaming analytics platforms like IBM, Software AG, SAS, SQLStream, and DataTorrent using various metrics. The table below shows the rating received by Gathr.

Gathr rating on various parameters by Bloor Research

“One of the major strengths of Gathr: the ability to interact with a multitude of different types of analytics (event, batch, micro-batch) and streaming engines (Spark, Storm, Flink) using the same interface and applications.” – Daniel Howard, Bloor Research

Market map of streaming analytics platform

Source: https://www.bloorresearch.com/technology/streaming-analytics-platforms/

The report validates our focus on the use of popular open source big data technologies such as Apache Spark and Apache Storm for real-time data insight. Gathr has also been recognized by Forrester, Aragon Research, Gartner, and Datanami in the past.

With a powerful visual IDE and 150+ drag-and-drop Apache Spark operators, Gathr can build and run Apache Spark based big data, stream processing, and machine learning applications up-to 10x faster than hand coding.

The platform provides end-to-end, 360-degree data processing, including ingestion, cleansing, transformation, blending, loading and visualization (via real-time dashboards), and analytics.

You can also sign up here for your free trial of Gathr.

Detect and prevent insider threats with real-time data processing and machine learning

Posted on by Amit Assudani

Insider threats are one of the most significant cybersecurity risks to banks today. These threats are becoming more frequent, more difficult to detect, and more complicated to prevent. PwC’s 2018 Global Economic Crime and Fraud Survey reveals that people inside the organization commit 52% of all frauds. Information security breaches originating within a bank can include employees mishandling user credentials and account data, lack of system controls, responding to phishing emails, or regulatory violations.
Ignoring any internal security breach poses as much risk as an external threat such as hacking, especially in a highly regulated industry like banking. Some of the dangers of insider threats in the banking and financial industry include:

  • Exposing the PII information of the customers
  • Jeopardized customer relationship
  • Fraud
  • Loss of intellectual property
  • Disruption to critical infrastructure
  • Monetary loss
  • Regulatory failure
  • De-stabilized cyber assets of financial institutions

Identifying and fighting insider threats requires the capability to detect anomalous user behavior immediately and accurately. This detection presents its own set of challenges such as appropriately defining what is normal or malicious behavior and setting automated preventive controls to curb predicted threats.

Machine learning-based real-time threat detection for banks

Posted on by Punit Shah

The business impact of the COVID-19 pandemic continues to unfold worldwide for the financial services industry. The “new normal” has not only given rise to unprecedented operational challenges, but also provided fertile ground for hackers and threat actors to take advantage of increased vulnerabilities.

In June 2020, the Internet Crime Complaint Center at the FBI reported a 75% rise in daily digital crime since the start of stay-at-home restrictions. These cyber-crimes are not only becoming more frequent, but also more difficult to detect and more complicated to prevent. Financial institutions like banks that run hundreds of sensitive customer-facing applications are at extremely high risk.

Why modernizing ETL is imperative for massive scale, real-time data processing

Posted on by Amit Assudani

During the past few years, a sea change occurred in the way enterprises acquire, process, and consume data. The exponential surge in the number of data sources and customer interactions fueled a major paradigm shift, with real-time stream processing and cloud technologies emerging as the backbone of intelligent decision making. This is driving businesses to re-look at traditional extract, transform, and load (ETL) platforms used to integrate data from multiple sources into a single repository. This article explores the need for ETL modernization and provides insights for evaluating ETL platforms and ensuring a seamless modernization journey.

Take your GenAI ideas to production today
Schedule a demo Start free trial