Fresh Perspectives And Stories
From Our Team

Discover how we're achieving our vision of unifying data in real-time by exploring our team's thoughts, ideas, and experiences.

Engineering

Running Windowing Queries in Stream Processing
August 15, 2023
|
Engineering

Running Windowing Queries in Stream Processing

Windowing queries in stream processing play a pivotal role in handling time-series data. This post unravels how to harness streaming-friendly window functions in queries with just using ANSI-SQL, emphasizing the importance of ordering for achieving optimal results in streaming datasets.

Sliding Window Hash Join: Efficiently Joining Infinite Streams with Order Preservation
July 28, 2023
|
Engineering

Sliding Window Hash Join: Efficiently Joining Infinite Streams with Order Preservation

The Sliding Window Hash Join (SWHJ) algorithm joins potentially infinite streams while preserving the order by building hash tables incrementally, storing only relevant rows from the build side that fall within a sliding window, allowing efficient processing of streams without materializing all data.

Probabilistic Data Structures in Streaming: Count-Min Sketch
July 13, 2023
|
Engineering

Probabilistic Data Structures in Streaming: Count-Min Sketch

The Count-Min Sketch uses hash functions to map streamed items into a 2D counter array. When processing the stream, items are hashed to incremented counters, frequencies are est. by taking the min count across rows for an item's hashes.

General-purpose Stream Joins via Pruning Symmetric Hash Joins
February 20, 2023
|
Engineering

General-purpose Stream Joins via Pruning Symmetric Hash Joins

Sliding window join for stream processing brings Datafusion a step closer to unified data processing. Find out how to efficiently join the streams with less memory usage and how to intelligently buffer both join sides.

Company

Modern Data Stack and the Data Chasm: Part 1
September 11, 2023
|
Company

Modern Data Stack and the Data Chasm: Part 1

The data ecosystem is rapidly expanding and fragmenting, posing integration challenges industry-wide. Many companies fall into a "data chasm", needing to abruptly scale their tools from 2-4 to 15-20, exacerbating complexity. Some organizations pioneered methodologies to cross this chasm and extract value. How can others navigate this data chasm?

AI / ML: The Race for Specialized Electricity Supremacy
April 5, 2023
|
Company

AI / ML: The Race for Specialized Electricity Supremacy

This blog post explores the AI/ML landscape, comparing it to a gold rush where the focus is on providing "specialized electricity" in the form of computing, storage, and networking resources.

Next Frontier: Action-Capable Intelligent Agents
February 13, 2023
|
Company

Next Frontier: Action-Capable Intelligent Agents

The world of AI and data is undergoing a rapid transformation. Enabling technologies are maturing to a level where we should be able to deploy action-capable, autonomous intelligent agents at scale. But what will it take to make this a reality?

We're not out yet, but we can keep you updated!