Message Broker

Data Storage and Sources
Updated on:
September 7, 2023
Glossary
/
Message Broker

What is a Message Broker

A message broker is software that enables asynchronous messaging and communication between distributed applications and microservices. It serves as an intermediary that decouples message producers from consumers.

Message brokers receive, temporarily store, and route messages using predefined protocols like AMQP or MQTT. This facilitates resilient delivery even with intermittent connectivity. Message brokers form a core part of modern event-driven architectures. They are commonly used with time-series databases and data warehouses.

What does it do/how does it work?

A message broker manages message delivery, transformation, and routing between applications. Producers send messages to the broker, which buffers them in a queue. It then delivers the messages to target consumer applications based on rules and mappings.

The broker handles connection management, message persistence, retries for reliability, and other complex tasks. It can route, transform, or split messages based on business logic. Message brokers typically support common protocols like JMS, AMQP, STOMP for interoperability.

Why is it important? Where is it used?

Message brokers allow building large-scale distributed systems by decoupling components. This enables independent scaling and resilience. Buffering aids with surges in volumes. Routing provides flexibility.

Use cases include automated workflows, real-time data pipelines, IoT systems, service integration, and microservices. Message brokers are crucial in finance, e-commerce, transportation, and other domains dealing with high volumes of events and data. They integrate systems and enable complex architectures.

FAQ

What are the main capabilities of a message broker?

Message brokers provide asynchronous messaging capabilities to enable decoupled integration between applications and services. They act as intermediaries that handle routing, buffering, persistence, translation, and security for messages between endpoints. Key capabilities include:

  • Message transport with guaranteed delivery.
  • Message queueing, buffering, and persistence.
  • Routing messages to appropriate destinations.
  • Translating between message formats, applying simple transformations to message content.
  • Handling security, auditing, and access controls.

When should you use a message broker?

Message brokers allow asynchronous event-driven communication between applications and can simplify integration between disparate systems. They are well suited for:

  • When you need to decouple applications and/or services and communicate asynchronously.
  • To integrate heterogeneous systems with different message formats.
  • To handle spikes in message loads with queueing and buffering.

What are key challenges with message brokers?

While simplifying integration, message brokers still need to handle inherent complexities around scale, reliability, and security:

  • Complexity of distributed systems.
  • Message ordering, duplicates, and loss scenarios.
  • Security, access control, and compliance management.
  • Performance tuning and optimization for high-throughput and/or low-latency use cases.

What are examples of popular message brokers?

References


Related Entries

Time-series Database (”TSDB”)

A time-series database (TSDB) is a database engineered and optimized for handling time-series data, where each data point contains a timestamp.

Read more ->

Fresh Perspectives And Stories
From Our Team

Modern Data Stack and the Data Chasm: Part 1
September 11, 2023
|
Company

The data ecosystem is rapidly expanding and fragmenting, posing integration challenges industry-wide. Many companies fall into a "data chasm", needing to abruptly scale their tools from 2-4 to 15-20, exacerbating complexity. Some organizations pioneered methodologies to cross this chasm and extract value. How can others navigate this data chasm?

Running Windowing Queries in Stream Processing
August 15, 2023
|
Engineering

Windowing queries in stream processing play a pivotal role in handling time-series data. This post unravels how to harness streaming-friendly window functions in queries with just using ANSI-SQL, emphasizing the importance of ordering for achieving optimal results in streaming datasets.

Sliding Window Hash Join: Efficiently Joining Infinite Streams with Order Preservation
July 28, 2023
|
Engineering

The Sliding Window Hash Join (SWHJ) algorithm joins potentially infinite streams while preserving the order by building hash tables incrementally, storing only relevant rows from the build side that fall within a sliding window, allowing efficient processing of streams without materializing all data.

We're not out yet, but we can keep you updated!