We’re all familiar with sending a message using a tool like WhatsApp. We can see the message we’ve sent, receive a notification that it’s been delivered, and then wait for the ticks confirming that the recipient has seen it. Eventually, a reply might come through.
The important thing is that, while all of this is happening, we can get on with whatever else we want to do – or even send a message to someone else.
Systems can communicate in a similar way (although they tend to have fewer anxieties about whether their message has been read but not replied to!).
A real-world example
One of our customers recently implemented a new system to capture information about people accessing their services, along with details about the work those customers wanted carried out.
However, the customer also had a whole host of historic reporting and linked processes that relied on their existing in-house database. Rewriting the entire technology stack was considered too high-risk, so instead, the supplier of the new system suggested setting up a messaging process to notify the existing internal database whenever new information was available.
The Infotex Systems team were asked to help, working alongside the customer’s internal data team, to build the integration that would process these messages and send the relevant information to the appropriate internal systems.
Using Microsoft Azure Service Bus
The chosen technology for the messaging was Microsoft Azure Service Bus.
Azure Service Bus is designed to receive messages from a remote system and store them until a consumer, usually a piece of application code, reads and processes them.
The sending system can simply send a message to Service Bus without having to wait for the receiving system to acknowledge or process it. This allows the sending system to continue with its core tasks rather than waiting for a response.
Azure Service Bus can be configured in a couple of different ways. Queues allow each message to be processed by a single consumer, while Topics and Subscriptions allow a message to be made available to multiple consumers, each of which can take its own action.
In our example, there were two main destinations for the data. We therefore configured a Topic with two Subscriptions. Each destination had its own consumer, allowing the two systems to read and process messages independently.
This separation was important because it meant that if something went wrong while processing messages for one destination, the other could continue operating unaffected.
Processing the messages
We assisted with configuring the Subscriptions to ensure that the right types of messages were delivered to the right destinations.
We then wrote an Azure Function App for each of the internal systems. These functions would:
- Read a message from the relevant queue.
- Transform the message into data that the internal system could accept.
- Update the internal system’s database with the new information.
The customer’s internal data team handled the database behaviour itself.
What happens when something goes wrong?
An important part of creating a consumer is deciding what should happen if a message cannot be processed successfully.
We implemented a retry approach. If a failure was caused by a temporary or transient issue, the message processing would be attempted again.
If the message could not be successfully processed after ten attempts, it would be moved to another part of Azure Service Bus called the Dead Letter Queue. At the same time, the support team would be notified that they needed to investigate the reason for the failure.
Once the issue had been resolved, the support team could use the built-in Service Bus functionality to resend the message for processing.
This approach means that a temporary problem doesn’t immediately cause data to be lost, while messages that genuinely need investigation are separated out for the support team to deal with.
When is messaging useful?
From an integrations perspective, messaging tends to be particularly useful in high-volume, enterprise-level applications.
The “send and forget” approach allows the sending system to continue operating while receiving systems process messages in their own time. It also provides a degree of resilience: different receiving systems can process messages independently, and failures can be isolated and dealt with without necessarily affecting the rest of the integration.
Messaging can also make integrations easier to evolve over time. New consumers can be added without requiring significant changes to the system that originally sent the message, while existing consumers can be updated or replaced independently. This creates a more loosely coupled architecture, helping systems remain flexible as business requirements change.
There are some trade-offs. Messaging introduces additional components to configure, monitor and support, and because processing may happen and because processing may happen some time after a message is sent, the receiving systems won’t always be instantly up to date. For most business processes a short delay is perfectly acceptable, but it’s worth considering if a user expects to see their changes reflected immediately. Messages can also arrive out of order or occasionally be delivered more than once, so consumers need to be designed to handle this gracefully. Finally, tracing a problem through several separate components can be more involved than debugging a single, direct connection, which makes good logging and alerting essential.
For the right scenario, though, these trade-offs are well worth it. Where reliability, scale and flexibility matter, messaging provides a robust foundation that allows systems to work together without being tightly bound to one another.
