We will build an event-driven, fully functional workflow in this tutorial. KombuWe treat messaging as an architectural core capability. As we set up exchanges and routing keys for background workers, concurrent producers, background worker, etc., it allows us to view a real distributed application. We see that asynchronous processing and message routing give production microservices the power they need every day. See the FULL CODES.
The!pip installer kombu
Threading, import
import time
import logging
import uuid
import datetime
Import sys
Import of kombu Connection, Exchanges, Queues, Producers, Consumers
ConsumerMixins imported from kombu.mixins
logging.basicConfig(
level=logging.INFO,
format="%(message)s",
handlers=[logging.StreamHandler(sys.stdout)],
force=True
)
logger = logging.getLogger(__name__)
BROKER_URL = "memory://localhost/"
Installing Kombu and importing the dependencies is our first step. We then configure logging to be able to see each message as it flows through the system. Set the URL of our in-memory Broker, which allows us to run all local processes in Colab and not need RabbitMQ. The setup is the basis for our distributed message workflow. See the FULL CODES.
media_exchange = Exchange('media_exchange', type="topic", durable=True)
task_queues = [
Queue('video_queue', media_exchange, routing_key='video.#'),
Queue('audit_queue', media_exchange, routing_key='#'),
]
Use wildcard patterns to define topic exchanges that allow messages to be routed flexibly. In addition, we set up two queues. One is dedicated to tasks related to video and another to audits that monitors everything. We can control messages in the system using topic routing. Click here to see the FULL CODES.
class Worker(ConsumerMixin):
def __init__(self, connection, queues):
"Self-connection" = "connection
self.queues = queues
False: self.should_stop
def get_consumers(self, Consumer, channel):
You can return to your original language by clicking here. [
Consumer(queues=self.queues,
callbacks=[self.on_message],
accept=['json'],
prefetch_count=1)
]
def on_message(self, body, message):
routing_key = message.delivery_info['routing_key']
payload_id = body.get('id', 'unknown')
logger.info(f"n⚡ RECEIVED MSG via key: [{routing_key}]")
logger.info(f" Payload ID: {payload_id}")
try:
If you have a routing_key that contains 'video:
self.process_video(body)
Elif "audit", in the routing_key
logger.info(" 🔍 [Audit] Logging event...")
message.ack()
logger.info(f" ✅ ACKNOWLEDGED")
Except Exception As e.
logger.error(f" ❌ ERROR: {e}")
def process_video(self, body):
logger.info(" ⚙️ [Processor] Transcoding video (Simulating work...)")
time.sleep(0.5)
We create a worker that runs in the background using Kombu ConsumerMixin. We inspect the route key in the callback for the message, then invoke the processing function and acknowledge it. The worker architecture allows us to consume messages in a clean and controlled manner. Take a look at the FULL CODES.
def publish_messages(connection):
Connector: Producer = Producer
tasks = [
('video.upload', {'file': 'movie.mp4'}),
('user.login', {'user': 'admin'}),
]
logger.info("n🚀 PRODUCER: Starting to publish messages...")
For r_key data, see tasks
['id'] = str(uuid.uuid4())[:8]
logger.info(f"📤 SENDING: {r_key} -> {data}")
producer.publish(
data,
exchange=media_exchange,
routing_key=r_key,
serializer="json"
)
time.sleep(1.5)
logger.info("🏁 PRODUCER: Done.")
Now we build a Producer that will send structured JSON data into the exchange using different routing keys. Each event is assigned a unique ID and we observe the way they are sent to different queues. It is similar to real-world publishing of microservices, where consumers and producers are separated. Take a look at FULL CODES.
def run_example():
with Connection(BROKER_URL) as conn:
worker = Conn(task_queues, Worker)
worker_thread = threading.Thread(target=worker.run)
worker_thread.daemon is True
worker_thread.start()
logger.info("✅ SYSTEM: Worker thread started.")
time.sleep(1)
try:
publish_messages(conn)
time.sleep(2)
Other than KeyboardInterrupt
pass
finally:
If worker.should_stop is True, then the answer to this question will be:
logger.info("n👋 SYSTEM: Execution complete.")
If the __name__ equals "__main__":
run_example()
The worker is started in a thread in the background and the producer is terminated in the thread in the foreground. The structure we have created allows us to run a distributed mini-system in Colab. By observing the logs, we see messages published → routed → consumed → acknowledged, completing the full event-processing lifecycle.
As a conclusion, we created a distributed, dynamic task-routing pipe that can process real-time events in lucidity and with accuracy. Kombu allows us to abstract away the complexity associated with messaging systems, while still allowing for fine-grained control of routing, consumption and worker concurrency. We gained an appreciation of the elegant design that event-driven systems provide as we watched messages flow from producer, to exchange, to queue, to worker.
Take a look at the FULL CODES. Check out our website to learn more. GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Join our Facebook group! 100k+ ML SubReddit Subscribe Now our Newsletter.
Asif Razzaq, CEO of Marktechpost Media Inc. is a visionary engineer and entrepreneur who is dedicated to harnessing Artificial Intelligence’s potential for the social good. Marktechpost was his most recent venture. This platform, a Media Platform for Artificial Intelligence, is renowned for its comprehensive coverage of news on machine learning and deep-learning, both technical and understandable to a broad audience. Over 2 million views per month are a testament to the platform’s popularity.

