One of the biggest challenges developers run into when coding large, scalable systems is automating batch processes and distributing workloads to optimize compute resource usage. More simply, intra-application and inter-system communications tend to become a bottleneck that affect the user experience, and there is no easy way to get around it. Well … There *was* no easy way around it.
Meet SoftLayer Message Queue.
As the name would suggest, Message Queue allows you to create one or more “queues” or containers which contain “messages” — strings of text that you can assign attributes to. The queues pass along messages in first-in-first-out order, and in doing so, they allow for parallel processing of high-volume workflows.
That all sounds pretty complex and “out there,” but you might be surprised to learn that you’re probably using a form of message queuing right now. Message queuing allows for discrete threads or applications to share information with one another without needing to be directly integrated or even operating concurrently. That functionality is at the heart of many of the most common operating systems and applications on the market.
What does it mean in a cloud computing context? Well, Message Queue facilitates more efficient interaction between different pieces of your application or independent software systems. The easiest way demonstrate how that happens is by sharing a quick example:
Creating a Video-Sharing Site
Let’s say we have a mobile application providing the ability to upload video content to your website: sharevideoswith.phil. The problem we have is that our webserver and CMS can only share videos in a specific format from a specific location on a CDN. Transcoding the videos on the mobile device before it uploads proves to be far too taxing, what with all of the games left to complete from the last Humble Bundle release. Having the videos transcoded on our webserver would require a lot of time/funds/patience/knowledge, and we don’t want to add infrastructure to our deployment for transcoding app servers, so we’re faced with a conundrum. A conundrum that’s pretty easily answered with Message Queue and SoftLayer’s (free) video transcoding service.
What We Need
- Our Video Site
- The SoftLayer API Transcoding Service
- SoftLayer Object Storage
- A “New Videos” Container
- A “Transcoded Videos” Container with CDN Enabled
- SoftLayer Message Queue
- “New Videos” Queue
- “Transcoding Jobs” Queue
The Process
- Your user uploads the video to sharevideoswith.phil. Your web app creates a page for the video and populates the content with a “processing” message.
- The web application saves the video file into the “New Vidoes” container on object storage.
- When the video is saved into that container, it creates a new message in the “New Videos” message queue with the video file name as the body.
- From here, we have two worker functions. These workers work independently of each other and can be run at any comfortable interval via cron or any scheduling agent:
Worker Two: Looks for messages in the “Transcoding Jobs” queue. If a message is found, Worker Two checks if the transcoding job is complete. If not, it does nothing with the message, and that message is be placed back into the queue for the next Worker Two to pick up and check. When Worker Two finds a completed job, the newly-transcoded video is pushed to the “Transcoded Videos” container on object storage, and Worker Two updates the page our web app created for the video to display an embedded media player using the CDN location for our transcoded video on object storage.
Each step in the process is handled by an independent component. This allows us to scale or substitute each piece as necessary without needing to refactor the other portions. As long as each piece receives and sends the expected message, its colleague components will keep doing their jobs.
Video transcoding is a simple use-case that shows some of the capabilities of Message Queue. If you check out the Message Queue page on our website, you can see a few other examples — from online banking to real-time stock, score and weather services.
Message Queue leverages Cloudant as the highly scalable low latency data layer for storing and distributing messages, and SoftLayer customers get their first 100,000 messages free every month (with additional messages priced at $0.01 for every 10,000).
What are you waiting for? Go get started with Message Queue!
-Phil (@SoftLayerDevs)