The Appeal and Controversy of ZeroMQ

In 2010, ZeroMQ was still a little-known networking library. Since then it’s grown to define a new set of patterns for writing scalable software. Recent projects like Axon, JeroMQ, NetMQ, NullMQ, Crossroads, and Nanomsg are inspired by, or directly copied from ZeroMQ.

What makes ZeroMQ so popular and controversial? I’ve watched and been part of this story for a long time. My little software firm, iMatix, has always built distributed software, as far back as 1990. We know how difficult and expensive it used to be to get code talking to code. In 2004 we started designing AMQP for JPMorganChase, and building the OpenAMQ messaging server. In 2007 we felt that wasn’t going to fly, and began building a brand new open source stack focused on performance, simplicity, and decentralization.

ZeroMQ’s appeal and controversy comes from how cheap it makes this task. It’s difficult to believe the first time you see it, and many people don’t believe it. It can be offensive to people who’ve spent years learning technologies, or millions of dollars to buy them, to hear, “well, actually there’s this software library that does all of this, faster, better, and cheaper by ten to a hundred times”.

Scalability is about cost: if you can bring the cost of something down, you can do more, or make more of it. Making distributed software was expensive before ZeroMQ came along. It made connectivity cheap by solving a bunch of the hardest technical problems–such as sending messages asynchronously in the background using a dedicated I/O thread without locks, or distributed a message to specific subscribers–and wrapping the results up in a simple API that looked a bit like “classic” BSD sockets.

Smart engineers and businessmen are learning and using ZeroMQ to build products like Storm, dotCloud, Mongrel2, ZeroRPC, SaltStack. When you make a new software product, you don’t really need to know how you’re going to make money from it if you have grossly inefficient competitors to attack. When MySQL got initial funding, their story to investors was, “There’s a $100 billion market for databases. We intend to reduce this to $10 billion, and take twenty percent of that”. People who still aren’t using ZeroMQ are paying too much to build distributed or multithreaded applications.

So what are ZeroMQ’s strengths and weaknesses? Let me start with its main weakness: it doesn’t do security. Like TCP or any other networking protocol, this has to be layered on top. It’s still something we’re learning how to do properly. Its strengths are its patterns and transports, which you use like building blocks to connect threads, processes, or boxes. These patterns have become a sort of common language for distributed systems.

The other strengths are, I think, its community and wide support for languages and platforms. When you read the ZeroMQ book you’ll see that the examples are in C. But the same patterns work in over 40 languages. I also really like using ZeroMQ to do multithreading. That used to be the nastiest part of programming but when you switch from locks and mutexes to messages, it’s like switching from a cold shower to a nice warm one.

Messaging is a difficult problem to understand, let alone solve.

Connecting two pieces, OK. But connecting five thousand? It’s mind-boggling. Thus, many people who are getting into messaging will use a product like RabbitMQ (which was the second AMQP product after OpenAMQ). RabbitMQ makes big problems simpler by putting a broker in the middle and mandating, “This is the centre of everything”. It’s like saying, “The centre of the Internet is Facebook”.  Simpler, if not totally accurate. RabbitMQ is a great piece of software but I’d never use it. It would be like using Subversion again when I’ve understood how Git works.

But the real reason why I recommend you download and try ZeroMQ, if you didn’t already, is the sheer pleasure of writing code using it. It takes a few days of coding and then you will experience a kind of enlightenment, and see new dimensions in your software. Not only can you make much more ambitious architectures, but you can make them easily, and the results are fast and solid.

For me, after working hard on OpenAMQ for years, and then having to kill the project as AMQP went off the rails, ZeroMQ saved me from professional burnout. It made coding fun again. It made me giggle as I saw 50-line C programs do stuff that used to be impossible with thousands of lines. I hope the book carries that enthusiasm forward and that you enjoy using and contributing to ZeroMQ as much as I have.

Pieter Hintjens started his first business making video games 30 years ago and has been building software products since then. Believing that, “the real physics of software is the physics of people”, he focuses on building communities through “Social Architecture”, writing, and building his businesses. He is the lead maintainer of ZeroMQ, and CEO of iMatix Corporation. Pieter speaks English, French, Dutch, and pieces of a dozen other languages. He plays with a West African drum group in Brussels, is a certified pistol instructor in Texas, and is currently learning Korean. He lives with his lovely family in Brussels, Belgium but travels extensively for work.