Rsyslog and ZeroMQ

As we get ready for the first deployments of our big data streaming platform, we’re pleased to announce another open source contribution. As we mentioned in our previous post, we’re big fans of Rsyslog. Well, we also have a lot of affection for a complementary component, ZeroMQ. So, let’s talk about Rsyslog and ZeroMQ. As many probably know, ZeroMQ is a mature open-source high-speed message library, currently supporting bindings in about 30 languages. We use ZeroMQ as a component of our real-time data streaming solutions.

For any project using ZMQ, the ability to utilize centralized logging via syslog is desirable. We have created input and output plugins for Rsyslog that allow ZMQ messages to be put into Rsyslog, or output from Rsyslog. We built our plugins using the czmq library (https://github.com/zeromq/czmq), which sits atop the ZMQ API. This enabled us to support the latest version of ZMQ (3.1), and the older stable version (2.1). This library was chosen as it hides the differences in the ZMQ 2.1 vs 3.1 API, and allows our plugin to work in both environments. Prior work (https://github.com/aggregateknowledge/rsyslog-zeromq) was written directly on top of the ZMQ API, and doesn’t support the current version of ZMQ. The output plugin supports Rsyslog V6 config format.

So, what do we allow you to do with the plugins?

  • BIND or CONNECT
  • Specify any number of worker threads
  • Specify thread affinities
  • Specify any zsockopt option
  • Input plugin supports SUB, PULL and XSUB
  • Output plugin supports PUB, PUSH and XPUB

Hammer on it! We want to see these plugins put through their paces. We are doing the same, and plan to continue to improve them over time.