Sunday, December 23, 2012

ZeroMQ Internal Architecture

Does Zeromq simply retry every so many milliseconds for the endpoint during disconnected operation?
Answer: Yes I believe so with exponential backoff.

Two cool things I got out of this article.

1. ØMQ's concurency model may a bit confusing at first. The reason is that we eat our own dogfood and use message passing to achieve concurrency and internal scalability. Thus, even though ØMQ is a multithreaded application you won't find mutexes, condition variables or semaphores meant to orchestrate the parallel processing. Instead, each object will live in its own thread and no other thread will ever touch it (that's why mutexes are not needed). Other threads will communicate with the object by sending it messages (called 'commands' to distinguish them for user-level ØMQ messages). Same way the object can speak to other objects — potentially running in different threads — by sending them 'commands'.

2. The requirements for messages are rather complex. The main reason for complexity is that the implementation should be very efficient for both very small and very large messages. The requirements are as follows:
For very small messages it's cheaper to copy the message than keep the shared data on the heap. These messages thus have no associated buffer and the data are stored directly in the zmq_msg_t structure — presumably on the stack. This has huge impact on performance as it almost entirely avoids need to do memory allocations/deallocations.
When using inproc transport, message should never be copied. Thus, the buffer sent in one thread, should be received in the other thread and get deallocated there.
Messages should support reference counting. Thus, if a message is published to many different TCP connections, all the sending I/O threads access the same buffer rather then copying the buffer for rach I/O thread or TCP connection.
The same trick should be accessible to user, so that he can send same physical buffer to multiple ØMQ sockets without need to copy the content.
User should be able to send buffer that was allocated by application-specific allocation mechanism without need to copy the data. This is especially important with legacy applications which allocate large amounts of data.

No comments:

Post a Comment