I personally like the fact that the codebase (up until a few months ago at least, when I last looked) is very clean and very understandable.
This probably has to do with both the transition from C++ to C, and the fact that Martin (the author) has a better idea of what he wants to make. In other words, he is (painfully) aware of the errors he made in his ZeroMQ and is doing it right from the first time in nanomsg. This leads to clearer and more consise code. The reduction of strange bugs and perhaps better performance adds a lot of icing on the cake as well. I look forward to using nanomsg in my future projects!
I've been totally conversed back to "C" from "C++" after working on a port of Metal Gear Solid to PC in 2000. I've had no problems understanding and reading through the code the Konami developers have written, it was using some kind of generic OO system in "C" with TCL-like passing of arguments (object* obj, int argc, const char*argv[]) - and passing text messages. Yes that was okay even on Playstation1 I guess.
Now I still prefer C++ at work for certain things as I'm simply too lazy to do it correctly in "C" - give me STL, Qt, any UI framework written in there, and it's easier to use. But for core/system libraries (zlib, png, jpeg, tiff, zmq, libpq, etc.) I'd rather have it in "C", at least on the interface part, but I've seen what hiding of C++ or .NET can do with a crazy debugger :)
It's important that the author of nanomsg is also the author of ZMQ. One can see nanomsg as something built on the lessons he learned from ZMQ, assuming he can avoid second-system syndrome.
Traditional message queues are actually composed of two rather different subsystem: the queue itself (basically a database) and the networking.
I would say we are on the path of separating the two. ZeroMQ does away with the centralised server holding all the queues. It still has local queues though.
nanomsg goes even further and gets rid of queues entirely. What's left it networking buffers, typically implemented at L4 of the network stack.
Nope. It's a platform for delivering packaged and reusable distributed algorithms, such as request/reply or publish/subscribe. The algorithms handle the issues like message routing, scaling up, handling partial failure etc.
Well, that's what I meant--technically, BSD sockets are themselves supposed to be "a platform for delivering packaged and reusable distributed algorithms".
In a friendly, ideal environment (like the Internet of the 1980s), req/rep is "just" TCP/SCTP unicast, pub/sub is "just" UDP multicast, and so on. Issues like "message routing [and] scaling up" are supposed to be handled on the link layer (i.e. using ARP, BGP, etc.); issues like handling partial failure, on the transport layer (if IP stacks weren't stuck in the 70s, we'd have such a thing as multicast TCP, and "send TCP packet to host set, wait for ACK from all hosts, and retransmit to failed hosts" would simply be its obvious send() system-call semantic.)
We've just screwed the "friendly, ideal environment' pooch by putting not-very-intelligent NATs and firewalls everywhere. So we have to, sadly, reimplement all these nice things on top of the few things we've kept working: TCP and UDP unicast from clients to "business-leased" IPs with open+addressable ports. The necessity of things like this, UPnP, and TURN/STUN/ICE sadden me sometimes, they really do; they're 100% examples of an inner-platform[1] created because software developers have gone on for so long only touching OSI layer 7, and thinking about the rest of the infrastructure as a black box, that now the application layer is the only place you can solve problems, because the infrastructure will no longer transport your packets if they do anything special below it.
So I say, please, look past your abstractions, programmers, and learn what makes the Internet tick! It's worth it to understand what your packets are doing--you might want a property that you can already get by asking something different of a lower layer. Or, even if not, you might just become--as Bret Victor says--very unsatisfied with the state of things. :)
You are oversimplifying it. REQ/REP is not the same as TCP, unlike TCP it works with multiple requesters and multiple repliers. Same with other patterns.
The main distinction between L4 and scalability layer is that L4 works between 2 nodes, scalability between arbitrary number of nodes.
There's the special case of UDP multicast, but the multicast itself feels more like a hack rather than a real L4 protocol.
> it was never meant to be reliable in the sense that messages were preserved in a database at each socket, was it?
No. It's meant to be a fast inter app messaging protocol, it's not a queue, it simply uses some in memory queues internally. It's not a queue, it's an improved socket library, having MQ in the name was clearly a mistake that he's now corrected.
It can be, but it generally isn't, look around the net and you'll see it's pretty clear that most think it's a message queue and it gets thrown into that mix when the topic comes up. People are always having to explain to others that it isn't a queue, that's bad naming of the product.
Now it's in kernel-space (TCP rx/tx buffers). That actually makes it a little more reliable as tx buffers are not necessarily dropped when the application fails.
I don't buy the `it's better because it's C rather than C++`. For example, "Number of memory allocations is drastically reduced as intrusive containers are used instead of C++ STL containers." can be solved in 3 ways:
1. use an intrusive container in C++.
2. Just use a `vector`, faster in most cases, regardless of complexity.
2. Prevent users from having to use Boost or the rest of the C++ nonsense
3. Users can now write trivial language bindings and link at runtime easily.
4. Profit.
~
C++ is a cool bro and all, but honestly the solutions you've listed make it annoying to work with this code outside of C++ on a platform with subgood template support.
Moving a simple utility library like ZMQ to C makes a great deal of sense.
Eventhough the backend for ZeroMQ was written in C++, the front-end, and the main language bindings as it were, are in C. That means it was already trivial to write language bindings.
No, the ZeroMQ author should be writing idiomatic code in whatever language he is using.
In his blog post, he complains that std::list<widget* > causes 2 allocations per insertion.
Fine, because that's not idiomatic C++. The C++ way would be to use std::list<widget>, since there's no need for the extra indirection.
Note that this would only be a good choice if you for some reason really need a list, std::vector is mostly preferred due to its better cache behaviour and requires even fewer allocations.
Secondly, if he needs to remove list items by their address, there's no reason he couldn't use an intrusive list in C++. The article I linked shows an implementation of one, and there's always boost::intrusive::list if you don't want to roll your own (and you shouldn't).
Extra indirection is needed when the objects are non-assignable, e.g. if there's a thread running inside the object, if it owns a fd or similar. std::vector has O(n) complexity for a lot of operations, so it's not an option.
If you can use C++11, this becomes a non-issue with move semantics.
Ownership of internal resources can be transferred, and emplace_back even allows constructing your object in place.
As others said, the O(n) really becomes more like pseudo-O(1) due to cache effects unless you have elements the size of your cache lines (in which case prefetching still helps) or you're only fetching one element at a time and then triggering a context switch (as in a scheduler).
I'm not bashing your library, and if you prefer to use C then that's great, but it's kind of unfair to blame it on the language in the first place.
Well, this is just precious. When the first C++ recommendation turned out to be not applicable, the answer is:
1) He can solve the issue by requiring C++11 (!!)
2) Just rely on cache-effects making it pseudo-O(1), unless
(insert various cases where it might blow).
Sure, he can jump through 200 hoops to use C++, special features, rely on CPU cache behavior, finetune to death, etc -- or, you know, he can use C and be done with it...
Languages are designed for specific purposes. I am claiming that C++ is not the best language for system development. The fact that most OSes are not written in C++ is a good indication of the fact. Still, C++ is great for rapid development & corporate development.
OS X drivers are "a restricted subset of C++"[1] that restricts you from using Exceptions, Multiple inheritance, Templates, and Runtime type information (RTTI). Also, not all drivers use IOKit.
That is one of the problems with C++. In "C" land you also might be put to restrictions - for example don't use malloc()/realloc()/free() but some other functions, or don't use the provided assert() but something else, or no Xxxprintf() functions, etc - e.g. most of them are around function calls, that could be caught at preprocessor, compiling, or even runtime level.
Now C++ comes with new/delete, dynamic_cast<>, throw/catch/etc, RAII, always export "C" interface (to avoid mangling) - While all of these are useful, due to the constraint where the language has to be used they can be forbidden. It's no longer functions that you are restricting from being called (as in "C"), you are forbidding to use parts of the language that are considered by many as vital, and it's basically a different language at this point.
I'd like to know what restrictions cause you not to be able to use RAII.
Even if that's the only C++ feature that a project were to use, I'd still consider it a net gain. Anything that makes your program less susceptible to manual resource management bugs in "goto cleanup"-style C is worth using. This doesn't just include memory leaks, but more insidious problems like unclear ownership of various system resources (sockets, file handles...). Things not even a GC would help you with in a long-running server process.
There are several popular "C" libraries that use them - jpeglib comes to mind, but there are others. Also certain language's runtime environment uses it. There is a way to handle it, but you need to take care.
Even in C, setjmp/longjmp leave all non-volatile automatic storage duration variables on the current stack frame in an indeterminate state if they are modified between the two calls.
To get defined behaviour with regards to destructors, you simply have to make sure that the function calling setjmp doesn't internally use RAII. Call a wrapper function to do the setjmp and you're safe.
Of course it's a whole other question whether using setjmp/longjmp to implement "exception handling in C" is a good idea in the first place.
Unfortunately, the complexity argument is generally bullshit and you really need to profile. It turns out multiple very respected authors have found under a typical work load, `vector` performs very well on a lot of machines.
Since you never traverse the list, the order of elements obviously doesn't matter. In that case, you can just swap the element to remove with the last element.
A vector will still give you better cache-locality and also amortize the number of allocations.
Yes, that's exaclty what ZeroMQ is doing. Then check how the code looks like. It's a mess. Intrusive containers deliver same performance characteristics and the code actually looks clean.
Nothing. But if he is going to write C in C++, then what is the point of writing C++? Especially sticking to C gives substantially increased flexibility in terms of language bindings, embedded usage etc.
I've probably spent more years writing C++ code than C code, but I'm all with him on this - if what you need to do for whatever reason don't need/benefit all that much from C++ when accounting for all constraints, then choosing C as being the lowest common denominator between other languages is a very good choice for a library.
Yes, the world does not end. For small objects and `vector`s, it's probably quicker, because 1) you found it quicker due to cache locality and 2) there were no system calls to release memory.
So, what you say is:
I don't care to implement an efficient solution, because the CPU cache will fix it anyway.
That way of thinking is well known as "The Java way of problem solving" (TM)
That's not what he said, and his solution is not "inefficient." CPUs have no emotions; if it runs faster due to cache, it's simply the better solution.
For the use case required by ZeroMQ, it's faster in all cases, because swap-with-last and shrink is O(1). Sure, if CPUs didn't have caches, a list might be faster.
Do you see where this is going? The C way of solving problems consists of adding one more level of indirection (ever heard of 3-star C programmers?), because it becomes more efficient asymptotically that way. Nevermind the hidden factor of 100-1000 due to pipeline stalls and cache misses.
Even the code in the article makes heavy use of C++ features (templates, resource management) and that's not even talking about the Boost intrusive library that gets mentioned, which is knee-deep in template effluvial...
Never mind that I've seen plenty of C code with wrapping structs for data structures, too. Intrusive vs. non-intrusive is a design choice in both languages.
The point of intrusive lists is to allow access to the link when you provide only the contained object.
The way you're thinking of would theoretically work, but you'd have to make sure the compiler isn't adding fancy padding that would break your pointer manipulation and type casting.
boost::intrusive::list on the other hand, works in a type-safe manner by having your class either inherit a list hook, or provide it as member and then pass the list implementation a member pointer as template parameter so that it can be accessed.
That's one way to look at it, but certainly not the only one. Back in 2007, when he started ZeroMQ, you couldn't build on C++11 features. Furthermore, if you're targeting slightly less common platforms, then C++ runtime support and recent C++ features is always a hit-or-miss issue.
For a library that is at such a basic level of infrastructure (which, if it's stable enough and popular enough deserves to be kernel-level) C++ is probably more trouble than it's worth.
The higher you go app the stack, the more reasonable it becomes to go farther from C (high enough, and even ruby makes sense - though it wouldn't for 0mq/nanomsg)
In C++ it's even easier than in C to safely use stack-based allocation (thanks to RAII) so this it's better because it's C rather than C++ is a huge warning sign.
The purpose of C++ is to provide experts with a toolbox to do anything without any compromise. This is why the language is hard to master and very easy to misuse. It's a two edged sword literraly. Although in C++ one could say it's more like a two-edged chainsaw.
In this case, I would simply say that the author problem is more about how he relates to C++ than about C++ itself.
Saying C++ is abrasive is a valid point, but saying it cannot be as fast as C or it cannot do "something" is stratospherically stupid.
The original post only claims that it makes his life easier, and then makes two further claims about the C implementation without stating you can't achieve the same in C++.
He then links to a post where he goes into more detail, including _explicitly pointing out_ you can do the same in C++, but that he believes that the language encourages you to think in certain ways that negatively affects these factors.
It seems to me that he is stating his reasons for why he believes C is the best choice, and particularly for him, without making any claim that it is objectively better.
In other words, unless I've completely missed something, you are setting up strawmen.
I was referring to his original rant on C++: http://250bpm.com/blog:4 and http://250bpm.com/blog:8. They are pretty misinformed. He claims in C++ the erasing solution is O(n) where it is O(1) in C. This doesn't make any sense of course.
The comments you are referring to where he moderates his position are resulting from the comments he received from the C++ community regarding his incorrect statements.
But then if you have some 3rd party library code using setjmp/longjmp (libjpeg and others) this would no long work with RAII - no unwinding would be done. That is if the your library calls a "callback" which calls one of the said 3rd party libs (or your code) with longjmp
i'm pretty sure the real reason for C is that the author is just more familiar with it (which is a fine point imo)
now there's other things i'd criticize, like nanomsg is a trademark of the company.. so if you repackage it i'm suspecting you have to rename all the stuff, for example.
Why would you be pretty sure he is more familiar with C when most of his past projects were written in C++? Not to mention that he's written two blog posts explaining his reasoning...
I'm not sure the trademark is a big deal, it's probably to prevent misuse like what happened with the Linux trademark (which took a few years to resolve). I doubt he's doing it to make it more annoying to use.
The author appears to care about performance. That is why I'm surprised to see the choice of switching to level-triggered interrupts instead of edge-triggered. This really defeats one of the most important performance improvements of epoll().
See `man epoll' for an excellent explanation of the difference.
The problem is that embedding an edge triggered file descriptor into an event loop that expects level triggered is a HUGE pain in the ass. I wrote two articles on the matter:
"iMatix CEO Pieter Hintjens registered the zeromq.org domain in May 2007 and started the ØMQ project together with Martin Sustrik, who was its architect and lead developer through March 2012.
[..]
In March 2012, due to disagreements over control of the project, some of the original developers forked ØMQ as Crossroads I/O. Martin Sustrik has started nanomsg, a reboot of the ØMQ core library."
Things that are normally valid C++ code that would get you into trouble if you are library (some of them you are static one):
- Globally constructed objects, called before main(). Yes you might say, don't use them, but once you are in C++ land you can't be safe that everything else you've decided to use does not use them.
- Throwing exceptions. This is mostly debugger pain. If you've used (again) without knowledge underlying code/library that uses exceptions (and note, you don't) then you might be inadvertently throwing them to the user.
- RAII does not work with longjmp/setjmp. If you have callback mechanisms, and your client does longjmp on you, this won't work with any RAII done objects.
- C++ runtime. You would be requiring this to be loaded, which might become nuisance in situations where your main application never used C++, you wrote a plugin/shared lib using C++ code, and now there is the requirement. Not a big deal after all.
- C++ runtime again but for MSVC (although it's for the C runtime too to a point). Now while you can make code that is more independent from the "C" runtime for MSVC, it's harder to maintain this for C++ - you might run into issue where mixing dynamically (shared) libraries would have to use different (and even more different through WinSxS) MSVCPxxx.DLL libs (C++ runtime). In all fairness, this is more of a Microsoft mishap, and not caring about backward binary compatibility, but hey it's there.
- Most of all, no matter what you do - you can't expose the really useful C++ things - like stl or qt5 or something else's objects/vectors/strings/maps/etc. Well that's obvious - off course you can, but then you are basically forcing the person using your library to go that route. If your interface is in "C" not only he longer has this pressure, he can use it easier through FFI in plethora of other language/runtimes without writing wrappers, and minimizing the number of precompiled binaries that he has to provide (no linking statically to C++ runtime would turn out bad at the end).
- Case in point - STL's ITERATOR_DEBUG_LEVEL in MSVC. By default it's set to 2, which means make extensive checks of every map<>, vector<>, etc. once they are released. This severely slows down the debug. It's useful, but it's not feature that can be turned on or off. Most of 3rd party libraries, are playing safe and don't do anything special, e.g. they expect ITERATOR_DEBUG_LEVEL=2. But in your production code you might want even your Debug versions to be somewhat faster. Most of your day won't be spent tracking down map<>, vector<>, but other things. As such you go ahead and compile ITERATOR_DEBUG_LEVEL=0, and then .... well you can not link anymore directly to the said 3rd party libs. You either have to precompile them, which might be fine, until you are given Middleware that is not compiled this way. And in all cases it's very important that all these libs use at the end just one runtime, not several (as it happens when you have tons of MSVCRxxx.dll/MSVCPxxx.dll in your running image).
- And there is more. For example for statically build
libraries, global constructors might be lost, if there is no symbol referring to them used by the main application. So you might lose some important initialization step.
I agree with your points, but the setjmp/longjmp issue is largely orthogonal to RAII.
If you've got code in your library that you need to run after invoking a user callback (like free or mutex unlock) and the user does a longjmp on you, it doesn't really matter if the code you want executed is done automatically from a destructor (as in RAII) or manually. If you want to allow the user to do a longjmp from a callback you have to write the code carefully to handle this, be it C or C++.
I just saw someone use a globally constructed object to instantiate some global variables and it seemed like a neat solution, but I don't really know C++. What causes it to go wrong?
It depends on compiler / librarian / linker. I'm talking from experience here, not from deep knowledge or understanding the problem.
So basically what happens, is that if you have a lone variable globally instantiated, but no one refers to it, it might be stripped from the final linking, and as such any constructors/destructors associated with it would not be called.
Now if you are linking directly to the .obj file where the variable is declared it most likely would be put into account, but if the .obj is in a .lib (.a) then it might not.
Again, do some experiments, try it out.
There is another nastiness with globally initialized variables, that one should not expet some kind of order when the things are initialized. We've had code in the past (Dreamcast) where globally initialized variables, which were assigned the values of other globally initialized variables and then some operation on them would not always work: e.g. this at the top level:
float x = 10;
float y = x + 20;
It worked most of the time, and then if someone started moving variables to other object files, it stopped.
I am hoping for some kind of solution for the DoS problems present in ZeroMQ (and probably in nanomsg as well) when used in an Internet facing application. For example the possibility to subscribe to an unlimited number of prefixes in the PUBSUB model leads to memory explosion on the server side.
I'm curious about the differences mentioned in the threading section, "In nanomsg the objects are not tightly bound to particular threads and thus these problems don't exist."
Does this mean that threading is still used/required, or could nanomsg use non-blocking techniques?
Threading is required as there's processing going in the background. Not sure what you mean by non-blocking techniques. Lock-free algorithms maybe? These are used in ZeroMQ as well.
They probably mean async/evented techniques. For example, provide a function to get a FD that select()s as readable when stuff needs background processing, and a do_background_step() method to call, so it can be hooked into a single threaded event loop.
Yuck. Why would anyone want that kind of stuff. nanomsg does all its processing in the background threads, same way as TCP does. I doubt anyone would ask to have to call do_background_processing_step() to make TCP work.
Also, that kind of thing compromises internal working of the library, if the user doesn't call do_background_step() fast enough. Timers may be delayed that way etc.
In one of the linked blogs, he implies that Windows OS uses MIT licensed software.
Is this true? Do windows 7/8 have any lists of open source software projects that they incorporated into the operative system? It sound as a really interesting read if such was the case.
Microsoft's DNS server was forked from BIND 4, Merchant Server was an acqusition that run on Python in it's first incarnation and as dalke mentions various networking infrastructure came from the BSDs.
The Wikipedia article on Microsoft's DNS server states that bind 4 was not used, and that the Microsoft's DNS server was fully written by Microsoft. Sadly, it doesn't have a source, so it kind of leaves the question open.
Out of curiosity, was libuv ever considered as the underlying i/o library? Looks like some problems at a quick glance (named pipes for instance) are already solved there.
I'm interested to see how the Nanomsg filesystem state model compares to the ZMQ context state model. I'm a little bit worried about using Nanomsg in a multithreaded environment.
The two are equivalent. The only difference is that in ZMQ the global state is created upon zmq_init() call. In nanomsg it's created upon first nn_socket() call.
Is there a resource talking about thread safety in Nanomsg? I was writing a Haskell client library a little while ago and couldn't quite figure out how to treat threading, though I haven't had a chance to get back to it in a bit.
The inproc fixes in nanonmsg are enough for me to start looking at it -- his post explicitly hit on (and claims is fixed) the two most annoying parts of inproc://
This probably has to do with both the transition from C++ to C, and the fact that Martin (the author) has a better idea of what he wants to make. In other words, he is (painfully) aware of the errors he made in his ZeroMQ and is doing it right from the first time in nanomsg. This leads to clearer and more consise code. The reduction of strange bugs and perhaps better performance adds a lot of icing on the cake as well. I look forward to using nanomsg in my future projects!
EDIT: github link, https://github.com/250bpm/nanomsg