> warp is currently able to preprocess many source files, one after the other, in a single command. [...] warp is set up/torn down only once for the entire set of files, rather than once for each source file.
I'd like to learn more about this. I spend a fair amount of time building on HPC systems. Frustratingly, compiling on a $100M computer is typically 50x slower than compiling on a laptop due to the atrocious metadata performance of the shared file system. Configuration is even worse because there is typically little or no parallelism. Moving my source tree to a fast local disk barely helps so long as system and library headers continue to reside on the slow filesystem. A compiler system that transparently caches file accesses across an entire project build would save computational scientists an enormous amount of time building on these systems.
This is why I asked Walter about use of asynchronous idioms in Warp. It ought to help even if there is no parallelism involved. It is perhaps a TODO. Walter would be able to elaborate.
So your build speed is limited by metadata file accesses to network storage? Your network/storage must be really bad then. How about mirroring all the needed headers locally before building? You could establish that as a makefile rule.
Could you describe in a bit more detail how the ranges-and-algorithms style applies to Warp? What are the major algorithms that you are gluing together? What is the high-level design of Warp?
> how the ranges-and-algorithms style applies to Warp?
First off, a text preprocessor is a classic filter program, and ranges-and-algorithms is a classic filter program solution. Hence, if that didn't work out well for Warp's design, that would have been a massive failure.
The classic preprocessor design, however, is to split the source text up into preprocessing tokens, process the tokens, and then reconstitute output text from the token stream.
Warp doesn't work like that. It deals with everything as text, and desperately tries to avoid tokenizing anything. The ranges used are all ranges of text in various stages of being preprocessed. A major attempt is done to minimize any state kept around, and to avoid doing memory allocation as much as possible.
Warp doesn't use much of any classic algorithms. They're all custom ones built by carefully examining the Standard description of how it should work.
So is the main data processing pipeline broken into several decoupled stages ("algorithms") that pass buffer pointer/length pairs between them ("ranges")? Sorry if I'm misunderstanding, I'm not very familiar with these terms as they are used in D.
What are the main stages/algorithms that constitute the processing pipeline?
I'm mainly trying to understand the high-level roadmap/design enough that I can use the source code itself to answer my more detailed questions. :)
The main stages correspond roughly to the "translation phases" described in the Standard. If you're familiar with those, such as \ line splicing, the source code will make more sense. There are also things like "Rescanning and further replacement" mentioned in the spec that are implemented, for example, by macroExpand. I tried to stick with terminology used in the Standard.
Since Warp is meant to be a drop-in replacement for GCC's cpp, do you plan to include a traditional (pre-standard) preprocessing mode? This has been a source of some agony in FreeBSD, preventing clang's cpp from fully replacing GCC cpp on some ports (mostly X11 related due to imake). So far the partial solution has been ucpp.
It seems that it's not so much about pre-1989 "code," but rather Makefiles. ANSI C preprocessors like yours will convert tabs to spaces, and Make doesn't like that. It seems reasonable to let this use case be handled the way it always has been: by using an old tool that still works.
The other major area of difference seems to be that pre-ANSI people used foo//bar to paste tokens (whereas now we use ##). If we're talking about C, that's easy to update; apparently some Haskell folks can't do that for their own reasons. Again it's a use case which is not preprocessing of C or C++, so it seems OK to ignore it if you're implementing a C preprocessor (as opposed to a generic macro expander usable with Makefiles and Haskell).
I wrote a 'make' program in the 80's (I still use it http://www.digitalmars.com/ctg/make.html) and it has a macro processor in it, but it is not like the old C preprocessors.
Warp does support the obsolete gcc-style varargs, but other obsolete practices it discards.
In any case, I haven't seen any Makefiles big enough to benefit from faster preprocessing.
According to your bio you're a lapsed ME, like myself. Do you think not coming from a CS background gives you an opportunity for novel solutions in compiler development work?
[My first real project used Zortech C++ for OS/2, thanks for the fond memories...]
I think not coming from a CS background has resulted in me reinventing the wheel on several occasions. (I thought I'd had a brilliant insight, only to be told it was well known.)
I'm credited with a few innovations (like NRVO), but for the most part I just put together novel combinations of other peoples' ideas :-)
On a related topic could you tell us about coroutines/fibres in D ... How are they implemented ? Can they be called from C ? are they are used in D's standard library, examples of a few notable use cases (I guess one would be Vibe.d), examples of asynchronous idioms in D.
Since that's a lot of questions, pointing to documents would be fine too.
Do you employ any detection of boost preprocessor library use, so that e.g. BOOST_PP_FOREACH gets processed as a literal loop rather than a large expansion?
Nope. Boost is just another bucket of chum as far as Warp is concerned. I did use Boost extensively to validate correct operation of Warp and profile execution speed.
Hard problems for C++ compilers are keeping track of where a piece of code originated and providing good diagnostics in code expanded from macros. Can warp help with this or was speed such a paramount goal that the rest of the tool-chain cannot provide good diagnostics as soon as macros are involved?
Will Warp's design allow modes to be added, where it can be be used as a preprocessor replacement for compilers other than gcc (msvc, clang)? Would you accept such patches? I know mcpp (the only other standalone c preprocessor I've seen) tried to do this.
Other than the speed processing, is there anything about how Warp operates that would make it easier to implement a distributed build system? It seems like the preprocessor can sometimes play a role in whether builds are deterministic enough to be successfully cacheable (__FILE__ macros and such).
Warp only predefines the macros required by the Standard. For all the other macros predefined by cpp, it is passed those definitions on the command line via a script file.
The same should work for any other compiler, provided they don't use preprocessors with custom behavior.
I don't see any reason why Warp cannot be used in a distributed build system.
