But also big props to Andreas for speaking so candidly about the back story of Serenity. I feel like there are certain things we don't talk about in tech and that can change.
I love the idea that programming, esp side projects, can be a therapeutic thing.
Guest here. Thanks for having me on your show, Adam!
I'm definitely a fan of talking more about things that we've been hiding from each other.
On that topic, one of the most common questions I get personally is "which drugs did you use in the past?" and I recently made a video[1] answering that question. It was a difficult thing to get through, but the overwhelmingly supportive response has been amazing.
For anyone who's struggling and is afraid to open up about it, know that there's a community of millions around the world who will welcome you with open arms.
I wanted to watch this video for a minute or so (very shallow of me) but ended up being stuck in it for the whole 29 minutes. Very interesting and hard story to tell but also extremely inspiring to see what hell have you been through and was able to mature to the point to where you became an inspiration for many others. Well done.
Now more technical questions ... for my own sake of understanding and learning.
What would you say that were most important aspects of re-implementing (subset of?) C++ runtime that must be appropriate for in-kernel usage?
Where would I look in the source code to learn about SerenityOS paging algorithms, and virtual memory management in general?
> What would you say that were most important aspects of re-implementing (subset of?) C++ runtime that must be appropriate for in-kernel usage?
Something that I messed up originally and we're now correcting incrementally is the handling of allocation failures. There are many places in a kernel where errors don't have a natural propagation path (IRQ handling for example), so operations that may fail (like allocation) must be avoided.
In general, heavy use of C++ features like templates and RAII has been very helpful for making kernel development productive.
> Where would I look in the source code to learn about SerenityOS paging algorithms, and virtual memory management in general?
> Something that I messed up originally and we're now correcting incrementally is the handling of allocation failures.
What in particular? Can you use exceptions reliably in the kernel land? I presume not unless you pre-allocate some hard-to-exhaust space for them.
Have you maybe thought of using LSBs to encode an allocation failure within the pointer itself? It's an entertaining idea I had for some time now. In 64-bit byte-addressable architectures one can take advantage of suitable alignments and have 3 least significant bits used to encode 8 different states. In a similar vein one could also take advantage of MSBs given the virtual address memory space limit of nowadays CPUs.
We don't use C++ exceptions anywhere in SerenityOS. Instead we use an ErrorOr<T>[1] return type for functions that may fail. These are then automatically propagated by use of a TRY() macro[2].
I haven't thought about encoding allocation failure as a low pointer value. I suppose you could, but it'd only work for pointers with sufficient alignment. Character/byte pointers often start at unaligned offsets when not coming directly from an allocator. Either way, I'm not sure compressing this information would solve any real problem we're having.
Then I might have misunderstood you with the issue of propagating errors. In IRQ, as you mentioned, it wouldn't be very practical to blow out the system just because your error propagating mechanism depends on yet another thing that can miserably fail. That's why I thought of tagged pointers, given the sufficiently aligned pointer operation of marking the action as failed cannot be unsuccessful. Nice thing about it is that you also get atomicity basically for free.
> Character/byte pointers often start at unaligned offsets
Yes, but the address of that pointer is still 8 bytes long and idea of using 3 LSB of that address to encode the state is still applicable, isn't it?
> > Character/byte pointers often start at unaligned offsets
> How's that possible? Isn't that an UB?
"Unaligned" was not the best word to use. What I really meant was byte-aligned:
char* s = strdup("Hello world!");
char* p = &s[1];
Here, `p` is byte-aligned and will have a 1 in the least significant position. So a scheme that uses low bits to encode allocation failure would have to work hard to avoid confusing byte-aligned pointers with allocation-failure-signalling pointers.
> Unaligned" was not the best word to use. What I really meant was byte-aligned
Yeah, I figured just few moments after I replied what you actually meant. That's why I edited the post above.
Well, to make it more robust one would have to wrap it in it's own pointer type but I think it's doable. I don't know whether or not it is a wise choice to make it a dependency in the OS though :)
It's been inspiring watching SerenityOS come to life with every update.
One of the things I look forward to at the start of every month is SerenityOS's monthly update, this was the latest January 2022 update where they've done a port of Half Life and their continual improvements to their libGL implementation:
https://www.youtube.com/watch?v=DL-RCV5a-C4
Due to the sheer gravity of the task it may not ever become a mainstream OS but because its entire source code is built from scratch and all maintained within the same repo, it's easily able to prototype new OS features you're unlikely to see anywhere else. In that regard it's possibly the easiest codebase to prototype new OS features given you can make changes to any part of the OS & view changes immediately.
This is a very important role for alternative OSes.
What makes Serenity OS and others like it useful for this purpose is that they have much less unnecessary complication, so that much more of the effort can go into actually experimenting with the feature, as opposed to trying to slot it in compatibly with all the historical baggage.
Another useful quality is how they can demonstrate that a language some insist is unsuitable for kernels is, in fact, much better suited. You cannot write a whole competitive desktop OS almost single-handed without some decisive advantages. Serenity OS is coded in modern C++, and leans hard on its strong support for writing cleaner code.
An OS kernel, besides providing a platform, is a big program. Any big program makes heavy demands on programmer and language, independent of its purpose.
There's nothing controversial about writing kernels in C++ though. Windows, Apple, BeOS, etc - basically every commercial OS has used C++ or a subset in the kernel for decades. Linux is an outlier.
> Due to the sheer gravity of the task it may not ever become a mainstream OS but because its entire source code is built from scratch and all maintained within the same repo, it's easily able to prototype new OS features you're unlikely to see anywhere else. In that regard it's possibly the easiest codebase to prototype new OS features given you can make changes to any part of the OS & view changes immediately.
Isn't this a bit like what FreeBSD and other similar OSes do? To have a tightly integrated/tested base system with a set of software that it has, while maybe optionally supporting third party ports.
Now, in practice you almost always end up with binary distribution of packages rather than building everything from source due to time/resource constraints, but there is definitely an argument to be made about less fragmentation, like much of the GNU/Linux software has.
Maybe, but I've not seen anyone iteratively make changes to all layers of the OS and get instant feedback as Andreas frequently does.
The Diablo port is a good example of this, whether it's missing C++ APIs, unsupported lang features, missing SDL port impl, toggling kernel features - it all gets done in a 1hr sit down session to produce a shiny new Diablo port.
Another nice quality of SerenityOS is that the entire OS is written in modern C++ with everything being written from scratch to use the standard AK libraries which the entire OS and components use, i.e. there isn't any existing App that brings in their own string types or other utils.
Andreas is such a good dude. I highly recommend his youtube channel, not just for the tech content but even more importantly for the important musings about life, overcoming your demons and overcoming that voice in your head always trying to pull you down https://www.youtube.com/c/AndreasKling/videos.
This was a very powerful episode. The strongest part for me was the description of Apple's developer workflow or philosophy, where you can look through the stack all the way through. When I first joined the JVM on Linux camp after mostly being in the .NET world on Win, seeing the system all the way through was a revelation, incredibly empowering.
Andreas with Serenity not only sees the system through but can modify any part of the layer to make the abstraction of each layer just right.
It's something that really doesn't get talked about enough. So much of modern tech is just abstraction on top of abstraction, being able to get in there and play around in those same spaces as your tools is important, though.
I've been doing this with Flutter recently. Turns out it is both a toolkit and a package/platform. If you take out all the 'flutter' code, you're left with something that looks like sdl or glfw, just written in Dart. And it 100% has made me a better Flutter dev to do so. I now get a lot more of _how_ Flutter's drawing loop works, and how the framework beneath reduces all of app startup to one `runApp` call.
Not being able to inspect the insides of a tool easily is a major turn off to me on anything.
While I agree with you, that is an impressively humble attitude. Having watched some of the early history, you can take at least partial credit for building some of those contributors as well.
It was a different time, but Linux did not come so far so fast.
Yeah it's a state of mind, you do something for and with a community and not exclusively for your personal gains. In the US that was often compared to communism (just listen to the questions from reporters in the early day of Linux)
It's not a political thing, maybe more as a side effect of living in a society that promotes well-thougth actions and good planing and gives you the tools to achieve things.
That being said, I consider the the Scandinavian "socialism" the perfect combination of communism-light and capitalism-light: you can get rich if you work hard, but your boss can't get extremely rich by paying you far below a living wage. In the long run it's a win-win for everyone.
Adding to what nix23 said, it gets made political by those who cannot comprehend social norms and cultural differences outside of their own experience and can only interpret "different" as "antagonistic".
> living in a society that promotes well-thougth actions and good planing and gives you the tools to achieve things.
I know, but the US-Press always try to make something political, even when it's a cultural thing (and the only way to survive in the past..working together).
>That being said, I consider the the Scandinavian "socialism" the perfect combination of communism-light and capitalism-light
Absolutely on your side...and as someone from Switzerland i think we tend more and more towards the Scandinavian model.
> Yeah it's a state of mind, you do something for and with a community and not exclusively for your personal gains.
That's just as common in the U.S. - but you need a strong community of practice with high standards of achievement to really achieve this mindset. (In the real world, this would of course be an actual, physical community bound to a well-defined place, ultimately with similarly strict norms of acceptance, behavior etc.) And that can get in the way of contemporary norms against perceived "gatekeeping". Though it's interesting to see that some communities of practice (e.g. the one which is forming around Rust and Redox OS) are being more successful in navigating this challenge.
He also used to post about it on 4chan's technology board, especially on the daily programming thread. Plenty of pictures and videos showing off progress as well as discussion of implementation details.
His recent pet sub-project seems to be getting Discord working in their ground up home grown web browser, that series just started with only the first video being posted so far so a great time to hop in and watch some of the development flow.
The original author, Terry Davis, died in 2018. There have been numerous forks, and ZealOS is one of the active ones. But TempleOS proper will remain as it is.
I first learned about Andreas from seeing one of his videos about TempleOS [1][2] when I was first playing with TempleOS and learning (Holy)C. There are a few forks of TempleOS around, I made a post about one a couple months back. [3]
I wonder what it would take to bring Linux to this state, where while running a program, you can decide to look at the source code, check in the debugger what is going on, apply a change and run the program as if you used the precompiled version of it with very little effort. So you could make changes to programs with similar manual effort as if you changed a config file. And when updating the programs, the package manager would check for your manual changes, if they can be applied without merge conflicts. Upstreaming the changes would also be a simple standardized command. But you wouldn't have to, just to be able to run the changed software without friction.
Andreas ability to interweave deep tech with life and [vulnerable] humanity is truly a gift. It is rarely that the two come together this way. Thank you for all the time you put into Serenity[OS|HumanityOS :)].
And it is /completely/ from scratch - its own libc and libc++, own kernel, ui framework, browser (and I mean from scratch not Yet Another WebKit/Chromium wrapper), etc
There are now a few regular core contributors now so it's not just one man vs the world
I've not played with the OS myself but I occasionally watch a live stream and ask him questions. He talks through stuff off the cuff and you know he is a genuine dude.
what for this os? it can't be used for iot, for desktop or server so i can't understand the main idea why i need to use it or became a sponsor for this project...
But also big props to Andreas for speaking so candidly about the back story of Serenity. I feel like there are certain things we don't talk about in tech and that can change.
I love the idea that programming, esp side projects, can be a therapeutic thing.