Writing a toy OS is something that while seemingly intimidating, is actually pretty approachable. My mind was blown when I came across Lions' Commentary on UNIX as a HS sophomore, and I ended up spending the whole summer working on some dinky Unix'y clone.
I do OS development and most of my coworkers had similar experiences.
It's funny, but this could have been me. I also wrote a toy OS in 2001 and I learned by using many of the same resources you listed.
One thing that makes this all SO much easier these days is the Virtual Machine. I remember back in '01 I didn't really know much about them and so, everytime I wanted to test my OS, I would dd it to a floppy and then boot it on a crappy laptop I had lying around. I probably would have stuck with it had I used a VM and some scripts.
What would you say about writing an OS on something like an Arduino ? Without mucking with ASM, you could at least give it the ability to run multiple processes and implement some IPC. I've been trying to do something of the sort but I've got other high-priority stuff to get out of the way first.
I've actually been writing a small OS-like system for the PIC24 line of 16bit microcontrollers to make my own life a little easier. Its written in a mix of C and PIC24 assembly. I'm trying to write as much as possible in assembly (without making the task too difficult), so that I know exactly how many cycles a given function takes (which will be useful for RTOS features; but I'm doing MIDI/audio stuff, so low (or at least, deterministic) latency is important to me). Its still lacking most important features, but so far it can do premptive multitasking and contains a few handy utilities (cycle accurate microsecond and millisecond delays, stuff for configuring peripherals, task management (create, start, suspend tasks) and the start of a driver interface).
At the moment, IPC is very simple: since there is no memory protection, tasks can access each others memory. I have a very simple mutex implementation (basically, a spinlock where tasks yield if the mutex is locked) to protect shared memory, but so far thats it. I plan on supporting a maibox/messaging system sometime soon. I still have not got a memory management system though, so tasks need to know ahead of time what memory they may use. Currently, tasks are all within the one program binary (ie, no runtime program loading yet), so the C compiler can handle most of this, butI cannot use malloc unless I temporarily turn off premptive multitasking. I cant just turn off interrupts because PIC24's only allow you to turn off interrupts for a known number of cycles and I don't know how many cycles malloc may take.
I have already got some UART code to interface with a bluetooth module, and I plan on writing a little shell for it soon, so that I can connect to it from a laptop/desktop over bluetooth.
I also want to expand the scheduler/multitasking to have more RTOS-like features: task priorities, higher priority tasks pre-empt lower priority tasks, deadlines (and an error system for when deadlines are not met) etc
Another thing I would like to add reasonably soon is an SD Card interface and FAT driver, so that I can load files/programs/configuration from SD cards.
Finally, I need to finish the driver interface and write some drivers for my most commonly used peripherals (planned devices I plan to support for my own use: various types of buttons, analog input devices (sliders, pots), seven segment displays, LCDs, SRAM modules, USB devices (some of the devices I am developing for have USB OTG), an RF module I have, a way for two devices running my OS to talk to each other over SPI, eventually ethernet).
Not terribly large. Not more than 2 or 3 KB of program memory right now (and probably half the amount of data memory). I expect it to grow to maybe 10 KB of program memory (not including user programs, and later when I have loads of drivers, I expect to only include the ones actually used - or dynamically load them off an SD Card).
The PIC24's are fairly beefy devices. 16bit, 40MIPS (for the HJ's, 16 MIPS for the F's and 32 for the dsPIC), a few different I/O pin options in case you need lots of pins... The PIC24H range of microcontrollers I'm currently using have from 12 KB up to 256 KB of program memory and from 1 KB to about 16 KB of data RAM. The PIC24F range have the same range of flash and up to 96 KB of RAM. The newly released PIC24E has 256 KB and 512 KB program memory versions and come in 28 KB and 52 KB data variants (and run at 60 MIPS).
So, currently, they have more than enough space available to comfortably fit anything I throw at it, but if I start to run out, I will 1) add additional external SRAM and/or flash, 2) add the ability to dynamically load programs from SD Cards and 3) have a way of stripping out features that aren't used (since its a microcontroller, its unlikely that a driver will ever be needed if its not needed from the start, unless its a driver for, eg, a USB (or some other hot-pluggable or easily replaceable) device, since most devices would be permenantly connected.)
It's a great idea. Having the OS run on a ROM directly would make things easier...
For me, the worst part of writing an OS to x86 is that you have to start with a bootloader, a primitive filesystem, change CPU modes... all that before you get to the fun part (memory management, threads, running programs, improving your filesystem). Then I discovered LILO and GRUB, but it was too late!
That sounds like a really cool idea. :) Do you have plans to release it? This combined with Ruby on Arduino (http://news.ycombinator.com/item?id=2774170) could make a nice combination. Anyway good luck on your project.
Writing an OS kernel is a great experience. You will be a stronger programmer at the end.
Another good resource is CMU's course on the subject: http://www.cs.cmu.edu/~410/
You could help out coding Haiku OS (www.haiku-os.org) which is an x86 clean-room open source continuation of the once commercial BeOS being developed by a committed team. The emphasis is on native code, fast, lightweight and being super-responsive. You can have 10 videos playing at once (I don't know how much, but it's designed to never block a user's actions) and it doesn't bat an eye. It has binary compatibility with BeOS and new apps are being created for it all the time (avoiding the chicken-and-egg problem). There is a variety of them (some open sourced professional apps). It is not UNIX-like, but it has POSIX compatibility. Haiku has a journaling-file system which is fast and supports user-definable attributes (Be engineers brought a subset of it to Apple with them developing Spotlight and Microsoft had been working on a similar feature). Stack and Tile (haikuos.info/video/stack-and-tile.html) is a cool windows management feature unique because of tabs for window titles. The OS API is said to be the simplest and a joy to use.
Some of the languages available:
C,
C++,
Python,
Ruby,
Perl,
Lua,
Haskell,
Scheme,
Squirrel,
Yabasic,
bash (comes with a bash shell),
zsh
Some of the developer tools available:
Paladin (its own IDE), Git, Hg (Mercurial), GNU Compiler Collection, Vim, QEMU, Bochs
Some of the technologies used:
Multi-process WebKit browser,
OpenGL,
OpenAL,
SDL and Qt libraries,
Gutenprint and CUPS for printing,
FreeBSD network drivers,
Haiku Vector Icon Format
On the way:
An ARM port,
Clang with the already working LLVM,
OpenJDK,
Gallium3D,
package management,
more language bindings for the API
Included below is ways to contact, lessons and software sources:
Back in the days I also managed to write a bootloader and handle interrupts from the keyboard. I found GeekOS to be a great source to learn from. I even contributed a patch. http://code.google.com/p/geekos/
As someone who has built his own operating system I wouldn't recommend going at it from scratch. I got much more useful information install Gentoo from scratch. I will never use that ASM knowledge I learned in real life. However, I do use the knowledge of how Linux works from the ground up on a daily basis.
Not really... you now know how to install Gentoo, a rather eccentric Linux distribution. To know how Linux works "from the ground up", you'd need to make a start of understanding the kernel source from top to bottom. Then you can go in to learning about the userland things, such as the init system (which is different in Gentoo from most other Linux distros).
Doing a stage 1 install of Gentoo or some similarly masochistic distro-installation trick (Linux From Scratch, hurrah!) means you're good at reading an install guide. After you install Gentoo, could you immediately go and write a kernel driver, or even set up NFS, since you know Linux from the ground up?
I do OS development and most of my coworkers had similar experiences.