So, here's my experience in another creative process.
I'm an artist. The vast majority of my thinking and planning time happens with a sketchbook or drawing program open in front of me, and me rapidly generating rough sketches and/or story notes. I have learnt over the years that, sure, I can sit there staring off into space trying to work stuff out in my mind's eye all I like, but things go a lot faster if I work it out by creating a rapid feedback loop between what I'm sketching out in front of me, and how I want it to look when it's done.
Sometimes when I am very stoned I can enter a state where I'm holding a nearly complete image of the whole drawing in my mind's eye and "tracing" it onto the paper. It's a pretty cool place to be in. But it's super fragile; interrupt me and I'll lose it, with an incoherent fragment of a drawing. So normally, I don't work like that at all - I'll quickly draw the roughest version possible of the important parts of the drawing, and iterate on that until it's done, or until I decide this approach is not working, and try another one. My ideas are in tangible form as soon as possible.
When I'm writing (I mostly make comics, so I get to draw AND write), it's the same thing. As soon as I have the germ of a baby idea that sounds okay, I pull out paper or Evernote or whatever and write it down.
Interruptions still suck. Interruptions can still take time to recover from. But I don't have to spend a half an hour building up an intricate mental model of what I want to do afterwards from scratch, because I have a pile of roughs and notes that documents the process of getting to where I was.
So my suggestion, from how I practice my craft: As soon as you realize you're getting into complex territory, open up a notebook and start scribbling questions and answers and inspirations to yourself, draw little diagrams (or big ones). And mess up your copy of the code with a ton of comments - what's this section doing, what do I need to investigate further, here's some pseudocode for how I might fix this, etc, etc. Document as much of your thinking as possible, put as much of the headspace you're generating outside of your head as you can so that it's easier to pick up the pieces after an interruption.
My problem, as a programmer, is that I can’t easily express what I’m thinking on screen/paper. I already do draw out a lot of it, when I can, but its not enough. I already try to document what’s in my mind before I commit it to code, but the “house of cards” mental aspects don’t go away, so there’s really not much difference between distracting me from documenting my thoughts and distracting me from writing code, both are equally as detrimental.
It depends on what I’m working on, of course, so I think its essentially when problems become too abstract, then I can’t easily translate my in-my-mind understanding to paper/screen.
I did, however, enjoy my time playing with Max/MSP (a visual programming language) for the exact reasons you mentioned: I could think “out loud” so to speak, at least to a larger degree than I can in text. Pseudocode helps, a bit, but I’ve always found boxes-and-lines better to focus my thoughts. Incidentally, whiteboards are one of my favourite programming tools :)
When I get in the zone, especially at the very beginning of coding for a new project, I can't type or write fast enough to keep up with everything in my head. It's on all levels - high level of hope the whole system works, down to the minute details of how some of the smallest components work. I typically bounce around, flushing out enough details of a small component to be sure it works conceptually, and then going back to the broader picture to fit it in. I'm rushing to get as much documented as quickly as I can.
This includes a mix of writing code, drawing on a whiteboard, sketching on paper, and taking point form notes.
Even as I get writing placeholders for unit tests, it often leads to thinking of another case of situations that goes back to inform the high level design.
I think also the motivation can be a factor. When I start getting into this mode, it's highly motivating to keep going. If I get so interrupted as to lose some of what I had figured out, I find it can be hard to build up that motivation to even get going again. It's actually demotivating to start into it, run into a new problem, then realize I had actually already thought of it and the work I just did has to be changed - whereas if I had kept going the first time that wouldn't have happened.
I'm an artist as well as a programmer. It looks like one of the things you're not appreciating is the sheer size of most commercial codebases. It's not possible to keep it all in your head, or even scribble meaningfully about it in a notebook. It's barely possible to keep the bare minimum you need in your head, and that can take a while to build up -- and then poof, it's (somewhat) gone because your coworker can't Google a simple question.
If it was just the size, it would actually still be manageable with a notebook -- but it's not just size. Every single character on every single line matters, to some non-negligible extent. If you could condense that into a notebook, you'd condense the code in the first place.
Imagine you (and your ancestors) had written five hundred novels. Now you're making a significant change in the storyline. You need to be DAMN SURE you didn't miss anything anywhere, or created any plot holes. You can either take a million notes as you go, defending against all possible interruptions, taking obscene amounts of time; or just try and focus and get it done.
Apologies HN for the brief comment, but I wanted to make excited noises about how similar your process echoed with how I'd describe my own (especially the "magic time where I try to take the whole problem space into my head" aspect) but within programming. Although, to be fair, even when I'm doing an isolated component on paper programming seems to necessitate that I keep the "local state" up to some reasonable logic isolation boundary in my head as well. It's oddly reassuring to find that I'm not as inefficient/off in left field (with my thought processes) as I thought I was, for whatever reason this isn't a conversation I've had much with other peers.
I'll try to make this post more useful than self congratulatory by echoing: Notes notes notes. Whiteboards, sticky notes, and a giant perpetually updated todo list in an email draft (discount-evernote :) ), have been my saving graces, even while my memory gets worse as I age.
As soon as we have an idea we immediately open up an IDE, and then get a GIT repository going, and get Yarn to prepare all the libraries we need, and get a quick little back end database api set up, and then a quick little front end framework and then we are...
I find setting up the environment a nice warm up routine that I wouldn't want to skip. Also, it takes just a few minutes, which is probably less than searching for that "template project" that I prepared those weeks / months ago. I guess everyone has their own creative process... :)
> My ideas are in tangible form as soon as possible.
In programming terms the tangible are the concrete things... the code on the screen in tangible, but when you start inferring meaning, concepts, origin, relationships etc in your head (or the reverse - doing that before you write some code) - that stuff is the quantum probability field that you need piece and quiet for, when that guy comes and asks you a question it collapses and you start over - if your very lucky you might get some state back and it wont take quite as long.
But here's the thing... the more complex the problem the longer you need to stay in that state - building it up larger and then refining it and so on, and the more important quiet becomes. Sometimes I don't mind noise if i'm just hoping from one "tangible" bit of code to the next doing some fairly mindless debugging that doesn't take a whole lot of insight, but most of the time i'm not.
But it's hard we all need to communicate eventually, constant meetings are no good, I find the best thing is this new instant messenger trend... not because I like being pestered but because I can turn off the notifications and then go and check it when i'm done on my current chunk of thinking.
> I find the best thing is this new instant messenger trend... not because I like being pestered but because I can turn off the notifications and then go and check it when i'm done on my current chunk of thinking.
No actually, there is value in instant messaging, and I admit I wasn't exactly enthusiastic about it at first... if you are available it's very productive having instant communication, very short back and forth without assumptions (strangely having it less immediate than a phone call and more more than an email _is_ advantageous, it brings clarity, forcing you to think more before putting responses into words, but immedite enough to not encouraging monologue and digressions at the same time).
If you aren't available, you can just collect little messages - better still the person who sent them can edit and refine their messages after they realise you aren't immediately available - you can't really do any of that with email.
What I still don't quite get though, is why anyone should be paying for this.
As someone who does both (artistic, design type stuff, and also coding), I can confirm that it doesn't work like that for coding. I do draw stuff, arrows, blocks, layers, whatever can help me, but everything pales in comparison to sitting get still, with your eyes closed, and holding all that abstract construction in your head. Especially since I can be imagining days/weeks/months of work in one go. Artists have a huge advantage here.
That's why personally I'm a big believer in functional visual/feedback based tools for programming, like touch designer, grasshopper. Strictly for code there's Light table, but it's pretty far off of real visual programming tools.
These help a lot in offloading some of that brittle house of cards construction you have to store in your head.
Trying to write down your thoughts can be considered a form of interruption too. It's a bit of a trade off or risk control. How much time to conceptualize it versus interrupting the conceptualization and writing down an inferior version of the idea?
Iteration does help improve on the inferior version, but the risk is that you might be iterating on a local maximum instead of the more global maximum you could've conceptualized.
This goes for programming too. Plus writing things down forces you to explicitly examine your thoughts, abd helps you catch contradictions before spending valuable time implementing it and realizing what you write was bogus.
That's enlightening. I always thought artists had the whole paint in mind before starting, never considered that it wasn't like that. Arguably it was one of the main deterrents to do any kind of drawing: it took too much effort to have the whole picture in mind!
I'm happy to see that I wasn't entirely out of my mind...
I'm an artist. The vast majority of my thinking and planning time happens with a sketchbook or drawing program open in front of me, and me rapidly generating rough sketches and/or story notes. I have learnt over the years that, sure, I can sit there staring off into space trying to work stuff out in my mind's eye all I like, but things go a lot faster if I work it out by creating a rapid feedback loop between what I'm sketching out in front of me, and how I want it to look when it's done.
Sometimes when I am very stoned I can enter a state where I'm holding a nearly complete image of the whole drawing in my mind's eye and "tracing" it onto the paper. It's a pretty cool place to be in. But it's super fragile; interrupt me and I'll lose it, with an incoherent fragment of a drawing. So normally, I don't work like that at all - I'll quickly draw the roughest version possible of the important parts of the drawing, and iterate on that until it's done, or until I decide this approach is not working, and try another one. My ideas are in tangible form as soon as possible.
When I'm writing (I mostly make comics, so I get to draw AND write), it's the same thing. As soon as I have the germ of a baby idea that sounds okay, I pull out paper or Evernote or whatever and write it down.
Interruptions still suck. Interruptions can still take time to recover from. But I don't have to spend a half an hour building up an intricate mental model of what I want to do afterwards from scratch, because I have a pile of roughs and notes that documents the process of getting to where I was.
So my suggestion, from how I practice my craft: As soon as you realize you're getting into complex territory, open up a notebook and start scribbling questions and answers and inspirations to yourself, draw little diagrams (or big ones). And mess up your copy of the code with a ton of comments - what's this section doing, what do I need to investigate further, here's some pseudocode for how I might fix this, etc, etc. Document as much of your thinking as possible, put as much of the headspace you're generating outside of your head as you can so that it's easier to pick up the pieces after an interruption.