Read this book https://ccrma.stanford.edu/~jos/pasp/. It's a hefty tome but kind of the bible of the various approaches to audio signal processing and synthesis.

There's no one method to rule them all, they all have tradeoffs. So there is no "ray tracing" of audio (well actually, it's ray tracing, but for a lot of reasons it's prohibitive to actually do that in a way that sounds good).

> What is this field called?

Audio synthesis. Popular journals with papers covering the topic are DAFX (digital audio effects) and jAES (journal of the audio engineering society). To a lesser extent, the IEEE transactions on audio signal processing.

> I am assuming it has something to do with the physics field of acoustics?

You can find plenty of sources in acoustics journals/text books - but this is like comparing the needs of mechanical and electrical engineers to game engine designers. At a surface level there is crossover, and cross pollination of techniques/tools, but the needs are fundamentally different.

PASP is awesome but it just scratches the surface. You will need to dive into papers to make any headway - there is a ton of research but there is no "toolkit" for doing it because it's comparatively a niche thing to do. You will probably also want to reference the author's other books on digital filters and computer audio. There is also no consensus on the "best" techniques, so you read papers until you find a technique you can adapt to the modeling framework you've built, implement, repeat.

Physical modeling can be done on a modern computer in realtime (for example, my VST instrument models a 6-string guitar, with a litany of modeling features, at about 3-4x realtime at 96kHz, and I have plans to expand that to a whole piano at roughly the same computational cost). But you will have to be quite good at DSP engineering to be able to meet the realtime requirements - this is similarly high-performance as graphics programming or any other kind of modeling code. You can also build toy models in Python or MATLAB and run them offline, but this is not nearly as much fun.

There is STK Toolkit by Julius O. Smith - which has a stunning violin model. But because I havent learnt C++ i dont know how stunted it is - as the IP is owned by Yamaha - but the code is still available, i dont know how that works.

The type of synthesis you're refering to is "physical modelling" - but from a physics and mathematics perspective - my understanding is that this is refered to as DSP (Digital Signal Processing - which all the resources pointed to here are) computational physics, numeric analysis, fluid dynamics - and most likely a few other names i'm yet to stumble upon.

The closest thing to what you mention is: Modalys by IRCAM (Institute for Research and Coordination in Acoustics/Music)

https://forum.ircam.fr/projects/detail/modalys/

From the readme:

Modalys is IRCAM’s flagship physical model-based sound synthesis environment, used to create virtual instruments from elementary physical objects such as strings, plates, tubes, membranes, plectra, bows, or hammers.

It is also possible to create objects with more complex shapes out of 3D meshes, or using measurements, and Modalys does all the hard computational work for you, bringing them to life and making them sound.

You can run Modalsys inside Ableton's flagship Suite.

I'm very passionate about this area of music/DSP...one day i hope to get involved in it :-)

There is a book on DSP (theory) by Steven W. Smith which you can read online: http://www.dspguide.com/pdfbook.htm

still working up to the math i need to get this one :)

I found it surprisingly approachable -- yes it discusses convolutions, but it also teaches you what that means in a very step-by-step manner, with nice illustrations.

im still pre-calculus! :P And i feel i need absolute fluency before i start haha. Thanks for the encouragement :-)

Playing with audio processing and generation is a great way to embed that knowledge as you learn.

> Like howling of wind, the sound from musical instruments

The typical technique used to model these is a "digital waveguide" - treating the wind instrument or string as a one-dimensional tube. The wave propagating down the tube is sampled at your sample rate spatially, so the computer model consists of a pair of queues of samples propagating in each direction. A typical guitar string might have a pair of buffers 300-600 audio samples long, and you move pointers into those buffers every cycle to model the propagation. I'll link to PASP as others are doing for a diagram:

https://ccrma.stanford.edu/~jos/pasp/Ideal_Acoustic_Tube.htm...

At the end of the waveguide (typically where the string is attached to the bridge, or the sound hole of a tube is), the traveling wave encounters an impedance discontinuity, exactly like an electrical wave on a transmission line would, except the units are different. This discontinuity causes the energy in the wave to reflect, and as this repeats, the string vibrates.

That's the core technique. There is so much research on top of this, backed by hard math, to model other nonlinear behaviors of strings, to add other components to the model, etc. I find it fascinating how physical modeling is very well studied within a very small circle of researchers, and nearly nobody else has heard of the concepts.

Happy to discuss further, either here or over email, if you or anyone else has questions!

Engine Simulator is an interesting example of this type of work: https://www.youtube.com/@AngeTheGreat

Trumpet Simulator is also very impressive though source not presently available

https://www.youtube.com/watch?v=rGNUHigqUBM

Engine Simulator is not source available either, but there are plans to open source it.

Yeah, was also going to mention this YouTube channel. It’s really good and he does everything from scratch.

There's quite a bit of this in the video & PC game world. Early on, technologies like Aureal A3D (https://en.wikipedia.org/wiki/Aureal_Semiconductor#A3D) or Creative EAX (https://en.wikipedia.org/wiki/Environmental_Audio_Extensions) tried to do simple simulations of reverb and spatial audio. The gave way to OpenAL (https://en.wikipedia.org/wiki/OpenAL), I guess more recently it's Wwise (https://www.audiokinetic.com/en/wwise/wwise-spatial-audio/).

The Wwise page is especially interesting, with a few paragraphs on audio positioning and reflections simulated inside level geometry.

A few companies have commercialized the concept of physical audio modeling. Audio modeling allows for complex control surfaces (e.g.: https://roli.com/products/seaboard/rise2) that use MPE (Multi polyphonic expression).

https://audiomodeling.com/. Creates the SWAM technology and provides a whole bunch of physically modeled instruments, like horns, winds, and strings.

For more fundamental sounds check out Ableton Suite

https://www.ableton.com/en/live/compare-editions/#software-i...

Ableton provides a number of physically modeled instruments. Tension = string, collision = mallet/percussion, and a few more.

Both are quite expensive and require a somewhat powerful machine.

While these sound cool, a sampler, that is a synth that plays recorded versions of the real instrument, sounds much more "real" than a modeled version. That's thanks to the Fourier transform. https://www.andreinc.net/2024/04/24/from-the-circle-to-epicy...

A good place to start on sound in general is at Bartosz Ciechanowski's blog on sound: https://ciechanow.ski/sound/

Tension and Collision were made by Applied Acoustics as far as I know, these are their „originals“ (although in terms of value you probably get more with Ableton)

https://www.applied-acoustics.com/chromaphone-3/ https://www.applied-acoustics.com/string-studio-vs-3/

This is essentially sound design from first principles. There's a good book here: https://www.amazon.com/Designing-Sound-Press-Andy-Farnell/dp... Note that the software used (Pure Data) can be replaced by another high-level language (SuperCollider: https://supercollider.github.io/) pretty easily. I know of no "tool" to do what you want because there are few things that are universal to different kinds of natural and unnatural sound. (Note: study acoustics and psycho-acoustics to better understand why the former is true.)

I was going to post the same link to Andy Farnell's book, it's one of the finest resources for procedural audio that I've found. A bit later I will post some links to software synthesizers based on simulating specific physical processes.

> the sound from musical instruments

Pianoteq (https://en.wikipedia.org/wiki/Pianoteq) is a physically modeled collection of instruments (mostly pianos). Runs even on a Raspberry Pi and sounds like the real deal without gigabytes of prerecorded samples. Super impressive what physical modeling can achieve.

It's called Physical Modelling, and it's a big sub-branch of audio synthesis. A very good intro is Perry Cook's "Real Sound Synthesis" book.

To understand the physics behind the software, the field you need to read is called Acoustics. A great intro (and the textbook we used in grad school) is "The Science of Sound" by Rossing et. al.

(And +1 to Julius Smith book recommendations).

Some good posts here. Another term to look into is auralization (often discussed in reference to room acoustics). I have done some research into architectural acoustics and real-time audio rendering tools and am happy to discuss further (though it is a big field and I definitely don’t know it all), anyone reading this should feel free to shoot me a message.

Hi - your email is not in your profile :) I'd love to chat!

This is called "physical modelling". Creating sound from a physical simulation is popular for things like plucked strings, but it's very difficult for more complicated instruments like harmonicas

The field you're looking for in audio is known as physical modeling, it's a facet of the broader field of audio synthesis (along with subtractive synthesis, FM synthesis, granular synthesis, etc).

Physical modeling is primarily used to simulate musical instruments or create non-existent instruments like a 40 ft wide frame drum or a saxophone made of rubber.

I don't see it used that often to simulate non-musical noises, but I wouldn't be surprised if there were other people out there doing it.

Search "physical modeling" on dafx.de: https://www.dafx.de/paper-archive/search.php?q=physical+mode...

You might also like to search "analog modeling" or "virtual analog" for models of analog electronic circuits (distortion pedals, filters, etc.)

Book about modeling, eg., car engines in Pure Data (eg., for games): https://mitpress.mit.edu/9780262014410/designing-sound/

You might check out the source code to Émilie Gillet's (of Mutable Instruments) Rings and Elements modules: https://github.com/pichenettes/eurorack

"Designing the Make Noise Erbe-Verb" Reverb Design Lecture": https://www.youtube.com/watch?v=Il_qdtQKnqk&t=18s

Sean Costello's (of Valhalla DSP) blog about mostly reverbs: https://valhalladsp.wordpress.com/

Well, howling of wind can be achieved with simply low-pass filtering white noise and varying the cutoff frequency.

What you are referring to is known as physical modeling, and there are many techniques employed, depending upon the sort of system one seeks to model.

Expressive amongst some others do sell some impressive physical modelling tech:

https://youtu.be/oTxgHJMpKsw?si=ibIX9pxxl3Sh--xy

If you want to delve deeper on the open source side with csound, super collider and more:

Csound - wg (waveguide) opcodes (e.g. wgpluck, wgbow)

https://csound.com/docs/manual/SiggenWavguide.html

SuperCollider - Stk (Synthesis ToolKit) plugins (e.g. StkBowed, StkPlucked)

https://github.com/supercollider/sc3-plugins

Faust - Functional audio processing

https://faustlibraries.grame.fr/libs/physmodels/

https://youtu.be/u8WTnQPzL2w?si=DY5J-ktLIYLdM_fB

Pure Data (Pd) - pmpd (Physical Modeling for Pure Data) library

https://puredata.info/downloads/pmpd

The pluckable string demo on ycombinator the other day is pretty cool https://news.ycombinator.com/item?id=40442595

Just simulating a string vibrating has a huge variety of sounds depending on how its harmonics pile up and where you sample from it (like where the pickup is on an electric guitar).

A lot of great suggestions in the thread, but I am surprised no one had mentioned Syntorial[0] yet. Especially given how widely recommended it is (seen it both on HN and elsewhere many times over the years). For context: it is over a decade old (if not more), but is constantly updated and improved, and none of it is outdated (not that fundamental sound synthesis can get outdated, but i digress).

It is basically a sound synthesis learning tool (and is also a synth in itself). The final goal of the whole thing is to essentially learn to be able to go from “sound in my head” to “sound in my synth” on your own. You start with a base sin wave (or a square pulse, i forgot), and you go from there. It goes quite a bit into the fundamental physics of it, like harmonics/frequency interaction, oscilattors of different types and how could they be used, different wave interactions, etc.

The tool is also good at giving practical intuition tips and not handholding you the entire time, so you get a good amount of room to experiment and try things to get it right. Which is imo the most invaluable part of the whole tool, as it doesn’t just tell you exactly what to do every time, but gives the exact amount info you would need to get there yourself.

It is very “from the first principles,” in a sense that you start with a basic oscillating wave, and everything just naturally builds up on top of it. I really liked how it basically simulated the process of discovering all that on your own (but in a much more efficient and educative way). It blew my mind at the time, not in the last part because it straight up opened my eyes to the fact that all a sound ultimately is is a collection of very basic oscillators and waves that get DSPd and “basic physics’d” into a sound with a unique harmonic profile. It is difficult to unsee that revelation and not think of sounds that way afterworks, which affirms to me that it was indeed fundamentally shifting in my understanding of the topic.

Disclaimer: I am yet to finish the full thing (planning to do so, eventually), but the “from the first principles” approach they use is great and sounds very much up your alley.

0. https://www.syntorial.com/

Man do I have a brilliant youtube channel to point you at: https://www.youtube.com/@AngeTheGreat/videos

this is incredible - my mind is completely blown at the idea of synthesising audio using full blown fluid mechanics.

I know companies like Siemens sell very enhanced acoustic modeling and simulation tools. The software is commercial, but if they use standards or open libraries you might be able to make use of them, at least as a starting point for research. https://plm.sw.siemens.com/en-US/simcenter/simulation-test/a...

Unreal Engine and Steam Audio may be worth looking into before you invest significant amounts of time into this.

If you need for game reasons, I would advise to inspect the code of OpenAL soft which has lots of interesting concepts implemented and also to read the documentation on Steam Audio - this one is also open source too, so you can take a look on valve's implementation here: https://github.com/ValveSoftware/steam-audio

I would at least take a look on how HRTF works as you need to account how humans experience sounds too when simulating audio things.

Some VCV Rack modules do physical modeling. You can also develop your own modules for VCV Rack, as it’s mostly open source.

The two that immediately come to mind are:

- the Tube Unit module from Sapphire: https://github.com/cosinekitty/sapphire/blob/main/TubeUnit.m...

…from the docs: “Tube Unit is loosely based on a physical acoustics model of a resonant tube, but with some fanciful departures from real-world physics to make it more fun.”

And

- Elastika (also Sapphire): https://github.com/cosinekitty/sapphire/blob/main/Elastika.m...

…from the docs: “The physics model includes a network of balls and springs connected in a hexagonal grid pattern.”

Another that comes to mind is Elements, from Mutable Instruments (called the Modal Synth, from Audible Instruments, in VCV Rack). It’s not necessarily a physical model, but more a physical imitation. You get control over bowing, blowing, and striking impulses, as well as the main structure of resonance… from the manual:

* Internally uses 64 zero-delay feedback state variable filters.

* Coarse, fine and FM frequency controls.

* Geometry: Interpolates through a collection of structures, including plates, strings, tubes, bowls.

* Brightness: Specifies the character of the material the resonating structure is made of – from wood to glass, from nylon to steel.

* Damping: Adds damping to the sound – simulates a wet material or the muting of the vibrations.

* Position: Specifies at which point the structure is excited.

* Space: Creates an increasingly rich stereo output by capturing the sound at two different points of the structure, and then adds more space through algorithmic reverberation.

https://pichenettes.github.io/mutable-instruments-documentat...

It's a bit old now, but you might be interested in Phya: Physical audio for virtual worlds

https://phyacode.wordpress.com/2016/01/30/phya/

https://repository.gatech.edu/bitstream/1853/50024/1/Menzies...

"Audio engines" exist.. they are called VSTs or plugins. The ones with the most physical modeling are echo and reverb plugins.. modeling a physical environment and then reproducing it digitally. Or noise reduction plugins - reverse engineering various "noise" sources and then subtracting them. Very math intensive.

There are also analog versions of this.. guitar pedals that create echo and reverb that "model" a room sound via physical internal parts

Download a free version of Chromophone or Logic Pro and play with physical modeling synths yourself for endless fun!

nVidia has had some implementation of path traced audio in their VR SKDs for quite a while. I don't think its that's popular, though.

https://www.youtube.com/watch?v=Ozhywx2YbzM

There's a course from SIGGRAPH 2016 about physically based sound that's fairly good: https://www.youtube.com/watch?v=4OGeAfyDa4Y

It can serve as a starting point.

I also recommend Bart Hopkin's book "Musical Instrument Design" - which is a practical crash course in DIY musical instrument making / mixed in with a below informal instroduction to acoustics/physics.

Look up the YouTube series work from AngeTheGreat. He is building a realistic physics based parametric audio simulation of car exhaust.

Perhaps more entertainment than education but he is a great place to start.

I can't be of much help here but

> I used to think that alot of physics was bs until I found a tutorial on Box 2D physics.

> The tutorials used real physics concepts, like momentum, coefficient of friction and others to simultate physical (mechanical properties of items on a computer).

Can you share these said tutorials if possible? Thanks!

Unfortunately I cannot find the links. It was about 10 years ago.

But it was about a Box2D physics JavaScript implementation

Can you share the tutorial you found?

Unfortunately it was a very long time ago about 10+ years ago.

I am unable to recover the link.

All I remember was that it was during the HTML5 wave & this tutorial was particularly about the JavaScript implementation of Box2D js

Could you indicate the tutorial on Box 2D physics you mentioned? I'm interested

Can you explain what you mean by “ I used to think that alot of physics was bs until I found a tutorial on Box 2D physics.”? Do you mean you thought physics as in the description of the real world was bs? Or physics as in computer physics engines? Did you not believe the physical laws were properly describing the real world?

There is the synthesis of sounds and there are a variety of models for that, such as FM synthesis, additive harmonic synthesis, physical modeling, wavetable synthesis, granular synthesis and more. Physical modeling is somewhat like ray tracing, modeling the materials and sound paths in math to imitate a real instrument (or any object) making sound.

There is modeling of acoustic environments, which is very much like the ray tracing of audio, dealing with reflections, damping, phase, refraction, dispersion and reverberation.

There is the physics of audio perception, how the ear works, how the brain perceives sound, psychoacoustics. What makes your ear "think" you are indoors or outdoors?

There are encoding standards, AAC, WAV, MP3, FLAC, which can also get into psychoacoustics.

There are algorithms, like DSP signal processing algorithms for audio which do reverb, EQ, echo, psychoacoustics, sound projection/direction, flanging, surround sound, and a lot more for shaping sounds.

There is the study of how to build optimal acoustic spaces, like concert halls and music studio performance rooms. More psychoacoustics, but some math and sensors also.

There is the art of trying to use all the above to accomplish something, a song, a soundtrack, a concert, a jingle.

So there are a lot of different aspects to audio, and I've probably left some out of the above list (voice, foley, sound effects, subsonic, supersonic,...). I'd suggest you first work on defining your problem better, what do you actually want to solve, and for what audience? There are many tools that do a lot of this already. If you start out with the most basic audio physics, how sound travels through a variety of materials, and what happens at transitions between materials you could probably spend many years working on just that (as a comprehensive system) before you even get to trying to generate sounds. I don't think there is a complete system that can be used to predict all aspects of sound propagation through materials (including air) because materials can be extremely complex, for example air has air currents, variations in humidity, air pressure is variable in time and space, and the sounds in air are also strongly shaped by reflections and the qualities of the surfaces it reflects from. Imagine the complexity of modeling all the trees, leaves, plants and rocks in a forest. You can "record" the effects of one specific space in a forest using impulse response recordings and apply it to sounds, but to model it from scratch and be able to recreate any forest location would probably be a lifetimes work, perhaps several lifetimes, unless you can find a clever way to do it (maybe generate artifical fractal landscapes based on parameters measured from real places, and then do acoustic ray tracing?)

Acoustic "ray tracing" is not like light ray tracing because sound is much lower frequency and diffracts, it bends around objects instead of bouncing off, and also bounces off. The phase of the interacting "rays" or waveforms as they reach the ear also matters a lot more with audio than with light which makes the computations more complex. In a light ray tracer light travels in straight lines from sources to virtual camera. In a sound ray tracer sound can bounce and refract and disperse from anywhere to reach the virtual ear. What is on the other side of a wall can matter to what a room sounds like.

Don't let my talking of complexity deter you though, look and maybe you'll find something nobody has thought of before. It might be a matter of plugging together some tools that already exist (e.g., a 3D definition of a room or outdoor space, sound source models, choosing a virtual ear location, and let the computer crunch for a few days.)

> I used to think that alot of physics was bs until I found a tutorial on Box 2D physics.

Haha. Thanks Newton for Angry Birds.