Missing: in polyphonic instruments, notes affect other notes. For instance, on a piano, a staccato C2 by itself is a totally different beast from a staccato C2 played while the dampers are off other keys, from a haunting reverb from notes in a higher register to deep overtones from notes in a lower register.
Most simulated pianos these days add the extra harmonics when the sustain pedal is pressed, but very few seem to get the harmonics from the dampers being lifted per-key; PianoTeq gets this right.
I have a Roland LX708 [1] that gets a lot of this right, like sympathetic resonance, and all the other artefacts of a real piano. It’s pretty damn good and it’s clear that modelling has come a long way, but still not as ‘rich’ as a real piano.
I wonder what the last mile is or whether acoustic modeling will always be stuck in a kind of ‘uncanny valley for sound’.
I assume in company it is close (enough), but Piano alone can sometimes live from exactly the non-musical effects. I spontaneously remembered listening to Hania Rani with good headphones [0].
You might like Nils Frahm, a master of the prepared piano [1].
He had a new piano made together with piano maker David Klavins, with a matching VST from Native Instruments [2]. It supports being "prepared" out of the box.
Other fantastic pianist in the same space: Clem Leek ("America") and Dustin O'Halloran ("Vorleben").
That resonance can be a defining element depending on the intent; the butteryness of the Rhodes style electric piano is due to individual pickups on each note picking up neighboring note resonance.
I think you're touching on the overall concept of harmonic and enharmonic overtones in sound synthesis.
I got a free piano from a lady that paid 25k for it back in 1980. Mice had lived in it while in storage, but I could tell it was in good shape.
Spent a summer going down the rabbit hole of tuning it myself. I found the Entropy Piano Tuner app to be quite helpful. Took me weeks to get it tuned to where I liked it.
My digital Yamaha doesn't even come close to capturing the resonance you get on a real piano. I find it hard to play digital pianos for anything other than simple melodies.
My brother has a fairly high-end Yamaha digital piano which is supposed to do these sorts of things, a model from something like ten years ago, but although it’s clearly better than my cheap Yamaha P45 which lacks this stuff altogether, it’s still utterly unrealistic. Nice as it is and sounds, I’ll still prefer my old upright almost every time just for the responsive feel and the proper resonance and acoustic interplay.
(Though in practice both of my pianos have been comparatively neglected since I got a Kawai DX1900 drawbar organ a year ago, which I think is rather amusing. Analogue electronics are really fun, the wear and failure modes are much more interesting than digital electronics like they’d use if making this kind of thing these days.)
Yeah, seeing Linux support on their list is very encouraging. I wonder if it’s common for people to embody this in an instrument that runs Linux—I can imagine that being about the most convenient.
I think the issue with a digital simulation of an acoustic is simply that an acoustic radiates in a full sphere from its body, interacting with a full surround environment, such that youre also hearing, subconsciously, the interaction of those vibrations with the full area, immediately.
Electronic speakers cant do this 100% generally speaking, especially when its not some really high-end audio installation. (I've been on projects where I was just in awe of what sound engineers can do with spaces) The sound would have to make a reflection or two before it actually hits all the same surfaces?
But most people are listening to music in a comparably humble setup and acoustic environment.
I am not an audio engineer, but this is my thought.
I'm not at home for a couple days, but http://cwillu.com/files/sympathetic-resonance.rot.mkv should give some sense: iirc I didn't pedal at all in this piece (or maybe only a little, honestly don't remember, it was a couple years ago now), but there are 3 or 4 keys held down silently.
Making synthesizers emulate the sound of real instruments was once a major goal of synthesis, but that has mostly faded outside of the rarefied world of physical modeling. Electronic musicians remain interested in sounding "warm" and "organic" (or the opposite), but it's rare to find someone who is really interested in making a synthesizer sound exactly like a guitar or a clarinet.
But when I was learning FM synthesis (one of the harder ones), I found it incredibly useful to go through the exercise of trying to make a synthesizer sound like a convincing gong, or reed instrument, or guitar string. In fact, I don't think FM really clicked until I did that.
This isn't about accurate recreations of acoustic instruments, it's about capturing some of the features of acoustic instruments in non-imitative patches to make them sound less sterile and simplistic.
A lot of (non-MIDI) computer music is either drones, granular clouds, or percussive blips.
There's an under-explored area of articulation and musical detail between those sounds and warm, mushy analog.
The DX7 was a good first take at that. But today you can have hundreds of oscillators stacked and patched in any configuration you want instead of just six.
You can actually use thousands of oscillators, which is what Magenta did with their differentiable DSP (DDSP) approach to modeling acoustic instruments including singing:
Feels like some better instruments can help with this, over the basic keyboard. Things with polyphonic aftertouch, or electronic wind instruments, or more advanced surfaces like the Linnstrument or Continuum.
"it's rare to find someone who is really interested in making a synthesizer sound exactly like a guitar or a clarinet"
Perhaps that's because it's quite difficult to do that with synthesizers? Because once you start using samples, you loss the ability to play and modulate without it sounding like a sample?
I guess my counterpoint would be the massive popularity of Omnisphere, which combines a sophisticated sample-based synthesizer with good filters and a few other types of synthesis. Or would you consider this "organic" but not "realistic/acoustic"?
Many synthesizers do not use samples. Physical modeling oscillators have been available in good synthesis engines for a long time. These generate sounds based on a physics model of real acoustic instrument designs and materials, dynamically reacting like real instruments. These used to be dedicated synthesizers but are now buried in a long list of oscillator models available on general purpose synthesizers.
I used to own a Korg Z1, which had one of the first physical modeling synthesis engines. While it could produce relatively realistic acoustic sounds, that is not what people ended up using these types of synthesizers for. Unlike a physical acoustic instrument, where the basic geometry, material properties, and other parameters are fixed, most parameters of the synthesized physical models could be dynamically modulated and manipulated like any other synthesis parameter.
Modulating the fundamental physics properties of an acoustic instrument at audio rates produces some really interesting timbres and effects that are not reproducible using any other type of synthesis or (obviously) acoustic instruments. Consequently, no one ended up using them for realistic acoustic sounds; it was much more interesting to use the synthesis engine to do physically impossible manipulation of the acoustic model to generate novel sounds.
These days almost everything in a synthesizer engine is based on modeling due to the inexorable increase in available computing power. Nonetheless, there is still considerable convenience and economy in using acoustic instruments or samples for many purposes. Just because we can create incredibly detailed and realistic physical models doesn't mean it is worth the effort and they often have terrible UI. I could see this as being something where AI could do a lot.
Omnisphere is a good example of where the term "sample-based" isn't entirely accurate (ignoring its modeled oscillators). While it does have a large and excellent sample library, many of the oscillator engines use the samples as spectral feedstock instead of as a sound to be played per se. The raw sample is not identifiable even though it imparts a characteristic quality on how the oscillator sounds. (Omnisphere is also massively popular because it is an excellent synthesis engine with an unusually good ease-of-use to power ratio. Still one of my all-time favorites.)
Problem: We don't really know what "realistic" quality means. Is it
the timbre? Is it the spectral dynamics? Is it a set of recognisable
behaviours? Does playing versus only listening to a simulated
instrument make a difference to the perception of "realism"? Sure we
can do naive A-B tests on audio, but they turn out to show unexpected
or "wrong" results.
Not that I'd encourage anyone to get involved in academia in its
current degenerate state, but for someone really passionate about this
I'd say this postdoc Diemo is heading at IRCAM looks like one of the
more fun, interesting and challenging projects around right now. FWIW
there's a similar programme running at Edinburgh with Stefan Bilbao
and Rod Selfridge testing quality of PDE synthesis and AI discovery of
parametric control.
High end realistic patches allow you to move mics around the virtual space, change mics, move musicians, control airflow, bowing speed and pressure etc. They sound REALLY good these days, almost indistinguishable from the real thing.
I think this article is more geared towards pure synthesis when you are using very few primary harmonics. Like stacked sine waves or something. They are impossible to eq because the frequencies are too pure and unnatural so you have to dirty them up.
Sample an elephant, synplant to synthesize it. As a non-singer synplant has allowed me to be on my tracks...you could never tell. Also, amseveral nembers of my friends and family are a) knocked out by the simple fun of synplant, and hearing/ looking at what ai believes constitutes their voices.
Not for DX7. It creates patches for it's own synth (2 oscillators, FM, envelopes, LFO, and effects). You can then look at how they are build and edit the parameters.
Just to give an example of "the rarefied field of physical modeling": SampleModeling/AudioModeling/SWAM instruments enjoy some popularity, I'd say they were the go-to demo when MPE controllers had a burst in popularity a couple of years ago. Most of them use a hybrid sampling-synthesis approach, and some of them use almost entirely synthesis. Astoundingly realistic when used correctly.
Personally, I'd say physical modeling still has a lot of unexplored potential, but that doesn't seem to be a popular opinion.
Agree with you there on both counts. Attempting to match physical instruments is a great learning exercise, but not usually that interesting creatively. However patches that live someone in the middle ground between naturalistic and synthetic are often the most interesting IMO.
i think its because sample based virtual instruments, while enormous in hard drive space, do a good job of approximating many classical instruments and have advanced enough to be useful without being difficult to implement.
I could learn a new synth or I could use the Spitfire Labs BBC Symphony Orchestra pack and spend a lot of takes playing with macro wheels and maybe some adjusting after to try to improve articulation to something more natural sounding. It would by a bear live but in the studio its easy.|
The synth would offer more customization but the sample one is good enough for most things, especially in a small/home studio setting.
Synth strings can be great, but in their own aesthetics. I'd argue the sample pack is much, much more realistic and rich sounding than any synth patch you can come up with. There's no easily programmable synth that can emulate the mixture of a dozen or so strings vibrating at slightly different pitches. Creating a single, somewhat realistic violin is almost impossible outside of very large physical models (there's a good model in Reaktor by Chet Singer). Even dedicated physical modelling synths fail at that task.
I'm trying to make FM synthesis mimic a particular sound but I find it very difficult to search for something that even approximates the sound I'm looking to emulate (an elephant's roar). Do you have any tips?
Edit - from the comments I found this paper which looks promising
Hard disagree with #18. If you don't pull your bass register into mono (or really close to it) it's going to sound like ass in small spaces, like a car, or big spaces, like a club or big stage.
I struggled for quite a while with getting bass to sound right on different systems and pushing the bass into mono helped a ton.
Also consider the length of the waves. If you're playing on a system in a club that has the subs 50 feet apart, you're going to get weird nodes from comb filtering and the bass will just disappear in certain places in the room.
I'm not aware of stereo bass having anything to do with vinyl, the issue is overall bass volume.
Most of the other ideas are pretty good and can be summed up as follows:
Humanize/randomize pitch, velocity, and rhythm
Use distortion to add extra harmonics
Don't use a flute patch 2 octaves below C or a contrabass patch 4 octaves above C, respect the "true" frequency range of the patch
Use appropriate panning
Use reverb
(Actually I disagree with the idea of using many different reverbs. Your ears will pick up on it and it will sound unnatural. IF you do this, turn up the reverb until you can hear it, and then dial it all the way back until it seems to disappear.)
>Don't use a flute patch 2 octaves below C or a contrabass patch 4 octaves above C, respect the "true" frequency range of the patch
If you're going for realism, then yes. If not - push the ranges, see what happens.
>(Actually I disagree with the idea of using many different reverbs. Your ears will pick up on it and it will sound unnatural. IF you do this, turn up the reverb until you can hear it, and then dial it all the way back until it seems to disappear.)
I believe it was Zappa, trying to rescue his LSO recordings, so he placed each section in a different 'atmosphere'.
Sorry, on mobile but I'm sure it's mentioned in (at least one of) his book(s).
> I'm not aware of stereo bass having anything to do with vinyl, the issue is overall bass volume.
It has to do with the needle coming out of the groove if low frequencies are out of phase. I agree with your point though. The lower a frequency gets, the less directional it gets. So, bass sounds in stereo are even more pointless for most music.
Rex Basterfield makes tons of free synthesised VST instruments, but created an interesting 'resonating sound box' effect earlier this year that would seem relevant here for the box tone:
Many of his instruments provide little details along the lines of those in the article (see his narrated YouTube vids), all freely-available here, most-recent at the bottom:
This is a pretty solid list, props to OP for putting his observations all in one place!
I definitely agree with 2,3 that clippy hard transients are cool (clipping is in general a bit underrated as a creative synthesis tool).
From my general experience playing and designing with the 'classical' synth approaches (additive, subtractive, FM/PM) nice results are the cumulative result of many tiny hacks of the kind listed, well-judged micro-aberrations.
The spice makes the dish, that is.
This makes design and testing patches a somewhat infinite, fiddly task. Or endless fun, depending on your point of view.
Obviously, if you want to productise your thing - ie you'd like other people to use it - this you now have a control problem: bake in dozens of parameters such that 'it just works' or make them all available and let the user figure it out themselves.
I find this article to be shallow and dismissive: even twenty years ago waveguides worked very well at modeling many acoustic instruments. If readers would like an introduction to waveguides this is one of the foundational papers: https://www.researchgate.net/publication/234809707_Theory_of...
I found it pretty reasonable. I've implemented both physical modeling and more traditional subtractive synthesis algorithms, and I fully agree with the author that physical modeling takes "more engineering tenacity" to implement correctly. Mastering simpler techniques before diving into physical modeling is something I would highly recommend, and that's exactly what the author is doing. Giving some degree of realism to subtractive synthesis is fun and can give great results too.
This is really interesting to me but there’s a few things here that are beyond my reach. Could anyone describe these in more detail or point to references?
* partials? Is that another word for harmonics?
* “Box tone”?
* Breakpoint synthesis? ChatGPT suggests this is a kind of interpolation but I don’t quite get it.
Any general references would be really appreciated. Thank you!
This is a topic close to my heart. In 2019 in the peak of lockdowns - i decided to indulge a backburner curiousity i've had for a decade and a bit - modular synthesis. I bought the recommended modules for Doepfer's rendition of a Trautonium (a ribbon input instrument) - which needs to be heard to be appreciated. One thing which this instrument showed me is how little pitch variation is utilised in synthesis. In this thread on modwiggler (modular synth forum) there's a very indepth convo about realistic bowing/string sounds.
I tried recording a rendition of Hans Zimmer's memoirs of a geisha main theme violin solo (scroll down to the youtube clip of the geisha).
The pitch being fully continuous and in control of the player injects incredible amounts of "realism" and "violin-ness" to the sound. The IR of violin body i added in this example is crude - BUT - it illustrates my point really nicely:
https://modwiggler.com/forum/viewtopic.php?t=68150&start=100
I think both the attack pitch envelope is crucial to get this right as well as the pitch bends induced by the player.
An even better illustration is in an even cruder synth i own - Unisyn's Pitch to CV converter has a VERY crude tri/square/saw wave - and the the pitch + volume - is the main carrier of expression - this is an example of me playing a bamboo flute into a mic - and having it converted to a saw wave synth patch through some reverb added in post - the flutiness is remarkable here imo:
I have to shout out Fricko modules - in Euro being really cool forward thinking modules which aim to inject some novel building blocks for building acoustic sounds:
https://fricko.home.blog/
I recently observed an interesting technique being used to radically alter the perception of synthesized sounds, by using the Pianoteq piano simulator, turning off all of the string and core piano sounds, and leaving only the level and key noise samples playing, then layering this on top of a synth sound - it had the amazing effect of making the synthesizer patch sound very realistic, as it added key noises and environmental (striker) noises to the background .. it was quite astonishing how this radically effected the perception.
Good move! Do you have a link for that? Would be interested to hear it
FWIW, the Roland D50 did something similar, and used PCM samples to model the attack/transients... you could then use subtractive synthesis for the body of the sound. I would guess Pianoteq's key/striker modelling is a lot subtler!
I recently bought Reason only so I could play with their new physical modelling synth "Objekt" (https://www.reasonstudios.com/shop/rack-extension/objekt/). It's the best of its kind that I have ever played with. It really feels like an uncategorized acoustic instrument. Highly recommended if you're into things like that!
I found this very interesting. I’ve done video synthesis and it creates quite nice results if you add a lot of similar analogue inspired tunings to very digital and simple oscillators. Slight randomness to everything, noise floor, over cranked signal that distorts and blooms, color channels mismatching, uneveness across screen, dirty color values of white and black, ringing, trailing etc.
Especially if everything is tuned so subtle that it’s just barely noticable.
About a decade ago, within my circle of amateur musicians, it was considered nearly impossible to achieve an ‘acoustic’ sound using a completely digital loop. The simplest method found to enhance the quality of a synthetic sound was to play it through a speaker and then capture it using a microphone.
In the end it may be easier to reprocess the synth version with AI trained on acoustic sounds, ideally with options to select different performers' styles. The long list of algorithmic tips there may indeed work, but could be expensive to code, test, debug, and tune.
Anyone interested in discussing these topics, I'd like to discuss more, since it's what I'm working on. My email is my profile. I'm also participating in discussion on the thread.
I always wondered about doing this with hardware. Surely there must be some other mechanism for accurately creating sounds then a flat magnetic speaker.
You can use a transducer to excite any physical surface. This is how plate reverbs work for instance.
You could stick one (or two) transducers to a guitar or piano and then mic it up, and get the sound of the body of that instrument and parasympathetic resonance of any undamped strings.
The body is quite easily modelled with an impulse response, but parasympathetic resonance in DSP is still not great
re-recording digital stuff by playing the sound through something analog is a good way of making things feel more 'real', and can impart really interesting, hard to get character to a track. It's just time consuming to do, as the analogue world really only works in real time.
Speakers, themselves, have an absurdly broad range of nonlinearity, even within a single speaker, the way it behaves at 10% power and the way it behaves at 90% power can be drastically different. That makes them fairly difficult to model. Impulse responses are the soup de jour, and do a good job, but, they only really model a slice of what a speaker can do, ie, a given amplitude.
Most simulated pianos these days add the extra harmonics when the sustain pedal is pressed, but very few seem to get the harmonics from the dampers being lifted per-key; PianoTeq gets this right.