- Is this more, less, or equivalently legal (copyright-wise) to just posting the youtube videos on youtube in the first place? It seems like the act of compositing them in this way would be significant, but maybe not?
- The normalisation of audio could use some work. SwitchCam seems to do something (the youtube volume on each clip was different for me), but it didn't quite work.
- I'd be really interested to know how automated this is, and how much human curation is required to get it right.
- Presumably, the next step is stitching the multiple videos together to make 3d models, allowing you to pan to places between the various camera operators. :)
- I'm no lawyer, but the advice we've received is that we're good on the legal side.
- You're on the money re: audio. It's miles better than it was 1 month ago, but we're still not 100% yet.
- There is a human curation step (~5% of the process), but that is mostly for removing really crappy videos. Searching/Synching/Sequencing is automatic.
- I dunno about next step(!), but i agree the idea is super interesting :)
I wonder how possible it would be both technically and legally to use the audio from the various angles as inputs into an audio processing step where you distill just the true music audio and clean it up. Ideally you'd have one 'cleaned' version of the audio playing and switching views just switches video.
This may be more legally grey than just piping through audio since you are actually producing a derivative work (even if it is mechanically produced), but the net effect would be awesome. You'd end up with better audio than any one person could record, and the more angles you get the higher quality you can make the audio.
Definitely a great idea and very well done for the first cut. You just need some Wilco on there.
It would probably be technically "possible" but I doubt that you'd end up with anything that sounds listenable.
The only bootleg audio that usually sounds decent are soundboard recordings, and when you find audience tapes spliced in with them the difference is immediately noticable.
I think theoretically you can make it quite good, but it's not going to be a simple task and you probably can't rely on purely the fans' recording. I think the best way would be to modulate them together based on a weighting depending on position or distance away from where the music is coming from, also discarding distorted recordings etc. That said, you probably only want this lightly mixed in along with a good recording from the desk, all adjusted for the viewing angle.
Maybe this is something that has been mentioned before, but I would love it if I could select a fixed audio from one of the videos. For most videos theres one audio track thats clearly the best and it would be great if I could listen to that one the whole time while switching camera views.
PS: Very cool stuff!
> There is a human curation step (~5% of the process), but that is mostly for removing really crappy videos.
Actually, I'd be tempted to make the process 100% automated, and open it up to the general public as soon as possible.
This has the potential to go extremely viral, and occasional "bad" videos could work in favour of this, by introducing an element of humour into the proceedings.
You may be on the legal side, but I'd argue so were Amazon and Google when it came to letting the users stream their own music from their own "cloud" accounts. That didn't seem to stop the music labels from "demanding" getting paid again for the streaming, too. So you should watch your back and don't give in to them, especially if you know you're on the legal side.
This certainly looks like a tortuous infringement in the UK (and I'd posit Europe). I didn't get a "only in the US" notice on YouTube though.
I can't see how this is possibly "fair use" - it's the complete work of music and the visual design of the set, choreography and show that is being reproduced in full in a commercial way. Unless the uploaders bought a license with their ticket to reproduce and distribute online and allow derivative works of those reproductions ...
Would be fascinated to read the letter from your IP lawyer justifying this?
>Supposed rights holders can easily contact Youtube with a DMCA request and they will take it down. //
This is largely irrelevant to the question of infringement and puts the onus on the owner to spot those infringing. Just because there's a ready way in which you can complain doesn't mean that the unlawful activity is somehow made lawful.
Note I'm making no comment here wrt the soundness or morality of said law.
Courts ruling on fair use are supposed to take into account "the amount and substantiality of the portion used in relation to the copyrighted work as a whole". Automated copyright detection schemes in YouTube may try to do some semblance of fair use by allowing snippets of copyrighted content under 30 seconds.
just a sidenote - if bittorrent software ensure that no peer-to-peer interaction exceeds 30 sec. for any given title, would MPAA/RIAA/etc... still be able to claim "illegal dowload"?
When they take into account fair use, they take the entire case into account, rather than following any hard and fast rules saying that using X% is "fair."
In other words, no, there's pretty much no way they'd buy any attempt to game the system so transparently. That's why you need a lawyer.
No, the courts take context into account. So for example, it might be perfectly legal to post 2 sentences from a book on your blog, if 10,000 blog posters conspired to post a different 2 sentences to share the whole book, that would be clearly illegal. Furthermore, the law is interpreted by Judges, who are people, not machines, and they tend to take a very harsh stance against people who try to game the system.
Yes it does. Just because the recording was permitted doesn't mean that the recorder has the rights to publish or replay the recording.
Another example: owning a DVD does not mean you can then stream the DVD over Justin.tv.
The thing about rights is that infringement must be claimed by the rightsholder for any action to be taken. Once a rightsholder claims infringement then the content/stream/video must be shut down or removed, and if the site or person continues to infringe, THEN action can be taken.
This is one of the good things about the DMCA and the reason YouTube, Soundcloud and even justin.tv can continue to operate. It is also exactly what the much maligned SOPA bill is trying to change - for the worse.
btw, you should sign a petition against that bill on votizen or similar.
They climbin in yo' windows, snatching your youtubes up.
The first video I picked had only one camera angle for the first two tracks of the concert. So it was an effort for me to understand how it works.
I would suggest you have a prominent link to a 'model video' that has multiple camera angles so new visitors can easily see it in action.
Very good job lads!
Of course, I presume the bigger play is for switchcam to be the default app that people use to record at concerts and other events. Smart!
Cool idea, but I was sad that it wasn't finally an implementation of an idea I've had for a long time. My idea is to actually use video and audio information from distinct sources to create a single video/audio stream that is of better quality and/or completeness than any of the constituent parts. Essentially, my idea would do to video what Photosynth does for photos.
That is just so much harder; though I haven't looked at the website yet, seems to be overwhelmed right now. Besides, I would want one consistent audio signal instead of one that varies in noise, volume, whatever. Video from different sources is alright though, since we are used to switching scenes and cameras all the time.
The idea I'm talking about should provide a single consistent audio signal. I know nothing about audio processing, but it seems like it should be possible to take multiple bad audio signals and combine them into one signal that's better than any constituent audio source. Perhaps one audio source captured low frequencies well, while another captured higher frequencies better.
This seems unlikely to be possible. If you haven't captured any frequencies above 15kHz (which an average cell phone mic is unlikely to do), no amount of averaging, filtering, or combining will get them back. There will also be a considerable amount of distortion, since concerts tend to be so loud that even one's ears are distorting. Good luck separating physical distortion in the mic, limiter distortion in the analog or DSP stage, and clipping distortion at the ADC.
I think the best you could do is use the video to determine where someone was standing, and try to reconstruct some of the stereo information based on multiple recorders.
If you haven't captured any frequencies above 15kHz (which an average cell phone mic is unlikely to do), no amount of averaging, filtering, or combining will get them back.
I think this is technically not quite true. If two cell phones right next to each other are both sampling at 15kHz, in the best case you could combine their samples to get an equivalent sampling of 30 kHz. (Best case meaning phone 1 samples exactly half way between phone 2's samples.)
In practice, however, you would have to account for positioning and the fact that the phones' samples aren't perfectly offset from one another. It would require an amazing engineering feat to overcome this challenge, but I think it's within the realm of physically possible.
This is unfortunately so unlikely to as to be practically impossible (currently!).
If the microphones, ADCs etc on both phones are incapable of capturing frequencies of above e.g. 15Khz below a certain range, combining those signals definitely won't bring you any closer to the original signal. You may be able to cancel out a fair bit of noise given enough processing but you won't get back what hasn't been originally captured by either device.
That's before you get into phase problems from trying to combine two signals. A likely outcome is that the amplitude of some signals are increased whilst some are decreased due to phasing issues.
/fuzzily remembered music tech degree. May be too fuzzy though!
Sorry to be off topic here... but this is why I don't understand HN sometimes - the post above isn't nasty, augmentative, hurtful or 'bad' in any way, but instead of people responding to the poster they've downvoted him.
Isn't downvoting for removing bad content, not trying to silence someone you don't agree with?
Thoughts/comments?
(edit - post is no longer showing as greyed out/downvoted - but still, any comments?)
There will always be idiots who downvote great comments. I have on occasion been such an idiot myself (it happens). I wouldn’t let me bother by that, it usually all works out in the end. (Remember: It can take only a single person to grey out a comment.)
15khz is actually is very high frequency, most adults can barely hear it. The audio from phones is probably much more band limited than that. But that's the least of the challenges. People listen to and enjoy highly band limited music all the time: laptop speakers might have a frequency response of 500hz - 4khz.
The more challenging problem is the distortion from the phones being overloaded, crowd noise, built in limiters, different sample rates and compression. It is true that phase relationships from a single sound captured by multple sources can be very problematic.
However, the further the mics are away from the source the less this a problem at least with "phaseyness." This annoying artifact is a type of comb filtering , and it's based on the fact that two mics close to a sound source can be thought of really capturing the "same" sound at slightly different times. If the mics are far apart, the sound is no longer the same: it's picking up reflections from a myriad of sources, the phase relationships within the frequency spectrum have been smeared and shifted by traveling through air. This negates a lot of phase problems. The more likely problem is cancellation in the low frequencies which can be ameliorated with time alignment.
Only if the two phones are under-sampling the signal.
Chances are that there is a low pass filter in front of the phone's ADC, blocking signals above the Nyquist limit from reaching the sampler. Assuming brick wall filters (ie perfect cutoff), combining the signals will reduce variance (noise) but not give any information on frequencies above the cutoff frequency of the filter.
Brick wall filters don't exist though. What you might see is a miniscule amount of signal in the filter's stop band. Combining the signal from many many phones might reduce the variable enough to give useful information for frequencies a tiny bit above the cutoff frequency.
A cool project would be to gather the audio from every networked microphone in an area (mobile phones, laptops, ...) and use beam-forming techniques to reconstruct the sound pressure field as a function of position. My guess is that the system would be sensitive enough that it could do amazing things like capture conversations though walls or from long distances.
In some cases you will have a good prior on the clean signal from studio recordings. Of course, "registering" a recording (or parts of a recording) to the video would be a formidable task in itself.
>take multiple bad audio signals and combine them into one signal that's better
The problem eventually comes down to the fact that "better" is subjective. We're in the murky realm of art here. Should your algorithm keep that fret noise or the squeaking of a vocalist's intake of breath? Are they "noise," or are they part of the performance?
>I know nothing about audio processing
Not wishing to be rude, but this much is very evident. Recording engineers position their microphones with millimetre precision in order to combat phase issues, and that is in an ideal studio scenario. Doing what you suggest is basically impossible.
Maybe I'm overstating it, you could probably do something and it'd be a nice bit of research, but you wouldn't get useful results in the way that you're imagining.
> Not wishing to be rude, but this much is very evident. Recording engineers position their microphones with millimetre precision in order to combat phase issues, and that is in an ideal studio scenario. Doing what you suggest is basically impossible.
Actually, that much I know, because I've done some amateur home recording. I know that, for example, when you mic a snare drum with two microphones that are pointed at each other, you have to put a phase inverter on one microphone. I also know my way around the basic processors for audio production (compressor, limiter, EQ, etc.).
What I don't know much about is the undoubtedly more advanced techniques which may or may not exist that could realize the idea I'm talking about. The best idea I can come up with is, if you had one audio source that captured the dynamics of a concert (perhaps from a phone that was far away from the house speakers), and another audio source that captured a clearer yet "smashed" sound (perhaps from a phone closer to the house speakers), perhaps you could apply a compressor to the second source that was keyed on the dynamics of the first. Again, I might be full of crap here.
I presume he means better in the sense that you'd try to remove overt noise, e.g. small conversations in the background, maybe wind noise. That sounds like it'd be possible to do. Improving a single video from multiple video sources would surely be impossible. Even with scene reconstruction etc you're not going to be improving the quality of any single video source...(?)
You can certainly automate crossfaded audio between multiple sources to try to get the cleanest copy, but it's hard. For instance, how do you decide whether it's noise or the letter "s" or the "chk" of a pick across muted guitar strings? The heuristics for "better than any constitutent audio source" can be extremely nuanced, algorithmically intensive, and still difficult to pin down, akin to speech recognition. Speaking purely to SaaS'y automated purposes, natch.
Typically what it seems you're talking about for audio here is similar to a matrix mix in the amateur/live audio world. People have been (manually) mixing soundboard audio with audience-recorded audio to improve the audio quality of recorded shows for some years now.
I don't know anything about audio processing algorithms, but (assuming there's a way), presumably with enough audio sources, there'd be a commonality between each one that describes the 'correct' sound. I.e. if there's different noise going on in each source (people talking around each microphone, at a gig, intermittently), you don't really need to decide which sound is 'clean' because you'd know which sounds are inconsistent...(?)
Sure, but it's determining what is "correct" that is the hard part. You could use a majority-rule if you have three or more sources, but the more additional sources are required starts getting into pretty niche territory and it still remains possible that the minority source is the most faithful one.
If both channels have a similar spike at the same frequency at the same time, it is probably part of the signal (not noise), so combine those, and dampen all others. This would cover your case, if the other channel had enough of the low/hi freq of the other to relate them. I reckon Shannon looked at exactly this in developing Information Theory (for telephone signals on flaky lines), and it's probably all textbook stuff now.
The thing is, sound doesn't travel all that fast when you consider the wavelengths of vocal-range soundwaves. Those spikes are not going to arrive at the same time on the different phones.
As ever with DSP, phase problems will be the ruin of you.
nice point, but they'd synchronize with an offset. I doubt absolute time would be used to synchronize the videos anyway; they'd be matched by content.
Or do you mean that different frequencies will travel at different speeds, enough to make (e.g.) high and low frequencies arrive at different times? Whoa, apparently it does (http://en.wikipedia.org/wiki/Speed_of_sound#Effect_of_freque...) but seems to be a small effect.
We've actually thought about this, but it's a low yield proposition until we've shown there's a market for what we have now. It's not impossible, but it is very hard to do :)
Very cool, just a slight niggle about the 'lightroom' colour scheme (dark area surrounding the video) which I see being poorly implemented time and time again: it doesn't really work when you have elements of bright white on the page, because they just end up looking even brighter next to the dark parts. The glaring white strips are distracting while watching the video, moreso than even an all white background.
TL;DR: lose the strips of white from the video viewing page.
Thanks for the feedback, I agree actually. We've been talking about changing the color scheme to either be all dark or all white but haven't pulled the trigger.
I had been thinking about something like this, but in the context of recreating riot crime scenes. After the last riot in Toronto the police received over 1M stills and thousands of hours of relatively low quality cell phone video. The main task for them is to connect different shots of individual offenders over time to build a) a coherent story demonstrating premeditation, b) an unbeatable description and connect it to some identifying info that might not necessarily be present at the exact time of the offense.
I am excited about the concept - was thinking about this very idea while watching Coachella streaming live this year and then seeing all the cams people were uploading.
This is awesome. My only suggestion is to allow the user to switch angles without changing the audio track. Since these are concerts, the audio track is going to be one of the most important things. Often times one video will have significantly better audio than the other ones. It'd be nice to be able to choose that one to stay constant. Alternately, it'd be awesome if there were a way for the bands themselves to upload the recorded concert audio.
I realize that often times getting the recorded audio for a concert can be a finicky matter, but maybe you could partner with one of the many music festivals that are cropping up to make it happen.
I went to add a concert with time/date but landed up getting lot of videos that didn't meet the criteria. The concert was part of Osheaga (a festival in Montreal) but the listing shows up as Parc Jean-Drapeau, where the festival is held. I think including the ability to indicate what event it was part of might narrow down your search a bit.
Since most of the 300 videos found were unrelated, I decided to go through manually and select only those which were relevant. By default, all 300 videos are checked with no easy way of unchecking them. Your average user isn't going to be able to just use a jQuery one-liner like I did, so this is something else you should consider!
Overall, great job and I look forward to seeing how you guys progress with this idea!
Thanks Ian, for the feedback and also for adding an event. I just saw those come through on the back end. We've put the processing on hold for the moment until the load becomes manageable - hope that's cool. We'll email you when they're up.
I love it. Any chance you would expand to more general "news" applications? The audio wouldn't be as good for syncing against, but I've always wanted an interface like this for watching protest videos.
I would love this, too. Fortunately, it has a name that can become more general, too. But first focus on promoting your service as it is, to this "niche", and later you can try growing into more general stuff.
Very impressive. Very cool. I assume this will eventually extend beyond concerts to all kinds of other live events (sports, plays, etc). This is going to be big.
What's also interesting is how much the effect of SOPA will have on such an awesome service like this. So disappointing to think about how the copyright holders would rather shut you down then come up with a great way to enhance the service and share revenue with you.
Pretty cool. One thing... after switching camera angles from the first camera I can switch between all other camera except I can never get back to camera 1.
Some of these concerts may have bootleg audio available, often recorded by audiophiles with decent equipment. It'd be amazing if supported syncing that up so you'd get great audio with multi-camera video
Nice. However - the timeline should be default. I was pretty meh'd until I found it, works great.
Now it would be nice with an underlying concert-track or / per song mp3. I don't like the crappy cellphonesound. I guess the syncing will be hard though, with the videos lag not very constant (most often, seems to work good now but I have a nice broadband connection)
So if you can do this with concerts, presumably you can do it with any collection of video shot in the same place, if there's enough audio in common.
That makes me think that all kinds of crowd video (protests, speeches, etc) should be stitchable, which seems like it would open larger (and potentially morally ambiguous) markets.
Pretty cool indeed. A question though: how much of a need is there for a service like this? Beyond the more mainstream artists and festivals, most of the artists whose shows I've seen have typically had more respectful audience members who didnt put recording with their cellphones ahead of enjoying the show.
Does anyone believe that this is done algorithmically? All the examples I checked seemed to be really poorly sync-ed, & as a DSP engineer I don't believe this task to be incredibly difficult...
would also be cooler to have a seating chart to click around to make sense of perspectives
It's all algorithmic but the flash player only seeks to the nearest key frame (every 2s) so that's why it isn't flawless. This can be overcome ... but I'd prefer to work on the seating chart thing - thanks!
This is one of the coolest things I have seen. While there are some little bits of feedback, I can admit that I spent the last 30 minutes watchings concerts, amazed at how great this felt.
MPAA: More of this, less of SOPA. If you give us great ways to get content, we will pay for it.
Absolutely one of the better ideas I've seen lately! An original idea with a lot of wiggle room for future growth.
I think it could be pretty cool if users had the ability to reference a higher quality audio source (when available - something on Archive.org or uploaded mp3s).
Sorry about that. It only happens when the JS borks out completely (which is rare), when JS is turned off, or when you're using a really old browser (which i assume you're not).
If you're inclined (I completely understand if not) can you tell us the error that the console in Chrome or Firebug gives you?
Interesting, thanks for poking around! I don't think we use anything dodgy (except maybe Facebook libraries) but I'll see if I can set up that configuration and debug it. Really appreciate you taking the time to write this up. Cheers, Chris
I'm having the same problem. I think it's a bad idea to use Facebook libraries for critical functionality. A lot of people (myself included) block Facebook resources from loading on external sites to decrease their ability to track user activity.
Thank you! I think you must be right about the adblock thing. We once had a similar thing with a Chrome plugin that borked things. We'll look into it; thanks again.
Is music a requirement? I'm trying to add Henry Rollins' spoken word at Coachella from April 18 2009, but your YouTube search always seems to fail. I can see the videos on YouTube just fine if I search for myself.
I had the idea to do this (albeit, manually) a while ago, syncing the audio from Justice's "A Cross The Universe" with Youtube video of the concert it came from.
Really cool to see it done, though!
Watching Ocean's 11 last night it occurred to me that other contexts in which intelligent multi-camera stitching might be profitable are casino security and retail loss prevention.
this is awesome. seriously. maybe you could package the technology in a way that lets other platforms use this without youtube. i am reminded of the original Color app.
some techie questions, how does it work? how do you get to know which videos are actually part of a specific event? how do you match the video with the actual song?
This is awesome for two reasons: 1) it's just plain cool and 2) it finally gives a reason to tolerate annoying people who hold up their cell phone cameras during concerts.
It's funny because watching a concert I thought how cool it was that when someone had a camera or phone in my line of sight I could simply switch to another unobstructed view. Then I realized the chances were good that my new view was coming from the very camera I switched views to avoid.
- Is this more, less, or equivalently legal (copyright-wise) to just posting the youtube videos on youtube in the first place? It seems like the act of compositing them in this way would be significant, but maybe not?
- The normalisation of audio could use some work. SwitchCam seems to do something (the youtube volume on each clip was different for me), but it didn't quite work.
- I'd be really interested to know how automated this is, and how much human curation is required to get it right.
- Presumably, the next step is stitching the multiple videos together to make 3d models, allowing you to pan to places between the various camera operators. :)