Conventional storytelling in visual media is all about directing the viewer's attention to where you want it.
Back in the 1990s I did some work with VRML and found that with the user interfaces of the time the user would usually turn his back to what he was supposed to see and then not be able to find it.
If you wanted to break the "looking ahead" pattern you'd want to design the scene so that important events happen in different places; one thing you will need is good audio cueing, and "stereoscopic" sound in headphones rendered by HRTF won't be good enough -- you probably want 22.4 sound, but a properly mixed 5.1 track and an optimal speaker system might be enough.
Also I have no idea how you light a 360 degree scene, both in the sense of "everything looks good" and "you don't see the lights".
Even in outdoor scenes, Hollywood-quality work requires the use of lights, reflectors and other tricks -- if you use real actors and don't light them well they'll go to the union and complain.
There is a large precedent in video game storytelling but even there, critical moments in the story are usually in cut scenes or rendered using "Visual Novel" conventions.
Half Life 2 was really good for telling the story while keeping the player in the action. Even during scenes where characters were just talking to you, you could play with the scenery and move around and pick things up. There'd be several people talking in a room and the game would expect you to turn around to face the person who is talking, then turn back around to see the other person respond.
It was a natural step from HL1 as players became more comfortable navigating 3D game worlds. I'm wondering if they're waiting for that moment in VR before choosing to make another story-heavy game like that.
Another thing that Half-Life 2 did, in order to turn the player's attention to an interesting bit of scenery was to position an enemy solider near that scenery. The solider would shoot at the player, and most people would turn towards the threat... And got their attention drawn to their setpiece.
They'd also use clever geometry to guide the eye toward important sights. A plank sticking out of a pile of debris can do wonders for guiding the viewer
This "cleverness" of HL2 is what ruined it for me. The environment felt heavily contrived, and destroyed my desire to explore, because I felt funnelled. It was worse than what came before, because before, exploring was rewarded by finding nooks and crannies, and hidden passages. With HL2, increasingly you were forced to find the alternative routes (when the main path was blocked), which killed my desire to look for them.
See and I feel somewhat opposite, so I love that we have such a diversity of choice in games. For me I've always had a hard time getting into games where there was a ton of pointless stuff lying around. Like a game that lets you pick up a wooden fork from someone's table... is this going to be important later? Why else would they let me pick it up? I'm going to take it because it must be worth something. And it's still in my inventory when I finish the game. Ooh, a book I can read in the game, must be important... nope, it's just a story you can read, because I guess the story of the game isn't good enough to keep players interested.
Games like Skyrim and GTA I like because most of the time the stuff you find hidden here and there are interesting things happening in every corner you look. But some of the modern takes on CRPGs like Divinity: Original Sin, I have a hard time with. They encourage you in tooltips and loading screens to talk to everyone, try to trade with everyone, because some random person might be super important. But 90% of them are useless. So I waste how much time searching empty chests and pointless NPCs to find maybe one out of 100 that's actually important?
I don't like 100% on-rails stuff, but I do like a little guidance. I don't play games to waste time, I play games to be entertained. I really liked HL2's guidance. It's one of those games where you play it as a cinematic experience, not to explore an vast and complex world. There are plenty of games on both sides of the coin.
HL2 made heavy use of going the "path less traveled" to find drop boxes and ammo stashes.
In fact, there were parts of the game that told an "ambient story" that were totally skippable during the driving scene along the coastal highway. The only reason you get out of the car is if your curiosity to see what's in that damn house gets the better of you. Otherwise it is total inconsequential.
I think you dont quite remember the context -- before Half Life 1's continuous 3D story telling, games would cut to video to move the story forward.
It was actually a revolutionary new story telling technique at the time that did "keep the action going" because you could explore the room as the characters talked.
I was pointing at HL2. As far as I can remember HL1 had like 2 or 3 lock in sequences (besides the loooong intro), while HL2 stops everything whenever someone talks and episodes have sequences that even lock the camera.
You're right, when I say action here I actually mean interactive. It doesn't play a pre-recorded video clip, it lets you keep some amount of control over your character and keeps you in the game world.
I'm glad someone mentioned Half Life since looking around in the various "someone is talking and you're not going anywhere" scenes was the first thing I thought of while reading this article.
22.4 sounds utterly ridiculous. Bass is omnidirectional, so the .4 is largely pointless, and 22 speakers in a room sounds like an impossible nightmare for the average consumer to set up and an inevitable mess of diffuse confusion and comb filtering in terms of actual delivery. I hope you wouldn't expect content creators to mix 22 spatial channels, either - 5 is painful enough.
Watching video has been a passive activity since its inception (with some modest exceptions). 360 video maintains all that passivity in its filmmaking, but adds an unnecessary and inconvenient (even if cool) piece of interactivity: turning your head. You can't even move around! It's cool, but it's largely a gimmick for most videos as they are now. Maybe filmmakers will develop new techniques for this technology, maybe not.
Games and other similar interactive content seem like a much better use of the technology.
As someone who had the opportunity to shoot a few of these last year [0], I wholeheartedly agree. There's so much left wanting out of the experience beyond the gimmick of the wraparound video. And, I'm not sure that films/docs translate 1:1 into VR. As you mentioned, interactive content (and thus new technology enabling this content) seems like a much better bet. You're already starting to see it with frameworks like ForgeJS from GoPro. I'd love to see the ability to export this type of content from the various engines (like Unity) into YouTube/Facebook. The only reason I still feel it's still more than a valid venue for content is the reach and access they've enabled for regular, mobile users as opposed to Oculus/Vive/etc.
> but it's largely a gimmick for most videos as they are now
Absolutely agree. There has been exactly one video where I was constantly spinning around 360 and it was the Lion King musical 360 video (1).
The reason it kept my interest was 1. things are happening 360 the entire time, and (more importantly) 2. I got to see a view that I had never seen before (that is, looking from the stage into the audience). If the Lion King video had been shot in an open field, I doubt I would have been spinning around nearly as much.
TBH, I find 360 video tiresome. It gives me a feeling of anxiety (low level annoyance, not full blown panic) that I might be missing something interesting if I'm facing the wrong way. I really don't find the payoff to be worth it. I feel like 360 video is going to end up like 3D TV.
360 will not be the majority of future VR content. 360 is mostly a gimmick. yeah, sure, it is exciting to experience your first 360 video but the truth is that when we are viewing content, there is an intended focal point and plot.
You get higher fidelity content given a smaller space as well. People will choose the higher fidelity option where they are can conveniently view from their couch without turning around. Its a chore. We like to "veg" and just enjoy the movie/game.
Even the entire 180 sphere may be too much. People don't want to have to look straight up or straight down. There will be a sweet spot or "sweet zone" as more content starts to utilize VR devices.
Stage plays have some 'research' into this area. People want some separation between the 'action' and where they are sitting so they don't have to move their necks much. This is one of the reasons why the orchestra pit is where it is, acoustics also plays a significant role.
Interesting. My experience is hardly a representative sample, but almost all theater I've seen recently has tried to 'break down the 4th wall' (i.e., the invisible one between the actors and audience on a traditional stage). Some examples:
* The play is set in a fancy nightclub, with half the audience sitting on the stage and the actors also moving around the rest of the audience.
* The play is set in a restaurant, and the servers are actors (unknown to the audience until the play starts and your waiter starts participating).
* The play is staged in a large room; the audience moves to different parts of the room, among the set.
* The stage is essentially a long, wide hallway between two long rows of the audience.
On a related note, I've seen many plays where the actors engage the audience members directly, asking questions, having them participate; in one we were the class for a pair of teachers. What I've seen has worked very well; it's not corny or hackneyed, but smoothly integrates with and enhances the performance.
I've seen it so much recently that it can't be coincidence; it must be a trend in the theater.
There still tends to be a focal point, in my experience. Yeah they'll force you to look somewhere other than the "stage" (which may, in some venues, be a pretty amorphous thing but is still probably in one general area) to create or break tension, but it's not a constant gawk-fest where you're looking all over the place all the time.
Good theater doing this sort of thing uses it for emphasis. To force attention to shift or change or to make it clear to you that "something has changed".
Likewise, audience participation is there to give the audience the sense of being active participants, but still without demanding too much of them (unless it's really active improv or something). It has particular power, I think, in plays that are somewhat confessional in nature. Drawing the audience in to the conspiracy/shame/excitement is powerful there. But if it's a standard narrative I think it bogs things down a bit.
> There still tends to be a focal point, in my experience
Agreed. This one and the others are all excellent points.
In the past I found these endeavors to be gimmicky, but recently it seems like someone has found the winning formula. Despite being completely comfortable in the theater, I still find that the recent attempts have eliminated a last bit of reserve and distance in my mind.
I'd love to know what the new theory is; what is the model or idea that all these productions are embracing? It can't be coincidence.
One of the most interesting 360 VR experiences I had recently was watching an SNL skit. The point wasn't to watch the skit itself, it was experience SNL as an audience member with the best possible seat. You could see the cue-card guy, the audience, the cameras, the stage hands, the other parts of the stage. It was pretty interesting.
In the same day I watched a short animated film in 360 and the experience was just annoying; it wasn't always clear where to look and at some times 50% of the action or character expression was happening behind you. It did not enhance the experience at all.
We believe the reason why so few people turn around and stay long enough to look behind them is due to stylistic choices made my filmmakers. Most videos orient users at 0° along the y-axis and keep the action focused there throughout the video
Absolutely this can be a contributing factor. Something we are interested in exploring in the future is "how" people are watching these videos. Are they sitting down, lying down on a bed, turning around in a swivel chair, etc.
However, we did notice that the majority of videos with lots of action tend to center their content at 0 degrees.
One rather awesome video you can look at is Senza Peso. It's implemented as a 'game' (though it's completely on rails), and you need a VR headset of course, but it's really well made.
Of course, most of the action still happens in front and to your sides. Not so much behind. That seems like a reasonable compromise with viewer attention, though.
There's a also a 2D opera version of the same concept, which you can find on YouTube. The VR version basically puts you in the role of the movie's (main? viewpoint?) character.
> We believe the reason why so few people turn around and stay long enough to look behind them is due to stylistic choices made my filmmakers.
I believe the reasons why so few people turn around and stay long enough to look behind them are the same reasons why they don't do so in real life.
Sure, if there are distracting/scary noises behind you you turn around, but otherwise you turn your body and concentrate your attention on something specific. And within that, at the less macro scale you foveate on the specific area of attention (hence their observation about the wider FoV of a headset...which isn't completely well attested IMHO).
So the filmmakers are doing what people do and want.
So, basically what they're saying is that 360 content creators aren't making content that utilizes the full space well. People don't want to see a story unfold in 360 degrees, especially if they're sitting.
Not to put too fine a point on it, but we recently created a 360 video of a "Goat Yoga" session. Everyone involved in the yoga was in a circle around the camera with all the goats in the middle. The people watching the video are encouraged to stand at the beginning of the video or follow along with the yoga instruction. There's no narrative and yet we have analytics that show multiple people have viewed the video several times because they are the center of the action and it doesn't matter where they look, they still get a similar, but somehow custom, experience.
This doesn't really surprise me. It's a bit like sitting on a tour bus I bet. Most of the interesting things are in the front and to the sides, so almost no one will be turning around, or looking up. This doesn't mean the full 306 degrees shouldn't exist however, just that human nature is always more excited about what's coming up rather than what's behind.
It fascinated me to see the data that people average their gaze slightly downward (expected, we are taller than most our prey), and slightly to the right. Do we all have mild torticolis?
People also don't look up or behind them and especially don't look up and behind which is why rides e.g. at Disney leave pretty much all the rigging out in the open above and behind. Which makes the rides extra interesting if you know where to look.
I wonder if it's an artifact of the resting sitting position when driving a vehicle.
A dataset of where the resting head position is for people from locales which drive on different sides of the road, and of people with differing hand-dominance would be interesting.
Could equally be influenced by UI familiarity - MS Windows, KDE, Android, at least, default to system tray and clock on the right. Mouse hand, or related artefact of dominance of right-handedness ...
I'd expect a lot of VR users to be young gamers who don't yet drive?
Re: slightly to the right, an anecdote—when I got laser eye surgery, I felt a short jolt of pain when my left eye was lazed that I didn't feel in my right. I mentioned this afterward; the nurse said that this is very common, it's always the left eye that hurts more (regardless of which eye is done first, etc.), and nobody is quite sure why.
That's all to say: the wiring of our optic nerves and visual cortices are seemingly quite symmetric—but there's more to the eye than its connection to the brain.
It drives me crazy to see 360* videos made like they are still flat.
The thing that bothers me the most is hosts that move... I don't want to move my head to watch the show!
Graphics also bother me.
If I had my choice: Minimal on-screen graphics and if any, they need to be duplicated and actually used in a 360* way ("Look behind you to see x") and no hosts, just narration.
Personally I think that there needs to be a separate "layer" for UI elements in 360* video encodings.
the whole thing of "show the title in 4 different places around you so you won't miss it" seems like a poor hack, and it feels like being able to include a separate "stream" which will be pinned to your view would alleviate a lot of the problems.
If you want to get fancy you could allow some programmatic interaction between the "view" and the "world" to give some kind of stylistic hint on where the user should look, but i'm not really sure how that would work, and at what point do you stop making a movie and start making a videogame?
I think producers have to rethink it from the ground-up rather than trying to force a model for one format into a new format.
I've seen a few 360* videos that are well-made and were obviously made specifically for 360 but others are just like "We got a 360 camera, then stood in front of it and did the same thing."
It would add to the verisimilitude of wearing VR goggles if your in-scene avatar were wearing some clunky HUD goggles.
The 3D scene can be rendered as usual, and any "must see" information can be rendered onto an obstructing-but-transparent 2D surface attached to your avatar's nose.
It's great for sports where the camera is attached to the athlete. For example mountain biking, paragliding etc. Sure, that's a small niche, but 360 isn't useless.
Ockham's razor - perhaps people don't look behind them for two obvious reasons: (1) People rarely consume movies standing up (2) since you're not standing up, you'll probably be sitting down in which case craning your neck to see the action all the time is painful. Especially for 'pure movies' where there's no interaction with the scene, I think this is equally as likely as decisions by the filmmakers (in unfamiliar territory).
When I've played with VR at friends' houses, most of the time we've played sitting on the sofa or sat in computer chairs. There's a bit of craning, e.g. when playing flying games, but often you only need to do a bit of left/right head movement.
I would imagine in plain sight, as extras, or just behind a door/prop. 360 is a bit new, but framing things out of a shot using line-of-sight is not. I also imagine with 360 you would need more cleanup in post for little things not noticed when shot. It leads to a new way of dressing a set that is more immersive, and I kinda like where it heads.
This reminds me a lot of some of the first home movies that were made, where people would just stand completely still in front of the camera, as they didn't really understand what the new device was capable of.
One my favorite 360 videos is this Charity Water short documentary. https://www.youtube.com/watch?v=nlVIsVfWwS4. Specifically because it's very interesting and immersive to look around.
Another good one is the SNL Seinfeld monologue https://www.youtube.com/watch?v=6HS9h4xFRww. Because of the audience participation you crane your neck around to see who's talking.
I thought the Seinfeld one was great for a few minutes, and then just wound up watching the rest on the screen that's behind you ;-) But I did appreciate the camera direction more
Last 360 video I watched I turned around and there was nothing there (just landscape and no action or narrative) so not surprised at all. Story tellers just aren't accustomed to using the whole field.
It would be interesting if there were an entirely separate scene playing out - but stuff only happening in one place is a limit of physical bodies, surely.
Something most people don't realise is these 360 videos are typically Monoscopic. You don't get any feeling of depth because a lot of 360 cameras are a single lens.
I much rather prefer 180 Steroscopic, these are videos that are two videos either Side-by-Side or Over/Under. Then mapped to each eye.
180 fills enough space which is supported by these findings in the post but give you a much better feeling of immersion.
It's "easy" for monoscopic because the camera is just a point in the center, but once you add in eye separation it needs to be in a physically different position depending on what direction you're looking.
Conceivably Lytro's light field camera could give you stereostopic 360 video, but the amount of data required for separate eye perspectives in every direction is colossal. The camera apparently captures video at something like 300 GB/s.
Common-sense (WAGging) says that IRL, we don't use a full 360-degree view, either. Maybe in a warzone, where it's a survival trait, but most days, we focus on what's in front of us.
We also use subconscious cues to let us know important things are happening outside of our FOV and forces us to change it. Has VR reached the point where it can make those cues possible?
Maybe, one should evoque the possible truth that, how can I say.... well simply that 360° videos are just not needed for tha majority of users.
I sure may be wrong but I feel that in the brotwser/or/ youtube, 360° videos are brotwsernkç noonsense.brotwsernk
In virtual reamity helmet, I woukd bet a dollar ororbut two for its success, but not my flat...
A lot of adult video using this sort of technology often only bothers with 180°. For obvious reasons. But it does also mean that the files (and video frames) can be a lot smaller/better quality.
I'm developing a 360 degree VR video player for JauntVR, and I've just implemented an experimental "roll-to-yaw" interface based on using the "head bobble" gesture [1] (roll) for turning around (yaw). ¯\_(ツ)_/¯
It seems to work ok, but I haven't yet done much user testing yet to see if it makes you puke or gives you a sore neck. It's so obvious I wouldn't think of patenting it, and I'd much rather share the idea and benefit from other people's experience.
The jury is still out on whether it sucks or not, and you really have to try it to know what it's like, but I'd appreciate feedback from anyone who's seen or tried anything like that, wants to hack it up and try it themselves, or even links to research, thought experiments or rules of thumb anyone has which might apply.
It's very simple: roll your head to the right to yaw to the right, roll your head to the left to yaw to the left. Of course there is a neutral "dead zone" so you have to roll enough past the threshold to start turning, and the speed you turn ramps up as you roll even more. The ramp can plateau at a maximum speed to keep you from spinning too fast. The range of the dead zone and the slope and plateau of the ramp are configurable parameters, which need to be tuned.
Head roll or bobble isn't a common gesture in my culture, so I thought it would be good to use for turning around, since you wouldn't do it accidentally very often (once it's properly tuned), and it's easily reversible. However the head bobble is more common and has meaning in Indian culture! [2] So I'd like to hear from anyone who bobbles their head on a regular basis!
As well as tuning the ramp parameters, it is possible to filter and debounce the gesture so you didn't turn around accidentally when you bobbled your head expressively.
The roll-to-yaw interface requires more tuning and experimentation of course, but since rolling your head is a much less common gesture than pitching up and down and yawing left and right, it seems useful to use it for turning (yaw) around.
I think it's important to have a way to turn your virtual body around without actually turning your physical body, so you can still comfortably view the entire 360 degrees of immersive video while sitting down or attached to the computer by wires.
Here's one of my favorite papers that has some great ideas about pragmatic issues which apply to all kinds of user interfaces: Buxton, W. (1983). Lexical and Pragmatic Considerations of Input Structures. Computer Graphics, 17 (1), 31-37. [3]
"PRAGMATICS
In examining the two studies discussed above, one quickly recognizes that the effect of the pragmatic level on the user interface, and therefore on the user model, is given very little attention. Moran, for example, points out that the physical component exists and that it is important, but does not discuss it further. Foley and Van Dam bury these issues within the lexical level. Our main thesis is that since the primary level of contact with an interactive system is at the level of pragmatics, this level has one of the strongest effects on the user's perception of the system. Consequently, the models which we adopt in order to specify, design, implement, compare and evaluate interactive systems must be sufficiently rich to capture and communicate the system's properties at this level. This is clearly not the case with most models, and this should be cause for concern. To illustrate this, let us examine a few case studies which relate the effect of pragmatics to:
1) pencil-and-paper tests of query languages
2) ease of use with respect to action language grammars
3) device independence"
I'm out of the industry now, but one more intuitive option we implemented was to magnify pan rotations, such that a user only has to turn their head through 120º (tuneable) to turn a full 360º. Thus a seated user doesn't have to strain their neck to get the full experience. Even in immersive VR, we aren't especially bothered by the lack of 1:1 rotation.
Wow, eyesee, what you did there looks cool at first glance! Thank you for sharing your idea -- I will try it out and see how it works in different situations.
Where people look in VR depends on the content. That's all there is to it.
That blog post is terrible advertising for your company, as it shows that you will gladly create bullshit content around a trivial fact to boost sales.
How is that "all there is to it?" As a content creator, don't you want data on how users are choosing to interact with your content? It's like saying "Where users click on your ecommerce site depends on the contents of the page. That's all there is to it (why use analytics?)."
Nonsense. It also depends on many other factors, like how far you can turn your head comfortably, if you're standing or sitting in a swivel chair, and whether you're using a wireless headset or if you might pull the computer off of the table or strangle yourself with wires if you spin or turn around too much.
Conventional storytelling in visual media is all about directing the viewer's attention to where you want it.
Back in the 1990s I did some work with VRML and found that with the user interfaces of the time the user would usually turn his back to what he was supposed to see and then not be able to find it.
If you wanted to break the "looking ahead" pattern you'd want to design the scene so that important events happen in different places; one thing you will need is good audio cueing, and "stereoscopic" sound in headphones rendered by HRTF won't be good enough -- you probably want 22.4 sound, but a properly mixed 5.1 track and an optimal speaker system might be enough.
Also I have no idea how you light a 360 degree scene, both in the sense of "everything looks good" and "you don't see the lights".
Even in outdoor scenes, Hollywood-quality work requires the use of lights, reflectors and other tricks -- if you use real actors and don't light them well they'll go to the union and complain.
There is a large precedent in video game storytelling but even there, critical moments in the story are usually in cut scenes or rendered using "Visual Novel" conventions.