Support Ukraine

I don't know what it is with winters, but it seems to trigger feature additions to Bigshot[a], my HTML5 library for zoomable images and VR panoramas. Maybe it's the darkness or the cold weather that drives hacking - or maybe it's the holidays.

Either way, this time I thought I'd try implementing animated textures in the VR panorama renderer, or in layman's terms: 360 degree video. It is still extremely experimental, as HTML 5 video requires a lot of fallback logic due to basically not being standardized, but here's something that kind of works in Chrome for Windows.

​1. Demo

Warning - this may not work on your device. I've tested it on Chrome on Windows - nothing else. That's what experimental means. Without further ado:

[b]
Bigshot VR Video Demo[c]

​2. Implementation

The implementation was quite straightforward. The texture cache fires an even whenever the texture data has been updated - usually by a new image tile having been loaded - and that in turn forces a render of the VR panorama. In this case, I simply opened up a video stream and then fired these update events at a suitable rate.

Turning a video into a WebGL texture was easy (albeit inefficient) - the HTMLVideoElement[d] can be drawn on a HTMLCanvasElement[e] using the CanvasRenderingContext2D.drawImage[f] method. The canvas can then be used in a call to WebGLRenderingContext.texImage2D[g].

​2.1. The Cube Map

When storing a cube map, you have basically two options regarding how you treat the faces - you can store each face separately, or store them all together. For static images, the separate way is the right way, as far as I can tell: It lets you load just as much as you need of the cube map. For video, I ended up storing them all together, as I could not reliably synchronize six video streams. This is what the cube map looks like:

I chose a simple layout with six squares mostly because it meant the best packing of image data. The squares have an 8px border around them so as to avoid compression artifacts from neighboring squares. With 512x512 pixel faces, I end up with a 1584x1056 video in 3:2 aspect ratio. (Not all video formats support such an odd resolution.)

​3. Rendering in Blender

To get the sample video I rendered a little Christmas scene in Blender.

You can see the box of cameras in the 3d view, and in the object list as Camera.B, Camera.F, ..., and so on. The first step was to render a video from each of these cameras, then assemble them into the video cube map.

​4. HTML 5 Video

Video on HTML is broken. The inability of the browser vendors to agree on a single format that is guaranteed to play means that we're right back in pre-web standards days. In no particular order:

• Every browser does it differently. Just how the video element works is subtly different among the browsers.

• Every browser supports different video formats - this goes for different versions of the same browser as well.

• The video will only loop in Chrome if the web server responds with 206 Partial Content when the video player requests a range of bytes.

• As an illustration of how bad it is, HTMLMediaElement.canPlayType[h] doesn't return true or false, but rather the strings "probably", "maybe" or "". Indeed, that was the empty string signaling "no". There is no reliable way to find out if a video format is supported, even by attempting to load it.