Native immersive 360° VR video playback on Android with Spectaculum

Playback of immersive 360° video on Android is usually done in a WebView with an HTML5 video player. This tutorial demonstrates how to display 360° video in a native view widget to save the overhead of a whole browser stack. This is done by using the versatile Spectaculum view widget for video rendering and the popular ExoPlayer for video decoding. Both of these libraries are open source under the Apache 2.0 license and available on GitHub and the JCenter repository.

Spectaculum is an OpenGL ES accelerated view widget for visual media content on Android, that comes with an effects architecture to process and transform visual content. For the UI presentation layer, we are going to use the SpectaculumView together with the ImmersiveEffect to render 360° content in equirectangular sphere projection correctly on screen. This effect supports rendering of monoscopic and stereoscopic 3D content in side-by-side and top-and-bottom flavors. We are also going to add ImmersiveTouchNavigation to let the user rotate the viewport with touch gestures. For VR, there is also an ImmersiveSensorNavigation available to navigate the viewport by device rotation sensors, but it is still in experimental stage and therefore not getting much attention in this tutorial. For video decoding and playback handling, we are going to use ExoPlayer 2, the newest incarnation of the de facto standard Android media player for everything where the native VideoView does not suffice. This player is developed by Google and used in various widespread apps, e.g. YouTube.

Required Steps

This tutorial is kept very simple by explaining the few important lines of code necessary to use Spectaculum for video rendering with ExoPlayer. The full sources of the final app are available on GitHub as Spectaculum-Example-Immersive. To construct the working app, we need to do the following:

1. Create a project with an empty Activity in Android Studio.
2. Add dependencies for Spectaculum, Spectaculum’s Immersive module and ExoPlayer to the app’s build.grade file.
   

   compile 'net.protyposis.android.spectaculum:spectaculum:1.1.0'
   compile 'net.protyposis.android.spectaculum:spectaculum-effect-immersive:1.1.0'
   compile 'com.google.android.exoplayer:exoplayer:r2.0.2'

3. Add SpectaculumView to the activity’s layout.
   

   <net.protyposis.android.spectaculum.SpectaculumView
       android:id="@+id/spectaculumview"
       android:layout_width="fill_parent"
       android:layout_height="fill_parent"
       android:layout_centerInParent="true" />

4. Configure SpectaculumView for immersive content.
   

   // Get view references from layout
   mSpectaculumView = (SpectaculumView) findViewById(R.id.spectaculumview);
   
   // Setup Spectaculum view for immersive content
   ImmersiveEffect immersiveEffect = new ImmersiveEffect(); // create effect instance
   mSpectaculumView.addEffect(immersiveEffect); // add effect to view
   mSpectaculumView.selectEffect(0); // activate effect
   
   // Setup Spectaculum immersive viewport touch navigation
   ImmersiveTouchNavigation immersiveTouchNavigation = new ImmersiveTouchNavigation(mSpectaculumView);
   immersiveTouchNavigation.attachTo(immersiveEffect);
   immersiveTouchNavigation.activate(); // enable touch navigation

5. Handling SpectaculumView‘s input surface to ExoPlayer and updating the view’s resolution. Here it is important to use getInputHolder(), which is the input to Spectaculum’s visual processing pipeline, instead of getHolder() like used on a normal SurfaceView.
   

   // Set Spectaculum view as playback surface
   player.setVideoSurface(mSpectaculumView.getInputHolder().getSurface());
   // Attach listener to listen to video size changed events
   player.setVideoListener(new SimpleExoPlayer.VideoListener() {
       @Override
       public void onVideoSizeChanged(int width, int height, int unappliedRotationDegrees, float pixelWidthHeightRatio) {
           // When the video size changes, we update the Spectaculum view
           mSpectaculumView.updateResolution(width, height);
       }
   
       @Override
       public void onRenderedFirstFrame() {}
   
       @Override
       public void onVideoTracksDisabled() {}
   });

Like mentioned before, the above code are the excerpts that are required to use Spectaculum with ExoPlayer. The ExoPlayer initialization and some glue code are left out. You can find the full source code in MainActivity.java.

Virtual Reality

To convert this example into a VR video player, you would replace ImmersiveTouchNavigation with ImmersiveSensorNavigation, but this class still needs some fixing and fine tuning and help is very appreciated. Alternatively, the ImmersiveEffect offers setRotationX/Y/Z methods to set the rotation programmatically in degrees, and a setRotationMatrix method to set the rotation matrix directly. These methods can be used to rotate the viewport and create VR experiences from various sensor sources.

Stereoscopic 3D

For stereoscopic video playback, pass ImmersiveEffect‘s setMode method one of the Mode enum’s STEREO_SBS or STEREO_TAB values for side-by-side and top-and-bottom encoded video.

Result

I suggest checking out the example app from GitHub and running it in Android Studio 2.2. The app needs internet access to stream a hard-coded video, which by default is the Orion360 Test Video. Run the app from Android Studio, and wait until the video is loaded (the first frame is shown). You can then navigate the viewport by swiping the screen.