360 Audio

Spatial audio support is currently available using Facebook Audio 360 on Windows desktop and Android. On Windows, only Windows 10 and above is supported and the Media Foundation video API must be selected. On Android the ExoPlayer video API must be selected. The video files must be using a MKV file container and audio must be using the Opus codec encoded with Facebook Audio 360 tools.

Note

Unfortunately Meta/Facebook are no longer updating their "Facebook Audio 360" technology. The download links for the Facebook 360 Spatial Workstation authoring tools are currently broken but apparently will be made live soon.

The best way to encode the video is to use the FB360 Encoder tool which comes as part of the FB360 Spatial Workstation. Set Output Format to “FB360 Matroska (Spatial Workstaton 8 channel)” and then set your video and audio source files and encode your video. This should create a MKV file with 10 channels of Opus audio.

alt_text

The settings are located under the Audio section of the MediaPlayer component. The “Head Transform” field must be set to the transform that represents the player's head so that rotation and positional changes affect the audio rendering. Usually this is the main camera.

“Enable Focus” can be enabled when a specific region of audio in the 360 field needs to be given focus. The rest of the audio has its volume reduced.

Next the Facebook Audio 360 support must be enabled for each platform that needs it via the “Platform Specific” panel. Currently it is only available on Windows desktop and Android.

alt_text

The Channel Mode must be set to the channel encoding mode used when creating the video. Currently this can not be determined automatically. The default is TBE_8_2 which means 8 channels of hybrid ambisonics and 2 channels of head-locked stereo audio.

// Set the head transform for audio
mediaPlayer.AudioHeadTransform = Camera.main;

// Set the MediaPlayer for spatial audio on Android - ExoPlayer API only
mediaPlayer.PlatformOptionsAndroid.videoApi = Android.VideoApi.ExoPlayer;
mediaPlayer.PlatformOptionsAndroid.audioOutput = Android.AudioOutput.FacebookAudio360;
mediaPlayer.PlatformOptionsAndroid.audio360ChannelMode = Audio360ChannelMode.TBE_8_2;

// Set the MediaPlayer for spatial audio on Windows - Media Foundation API only
mediaPlayer.PlatformOptionsWindows.videoApi = Windows.VideoApi.MediaFoundation;
mediaPlayer.PlatformOptionsWindows.audioOutput = Windows.AudioOutput.FacebookAudio360;
mediaPlayer.PlatformOptionsWindows.audio360ChannelMode = Audio360ChannelMode.TBE_8_2;

Encoding with Facebook 360 Spatial WorkStation

More information on encoding etc can be found on the Facebook Audio 360 website at: https://github.com/facebookarchive/facebook-360-spatial-workstation.

Alternative steps for encoding manually

  1. Create a WAV file with the audio format they need (Eg 9 channels ambisonics with 2 channels of head-locked audio will require a 11 channel WAV file with the 2 head-locked channels at the end)

  2. Use Opus tools, as described here to convert the WAV file to Opus: https://opus-codec.org/downloads/ https://github.com/facebookarchive/facebook-360-spatial-workstation/blob/main/docs/Documentation/SDK/Audio360_SDK_GettingStarted.html#encoding-opus-files

  3. Use ffmpeg to mux this opus file into the video container (ensure that the video file doesn’t have any audio first):

    ffmpeg -i audio.opus -i video.mp4 -c:a copy -c:v copy audio_video.mkv

  4. In AVPro Video specify the required channel map

Converting existing ambisonic videos

It is also possible to convert existing ambisonic videos so they are compatible. For example if you have an existing MP4 file with 4-channel 1st order ambisonic audio, then it is possible to convert this into the above format (MKV container with Opus audio) using a tool like FFMPEG. Simply put the following command in a .BAT file and then drag your MP4 into the batch file:

ffmpeg -y -i input.mp4 -c:v copy -acodec libopus -mapping_family 255 output.mkv

This should then generate an MKV file that you can play with AVPro Video. All that remains is to set the channel mapping in the MediaPlayer component to AMBIX_4.