Combining video, audio and mocap

Overview

Teaching: 5 min
Exercises: 20 min

Questions

• How is it possible to analyze video, audio and mocap together?

Objectives

• Learn to create and manipulate MGT structures

This example is showing how it is possible to use MGT for the combined analysis of visualization of video data together with video and audio. As opposed to the previous example, this requires the use of the MGT data structure for handling synchronization of the data streams.

Reading in multiple data files

We start by reading in three files from the same recording session: video (.mp4), audio (.wav) and motion capture data (.tsv):

mg = mgread('pianist.avi','pianist.wav','pianist.tsv');

This leads to the structure mg containting the following information:

These are three very different types of files, and they therefore have to be treated differently inside of Matlab. The aim is that MGT should take care of most of these differences, so that you can work on the MGT structure directly.

Trimming the structure

The benefit of using an MGT structure is that it will handle the trimming of all related files and data:

mgseg = mgreadsegment(mg,10,30);

Here mgseg is a new (and complete) structure that can be used for further processing.

mgsegmo = mgmotion(mgseg,'Diff','Regular',0.1);

Plotting together

The function mgvideoplot plots the horizontal and vertical motiongram combined with mocapgram over time.

mgvideoplot(mgsegmo);

Next, we can investigate the audio recording. mgwaveplot plots the waveform of audio, root mean square, and their spectrum.

mgwaveplot(mgsegmo);

Periodicity

Next,the periodicity of movement might be investigated from the quantity of motion. Let’s have a look at it.

[per,ac,eac,lag] = mgautocor(mgsegmo,'video');

The autocorrelation function for the data shows periodicities:

plot(lag, ac), xlabel('Period / secs')

The enhanced autocorrelation function for the same data looks like this:

plot(lag, eac), xlabel('Period / secs')

Statistics

If we want to look at some features of motion image, function mgstatistics can generate first order and second order features.

features1 = mgstatistics(mgsegmo,'Video','FirstOrder');
features2 = mgstatistics(mgsegmo,'Video','SecondOrder');

Key Points

• An MGT struct can be used to organize and manipulate different types of data.