Merging Audio and Video in Native iOS

The goal of this article is to demonstrate how to create a video by merging a video and audio file in an iOS application.

In the application pictured below, video and audio are merged to create a single video using the built-in AV Foundation framework. Before going through the code, we’ll discuss some of the basic classes and their methods and properties.

AVMutableComposition and AVMutableCompositionTrack

These classes are subclasses of the AVComposition class. An object of AVComposition combines multiple file-sources and processes media data from multiple sources together.

AVComposition is a collection of fixed tracks. To make any changes within a track, you need an object of type AVMutableComposition. AVMutableComposition is a collection of AVMutableCompositionTrack that lets you to insert, remove, and scale track segments. Some of its useful methods and properties are listed below:

Methods

• insertTimeRange:ofTrack:atTime:error: – Inserts a time range of a source track.
• addMutableTrackWithMediaType:preferredTrackID: – Adds an empty track to the receiver.

Properties

• Tracks is an array of AVMutableCompositionTrack objects conatained by AVMutableComposition.
• naturalSize is an encoded or authorized size of the visual portion of the asset.

AVURLAsset

This class is the subclass of the AVAsset class. AVAsset is an abstract class that represents timed audiovisual media such as videos and sounds. Itscontains the collection of tracks that are intended to be processed together. AVURLAsset is used to initialize an asset. Some of its useful methods and properties are listed below.

Methods

• initWithURL:Options: – Initializes an asset referenced by a given URL.
• URLAssetWithURL:Options: – Returns an asset referenced by a given URL.

Properties

• URL is the URL with which the asset was initialized. This is a read-only property.

AVAssetExportSession

An object of this class transcodes the contents of an AVAsset source object to create an output of the form described by a specified export preset. Some of its useful methods & properties are listed below.

Methods

• initWithAsset:presetName: – Initializes an asset export session with specified asset and preset.
• exportSessionWithAsset:presetName: – Returns an asset export session configured with a specified asset and preset.
• exportAsynchronouslyWithCompletionHandler: – Starts the asynchronous execution of an export session.

Properties

• OutputURL – Contains the URL of the export session’s output.
• presetName – The name of the preset with which the session was initialized. This is a read-only property.
• outputFileType – The type of the file to be written by the session.

Building the App

Lets take a look at the code required to merge audio and video.

Creating the Project

Create a new Single View Application in Xcode and name it AudioVideoMergeDemo targeting only the iPhone.

Building the UI

Now that you have a blank storyboard, let’s put together the simple layout shown on first screenshot.

For that, you will use following controls:

1. View – name: vwMoviePlayer
2. Button – name: btnMergeTapped
3. Activity Indicator View – name: activityView

You’ll also need to place a Navigation Controller:

• Select your viewController in the StoryBoard.
• In your MenuBar : Editor embed a Navigation Controller

Now, your storyboard looks like:

Linking the Library and Adding Media Files

The basic requirement to merge video and audio is the AVFoundation framework, so you have to link this library to your project.

1. You may do so by clicking on the “General” tab and scrolling to the bottom under “Linked Framework and Libraries” and adding `AVFoundation.framework` and then press the “Add” button to link the framework to your project.
2. Next you’ll need any video file and audio file. I’m working with .MKV and .CAF in my example. Right click on the project folder and choose “Add Files to AudioVideoMergeDemo.”
3. Now select your audio and video file, making sure that the checkbox to copy items into the destination group’s folder is checked (if necessary) and press “Add” as shown below:

Writing the Code

Open the ViewController.h header file and import AVFoundation, MediaPlayer and the AssetsLibrary:

#import <AVFoundation/AVFoundation.h>
#import <AssetsLibrary/AssetsLibrary.h>
#import <MediaPlayer/MediaPlayer.h>

Connect your storyboard controls with this header file. Beneath the previous code, add the following code:

@interface ViewController : UIViewController
{
    MPMoviePlayerController *moviePlayer;
}

- (IBAction)btnMergeTapped:(id)sender;
- (void)exportDidFinish:(AVAssetExportSession*)session;

@property(nonatomic,retain)AVURLAsset* videoAsset;
@property(nonatomic,retain)AVURLAsset* audioAsset;
@property (weak, nonatomic) IBOutlet UIActivityIndicatorView *activityView;
@property (weak, nonatomic) IBOutlet UIView *vwMoviePlayer;
@end

Within the ViewController.m file, add a synthesize property which we’ll place beneath the @implementation line:

@synthesize videoAsset,audioAsset;
@synthesize activityView;
       @synthesize vwMoviePlayer;

And, set property for activityView and vwMoviePlayer in the viewDidLoad method under the [super viewDidLoad] line:

       [activityView setHidden:YES];
       [vwMoviePlayer setHidden:YES];

Let’s add a method for btnMergeTapped:

- (IBAction)btnMergeTapped:(id)sender {
[activityView setHidden:NO];
[activityView startAnimating];
[vwMoviePlayer setHidden:YES];     
[self performSelector:@selector(mergeAndSave) withObject:nil afterDelay:.6];
 }

Next, within the ViewController.m implementation file, copy the below code:

-(void)mergeAndSave
{
    //Create AVMutableComposition Object which will hold our multiple AVMutableCompositionTrack or we can say it will hold our video and audio files.
    AVMutableComposition* mixComposition = [AVMutableComposition composition];

    //Now first load your audio file using AVURLAsset. Make sure you give the correct path of your videos.
    NSURL *audio_url = [NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"Asteroid_Sound" ofType:@"mp3"]];
    AVURLAsset  *audioAsset = [[AVURLAsset alloc]initWithURL:audio_url options:nil];
    CMTimeRange audio_timeRange = CMTimeRangeMake(kCMTimeZero, audioAsset.duration);

    //Now we are creating the first AVMutableCompositionTrack containing our audio and add it to our AVMutableComposition object.
    AVMutableCompositionTrack *b_compositionAudioTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
    [b_compositionAudioTrack insertTimeRange:audio_timeRange ofTrack:[[audioAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0] atTime:kCMTimeZero error:nil];

    //Now we will load video file.
    NSURL *video_url = [NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"Asteroid_Video" ofType:@"m4v"]];
    AVURLAsset  *videoAsset = [[AVURLAsset alloc]initWithURL:video_url options:nil];
    CMTimeRange video_timeRange = CMTimeRangeMake(kCMTimeZero,audioAsset.duration);

    //Now we are creating the second AVMutableCompositionTrack containing our video and add it to our AVMutableComposition object.
    AVMutableCompositionTrack *a_compositionVideoTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
    [a_compositionVideoTrack insertTimeRange:video_timeRange ofTrack:[[videoAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0] atTime:kCMTimeZero error:nil];

    //decide the path where you want to store the final video created with audio and video merge.
    NSArray *dirPaths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString *docsDir = [dirPaths objectAtIndex:0];
    NSString *outputFilePath = [docsDir stringByAppendingPathComponent:[NSString stringWithFormat:@"FinalVideo.mov"]];
    NSURL *outputFileUrl = [NSURL fileURLWithPath:outputFilePath];
    if ([[NSFileManager defaultManager] fileExistsAtPath:outputFilePath])
        [[NSFileManager defaultManager] removeItemAtPath:outputFilePath error:nil];

    //Now create an AVAssetExportSession object that will save your final video at specified path.
    AVAssetExportSession* _assetExport = [[AVAssetExportSession alloc] initWithAsset:mixComposition presetName:AVAssetExportPresetHighestQuality];
    _assetExport.outputFileType = @"com.apple.quicktime-movie";
    _assetExport.outputURL = outputFileUrl;

    [_assetExport exportAsynchronouslyWithCompletionHandler:
     ^(void ) {

         dispatch_async(dispatch_get_main_queue(), ^{
             [self exportDidFinish:_assetExport];
         });
     }
     ];
}

Working with the Generated File

You can implement the following method in the VideoController.m if you want to perform any task on the final generated video:

- (void)exportDidFinish:(AVAssetExportSession*)session {
if(session.status == AVAssetExportSessionStatusCompleted) {
   NSURL *outputURL = session.outputURL;
   ALAssetsLibrary *library = [[ALAssetsLibrary alloc] init];
   if([library videoAtPathIsCompatibleWithSavedPhotosAlbum:outputURL]) {
     [library writeVideoAtPathToSavedPhotosAlbum:outputURL
             completionBlock:^(NSURL *assetURL, NSError *error) {
                                           dispatch_async(dispatch_get_main_queue(), ^{
         if (error) {
       UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"Error"  message:@"Video Saving Failed"  delegate:nil cancelButtonTitle:@"Ok" otherButtonTitles: nil, nil];
             [alert show];
         } else {
             UIAlertView *alert = [[UIAlertView alloc] 
      initWithTitle:@"Video Saved" message:@"Saved To Photo Album"      delegate:self cancelButtonTitle:@"Ok" otherButtonTitles: nil];
             [alert show];
             [self loadMoviePlayer:outputURL];
         }
      });
    }];
  }
}
audioAsset = nil;
videoAsset = nil;
[activityView stopAnimating];
[activityView setHidden:YES];
}

We call the loadMoviePlayer method in the above code. We can implement it as shown below:

-(void)loadMoviePlayer:(NSURL*)moviePath {
    moviePlayer = [[MPMoviePlayerController alloc]
    initWithContentURL:moviePath];
    moviePlayer.view.hidden = NO;
    moviePlayer.view.frame = CGRectMake(0, 0, vwMoviePlayer.frame.size.width,     
    vwMoviePlayer.frame.size.height);
    moviePlayer.view.backgroundColor = [UIColor clearColor];
    moviePlayer.scalingMode = MPMovieScalingModeAspectFit;
    moviePlayer.fullscreen = NO;
    [moviePlayer prepareToPlay];
    [moviePlayer readyForDisplay];
    [moviePlayer setControlStyle:MPMovieControlStyleDefault];
    moviePlayer.shouldAutoplay = NO;
    [vwMoviePlayer addSubview:moviePlayer.view];
    [vwMoviePlayer setHidden:NO];
}

Build the Project

Build and run the application. Press the “Merge and Save” button and the merged file will appear in your “Photo and Video” library in the simulator.

Where to Go From Here

Hopefully this project got your feet wet with the idea of merging audio and video on iOS. Below are some resources if you’d like to go further.

1. The GitHub project for this sample app.
2. Ray Wenderlich on How to Play, Record, and Edit Videos in iOS
3. How to merge Audio and video using AVMutableCompositionTrack from StackOverflow