Video Manipulation in iOS : Resizing,Merging and Overlapping Videos in iOS

We all have seen video merging applications on Appstore where you can add multiple videos one after one other or inserting videos between other videos.Like the Video Editor App on Appstore.

But what if we want to resize the Video and Merge Multiple videos so that they overlap Each other and both Videos are playing at the same time.Much like the Google Video and Skype Video where you can see yourself in a small video screen overlapping the video of the other participant.

Surprisingly there is very little support on how to implement this and so far i have not found a decent tutorial on how to achieve this,Thus writing this tutorial for any one else who wants to add same features to his application.

It might look hard but actually it is a very simple feature.You just have to dig a little deeper.Apple has provided a very clean and efficient way to manipulate resize and overlap your Videos and thiese features where also explained in WWDC2010.So by digging a little deeper you have to watch the WWDC2010 “Editing Media with AV Foundation” videos and start from there.Lucky for you you will find everything you need write here in this tutorial.

So lets get started.

Our Aim : We have 2 input .mp4 video files and we want to merge them into one single video such that one video overlaps the other and are playign simultaneously.

So here is how our final Video will look like.

Overlaping Videos

See how the smaller video is overlapping the bigger video and YES both are playing simultaneously as well.

How to Do this:

AvFoundation is really power full framework if you want to work and edit media in iOS.

Starting from iOS4.0 we can use AVComposition object combines media data from multiple file-based sources in a custom temporal arrangement, in order to present or process media data from multiple sources together. All file-based audiovisual assets are eligible to be combined, regardless of container type. The tracks in anAVComposition object are fixed; to change the tracks, you use an instance of its subclass, AVMutableComposition.

At its top-level, AVComposition is a collection of tracks, each presenting media of a specific media type, e.g. audio or video, according to a timeline. Each track is represented by an instance of AVCompositionTrack.
A higher-level interface for constructing compositions is also presented by AVMutableComposition and AVMutableCompositionTrack, offering insertion, removal, and scaling operations without direct manipulation of the trackSegment arrays of composition tracks. This interface makes use of higher-level constructs such as AVAsset and AVAssetTrack, allowing the client to make use of the same references to candidate sources that it would have created in order to inspect or preview them prior to inclusion in a composition.

So in short u have a AVComposition and you can add multiple AVMutableCompositionTrack to it. Each AVMutableCompositionTrack will have a separate video Asset.

Ok enough with the theory lets get started with some code.

//Here where load our movie Assets using AVURLAsset

AVURLAsset* firstAsset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource: @”gizmo” ofType: @”mp4″]] options:nil];

AVURLAsset * secondAsset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource: @”gizmo” ofType: @”mp4″]] options:nil];

 

//Create AVMutableComposition Object.This object will hold our multiple AVMutableCompositionTrack.

AVMutableComposition* mixComposition = [[AVMutableComposition alloc] init];

 

//Here we are creating the first AVMutableCompositionTrack.See how we are adding a new track to our AVMutableComposition.

AVMutableCompositionTrack *firstTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];

//Now we set the length of the firstTrack equal to the length of the firstAsset and add the firstAsset to out newly created track at kCMTimeZero so video plays from the start of the track.

[firstTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, firstAsset.duration) ofTrack:[[firstAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0] atTime:kCMTimeZero error:nil];

 

//Now we repeat the same process for the 2nd track as we did above for the first track.Note that the new track also starts at kCMTimeZero meaning both tracks will play simultaneously.

AVMutableCompositionTrack *secondTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];

[secondTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, secondAsset.duration) ofTrack:[[secondAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0] atTime:kCMTimeZero error:nil];

As you can see that we have added our 2 AVMutableCompositionTrack to our AVMutableComposition,but this is not the end.If you play the newly created composition at this stage you will only see the video of the first Asset.Thsi is because we have not added any instructions to our AVMutableCompositionTrack.Instructions such as video size position and transition effects.

To add these instructions AVFoundations provides us with AVVideoComposition Class.

AVVideoComposition contains an array of AVVideoCompositionInstructions .AVVideoCompositionInstructions object represents an operation to be performed by a compositor and AVVideoComposition object maintains an array of instructions to perform its composition.AVVideoCompositionInstructions itself contains an Array of AVVideoCompositionLayerInstruction defined for each layer.

■ Each AVVideoCompositionInstructions instruction describes the output video in terms of input layers (AVVideoCompositionLayerInstruction)

■ Each AVVideoCompositionLayerInstruction layer has an opacity and an affine transform and we will be using this affine transform to scale and position our videos.

Ok so i know this is a lot to digest so lets get into the code and see how it actually works.

//See how we are creating AVMutableVideoCompositionInstruction object.This object will contain the array of our AVMutableVideoCompositionLayerInstruction objects.You set the duration of the layer.You should add the lenght equal to the lingth of the longer asset in terms of duration.

AVMutableVideoCompositionInstruction * MainInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction];

[MainInstruction retain];

MainInstruction.timeRange = CMTimeRangeMake(kCMTimeZero, firstAsset.duration);

 

//We will be creating 2 AVMutableVideoCompositionLayerInstruction objects.Each for our 2 AVMutableCompositionTrack.here we are creating AVMutableVideoCompositionLayerInstruction for out first track.see how we make use of Affinetransform to move and scale our First Track.so it is displayed at the bottom of the screen in smaller size.(First track in the one that remains on top).

AVMutableVideoCompositionLayerInstruction *FirstlayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:firstTrack];

CGAffineTransform Scale = CGAffineTransformMakeScale(0.7f,0.7f);

CGAffineTransform Move = CGAffineTransformMakeTranslation(230,230);

[FirstlayerInstruction setTransform:CGAffineTransformConcat(Scale,Move) atTime:kCMTimeZero];

 

//Here we are creating AVMutableVideoCompositionLayerInstruction for out second track.see how we make use of Affinetransform to move and scale our second Track.

AVMutableVideoCompositionLayerInstruction *SecondlayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:secondTrack];

CGAffineTransform SecondScale = CGAffineTransformMakeScale(1.2f,1.5f);

CGAffineTransform SecondMove = CGAffineTransformMakeTranslation(0,0);

[SecondlayerInstruction setTransform:CGAffineTransformConcat(SecondScale,SecondMove) atTime:kCMTimeZero];

 

//Now we add our 2 created AVMutableVideoCompositionLayerInstruction objects to our AVMutableVideoCompositionInstruction in form of an array.

MainInstruction.layerInstructions = [NSArray arrayWithObjects:FirstlayerInstruction,SecondlayerInstruction,nil];;

 

//Now we create AVMutableVideoComposition object.We can add mutiple AVMutableVideoCompositionInstruction to this object.We have only one AVMutableVideoCompositionInstruction object in our example.You can use multiple AVMutableVideoCompositionInstruction objects to add multiple layers of effects such as fade and transition but make sure that time ranges of the AVMutableVideoCompositionInstruction objects dont overlap.

AVMutableVideoComposition *MainCompositionInst = [AVMutableVideoComposition videoComposition];

[MainCompositionInst retain];

MainCompositionInst.instructions = [NSArray arrayWithObject:MainInstruction];

MainCompositionInst.frameDuration = CMTimeMake(1, 30);

MainCompositionInst.renderSize = CGSizeMake(640, 480);

 

//Finally just add the newly created AVMutableComposition with multiple tracks to an AVPlayerItem and play it using AVPlayer.

AVPlayerItem * newPlayerItem = [AVPlayerItem playerItemWithAsset:mixComposition];

[newPlayerItem retain];

newPlayerItem.videoComposition = MainCompositionInst;

self.mPlayer = [AVPlayer playerWithPlayerItem:newPlayerItem];

[mPlayer addObserver:self forKeyPath:@"status" options:0 context:AVPlayerDemoPlaybackViewControllerStatusObservationContext];

And thats it.Run the code and you should see 2 videos running side by side overlapping each other.We can multiple videos and resize and rotate and transform them in any way we like thus adding extremely cool effects.

These concepts are hard to understand at first but once you get hold of the concept its real fun to play around with this.

For more details i would recommend see the “Editing Media with AV Foundation” presentation at the WWDC2010.It also come with AVEditDemo which guides you how to add cool transition and fade effects in merged video.

Full code for the above example is available here for download.


If you want any more help just send me an email or leave a comments.

You can follow me at twitter @Abdul_Azeem

 

About these ads

26 Responses to Video Manipulation in iOS : Resizing,Merging and Overlapping Videos in iOS

  1. msteinb says:

    Hi,

    Great post! I was trying to figure out how to do this. However, my client has a specific request. The larger video is going to be some content and the smaller one is going to be someone talking. The user has the option to show/hide the speaker. After scouring the docs I couldn’t find a way to have a specific track (i.e. the speaker video track) respond to user generated events.
    One option is to have multiple AVPlayers but that poses the problem of keeping them in sync.
    Another (farout) option is if there was a way to have specific tracks from one AVPlayer display on different AVPlayerLayers.
    Anyway, you have any ideas?

    • Abdul azeem says:

      Well u can use multiple avplayers .if ur making an app specific for ios u dont need to render a final output u can use avplayers to do so.but if u wana render a final output then it might not be possible to hide the smaller video.plz provide me with some more detail and i will see what can be done

  2. nad says:

    Hi !
    Very helpful and clear for a newbie like me :)
    Thanks a lot !

  3. srini says:

    hi,

    very nice tutorial. But why is that the code crashes when I try to play a different video other than the gizmo.mp4 .Also can I play multiple videos?
    Thanks,

  4. JD says:

    Hi, very helpful tutorial, thanks so much.

    I’m actually working with Xcode 4 and I’m trying to reproduce this tutorial, but I can’t connect the OUTLET
    @property (nonatomic, strong) IBOutlet AVPlayerDemoPlaybackView *mPlaybackView;

    I tried to connect to a simple UIView but Xcode does not allow me, any help would be nice.

    And I’m wondering if by using AVPlayer you have a limited number of video playing at once? I’ve heard about 4 but nothing official from Apple.

  5. AU says:

    Hi,

    This was very helpful, thanks! I’m able to change the video source, however, there is no audio playback. Any help on this?

  6. AU says:

    Can you tell me how I can modify the same for hosted content?

  7. abdul mannan says:

    Could you please explain
    MainCompositionInst.renderSize = CGSizeMake(640, 480);

    Because I am facing cropping issue due to this property(less view then the original video),

  8. dianne says:

    Hi Abdul,
    Your wordpress is the closest code to what I am trying to do. I am trying to play a separate audio file (several to choose from). Then use the video file. Then play them simultaneously. Then save as one file name. Can you please help me?
    Thanks in advance for all your help.

  9. The audio is not recored in this post !!

  10. vishal says:

    hi,
    this is very good tutorial, but i want to add selected frame(image) on video and also add selected audio file to that video and export final video with selected audio and frame and captured video……how to do this ?? please help me!!

    thank in advance…..Mail me on :-iosprofessionals@live.com

  11. Thanks a lot! I’m trying to do another thing with videos, and after 1 month I found this post that helped me a a lot achieving it!! Thanks!! :)

  12. veeru says:

    how to crop a second video frame ?

  13. jignesh says:

    Geat Post:
    Thanks….Saves my lot of time…”God Bless You”

  14. Parvez Belim says:

    Hi anybody know,If I have to crop the video(i.e all video frames) with particular dimension at particular location or coordinate(i.e. x,y).How to do that ???

  15. Emerson de Lira Espínola says:

    Hi, Abdul.

    I’ve got a question about the merging videos in iOS. I have a project that also merges videos and in the exportDidFinish method I also copy the result video in the camera roll.

    However I noticed that my app gets more space in the Settings -> General -> Use. Every time I run it and it merge and copy the new video to the camera roll, I can see its MB increasing in the Use screen. The bigger the video, the bigger the space increased. This way my app can reach 1GB (documents and data) in disk easily.

    My question is: how can I avoid it to increase its space in disk and keep its size as initial install?

    Thanks in advance.

  16. Ram says:

    Hi,

    I have problem after merging videos video’s sound was gone its not play video sound I want video sound otherwise its very fine. Can you please help me to keep video sound to as it.

    Thanks in advance.

  17. kiran yasala says:

    ) Before merging, we take the videos URLs(static videos or camera captured videos) from array, and store them into assets.2) In the processing of storing the URLs into assets,1) First, we take the first URL into ‘firstVideoAsset’, after that we take second URL into ‘secondVideoAsset’. Then merge the two videos and store the merged URL into again ‘firstVideoAsset’.2) After that, take the take third URL into ‘secondVideoAsset’. Them merge the two videos. Same process will go on until the loop ends.3) Send the final merged URL into MPMoviePlayerController to play the ‘preview’ of the merged video.
    Orientation issue:-
    1) While in the merging process, if we capture the video in ‘landscape’ orientation(i.e. LandscapeLeft), the captured video after merging, is coming in ‘Upside-down’ mode(i.e. reverse of captured video).2) The sizes of all the videos(static or camera captured) are not coming in same size after merging.3) If we capture the video using front camera, after merging all, the captured videos are coming in upside down.
    Need help from your side:-
    1) Before merging, if we capture the videos in any orientation mode, they have to come in only one orientation(i.e. LandscapeLeft or LandscapeRight) mode after merging.2) After merging all the videos, the sizes(height and width) of all the videos(static or camera captured) in any orientation mode, should come in same sizes.3) While using front camera, if we capture the video in Landscape left mode, after merging all the videos, the captured videos will come correct mode. But, when capture the video in Landscape right mode, the captured videos after merging all the videos will come in Upside down.(i.e. Reverse of original captured video).4) While using back camera, if we capture the video in Landscape left mode, after merging all the videos, the captured videos will come in Upside down.(i.e. Reverse of original captured video) . But, when capture the video in Landscape right mode, the captured videos after merging all the videos will come in correct mode.

  18. kiran yasala says:

    Hi Abdul Sir,

    I have problem that I mentioned in the following

    1) Before merging, we take the videos URLs(static videos or camera captured videos) from array, and store them into assets.

    2) In the processing of storing the URLs into assets,

    1) First, we take the first URL into ‘firstVideoAsset’, after that we take second URL into ‘secondVideoAsset’. Then merge the two videos and store the merged URL into again ‘firstVideoAsset’.

    2) After that, take the take third URL into ‘secondVideoAsset’. Them merge the two videos. Same process will go on until the loop ends.
    3) Send the final merged URL into MPMoviePlayerController to play the ‘preview’ of the merged video.

    Orientation issue:-

    1) While in the merging process, if we capture the video in ‘landscape’ orientation(i.e. LandscapeLeft), the captured video after merging, is coming in ‘Upside-down’ mode(i.e. reverse of captured video).

    2) The sizes of all the videos(static or camera captured) are not coming in same size after merging.

    3) If we capture the video using front camera, after merging all, the captured videos are coming in upside down.

    Need help from your side:-

    1) Before merging, if we capture the videos in any orientation mode, they have to come in only one orientation(i.e. LandscapeLeft or LandscapeRight) mode after merging.

    2) After merging all the videos, the sizes(height and width) of all the videos(static or camera captured) in any orientation mode, should come in same sizes.

    3) While using front camera, if we capture the video in Landscape left mode, after merging all the videos, the captured videos will come correct mode. But, when capture the video in Landscape right mode, the captured videos after merging all the videos will come in Upside down.(i.e. Reverse of original captured video).

    4) While using back camera, if we capture the video in Landscape left mode, after merging all the videos, the captured videos will come in Upside down.(i.e. Reverse of original captured video) . But, when capture the video in Landscape right mode, the captured videos after merging all the videos will come in correct mode.

  19. Reblogged this on MobileDesk and commented:
    Video Manipulation in iOS : Resizing,Merging and Overlapping Videos in iOS

  20. jatin says:

    Hey Abdul

    Thanks for this post, I am trying to write a video in a similar way but is there a way to increase or decrease the speed of video when you save it. Is it related to frame duration property?

    http://stackoverflow.com/questions/21084047/how-to-increase-speed-of-the-video-when-saving-it-by-using-avassetexportsession

    Thanks

    Jatin

  21. Raju says:

    i have merged three videos successfully but i couldn’t keep the video which stopped earlier than other videos…..ho could i fix this …………….

  22. darshan says:

    fantastic tutorial :)

  23. Jonathan says:

    Hi there,

    Yes, this is an outstanding tutorial and a great intro to customizing media assets in iOS.

    The one small issue I have is I also have lost the audio signal upon merge. Any thoughts on this? Otherwise, I’m able to make it work really well. Thanks!

  24. naveen says:

    hi ,

    how to track the time while video recording using avfoundaiton(AVCaptureMovieFileOutput)? I am able to track the time after completion of the video but how can we track time while recording. I have maintain the nstimer but it’s not giving the exact time. It’s shows the difference time from the video.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: