AVFoundation框架(六) 媒体数据的编辑- 读取与写入
2017-04-20 本文已影响261人
ValienZh
视频应用的大部分场景使用前面介绍的AVFoundation各种功能即可.但有时候会遇到特殊要求,这时就可能需要直接编辑处理媒体样本来解决.
1. 读取和写入
WechatIMG2.jpeg
AVAssetReader用于从AVAsset实例中读取媒体样本. 而对应的把媒体资源进行编码并写入容器文件的是AVAssetWriter类;
-
AVAssetReader:
AVAssetReader通常会配置一个或多个AVAssetReaderOutput对象,并通过其copyNextSampleBuffer方法访问音视频帧. AVAssetReaderOutput是一个抽象类,它主要是用来从指定的AVAssetTrack中读取解码的媒体样本.
AVAssetReader只针对带有一个资源的媒体样本.如果需要同时从多个基于文件的资源中读取样本, 可将他们组合到一个AVComposition中. 另外, AVAssetReader虽然是以多线程的方式不断读取下一个可用样本,但是依旧不适合实时操作,比如播放.
- ** AVAssetWriter:**
它通常配置一个或多个AVAssetWriterInput对象,用于附加要写入的媒体样本的CMSampleBuffer对象;AVAssetWriterInput可以被配置来处理指定的媒体类型,例如音频或视频,并且附加在其后的样本会在最终输出时生成一个独立的AVAssetTrack.
当使用一个配置了处理视频样本的AVAssetWriterInput时,通常会用到一个专门的适配器对象
AVAssetWriterInputPixelBufferAdaptor. 这个类在附加被包装成CVPixelBuffer对象的视频样本时可以提供最优性能.
另外, 可通过AVMediaSelectionGroup和AVMediaSelectionOption参数设置来创建特定资源.
AVAssetWriter支持自动交叉媒体样本 (音视频错开存储,方便读取), 为了保持这个方式,只有AVAssetWriterInput的readyForMoreMediaData属性为YES时才可以将更多新的样本添加到写入信息中.
AVAssetWriter有实时操作和离线操作两种情况;
-
实时:处理实时资源,比如从AVCapturerVideoOutput写入正在捕捉的媒体样本时,AVAssetWriterInput应该令
expectsMediaDataInRealTime属性为YES来确保readyForMoreMediaData值被正确计算. 从而音视频自然交错存储.优化实时资源数据写入器. -
离线: 当从离线资源中读取资源时, 比如从AVAssetReader读取样本buffer, 在附加样本 前仍然需要观察写入器输入的
readForMoreMediaData属性状态,不过可以使用requestMediaDataWhenReadyOnQueue: usingBlock:方法控制数据的提供. 这个block中代码会随写入器 输入 准备附加更多的样本而不断被调用.添加样本是,开发者需要检索数据并从资源中找到下一个样本进行添加.
1.2 离线非实时的读写示例
使用AVAssetReader直接从资源文件中读取样本,并使用AVAssetWriter写入一个新的QuickTime文件中.
// 0. 获取资源。
AVAsset *asset = [AVAsset assetWithURL:url];
// 1. 配置AVAssetReader
AVAssetTrack *track = [[asset trackWithMediaType:AVMediaTypeVideo] firstObject];
self.assetReader = [[AVAssetReader alloc] initWithAsset:asset error:nil];
NSDictionary *readerOutputSettings = @{
(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA)
};
AVAssetReaderTrackOutput *trackOutput = [[AVAssetReaderTrackOutput alloc] initWithTrack:track outputSettings: readerOutputSettings];
[self.assetReader addOutput: trackOutput];
// 2. 开始读取
[self.assetReader startReading];
// 配置AVAssetWriter , 传递一个新文件的写入地址和类型
self.assetWriter = [[AVAssetWriter alloc] initWithURL:outputURL fileType:AVFileTypeQuickTime];
NSDictionary * writerOutputSettings =@{
AVVideoCodecKey : AVVideoCodecH264,
AVVideoWidthKey : @1280,
AVVideoHeightKey :@720,
AVVideoCompressionPropertiesKey : @{
AVVideoMaxKeyFrameIntervalKey : @1,
AVVideoAverageBitRateKey : @10500000,
AVVideoProfileLevelKey : AVVideoProfileLevelH264Main31,
}
};
AVAssetWriterInput *writerInput = [[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeVideo
outputSettings: writerOutputSettings];
[self.assetWriter addInput: writerInput];
// 开始写入
[self.assetWriter startWriting];
之前介绍过
AVAssetExportSession也可以用来导出新资源文件, AVAssetWriter与其对比的优势在于,它可以在进行输出编码时有更多的控制,比如指定关键帧间隔,视频比特率,像素宽高比,H264配置文件等.
在完成AVAssetReader和AVAssetWriter对象设置之后,创建一个新的写入会话来完成读取写入过程. 对于离线模式,一般使用pull model方式 , 即当AssetWriterInput准备好附加更多样本时才从资源中拉取样本.
dispatch_queue_t dispatchQueue = dispatch_queue_create("writerQueue",NULL);
// 创建一个新的写入会话, 传入开始时间
[self.assetWriter startSessionAtSourceTime:kCMTimeZero ];
writerInput requestMediaDataWhenReadyOnQueue usingBlock:^{
BOOL complete = NO;
while ([writerInput isReadYForMoreMediaData] && !conmpelte) { // pull model . 这个代码块在写入器准备好添加更多样本时会不断调用. 在这期间,从trackOutput 复制可用的sampleBuffer,并添加到输入中.
CMSampleBufferRef sampleBuffer = [trackOutput copyNextSampleBuffer];
if (sampleBuffer) {
BOOL result = [writerInput appendSampleBuffer:sampleBuffer];
CFRelease(sampleBuffer);
complete = !result;
} else {
[writerInput markAsFinished];
complete = YES;
}
// 直到所有sampleBuffer添加写入完成. 关闭会话
if (complete) {
[self.assetWriter finishWritingWithCompletionHandler:^{
AVAssetWriterStatus status = self.assetWriter.status;
if(status == AVAssetWriterStatusComplete) {
// 写入成功
} else {
//写入失败
}
}];
}
}];
2. 绘制音频波形图
有些应用可能要显示音频波动. 使用波形图; 一般分三步.
- 读取:首先读取或解压音频数据.
- **缩减: **从资源中实际读取的样本数量会远比我们在屏幕上渲染的多的多, 所以必须缩减这个样本集合来方便呈现. 通常方法是降样本总量分为小的样本块,并在每个样本块中找到最大样本\所有样本的平均值或min/max 值.
- 渲染: 绘制缩减后的样本即可.
3. 捕捉录制的高级方法
之前介绍过通过AVCaptureVideoDataOutput捕捉CVPixelBuffer对象 再来OpenGL渲染制作视频特效. 但这有一个小问题, 使用AVCaptureVideoDataOutput就会失去AVCaptureMovieFileOutput来记录Output的便捷性. 无法记录就不能分享到其他地方. 下面我们使用AVAssetWriter自己实现从CaptureOutput中记录输出.
- 首先,第一步还是配置session.
-(BOOL)setupSession:(NSError **)error {
self.captureSession = [[AVCaptureSession alloc] init];
self.captureSession.sessionPreset = AVCaptureSessionPresetHigh;
if (![self setupSessionInputs:error]) {
return NO;
}
if (![self setupSessionOutputs:error]) {
return NO;
}
return YES;
}
//
-(BOOL)setupSessionInputs:(NSError **)error {
// Set up default camera device
AVCaptureDevice *videoDevice =
[AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
AVCaptureDeviceInput *videoInput =
[AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:error];
if (videoInput) {
if ([self.captureSession canAddInput:videoInput]) {
[self.captureSession addInput:videoInput];
self.activeVideoInput = videoInput;
} else {
NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"Failed to add video input."};
*error = [NSError errorWithDomain:THCameraErrorDomain
code:THCameraErrorFailedToAddInput
userInfo:userInfo];
return NO;
}
} else {
return NO;
}
// Setup default microphone
AVCaptureDevice *audioDevice =
[AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio];
AVCaptureDeviceInput *audioInput =
[AVCaptureDeviceInput deviceInputWithDevice:audioDevice error:error];
if (audioInput) {
if ([self.captureSession canAddInput:audioInput]) {
[self.captureSession addInput:audioInput];
} else {
NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"Failed to add audio input."};
*error = [NSError errorWithDomain:THCameraErrorDomain
code:THCameraErrorFailedToAddInput
userInfo:userInfo];
return NO;
}
} else {
return NO;
}
return YES;
}
//
-(BOOL)setupSessionOutputs:(NSError **)error {
self.videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
// kCVPixelFormatType_32BGRA格式适用于OpenGL和CoreImage.
NSDictionary *outputSettings =
@{(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA)};
self.videoDataOutput.videoSettings = outputSettings;
self.videoDataOutput.alwaysDiscardsLateVideoFrames = NO; // 由于要记录输出内容,设置NO,这样会给捕捉回调方法一些额外的时间来处理样本buffer.
[self.videoDataOutput setSampleBufferDelegate:self
queue:self.dispatchQueue];
if ([self.captureSession canAddOutput:self.videoDataOutput]) {
[self.captureSession addOutput:self.videoDataOutput];
} else {
return NO;
}
self.audioDataOutput = [[AVCaptureAudioDataOutput alloc] init]; // 捕捉音频内容.
[self.audioDataOutput setSampleBufferDelegate:self
queue:self.dispatchQueue];
if ([self.captureSession canAddOutput:self.audioDataOutput]) {
[self.captureSession addOutput:self.audioDataOutput];
} else {
return NO;
}
// 下面代码是调用第二步THMovieWriter类的地方.
NSString *fileType = AVFileTypeQuickTimeMovie;
NSDictionary *videoSettings =
[self.videoDataOutput
recommendedVideoSettingsForAssetWriterWithOutputFileType:fileType];
NSDictionary *audioSettings =
[self.audioDataOutput
recommendedAudioSettingsForAssetWriterWithOutputFileType:fileType];
self.movieWriter =
[[THMovieWriter alloc] initWithVideoSettings:videoSettings
audioSettings:audioSettings
dispatchQueue:self.dispatchQueue];
self.movieWriter.delegate = self;
return YES;
}
// 回调处理捕捉到的sampleBuffer
-(void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection {
// 这个方法在第二步中解释.
[self.movieWriter processSampleBuffer:sampleBuffer];
if (captureOutput == self.videoDataOutput) {
CVPixelBufferRef imageBuffer =
CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage *sourceImage =
[CIImage imageWithCVPixelBuffer:imageBuffer options:nil];
// 传递给屏幕显示图片.
[self.imageTarget setImage:sourceImage];
}
}
上面为
AVCaptureVideoDataOutput和AVCaptureAudioDataOutput使用了一个调度队列,对于示例是足够的, 但是如果希望对数据进行更复杂的处理,可能需要为每一个使用单独的队列.详细参考苹果示例代码RosyWriter.
- 之后就是实现记录输出的内容.这需要创建一个与
AVCaptureMovieFileOutput功能类似的新对象THMovieWriter.它使用AVAssetWriter来执行视频编码和文件写入;
// .h
#import <AVFoundation/AVFoundation.h>
@protocol THMovieWriterDelegate <NSObject>
- (void)didWriteMovieAtURL:(NSURL *)outputURL; // 定义代理提示什么时候影片文件被写入磁盘.
@end
@interface THMovieWriter : NSObject
- (id)initWithVideoSettings:(NSDictionary *)videoSettings // 两个字典用来描述AVAssetWriter配置参数和调度对象.
audioSettings:(NSDictionary *)audioSettings
dispatchQueue:(dispatch_queue_t)dispatchQueue;
// 另外定义开始和停止写入进程的接口方法.
- (void)startWriting;
- (void)stopWriting;
@property (nonatomic) BOOL isWriting;
@property (weak, nonatomic) id<THMovieWriterDelegate> delegate;
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer; // 在每当有新的样本被捕捉到时调用这个方法, 用于处理sampleBuffer.
@end
// .m
static NSString *const THVideoFilename = @"movie.mov";
@interface THMovieWriter ()
@property (strong, nonatomic) AVAssetWriter *assetWriter;
@property (strong, nonatomic) AVAssetWriterInput *assetWriterVideoInput;
@property (strong, nonatomic) AVAssetWriterInput *assetWriterAudioInput;
@property (strong, nonatomic)
AVAssetWriterInputPixelBufferAdaptor *assetWriterInputPixelBufferAdaptor;
@property (strong, nonatomic) dispatch_queue_t dispatchQueue;
@property (weak, nonatomic) CIContext *ciContext;
@property (nonatomic) CGColorSpaceRef colorSpace;
@property (strong, nonatomic) CIFilter *activeFilter;
@property (strong, nonatomic) NSDictionary *videoSettings;
@property (strong, nonatomic) NSDictionary *audioSettings;
@property (nonatomic) BOOL firstSample;
- (id)initWithVideoSettings:(NSDictionary *)videoSettings
audioSettings:(NSDictionary *)audioSettings
dispatchQueue:(dispatch_queue_t)dispatchQueue {
self = [super init];
if (self) {
_videoSettings = videoSettings;
_audioSettings = audioSettings;
_dispatchQueue = dispatchQueue;
_ciContext = [THContextManager sharedInstance].ciContext; // Core Image 上下文,用于筛选sampleBuffer来得到CVPixelBuffer
_colorSpace = CGColorSpaceCreateDeviceRGB();
// 封装的文件写入管理类.
_activeFilter = [THPhotoFilters defaultFilter];
_firstSample = YES;
NSNotificationCenter *nc = [NSNotificationCenter defaultCenter]; // 注册通知,当用户切换筛选器时调用.用于更新activeFilter属性.
[nc addObserver:self
selector:@selector(filterChanged:)
name:THFilterSelectionChangedNotification
object:nil];
}
return self;
}
- (void)startWriting {
dispatch_async(self.dispatchQueue, ^{ // 异步避免按钮点击卡顿.
NSError *error = nil;
NSString *fileType = AVFileTypeQuickTimeMovie;
self.assetWriter = // 创建新的AVAssetWriter
[AVAssetWriter assetWriterWithURL:[self outputURL]
fileType:fileType
error:&error];
if (!self.assetWriter || error) {
NSString *formatString = @"Could not create AVAssetWriter: %@";
NSLog(@"%@", [NSString stringWithFormat:formatString, error]);
return;
}
self.assetWriterVideoInput = // 创建新的AVAssetWriterInput,来附加从captureOutput得到的sampleBuffer.
[[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeVideo
outputSettings:self.videoSettings];
self.assetWriterVideoInput.expectsMediaDataInRealTime = YES; // 告诉应用这个输入 应该对实时性进行优化.
UIDeviceOrientation orientation = [UIDevice currentDevice].orientation;
self.assetWriterVideoInput.transform = // 这里是为捕捉时设备方向做适配.
THTransformForDeviceOrientation(orientation);
NSDictionary *attributes = @{ // 用于配置assetWriterInputPixelBufferAdaptor. 要保证最大效率,字典的值要对应配置AVCaptureVideoDataOutput时的值.
(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA),
(id)kCVPixelBufferWidthKey : self.videoSettings[AVVideoWidthKey],
(id)kCVPixelBufferHeightKey : self.videoSettings[AVVideoHeightKey],
(id)kCVPixelFormatOpenGLESCompatibility : (id)kCFBooleanTrue
};
self.assetWriterInputPixelBufferAdaptor = // 创建,用于提供一个优化的CVPixelBufferPool ,使用它可以创建CVPixelBuffer对象来渲染筛选视频帧.
[[AVAssetWriterInputPixelBufferAdaptor alloc]
initWithAssetWriterInput:self.assetWriterVideoInput
sourcePixelBufferAttributes:attributes];
if ([self.assetWriter canAddInput:self.assetWriterVideoInput]) { // 基本步骤,将视频输入添加到写入器.
[self.assetWriter addInput:self.assetWriterVideoInput];
} else {
NSLog(@"Unable to add video input.");
return;
}
self.assetWriterAudioInput = // 同上面Video的操作.
[[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeAudio
outputSettings:self.audioSettings];
self.assetWriterAudioInput.expectsMediaDataInRealTime = YES;
if ([self.assetWriter canAddInput:self.assetWriterAudioInput]) {
[self.assetWriter addInput:self.assetWriterAudioInput];
} else {
NSLog(@"Unable to add audio input.");
}
self.isWriting = YES; // 表示可以附sampleBuffer了.
self.firstSample = YES;
});
}
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer {
if (!self.isWriting) {
return;
}
CMFormatDescriptionRef formatDesc = // 这个方法会处理音频和视频两类样本. 所以要进行判断分类处理.
CMSampleBufferGetFormatDescription(sampleBuffer);
CMMediaType mediaType = CMFormatDescriptionGetMediaType(formatDesc);
if (mediaType == kCMMediaType_Video) {
CMTime timestamp =
CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
if (self.firstSample) { // 如果是刚开始处理的第一个sampleBuffer,则调用资源写入器开启一个新的写入会话.
if ([self.assetWriter startWriting]) {
[self.assetWriter startSessionAtSourceTime:timestamp];
} else {
NSLog(@"Failed to start writing.");
}
self.firstSample = NO;
}
CVPixelBufferRef outputRenderBuffer = NULL;
CVPixelBufferPoolRef pixelBufferPool =
self.assetWriterInputPixelBufferAdaptor.pixelBufferPool;
OSStatus err = CVPixelBufferPoolCreatePixelBuffer(NULL, // 从PixelBuffer适配器中 创建一个空的CVPixelBuffer,使用该PixelBuffer渲染筛选好的视频帧的output.
pixelBufferPool,
&outputRenderBuffer);
if (err) {
NSLog(@"Unable to obtain a pixel buffer from the pool.");
return;
}
CVPixelBufferRef imageBuffer = // 获取当前sampleBuffer的CVPixelBuffer.根据CVPixelBuffer创建一个新的CIImage. 并将它设置为活动筛选器的kCIInputImageKey值. 通过筛选器来得到输出的图片.
CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage *sourceImage = [CIImage imageWithCVPixelBuffer:imageBuffer
options:nil];
[self.activeFilter setValue:sourceImage forKey:kCIInputImageKey];
CIImage *filteredImage = self.activeFilter.outputImage;
if (!filteredImage) {
filteredImage = sourceImage;
}
[self.ciContext render:filteredImage // 将筛选好的CIImage的输出渲染到上面创建的CVPixelBuffer中.
toCVPixelBuffer:outputRenderBuffer
bounds:filteredImage.extent
colorSpace:self.colorSpace];
if (self.assetWriterVideoInput.readyForMoreMediaData) { // 如果视频输入的此属性为YES. 则将PixelBuffer连同当前样本的呈现时间都附加到assetWriterInputPixelBufferAdaptor适配器. 至此就完成了对当前视频样本的处理.
if (![self.assetWriterInputPixelBufferAdaptor
appendPixelBuffer:outputRenderBuffer
withPresentationTime:timestamp]) {
NSLog(@"Error appending pixel buffer.");
}
}
CVPixelBufferRelease(outputRenderBuffer);
}
else if (!self.firstSample && mediaType == kCMMediaType_Audio) { // 如果第一个样本处理完成且当前为音频样本, 则添加到输入.
if (self.assetWriterAudioInput.isReadyForMoreMediaData) {
if (![self.assetWriterAudioInput appendSampleBuffer:sampleBuffer]) {
NSLog(@"Error appending audio sample buffer.");
}
}
}
}
- (void)stopWriting {
self.isWriting = NO; // 让processSampleBuffer方法停止处理更多样本.
dispatch_async(self.dispatchQueue, ^{
[self.assetWriter finishWritingWithCompletionHandler:^{ // 终止写入会话. 并关闭磁盘上的文件.
if (self.assetWriter.status == AVAssetWriterStatusCompleted) {
dispatch_async(dispatch_get_main_queue(), ^{ // 判断写入器状态, 如果成功写入,回到主线程写入用户的 photos Library
NSURL *fileURL = [self.assetWriter outputURL];
[self.delegate didWriteMovieAtURL:fileURL];
});
} else {
NSLog(@"Failed to write movie: %@", self.assetWriter.error);
}
}];
});
}
// 用于配置AVAssetWriter. 在临时目录定义一个URL,并将之前的同名文件删除.
- (NSURL *)outputURL {
NSString *filePath =
[NSTemporaryDirectory() stringByAppendingPathComponent:THVideoFilename];
NSURL *url = [NSURL fileURLWithPath:filePath];
if ([[NSFileManager defaultManager] fileExistsAtPath:url.path]) {
[[NSFileManager defaultManager] removeItemAtURL:url error:nil];
}
return url;
}
