iOS 音视频采集及rtmp推流
2016-12-07 本文已影响1364人
JiangSy
项目暂时告一段落，也是一知半解，不过我的分享可以帮助我这样菜鸟了。
先来下知识结构：
1、h264
视频编码处理的最后一步就是熵编码，在H.264中采用了两种不同的熵编码方法：通用可变长编码（UVLC）和基于文本的自适应二进制算术编码（CABAC）。
2、aac
Advanced Audio Coding。一种专为声音数据设计的文件压缩格式，与MP3不同，它采用了全新的算法进行编码，更加高效，具有更高的“性价比”。利用AAC格式，可使人感觉声音质量没有明显降低
3、pcm
音频采集的原始数据，硬编码数据
4、yuv
视频采集的原始数据，硬编码数据
5、时间戳
直播音视频同步的关键参数
6、rtmp推流
直播的推流手段
一、我们首先要做的就是采集，我们需要采集硬编码数据yuv将其转化成h264，然后采集pcm数据，并将其转化成aac数据，并发送
1、视频采集，我们要采集最后要转化为h264编码的格式，需要用到VideoToolbox.framework及AVFoundation.framework
VideoToolbox.framework 的主要工作是编码，将yuv数据编码为h264。AVFoundation.framework的任务是采集yuv原始数据。
// 获取硬编码数据函数，一些初始化工作就不在这里熬述了，网上有很多
-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
}
（1）初始化VTCompressionSession。
VTCompressionSession初始化的时候需要给出width宽，height长，编码器类型kCMVideoCodecType_H264等。然后通过调用VTSessionSetProperty接口设置帧率等属性，最后需要设定一个回调函数，这个回调是视频图像编码成功后调用。全部准备好后，使用VTCompressionSessionCreate创建session。
// 这个函数是初始化
- (void) initEncode:(int)width height:(int)height bite:(int)iBite
{
dispatch_sync(aQueue, ^{
// For testing out the logic, lets read from a file and then send it to encoder to create h264 stream
// Create the compression session 注意h264函数
OSStatus status = VTCompressionSessionCreate(NULL, width, height, kCMVideoCodecType_H264, NULL, NULL, NULL, didCompressH264, (__bridge void *)(self), &EncodingSession);
NSLog(@"H264: VTCompressionSessionCreate %d", (int)status);
if (status != 0)
{
NSLog(@"H264: Unable to create a H264 session");
error = @"H264: Unable to create a H264 session";
return ;
}
// 码率是清晰度
// Set the properties
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue);
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_AllowFrameReordering, kCFBooleanFalse);
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, (__bridge CFTypeRef _Nonnull)(@(GOP_SIZE)));
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Main_AutoLevel);
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_AverageBitRate, (__bridge CFTypeRef _Nonnull)@(iBite));
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_ExpectedFrameRate, (__bridge CFTypeRef _Nonnull)@(FRAME_RATE));
VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_DataRateLimits, (__bridge CFTypeRef _Nonnull)@[@(iBite/8),@(1)]);
// Tell the encoder to start encoding
VTCompressionSessionPrepareToEncodeFrames(EncodingSession);
});
}
（2）提取摄像头采集的原始图像数据给VTCompressionSession来硬编码。
摄像头采集后的图像是未编码的CMSampleBuffer形式，利用给定的接口函数CMSampleBufferGetImageBuffer从中提取出CVPixelBufferRef，使用硬编码接口VTCompressionSessionEncodeFrame来对该帧进行硬编码，编码成功后，会自动调用session初始化时设置的回调函数。
dispatch_sync(aQueue, ^{
frameCount++;
// Get the CV Image buffer 提取摄像头采集的原始图像数据给VTCompressionSession来硬编码 也就是给VTCompressionSessionCreate来编码
CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
// Create properties
CMTime presentationTimeStamp = CMTimeMake(frameCount, 1000);
//CMTime duration = CMTimeMake(1, DURATION);
VTEncodeInfoFlags flags;
// Pass it to the encoder
OSStatus statusCode = VTCompressionSessionEncodeFrame(EncodingSession,
imageBuffer,
presentationTimeStamp,
kCMTimeInvalid,
NULL, NULL, &flags);
// Check for error
if (statusCode != noErr) {
NSLog(@"H264: VTCompressionSessionEncodeFrame failed with %d", (int)statusCode);
error = @"H264: VTCompressionSessionEncodeFrame failed ";
// End the session
VTCompressionSessionInvalidate(EncodingSession);
CFRelease(EncodingSession);
EncodingSession = NULL;
error = NULL;
return;
}
// NSLog(@"H264: VTCompressionSessionEncodeFrame Success");
});
（3）利用回调函数，将因编码成功的CMSampleBuffer转换成H264码流，通过网络传播。
基本上是硬解码的一个逆过程。
void didCompressH264(void *outputCallbackRefCon, void *sourceFrameRefCon, OSStatus status, VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer )
{
// NSLog(@"didCompressH264 called with status %d infoFlags %d", (int)status, (int)infoFlags);
NSLog(@"H264");
if (status != 0) return;
if (!CMSampleBufferDataIsReady(sampleBuffer))
{
NSLog(@"didCompressH264 data is not ready ");
return;
}
H264Encoder* encoder = (__bridge H264Encoder*)outputCallbackRefCon;
// Check if we have got a key frame first
bool keyframe = !CFDictionaryContainsKey( (CFArrayGetValueAtIndex(CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true), 0)), kCMSampleAttachmentKey_NotSync);
encoder->countFrame=encoder->countFrame+1;
// NSLog(@"dzf frameCount%d",encoder->countFrame);
if (keyframe)
{
// NSLog(@"dzf keyframe is true ");
CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
// CFDictionaryRef extensionDict = CMFormatDescriptionGetExtensions(format);
// Get the extensions
// From the extensions get the dictionary with key "SampleDescriptionExtensionAtoms"
// From the dict, get the value for the key "avcC"
size_t sparameterSetSize, sparameterSetCount;
const uint8_t *sparameterSet;
OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sparameterSet, &sparameterSetSize, &sparameterSetCount, 0 );
if (statusCode == noErr)
{
// Found sps and now check for pps
size_t pparameterSetSize, pparameterSetCount;
const uint8_t *pparameterSet;
OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pparameterSet, &pparameterSetSize, &pparameterSetCount, 0 );
if (statusCode == noErr)
{
// Found pps
encoder->sps = [NSData dataWithBytes:sparameterSet length:sparameterSetSize];
encoder->pps = [NSData dataWithBytes:pparameterSet length:pparameterSetSize];
if (encoder->_delegate)
{
[encoder->_delegate gotSpsPps:encoder->sps pps:encoder->pps];
}
}
}
}
CMBlockBufferRef dataBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t length, totalLength;
char *dataPointer;
OSStatus statusCodeRet = CMBlockBufferGetDataPointer(dataBuffer, 0, &length, &totalLength, &dataPointer);
if (statusCodeRet == noErr) {
// 发送数据
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;
while (bufferOffset < totalLength - AVCCHeaderLength) {
// Read the NAL unit length
uint32_t NALUnitLength = 0;
memcpy(&NALUnitLength, dataPointer + bufferOffset, AVCCHeaderLength);
// Convert the length value from Big-endian to Little-endian
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
NSData* data = [[NSData alloc] initWithBytes:(dataPointer + bufferOffset + AVCCHeaderLength) length:NALUnitLength];
[encoder->_delegate gotEncodedData:data isKeyFrame:keyframe];
// Move to the next NAL unit in the block buffer
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
// 你小子存的数据
[encoder->_delegate oneFrameEncodeEnd:keyframe];
}
}
值得注意的是一段视频的头部是sps pps 组成的，我们在这个函数中要检查头部信息，筛选普通信息进行封装发送推流。先发送头部数据再发送普通视频数据。
解析出参数集SPS和PPS，加上开始码后组装成NALU。提取出视频数据，将长度码转换成开始码，组长成NALU。将NALU发送出去。
发送视频头部信息代码
- (void)gotSpsPps:(NSData*)sps pps:(NSData*)pps
{
// NSLog(@"gotSpsPps");
frameCount2 = [_h264Encoder getFreameCound];
const char bytes[] = "\x00\x00\x00\x01";
size_t length = (sizeof bytes) - 1; //string literals have implicit trailing '\0'
NSData *ByteHeader = [NSData dataWithBytes:bytes length:length];
mysps = sps;
mypps = pps;
[mutableData appendData:ByteHeader];
[mutableData appendData:mysps];
[mutableData appendData:ByteHeader];
[mutableData appendData:mypps];
pos = pos + sps.length + pps.length + ByteHeader.length*2;
NSMutableData *mutableDataTem1 = [[NSMutableData alloc] init];;
[mutableDataTem1 appendData:ByteHeader];
[mutableDataTem1 appendData:mysps];
long tem1 = sps.length + ByteHeader.length;
[self sendData:sizeof(Byte)*tem1 data:(char*)[mutableDataTem1 bytes]];
NSMutableData *mutableDataTem = [[NSMutableData alloc] init];;
[mutableDataTem appendData:ByteHeader];
[mutableDataTem appendData:mypps];
long tem = pps.length + ByteHeader.length;
[self sendData:sizeof(Byte)*tem data:(char*)[mutableDataTem bytes]];
}
发送实体部分代码
- (void)oneFrameEncodeEnd:(BOOL)isKeyFrame
{
FrameData *frameData = [[FrameData alloc] init];
frameData.Iframe = isKeyFrame;
frameData.frame_len = (int) pos;
frameData.frame_seq = total_vseq;
frameData.stream_index = 0;
frameData.frame_data = (Byte *)malloc(sizeof(Byte)*pos);//new Byte[pos];
memcpy(frameData.frame_data,[mutableData bytes], pos*sizeof(Byte));
[_videoArray addObject:frameData];
total_vseq++;
//if(isKeyFrame)
//NSLog(@"add one h264 h264 h264 frame to videoArray---seq:%ld",total_vseq);
mysps = nil;
mypps = nil;
[mutableData resetBytesInRange:NSMakeRange(0, [mutableData length])];
[mutableData setLength:0];
pos = 0;
}
音频的采集发送
将采集pcm数据进行aac编码，网上应该有相关的代码可以学习
-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
static BOOL firstStartTimer = false;
static long num = 0;
if (connection == _audioConnection) {
NSLog(@"captureOutput audio");
char szBuf[4096];
memset(szBuf, 0, sizeof(szBuf));
uint32_t nSize = sizeof(szBuf);
// AudioStreamBasicDescription inputFormat = *(CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer))); // 输入音频格式
AudioStreamBasicDescription outputFormat = *(CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer)));
nSize = CMSampleBufferGetTotalSampleSize(sampleBuffer);
CMBlockBufferRef databuf = CMSampleBufferGetDataBuffer(sampleBuffer);
if (CMBlockBufferCopyDataBytes(databuf, 0, nSize, szBuf) == kCMBlockBufferNoErr)
{
int32_t nOffSet = 0;
while (nOffSet < nSize)
{
int outsize = 0;
char szOutBuf[4096] = {0};
int nInSize = 0;
if (nSize - nOffSet >= 640) {
nInSize = 640;
} else {
nInSize = nSize - nOffSet;
}
outsize = [ecdoer AACEncoderEncode:lHand inData:szBuf + nOffSet inSize:nInSize outData:szOutBuf maxOutSize:4096];
// [ecdoer AACEncoderClose:outsize];
if (outsize > 0)
{
[self sendAacDataLen:outsize data:szOutBuf ptsTime:0];
}
nOffSet += 640;
}
}
}
}
音频数据的发送
- (void)sendAacDataLen:(int) totalLength data: (char*) dataPointer ptsTime:(int64_t)pts{
int ret = WM_RTMPLIVESDK_InputData(WMRtmpLiveDataType_AAC, (const char* )dataPointer, totalLength, [self getNowTime]);
NSLog(@"~~~~~~~~~iAAc[%lld]",[self getNowTime]);
if (ret == 1) {
NSLog(@"~~~~~aac~~~~~sendData ret[%d] totalLength[%d]",ret,(int)totalLength);
}
// fail 1 success 0
}
rtmp推流网上也有很多代码，调用rtmplib 可以自己用c++封装一个库用来调用。
二、最后的陈述
这里就先不解释了大体采集发送的过程就是这样，还有一点视频采集发送音频采集发送的时间获取的是当前时间，测试的时候也可以写间隔20ms来测试延迟的问题，它的逻辑是发送一堆音频数据再发送一个视频数据，因为音频数据比较多，音频数据如果丢帧会感觉出来明显的卡顿，视频则不是，视频丢一帧人眼是很难发现的
有些详细的理论推荐大家看这篇博客。
http://www.jianshu.com/p/a6530fa46a88
