Android OpenGL ES 十五.ffmpeg+open

在上一篇的中，已经可以播放MP3音乐了，那么如何把声音加入的录制的视频中呢

总体思路如下：

1. 打包musicplayer项目，拿到编译后的.so文件
2. 把该.so放入到录屏的项目，集成进去
3. 仿照视频录制，把上一篇回调给java端的callBackPcm(byte[] pcmData, int size)，数据放入音频的mAudioCodec队列，然后开启线程用mMediaMuxer写入音频数据。

---

一. 打包so

使用gradlew assembleRelease打包命令，然后再build路径下拿到.so文件

image.png

二. 放入新的录屏项目中

要注意也需要放到新的项目的v8a路径

然后配置CmakeList.txt

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.
cmake_minimum_required(VERSION 3.4.1)

# 需要引入我们头文件,以这个配置的目录为基准
include_directories(src/main/jniLibs/include)

AUX_SOURCE_DIRECTORY(${CMAKE_SOURCE_DIR}/src/main/cpp SRC_LIST)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

# 添加共享库搜索路径
LINK_DIRECTORIES(${CMAKE_SOURCE_DIR}/src/main/jniLibs/arm64-v8a)

add_library(
        # Sets the name of the library.
        video-record
        # Sets the library as a shared library.
        SHARED
        ${SRC_LIST}
)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        # 链接额外的 ffmpeg 的编译
        video-record
        # 编解码(最重要的库)
        avcodec
        # 滤镜特效处理库
        avfilter
        # 封装格式处理库
        avformat
        # 工具库(大部分库都需要这个库的支持)
        avutil
        # 后期处理
        postproc
        # 音频采样数据格式转换库
        swresample
        # 视频像素数据格式转换
        swscale
        music-player
        # 链接 android ndk 自带的一些库
        android
        # 链接 OpenSLES
        OpenSLES
        # Links the target library to the log library
        # included in the NDK.
        log)

music-player在target_link_libraries中引入，其他的ffmpeg因为要调用到，也需要放入
SRC_LIST配置了，放一个空的.cpp文件就行，没有该变量还报错

注意事项：

这里是从music-player拿过来的，整个包的包名要和原来的项目保持一致，原来是什么包名不要动，要不然和native层代码对应不上。

三. 编写录制播放的音乐代码

1. 初始化mediaPlayer

//BaseVideoRecorder.java
public BaseVideoRecorder(Context context,EGLContext eglContext) {
      //...
        mediaPlayer = new DarrenPlayer();
}

public void initMediaParams(String audioPath, String outPath, int videoWidth, int videoHeight) {

        try {
            //...
            initAudioParams(audioPath);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

private void initAudioParams(String audioPath) throws Exception{
        // 设置mediaPlayer音频路径，并做初始化
        mediaPlayer.setDataSource(audioPath);
        mediaPlayer.setOnPreparedListener(new MediaPreparedListener() {
            @Override
            public void onPrepared() {
                //native准备好后，可以播放开启录制了
                start();

                //音频播放
                mediaPlayer.play();
            }
        });
        mediaPlayer.setMediaInfoListener(new MediaInfoListener() {

            int sampleRate;
            int channel;

            long audioPts;

            @Override
            public void musicInfo(int sampleRate, int channel) {
                //创建audio的codec
                initAudioCodec(sampleRate,channel);
                this.sampleRate = sampleRate;
                this.channel = channel;
            }

            @Override
            public void callBackPcm(byte[] pcmData, int size) {

                int inputBufferTrack = mAudioCodec.dequeueInputBuffer(0);
                if(inputBufferTrack >= 0){
                    ByteBuffer inputBuffer = mAudioCodec.getInputBuffers()[inputBufferTrack];
                    inputBuffer.clear();

                    inputBuffer.put(pcmData);

                    //0.41795918 *1000 000
                    audioPts += 1000000 * size * 1.0f / sampleRate * channel * 2;
                    Log.e(TAG, "callBackPcm: "+audioPts);
                    //数据放入mAudioCodec的队列中
                    mAudioCodec.queueInputBuffer(inputBufferTrack,0,size,audioPts,0);
                }
            }
        });
        mediaPlayer.setOnErrorListener( new MediaErrorListener() {
            @Override
            public void onError(int code, String msg) {
                Log.e(TAG, "mediaPlayer jni onError: code="+code+", msg="+msg);
            }
        });
    }

    private void start(){
        // 视频渲染开始
        videoRenderThread.start();
        // 视频编码开始
        videoEncoderThread.start();
        // 音频编码开始
        audioEncoderThread.start();
    }

callBackPcm会不断的循环被调用，拿到的数据放入到音频的mediacodec中，这样muxer那边能拿到

2. 设置音频Mediacodec的参数，需要使用aac_lc

    private void initAudioCodec(int sampleRate, int channel) {
        try {
            // 采样率，44.1khz，双声道，每个声道16位，2字节
            MediaFormat mediaFormat = MediaFormat.createAudioFormat(MediaFormat.MIMETYPE_AUDIO_AAC,sampleRate,channel);
            // 设置比特率96k hz
            mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE,96000);
            mediaFormat.setInteger(MediaFormat.KEY_AAC_PROFILE,MediaCodecInfo.CodecProfileLevel.AACObjectLC);
            // 设置输入数据缓冲区的最大大小
            mediaFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE,sampleRate * channel * 2);

            mAudioCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_AAC);
            mAudioCodec.configure(mediaFormat,null,null,MediaCodec.CONFIGURE_FLAG_ENCODE);

            audioEncoderThread = new AudioEncoderThread(recorderReference);

            mAudioCodec.start();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

3. 一般是init设定参数后，调用startRecord，这样等播放器准备好的方法回调了，就开始录屏和录音

public void startRecord(){
        // mediaPlayer的prepare有0.几秒的延迟，最好这里启动准备，准备好后启动音、视频录制线程
        // 如果这里用play（）,并启动start()方法的线程，会导致录制的视频黑屏
        mediaPlayer.prepareAsync();
    }

4. 录音代码

private class AudioEncoderThread extends Thread{

        WeakReference<BaseVideoRecorder> videoRecorderWf;
        private boolean shouldExit =false;

        private MediaCodec mAudioCodec;
        private MediaMuxer mMediaMuxer;
        MediaCodec.BufferInfo bufferInfo;

        long audioPts = 0;
        final CyclicBarrier startCb;
        final CyclicBarrier stopCb;
        /**
         * 音频轨道
         */
        private int mAudioTrackIndex = -1;

        public AudioEncoderThread(WeakReference<BaseVideoRecorder> videoRecorderWf){
            this.videoRecorderWf = videoRecorderWf;
            this.mAudioCodec = videoRecorderWf.get().mAudioCodec;
            this.mMediaMuxer = videoRecorderWf.get().mMediaMuxer;
            this.startCb = videoRecorderWf.get().startCb;
            this.stopCb = videoRecorderWf.get().stopCb;
            bufferInfo = new MediaCodec.BufferInfo();
        }

        @Override
        public void run() {
            while (true){
                try {
                    if(shouldExit){
                        onDestroy();
                        return;
                    }

                    // 返回有效数据填充的输出缓冲区的索引
                    int outputBufferIndex = mAudioCodec.dequeueOutputBuffer(bufferInfo,0);
                    if(outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED){
                        // 将mMediaCodec的指定的格式的数据轨道，设置到mMediaMuxer上
                        mAudioTrackIndex = mMediaMuxer.addTrack(mAudioCodec.getOutputFormat());
                        Log.e(TAG, "run:  audio mMediaMuxer.await before");
                        startCb.await();
                        Log.e(TAG, "run: audio mMediaMuxer.await after");
                    }else{

                        int index = 0;
                        while (outputBufferIndex >= 0){

                            Log.e(TAG,"outputBufferIndex:"+outputBufferIndex+" count:"+index);
                            // 获取数据
                            ByteBuffer outBuffer = mAudioCodec.getOutputBuffers()[outputBufferIndex];

                            outBuffer.position(bufferInfo.offset);
                            outBuffer.limit(bufferInfo.offset+bufferInfo.size);

                            // 修改视频的 pts,基准时间戳
                            if(audioPts ==0)
                                audioPts = bufferInfo.presentationTimeUs;
                            bufferInfo.presentationTimeUs -= audioPts;

//                                System.out.println(bufferInfo.presentationTimeUs);
                            System.out.println("writeSampleData mAudioTrackIndex:"+ mAudioTrackIndex);
                            // 写入音频数据
                            mMediaMuxer.writeSampleData(mAudioTrackIndex,outBuffer,bufferInfo);

                            // 释放 outBuffer
                            mAudioCodec.releaseOutputBuffer(outputBufferIndex,false);
                            outputBufferIndex = mAudioCodec.dequeueOutputBuffer(bufferInfo,0);
                        }
                    }
                } catch (Exception e){
                    e.printStackTrace();
                }
            }
        }

        private void onDestroy() {
            try {
                if (mAudioCodec != null){
                    mAudioCodec.stop();
                    mAudioCodec.release();
                    mAudioCodec = null;
                }
                //和videoEncoderThread的退出，避免一方退出了，一方还在运行，需保持一致
                Log.e(TAG, "run: audio stopCb.await() before");
                stopCb.await();
                Log.e(TAG, "run: audio stopCb.await() after");
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

        public void requestExit() {
            shouldExit = true;
        }
    }

    private class VideoEncoderThread extends Thread{

       //...
        CyclicBarrier startCb;
        CyclicBarrier stopCb;

        /**
         * 视频轨道
         */
        private int mVideoTrackIndex = -1;

        public VideoEncoderThread(WeakReference<BaseVideoRecorder> videoRecorderWf){
       //...
            this.startCb = videoRecorderWf.get().startCb;
            this.stopCb = videoRecorderWf.get().stopCb;
            //...
        }

        @Override
        public void run() {
                //...
                    if(outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED){
                        // 将mMediaCodec的指定的格式的数据轨道，设置到mMediaMuxer上
                        mVideoTrackIndex = mMediaMuxer.addTrack(mVideoCodec.getOutputFormat());
                        mMediaMuxer.start();
                        //等待audioEncoder，一起往 startCb.wait（）后面走
                        Log.e(TAG, "run: video mMediaMuxer.start()");
                        startCb.await();
                        Log.e(TAG, "run: video mMediaMuxer.await after");
                  //...
        }

        private void onDestroy() {
            try {
               //...
                Log.e(TAG, "run: video stopCb.await() before");
                stopCb.await();
                Log.e(TAG, "run: video stopCb.await() after");
                if (mMediaMuxer != null){
                    mMediaMuxer.stop();
                    mMediaMuxer.release();
                    mMediaMuxer = null;
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

     
    }

这里使用CyclicBarrier 来保持线程之间步调一致，因为mMediaMuxer需要开启后，才能写入，而开启是在一个线程中，还有释放也是需要均结束后再释放，避免一方尚未结束，一方已经释放了。

5. 流程退出
   停止音频播放，停止录制线程。

public void stopRecord(){

        mediaPlayer.stop();

        videoRenderThread.requestExit();
        videoEncoderThread.requestExit();
        audioEncoderThread.requestExit();
    }

代码在Android OpenGL ES 十三.MediaCodec录制视频(转载整理)下方已有，集成在一起了。