ijkplayer秘籍androidMedia

MediaCodec

2017-05-23  本文已影响426人  thebestofrocky

原文：https://developer.android.com/reference/android/media/MediaCodec.html

MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components. It is part of the Android low-level multimedia support infrastructure (normally used together with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack.)

MediaCodec 类可以处理低层级的多媒体编解码器，如：encoder/decoder。它是Android低层级的多媒体支持基础的一部分（通常和MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack 一起使用）

Screen Shot 2017-05-23 at 10.55.42 AM.png

In broad terms, a codec processes input data to generate output data. It processes data asynchronously and uses a set of input and output buffers. At a simplistic level, you request (or receive) an empty input buffer, fill it up with data and send it to the codec for processing. The codec uses up the data and transforms it into one of its empty output buffers. Finally, you request (or receive) a filled output buffer, consume its contents and release it back to the codec.
总的来说，codec 处理输入的数据，生成输出数据。异步处理数据，并且使用一系列的输入输出缓冲(buffer). 在一个简化的层次， 你请求（或者接收）一个空的输入buffer, 填充数据后把它交给codec进行处理。codec用完数据后把buffer转换成它的里面的众多空的输出buffer中的一个。最后，你请求（或者接收）一个充满数据的输出buffer，提取出它里面的数据，然后把它释放回codec。

Data Types

Codecs operate on three kinds of data: compressed data, raw audio data and raw video data. All three kinds of data can be processed using ByteBuffers, but you should use a Surface for raw video data to improve codec performance. Surface uses native video buffers without mapping or copying them to ByteBuffers; thus, it is much more efficient. You normally cannot access the raw video data when using a Surface, but you can use the ImageReader class to access unsecured decoded (raw) video frames. This may still be more efficient than using ByteBuffers, as some native buffers may be mapped into direct ByteBuffers. When using ByteBuffer mode, you can access raw video frames using the Image class and getInput/OutputImage(int).

数据类型

Codec对三种数据进行处理：压缩数据，音频原始数据和视频原始数据。三种数据都能够用ByteBuffers进行处理，但是为了提高codec处理能力，在处理视频原始数据时，你应该使用Surface。Surface使用native的视频buffer，不需要映射或都复制到ByteBuffers;因此，效率更高。通常使用Surface时，你无法接获取原始视频数据，但是你可以使用ImageReader类来获取未加密的解码后的（原始）视频帧。即使这样也比使用ByteBuffers的效率更高，因为一些native buffers可以被映射到ByteBuffers. 当使用ByteBuffer模式时，你可以用Image， getInput/OutputImage(int)获取到原始视频帧。

Compressed Buffers

Input buffers (for decoders) and output buffers (for encoders) contain compressed data according to the format's type. For video types this is a single compressed video frame. For audio data this is normally a single access unit (an encoded audio segment typically containing a few milliseconds of audio as dictated by the format type), but this requirement is slightly relaxed in that a buffer may contain multiple encoded access units of audio. In either case, buffers do not start or end on arbitrary byte boundaries, but rather on frame/access unit boundaries.

压缩 Buffers

输入buffers(解码器使用）和输出buffers(编码器使用）包含根据格式压缩的数据。如果是视频，那么就是压缩后的一帧。如果是音频，通常是一个access单位（一个编码过的音频片段，一般来说包含几毫秒的音频），但也可能几个access单位。不论是哪种情况，buffers 不会在随意的byte边界开始或结束，而是以frame/acess为单位来划分起始和结束位置。

Raw Audio Buffers

Raw audio buffers contain entire frames of PCM audio data, which is one sample for each channel in channel order. Each sample is a 16-bit signed integer in native byte order.

原始音频buffers

原始音频buffers包含所有PCM音频数据帧。PCM音频数据是一个按通道的顺序对所有音频通道的采样结果。每个采样是一个16位的带符号的整数。

 short[] getSamplesForChannel(MediaCodec codec, int bufferId, int channelIx) {
   ByteBuffer outputBuffer = codec.getOutputBuffer(bufferId);
   MediaFormat format = codec.getOutputFormat(bufferId);
   ShortBuffer samples = outputBuffer.order(ByteOrder.nativeOrder()).asShortBuffer();
   int numChannels = formet.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
   if (channelIx < 0 || channelIx >= numChannels) {
     return null;
   }
   short[] res = new short[samples.remaining() / numChannels];
   for (int i = 0; i < res.length; ++i) {
     res[i] = samples.get(i * numChannels + channelIx);
   }
   return res;
 }

Raw Video Buffers

In ByteBuffer mode video buffers are laid out according to their color format. You can get the supported color formats as an array from getCodecInfo().getCapabilitiesForType(…).colorFormats. Video codecs may support three kinds of color formats:

原始视频buffers

在ByteBuffer模式下，视频buffers根据它们的颜色格式进行布置的。调用getCodecInfo().getCapabilitiesForType(…).colorFormats可以得到一个包含所有支持的颜色格式的数组。视频codecs可能支持三种颜色格式：

Accessing Raw Video ByteBuffers on Older Devices

Prior to LOLLIPOP and Image support, you need to use the KEY_STRIDE and KEY_SLICE_HEIGHT output format values to understand the layout of the raw output buffers.

在LOLLIPOP之前的设备上获取原始视频ByteBuffers

在LOLLIPOP之前的或不支持Image的系统版本中，你需要用KEY_STRIDE和KEY_SLICE_HEIGHT输出格式值来理解原始输出buffers的布置。

Note that on some devices the slice-height is advertised as 0. This could mean either that the slice-height is the same as the frame height, or that the slice-height is the frame height aligned to some value (usually a power of 2). Unfortunately, there is no standard and simple way to tell the actual slice height in this case. Furthermore, the vertical stride of the U plane in planar formats is also not specified or defined, though usually it is half of the slice height.

注意在一些设备上，slice-height值是0。这是说，要么 slice-height和帧高度一致，要么是帧高度与某个值（通常是2的指数）对齐之后的值。不幸的是，在这种情况下，没有一个标准简单的方法来告实际的slice height。Furthermore, the vertical stride of the U plane in planar formats is also not specified or defined, though usually it is half of the slice height.（不知道这句什么意思）

The KEY_WIDTH and KEY_HEIGHT keys specify the size of the video frames; however, for most encodings the video (picture) only occupies a portion of the video frame. This is represented by the 'crop rectangle'.
KEY_WIDTH和KEY_HEIGHT指定了视频的帧宽高；但是，对于大多数的encodings，视频（图片）只占了视频帧的一部分。这部分就是'crop rectangle'.

You need to use the following keys to get the crop rectangle of raw output images from the output format. If these keys are not present, the video occupies the entire video frame.The crop rectangle is understood in the context of the output frame before applying any rotation.
获取输出格式的原始输出图片的crop rectangle，需要用到以下的keys。如果没有提供这些keys，视频占据全部的视频帧。在旋转之前，crop rectangle在输出帧的上下文中进行‘理解’。

Format Key Type Description
"crop-left" Integer The left-coordinate (x) of the crop rectangle
"crop-top" Integer The top-coordinate (y) of the crop rectangle
"crop-right" Integer The right-coordinate (x) MINUS 1 of the crop rectangle
"crop-bottom" Integer The bottom-coordinate (y) MINUS 1 of the crop rectangle

The right and bottom coordinates can be understood as the coordinates of the right-most valid column/bottom-most valid row of the cropped output image.
右边和底部的坐标可以理解为裁剪后的图片最右面的列和最下边的行的坐标。

The size of the video frame (before rotation) can be calculated as such:
旋转前的视频帧的大小可以通过以下方式进行计算：

 MediaFormat format = decoder.getOutputFormat(…);
 int width = format.getInteger(MediaFormat.KEY_WIDTH);
 if (format.containsKey("crop-left") && format.containsKey("crop-right")) {
     width = format.getInteger("crop-right") + 1 - format.getInteger("crop-left");
 }
 int height = format.getInteger(MediaFormat.KEY_HEIGHT);
 if (format.containsKey("crop-top") && format.containsKey("crop-bottom")) {
     height = format.getInteger("crop-bottom") + 1 - format.getInteger("crop-top");
 }

Also note that the meaning of BufferInfo.offset was not consistent across devices. On some devices the offset pointed to the top-left pixel of the crop rectangle, while on most devices it pointed to the top-left pixel of the entire frame.
另外需要注意的是，BufferInfo.offset 代表的意义在不同的设备上并不一样。在一些设备上，offset指crop rectangle最左上方的像素值，而在另一些设备上是指整个帧的最左上方的像素值。

States

During its life a codec conceptually exists in one of three states: Stopped, Executing or Released. The Stopped collective state is actually the conglomeration of three states: Uninitialized, Configured and Error, whereas the Executing state conceptually progresses through three sub-states: Flushed, Running and End-of-Stream.

状态

在codec的生命周期中，逻辑上只处于三种状态中的一种：停止，执行和释放。停止状态包含三种状态：Uninitialized, Configured和Error；执行状态也含三种状态：Flushed, Running和End-of-Stream.

Screen Shot 2017-05-23 at 11.07.16 AM.png

When you create a codec using one of the factory methods, the codec is in the Uninitialized state. First, you need to configure it via configure(…), which brings it to the Configured state, then call start() to move it to the Executing state. In this state you can process data through the buffer queue manipulation described above.
当你用其中一个工厂方法创建一个codec时，codec处于Uninitialized状态。首先，你需要用configure(…)去配置它，使它进入Configured状态，然后调用start()方法使codec进入Executing状态。在Executing状态，你可以操作buffer队列进行数据处理。

The Executing state has three sub-states: Flushed, Running and End-of-Stream. Immediately after start() the codec is in the Flushed sub-state, where it holds all the buffers. As soon as the first input buffer is dequeued, the codec moves to the Running sub-state, where it spends most of its life. When you queue an input buffer with the end-of-stream marker, the codec transitions to the End-of-Stream sub-state. In this state the codec no longer accepts further input buffers, but still generates output buffers until the end-of-stream is reached on the output. You can move back to the Flushed sub-state at any time while in the Executing state using flush().
执行状态含三种状态：Flushed, Running和End-of-Stream.调用start()方法后，code立即处于Flushed sub-state, 持有所有buffers。当第一个输入buffer被dequeued，codec进入Running sub-state, 这个state会占据大部分生命周期。当你用一个end-of-stream标志queue一个输入buffer时，codec进入End-of-Stream状态。在这个状态，codec不再接收输入buffers，但是仍然在生成输出buffers，直到输出至end-of-stream。在Executing状态 ，任何时候你都可以调用flush()返回Flushed sub-state.

Call stop() to return the codec to the Uninitialized state, whereupon it may be configured again. When you are done using a codec, you must release it by calling release().
调用stop()方法使codec返回Uninitialized状态,然后它可以重新被配置。使用完codec，必须调用release()方法.

On rare occasions the codec may encounter an error and move to the Error state. This is communicated using an invalid return value from a queuing operation, or sometimes via an exception. Call reset() to make the codec usable again. You can call it from any state to move the codec back to the Uninitialized state. Otherwise, call release() to move to the terminal Released state.
极少数的情况下，codec会因为错误而进入Error状态 。这种况情下，queue操作会返回一个非法值，或者有时候抛出异常。调用reset()方法重置codec。在任何状态下都可以调用reset()方法使codec回到Uninitialized状态。

Creation

Use MediaCodecList to create a MediaCodec for a specific MediaFormat. When decoding a file or a stream, you can get the desired format from MediaExtractor.getTrackFormat. Inject any specific features that you want to add using MediaFormat.setFeatureEnabled, then call MediaCodecList.findDecoderForFormat to get the name of a codec that can handle that specific media format. Finally, create the codec using createByCodecName(String).

创建

使用MediaCodecList来创建一个特定MediaFormat格式的MediaCodec。当解码一个文件或者流的时候，你可以用MediaExtractor.getTrackFormat来获取相应的格式。插入任何指定的特性，使用MediaFormat.setFeatureEnabled，然后调用MediaCodecList.findDecoderForFormat来获取可以处理这种多媒体格式的codec。最后，使用createByCodecName(String)来创建codec.

Note: On LOLLIPOP, the format to MediaCodecList.findDecoder/EncoderForFormat must not contain a frame rate. Use format.setString(MediaFormat.KEY_FRAME_RATE, null) to clear any existing frame rate setting in the format.
注意：在LOLLIPOP, MediaCodecList.findDecoder/EncoderForFormat中的format不包含frame rate. 使用format.setString(MediaFormat.KEY_FRAME_RATE, null)来清空已经format存在的frame rate.

You can also create the preferred codec for a specific MIME type using createDecoder/EncoderByType(String). This, however, cannot be used to inject features, and may create a codec that cannot handle the specific desired media format.
你也可以使用createDecoder/EncoderByType(String)来创建针对特定MIME类型的codec。然后，这种codec不能用来插入特性，然后可能不能处理特定的多媒体格式。

Creating secure decoders

On versions KITKAT_WATCH and earlier, secure codecs might not be listed in MediaCodecList, but may still be available on the system. Secure codecs that exist can be instantiated by name only, by appending ".secure" to the name of a regular codec (the name of all secure codecs must end in ".secure".) createByCodecName(String) will throw an IOException if the codec is not present on the system.

创建一个加密的解码器

在KITKAT_WATCH（API Level 20)和之前，加密的codecs可能不在MediaCodecList里，但是仍然是可用的。在正常的codec名字后面添加".secure", 加密的codec只可以通过名字来初始化。createByCodecName(String)会抛出异常，如果codec在系统中不存在。

From LOLLIPOP onwards, you should use the FEATURE_SecurePlayback feature in the media format to create a secure decoder.
LOLLIPOP之后，你应该用通过media format 的FEATURE_SecurePlayback来创建一个加密的解码器。

Initialization

After creating the codec, you can set a callback using setCallback if you want to process data asynchronously. Then, configure the codec using the specific media format. This is when you can specify the output Surface for video producers – codecs that generate raw video data (e.g. video decoders). This is also when you can set the decryption parameters for secure codecs (see MediaCrypto). Finally, since some codecs can operate in multiple modes, you must specify whether you want it to work as a decoder or an encoder.

初始化

创建codec之后，如果你想异步处理数据，你可以设置一个callback。然后，使用特定的media format来配置codec。这时你可以为video生产者（生生原始视频数据的codec, 例如：视频解码器）指定输出Surface。在这时也可以为加密的codec设置解密参数(查看：MediaCrypto).最后，因为一些codec能在几种模式下工作，你必须指定它是做为解码器还是编码器。

Since LOLLIPOP, you can query the resulting input and output format in the Configured state. You can use this to verify the resulting configuration, e.g. color formats, before starting the codec.
从LOLLIPOP开始，在Configured状态，你可以查询输入输出格式。你可以使用这个功能来验证配置结果，例如：颜色格式，在还没有starting codec之前。

If you want to process raw input video buffers natively with a video consumer – a codec that processes raw video input, such as a video encoder – create a destination Surface for your input data using createInputSurface() after configuration. Alternately, set up the codec to use a previously created persistent input surface by calling setInputSurface(Surface).
如果你想用一个codec（比如一个视频编码器）来处理原始的视频输入buffer，在配置之后，调用createInputSurface() 来为输入数据创建一个目的地Surface。或者，调用setInputSurface(Surface)来设置一个之前已经创建好的输入surface.

Codec-specific Data

Some formats, notably AAC audio and MPEG4, H.264 and H.265 video formats require the actual data to be prefixed by a number of buffers containing setup data, or codec specific data. When processing such compressed formats, this data must be submitted to the codec after start() and before any frame data. Such data must be marked using the flag BUFFER_FLAG_CODEC_CONFIG in a call to queueInputBuffer.

指定编码的数据

一些格式，如AAC音频和MPEG4, H.264, H.265视频格式需要在实际数据之前添加一些含设置数据或者指定编码的数据的buffers做为前缀。在处理这种压缩格式时，这些数据必须在start()之后、在任何帧数据之前提交给codec。在queueInputBuffer时，这种数据必须用BUFFER_FLAG_CODEC_CONFIG标志。

Codec-specific data can also be included in the format passed to configure in ByteBuffer entries with keys "csd-0", "csd-1", etc. These keys are always included in the track MediaFormat obtained from the MediaExtractor. Codec-specific data in the format is automatically submitted to the codec upon start(); you MUST NOT submit this data explicitly. If the format did not contain codec specific data, you can choose to submit it using the specified number of buffers in the correct order, according to the format requirements. In case of H.264 AVC, you can also concatenate all codec-specific data and submit it as a single codec-config buffer.

Android uses the following codec-specific data buffers. These are also required to be set in the track format for proper MediaMuxer track configuration. Each parameter set and the codec-specific-data sections marked with (*) must start with a start code of "\x00\x00\x00\x01".

Screen Shot 2017-05-23 at 11.09.26 AM.png

Note: care must be taken if the codec is flushed immediately or shortly after start, before any output buffer or output format change has been returned, as the codec specific data may be lost during the flush. You must resubmit the data using buffers marked with BUFFER_FLAG_CODEC_CONFIG after such flush to ensure proper codec operation.
注意：如果codec马上或者start之后就flushed，在任何输出buffer或者输出格式变化被返回之前，指定编码的数据可能在flush的过程中丢失。在flush之后，你必须重新用BUFFER_FLAG_CODEC_CONFIG标记的buffers提交这些数据来确保正常的codec操作。

Encoders (or codecs that generate compressed data) will create and return the codec specific data before any valid output buffer in output buffers marked with the codec-config flag. Buffers containing codec-specific-data have no meaningful timestamps.

Data Processing

Each codec maintains a set of input and output buffers that are referred to by a buffer-ID in API calls. After a successful call to start() the client "owns" neither input nor output buffers. In synchronous mode, call dequeueInput/OutputBuffer(…) to obtain (get ownership of) an input or output buffer from the codec. In asynchronous mode, you will automatically receive available buffers via the MediaCodec.Callback.onInput/OutputBufferAvailable(…) callbacks.

数据处理

每个codec维护着一批输入和输出buffers，在API请求中， 这些buffer可以通过buffer-ID来指向。在成功调用start()方法后，客户端既没有‘拥有’输出也没有‘拥有’输入buffers。在同步模式中，调用dequeueInput/OutputBuffer(…)从codec中获取输入或者输出buffer。在异步模式中，通过MediaCodec.Callback.onInput/OutputBufferAvailable(…) 回调，你会自动收到可用的buffers。

Upon obtaining an input buffer, fill it with data and submit it to the codec using queueInputBuffer – or queueSecureInputBuffer if using decryption. Do not submit multiple input buffers with the same timestamp (unless it is codec-specific data marked as such).
收到输入buffer时，填入数据后用queueInputBuffer交给codec， 或者，如果是加密的，用queueSecureInputBuffer。不要同时提交多个有同要的时间戳的输入buffers（除非是指定codec的data).

The codec in turn will return a read-only output buffer via the onOutputBufferAvailable callback in asynchronous mode, or in response to a dequeuOutputBuffer call in synchronous mode. After the output buffer has been processed, call one of the releaseOutputBuffer methods to return the buffer to the codec.
在异步模式下，通过onOutputBufferAvailable 回调codec会返回一个只读的输出buffer，或者在同步模式下，响应dequeuOutputBuffer。输出buffer被处理后，调用releaseOutputBuffer来释放buffer给codec。

While you are not required to resubmit/release buffers immediately to the codec, holding onto input and/or output buffers may stall the codec, and this behavior is device dependent. Specifically, it is possible that a codec may hold off on generating output buffers until all outstanding buffers have been released/resubmitted. Therefore, try to hold onto to available buffers as little as possible.
当你没有被要求马上提交或者释放buffers给codec， 持有输入或者输出buffers可能使codec停止工作，最终结果可能与设备有关。特别地，codec可能取消生成输出buffers直到所有未完成的buffers被释放/提交。所以，尽可能地少持有buffers.

Depending on the API version, you can process data in three ways:
根据不同的API版本，你可以用以下三种方式来处理数据。

Screen Shot 2017-05-23 at 11.11.21 AM.png

Asynchronous Processing using Buffers

Since LOLLIPOP, the preferred method is to process data asynchronously by setting a callback before calling configure. Asynchronous mode changes the state transitions slightly, because you must call start() after flush() to transition the codec to the Running sub-state and start receiving input buffers. Similarly, upon an initial call to start the codec will move directly to the Running sub-state and start passing available input buffers via the callback.

异步处理中的buffers使用

从LOLLIPOP开始，较好的方式是在配置之前设置回调，异步处理数据。异步模式稍微改变了状态的变换，因为你必须在flush()之后调用start()，使codec处于Running 状态，接收输入buffers.同样， codec初始化开始之后会直接进入Running状态， 并通过回调传递可用的输入buffers.

Screen Shot 2017-05-23 at 11.11.59 AM.png

MediaCodec is typically used like this in asynchronous mode:
异步模式下，MediaCodec一般是这样使用的:

MediaCodec codec = MediaCodec.createByCodecName(name);
 MediaFormat mOutputFormat; // member variable
 codec.setCallback(new MediaCodec.Callback() {
   @Override
   void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
     ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
     // fill inputBuffer with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }

   @Override
   void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is equivalent to mOutputFormat
     // outputBuffer is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   }

   @Override
   void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     mOutputFormat = format; // option B
   }

   @Override
   void onError(…) {
     …
   }
 });
 codec.configure(format, …);
 mOutputFormat = codec.getOutputFormat(); // option B
 codec.start();
 // wait for processing to complete
 codec.stop();
 codec.release();

Synchronous Processing using Buffers

Since LOLLIPOP, you should retrieve input and output buffers using getInput/OutputBuffer(int) and/or getInput/OutputImage(int) even when using the codec in synchronous mode. This allows certain optimizations by the framework, e.g. when processing dynamic content. This optimization is disabled if you call getInput/OutputBuffers().

同步处理Buffers的使用

从LOLLIPOP开始，你应该使用getInput/OutputBuffer(int)和getInput/OutputImage(int)来获取输入和输出buffers，即使是在同步模式下使用codec.这样做，framework会有一些优化，比如在处理动态内容。你调用getInput/OutputBuffers()时就不会有优化。

Note: do not mix the methods of using buffers and buffer arrays at the same time. Specifically, only call getInput/OutputBuffers directly after start() or after having dequeued an output buffer ID with the value of INFO_OUTPUT_FORMAT_CHANGED.

注意：不要在同时使用buffers和buffer arrays的方法。特别是在start()之后马上调用getInput/OutputBuffers，或者是dequeued一个ID为INFO_OUTPUT_FORMAT_CHANGED的输出buffer.

MediaCodec is typically used like this in synchronous mode:
同步模式下，MediaCodec一般是这样使用的:

MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 MediaFormat outputFormat = codec.getOutputFormat(); // option B
 codec.start();
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
   if (inputBufferId >= 0) {
     ByteBuffer inputBuffer = codec.getInputBuffer(…);
     // fill inputBuffer with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }
   int outputBufferId = codec.dequeueOutputBuffer(…);
   if (outputBufferId >= 0) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is identical to outputFormat
     // outputBuffer is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     outputFormat = codec.getOutputFormat(); // option B
   }
 }
 codec.stop();
 codec.release();

Synchronous Processing using Buffer Arrays (deprecated)

In versions KITKAT_WATCH and before, the set of input and output buffers are represented by the ByteBuffer[] arrays. After a successful call to start(), retrieve the buffer arrays using getInput/OutputBuffers(). Use the buffer ID-s as indices into these arrays (when non-negative), as demonstrated in the sample below. Note that there is no inherent correlation between the size of the arrays and the number of input and output buffers used by the system, although the array size provides an upper bound.

同步模式下Buffer Arrays的使用（deprecated)

在KITKAT_WATCH（API Level 20)和之前，输入和输出buffers都是用ByteBuffer[] 表示的。成功调用start()之后，通过getInput/OutputBuffers()来获取Buffer数组。像下面的例子，采用buffer IDs来做为数组索引。注意，数组长度和系统中使用的输入和输入buffers数量并没有内在关联，虽然数组有一个上限。

 MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 codec.start();
 ByteBuffer[] inputBuffers = codec.getInputBuffers();
 ByteBuffer[] outputBuffers = codec.getOutputBuffers();
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(…);
   if (inputBufferId >= 0) {
     // fill inputBuffers[inputBufferId] with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }
   int outputBufferId = codec.dequeueOutputBuffer(…);
   if (outputBufferId >= 0) {
     // outputBuffers[outputBufferId] is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
     outputBuffers = codec.getOutputBuffers();
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     MediaFormat format = codec.getOutputFormat();
   }
 }
 codec.stop();
 codec.release();

End-of-stream Handling

When you reach the end of the input data, you must signal it to the codec by specifying the BUFFER_FLAG_END_OF_STREAM flag in the call to queueInputBuffer. You can do this on the last valid input buffer, or by submitting an additional empty input buffer with the end-of-stream flag set. If using an empty buffer, the timestamp will be ignored.

结束stream

当输入数据结束时，你必须通知codec, 在调用queueInputBuffer时标志BUFFER_FLAG_END_OF_STREAM.你可以在最后一个输入buffer进行标志，或者增加一个设置了end-of-stream标志的额外的空buffer。如果使用空buffer，时间戳会被忽略。

The codec will continue to return output buffers until it eventually signals the end of the output stream by specifying the same end-of-stream flag in the MediaCodec.BufferInfo set in dequeueOutputBuffer or returned via onOutputBufferAvailable. This can be set on the last valid output buffer, or on an empty buffer after the last valid output buffer. The timestamp of such empty buffer should be ignored.
Codec会持续返回输入buffers直到最后在dequeueOutputBuffer 或者onOutputBufferAvailable 返回的MediaCodec.BufferInfo中标记同样的end-of-stream来说明输出流结束。可能在最后一个输出buffer中标记或者在最后一个buffer后面添加一个空的buffer进记标记。空的buffer上面的时间戳会被忽略。

Do not submit additional input buffers after signaling the end of the input stream, unless the codec has been flushed, or stopped and restarted.
标记输入流结束后，不要再继续添加输入buffers，除非codec已经被flushed，停止或者重启了。

Using an Output Surface

The data processing is nearly identical to the ByteBuffer mode when using an output Surface; however, the output buffers will not be accessible, and are represented as null values. E.g. getOutputBuffer/Image(int) will return null and getOutputBuffers() will return an array containing only null-s.

使用一个输出Surface

使用一个输出Surface的数据处理与ByteBuffer模式下的数据处理几乎是一样的；但是，在Surface模式下，不能获取输出buffers（值都为null).例如：getOutputBuffer/Image(int) 会返回null, getOutputBuffers()会返回一个只包含null的数组。

When using an output Surface, you can select whether or not to render each output buffer on the surface. You have three choices:
当使用输出Surface时，你可以选择是否在surface上绘制每个输出buffer.你有三个选择：

Since M, the default timestamp is the presentation timestamp of the buffer (converted to nanoseconds). It was not defined prior to that.
从M(API Level23)开始，缺省时间戳是presentation时间戳（转换成nanoseconds）。在M之前是没有定义的。

Also since M, you can change the output Surface dynamically using setOutputSurface.
同样，从M开始，你可以使用setOutputSurface动态地改变输出Surface.

Transformations When Rendering onto Surface

If the codec is configured into Surface mode, any crop rectangle, rotation and video scaling mode will be automatically applied with one exception:

在Surface绘制时进行转换

如果Codec设置为Surface模式，任何矩形的剪裁，旋转和video缩放模式会自动地进行，但有一个例外：

Prior to the M release, software decoders may not have applied the rotation when being rendered onto a Surface. Unfortunately, there is no standard and simple way to identify software decoders, or if they apply the rotation other than by trying it out.
在M之前，在Surface绘制时，软解码可能不会进行旋转变换。不幸的是，没有标准而简单的方法来对软解码进行判断是否进行了旋转变换。

There are also some caveats.
下面是一些警告：

Note that the pixel aspect ratio is not considered when displaying the output onto the Surface. This means that if you are using VIDEO_SCALING_MODE_SCALE_TO_FIT mode, you must position the output Surface so that it has the proper final display aspect ratio. Conversely, you can only use VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING mode for content with square pixels (pixel aspect ratio or 1:1).
注意，当在Surface上绘制输出时，像素的宽高比是不被考虑的。这意味着，如果你正在使用VIDEO_SCALING_MODE_SCALE_TO_FIT模式，你必须确保输入Surface最终有一个合适的宽高比。相反地，当显示的内容是正方形的，你只能使用VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING模式。

Note also that as of N release, VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING mode may not work correctly for videos rotated by 90 or 270 degrees.
注意，当N(API Level 24)发布 ，videos旋转了90或者270度时，VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING模式可能不能正常工作。

When setting the video scaling mode, note that it must be reset after each time the output buffers change. Since the INFO_OUTPUT_BUFFERS_CHANGED event is deprecated, you can do this after each time the output format changes.
每当输出buffers改变时，必须重新设置video缩放模式。因为INFO_OUTPUT_BUFFERS_CHANGED事件已经deprecated，每次输出格式改变时，你可以进行重置。

Using an Input Surface

When using an input Surface, there are no accessible input buffers, as buffers are automatically passed from the input surface to the codec. Calling dequeueInputBuffer will throw an IllegalStateException, and getInputBuffers() returns a bogus ByteBuffer[] array that MUST NOT be written into.

使用一个输入Surface

当使用一个输入Surface时，不能获取输入buffers，因为它们被自动从输入Surface传递给了Codec。调用dequeueInputBuffer会抛出IllegalStateException，调用getInputBuffers()会返回一个假的ByteBuffer[]数组，而且这个数组不能进行赋值。

Call signalEndOfInputStream() to signal end-of-stream. The input surface will stop submitting data to the codec immediately after this call.
调用signalEndOfInputStream()来通知结束。调用这个方法之后，输入Surface就马上停止提交数据给Codec了。

Seeking & Adaptive Playback Support

Video decoders (and in general codecs that consume compressed video data) behave differently regarding seek and format change whether or not they support and are configured for adaptive playback. You can check if a decoder supports adaptive playback via CodecCapabilities.isFeatureSupported(String). Adaptive playback support for video decoders is only activated if you configure the codec to decode onto a Surface.

快进和适应播放的支持

视频解码器（或者一般的消耗压缩视频数据的codecs)在快进和格式变化上表现不一致，不论它们是否支持，并且被配置进行适应性播放。你可以通过CodecCapabilities.isFeatureSupported(String)来检查一个解码器是否支持适应性播放。只有你设置codec解码到Surface上，视频解码器的适应性播放才会被激活。

Stream Boundary and Key Frames

It is important that the input data after start() or flush() starts at a suitable stream boundary: the first frame must a key frame. A key frame can be decoded completely on its own (for most codecs this means an I-frame), and no frames that are to be displayed after a key frame refer to frames before the key frame.

Stream边界和关键帧

start()和flush()之后，输入数据从一个合适的stream边界开始是非常重要的:第一个帧必须是关键帧。一个关键帧可以被完全的解码（对于大多数的codecs来说是一个I-frame), 而且关键帧之后显示的帧不能指向关键帧之前的帧。

The following table summarizes suitable key frames for various video formats.
下表的列表概括了不同的视频格式的合适的关键帧。

Screen Shot 2017-05-23 at 11.17.57 AM.png

For decoders that do not support adaptive playback (including when not decoding onto a Surface)

In order to start decoding data that is not adjacent to previously submitted data (i.e. after a seek) you MUST flush the decoder. Since all output buffers are immediately revoked at the point of the flush, you may want to first signal then wait for the end-of-stream before you call flush. It is important that the input data after a flush starts at a suitable stream boundary/key frame.

不支持适应性播放的（包括没有解码到Surface的）解码器

为了解码和之前提交的数据（比如，快进后）不相临的数据，你必须flsuh解码器。因为所有的输出buffers在flush时立刻被废除了，所以在你调用flush前，你可能需要先通知，然后等待end-of-stream 。flush之后，输入数据从一个合适的stream边界/关键帧开始是非常重要的。

Note: the format of the data submitted after a flush must not change; flush() does not support format discontinuities; for that, a full stop() - configure(…) - start() cycle is necessary.
注意：flush之后提交的数据不能更改格式；flush()不支持格式的不连续性；对于不连续的格式，需要stop()-configure(...)-start()。

Also note: if you flush the codec too soon after start() – generally, before the first output buffer or output format change is received – you will need to resubmit the codec-specific-data to the codec. See the codec-specific-data section for more info.
还要注意的是：当你在start()之后马上进行flush--一般来说，在第一个输出buffer或者输出格式变换通知收到以前--你必须重新提交批定编码的数据给codec。更多信息，请查看codec-specific-data这一节。

For decoders that support and are configured for adaptive playback

In order to start decoding data that is not adjacent to previously submitted data (i.e. after a seek) it is not necessary to flush the decoder; however, input data after the discontinuity must start at a suitable stream boundary/key frame.

支持并配置适应性播放的解码器

开始解码一个和上一次提交的数据不相邻的数据（比如，快进），没有必要flush解码器。然而，非连续性的输入数据必须从一个合适的stream边界/关键帧开始。

For some video formats - namely H.264, H.265, VP8 and VP9 - it is also possible to change the picture size or configuration mid-stream. To do this you must package the entire new codec-specific configuration data together with the key frame into a single buffer (including any start codes), and submit it as a regular input buffer.
对于一些视频格式，比如：H.264, H.265, VP8 and VP9，是可能改变图片大小或者mid-stream的结构。要实现这一步，你必须对整个新的特定编码的配置数据和关键帧进行打包成一个新buffer(包括任何start codes),然作为一个普通的输入buffer进行提交。

You will receive an INFO_OUTPUT_FORMAT_CHANGED return value from dequeueOutputBuffer or a onOutputFormatChanged callback just after the picture-size change takes place and before any frames with the new size have been returned.
在图片大小改变后，并且在任何新的大小的帧被返回之前，dequeueOutputBuffer或者onOutputFormatChanged回调马上就会收到一个INFO_OUTPUT_FORMAT_CHANGED返回值。

Note: just as the case for codec-specific data, be careful when calling flush() shortly after you have changed the picture size. If you have not received confirmation of the picture size change, you will need to repeat the request for the new picture size.
注意：对于指定codec的数据，改变图片大小后马上调用flush()要非常小心。如果没有收到图片大小改变的确认，你需要重新提交新的图片大小请求。

Error handling

The factory methods createByCodecName and createDecoder/EncoderByType throw IOException on failure which you must catch or declare to pass up. MediaCodec methods throw IllegalStateException when the method is called from a codec state that does not allow it; this is typically due to incorrect application API usage. Methods involving secure buffers may throw MediaCodec.CryptoException, which has further error information obtainable from getErrorCode().

应对错误的情况

工厂方法createByCodecName和createDecoder/EncoderByType 在遇错的时候会抛出IOException，你必须进行catch或者声明异常。当codec在一个不允许的状态被调用时会抛出IllegalStateException；这是典型的API错误引起的。与加密的buffers有关的方法可能会抛出MediaCodec.CryptoException，使用getErrorCode()方法可以获取进一步的错误信息。

Internal codec errors result in a MediaCodec.CodecException, which may be due to media content corruption, hardware failure, resource exhaustion, and so forth, even when the application is correctly using the API. The recommended action when receiving a CodecException can be determined by calling isRecoverable() and isTransient():
内部的codec错误会抛出MediaCodec.CodecException，可能是因多媒体文件损坏，硬件错误，没有资源等，即使是正确地使用了API也是如此。遇到CodecException时，调用isRecoverable()和isTransient()可以决定合适的方法。

Both isRecoverable() and isTransient() do not return true at the same time.
isRecoverable()和isTransient()不可能同时返回true;

Valid API Calls and API History

This sections summarizes the valid API calls in each state and the API history of the MediaCodec class. For API version numbers, see Build.VERSION_CODES.

正确的API请求和API历史记录

这小节概括了在每个状态下正确的API请求，以及MediaCodec类的API历史。查看API版本号，请看Build.VERSION_CODES。

Screen Shot 2017-05-25 at 11.14.34 AM.png
上一篇下一篇

猜你喜欢

热点阅读