webrtc视频jitterbuffer原理机制（一）

从jitterbuffer取出frame，解码

在ViEChannel类中创建解码线程，在VCMReceiver类中调用jitterbuffer取出frame。

bool ViEChannel::ChannelDecodeProcess()
->int32_t Decode(uint16_t maxWaitTimeMs)
->int32_t VideoReceiver::Decode(uint16_t maxWaitTimeMs)
->VCMEncodedFrame* VCMReceiver::FrameForDecoding
```
先检查是否存在完整的帧，存在则取出，若不存在，则检查不完整的帧，满足条件则取出，不满足则不取出。
```
  bool found_frame = jitter_buffer_.NextCompleteTimestamp(
      max_wait_time_ms, &frame_timestamp);
  if (!found_frame)
    found_frame = jitter_buffer_.NextMaybeIncompleteTimestamp(&frame_timestamp);
```
在上一步取出frame_timestamp，再用frame_timestamp从jitterbuffer中取出帧数据
```
VCMEncodedFrame* frame = jitter_buffer_.ExtractAndSetDecode(frame_timestamp);
```
##接收到包，将包插入jitterbuffer
接收到包，将包插入jitterbuffer代码流程
```
void UdpTransportImpl::IncomingRTPCallback
->void UdpTransportImpl::IncomingRTPFunction
->void VideoChannelTransport::IncomingRTPPacket
->int ViENetworkImpl::ReceivedRTPPacket
->int32_t ViEChannel::ReceivedRTPPacket
->int ViEReceiver::ReceivedRTPPacket
->int ViEReceiver::InsertRTPPacket
->bool ViEReceiver::ReceivePacket
->bool RtpReceiverImpl::IncomingRtpPacket
->int32_t RTPReceiverVideo::ParseRtpPacket
->int32_t ViEReceiver::OnReceivedPayloadData
->int32_t IncomingPacket
->int32_t VideoReceiver::IncomingPacket
->int32_t VCMReceiver::InsertPacket
->VCMFrameBufferEnum VCMJitterBuffer::InsertPacket
```

##decodable_frames_，incomplete_frames_，free_frames_
decodable_frames_，incomplete_frames_，free_frames_的处理，主要在VCMFrameBuffer类中，其中包含的VCMSessionInfo _sessionInfo;为主要处理成员。

对于每一次收到的包，根据时间戳找到，当前帧在哪个队列中，在decodable_frames_或incomplete_frames_队列中，若不存在，则从free_frames_队列中给出一个空帧。
```  
VCMFrameBuffer* frame;
  FrameList* frame_list;
  const VCMFrameBufferEnum error = GetFrame(packet, &frame, &frame_list);```
```
// Gets frame to use for this timestamp. If no match, get empty frame.
VCMFrameBufferEnum VCMJitterBuffer::GetFrame(const VCMPacket& packet,
                                             VCMFrameBuffer** frame,
                                             FrameList** frame_list) {
  *frame = incomplete_frames_.PopFrame(packet.timestamp);
  if (*frame != NULL) {
    *frame_list = &incomplete_frames_;
    return kNoError;
  }
  *frame = decodable_frames_.PopFrame(packet.timestamp);
  if (*frame != NULL) {
    *frame_list = &decodable_frames_;
    return kNoError;
  }

  *frame_list = NULL;
  // No match, return empty frame.
  *frame = GetEmptyFrame();
  if (*frame == NULL) {
    // No free frame! Try to reclaim some...
    LOG(LS_WARNING) << "Unable to get empty frame; Recycling.";
    bool found_key_frame = RecycleFramesUntilKeyFrame();
    *frame = GetEmptyFrame();
    assert(*frame);
    if (!found_key_frame) {
      free_frames_.push_back(*frame);
      return kFlushIndicator;
    }
  }
  (*frame)->Reset();
  return kNoError;
}```

对于VCMSessionInfo中  complete_和decodable_的判定，每一次插入一个包，都要进行UpdateCompleteSession();对于帧的完整性进行检查。
```
  size_t returnLength = InsertBuffer(frame_buffer, packet_list_it);
  UpdateCompleteSession();
  if (decode_error_mode == kWithErrors)
    decodable_ = true;
  else if (decode_error_mode == kSelectiveErrors)
    UpdateDecodableSession(frame_data);
  return static_cast<int>(returnLength);
```
UpdateCompleteSession();检查是否有第一个包和最后一个包，并且都是按序的，即没有丢包，若满足这些条件，则判定为complete_ = true;
```
void VCMSessionInfo::UpdateCompleteSession() {
  if (HaveFirstPacket() && HaveLastPacket()) {
    // Do we have all the packets in this session?
    bool complete_session = true;
    PacketIterator it = packets_.begin();
    PacketIterator prev_it = it;
    ++it;
    for (; it != packets_.end(); ++it) {
      if (!InSequence(it, prev_it)) {
        complete_session = false;
        break;
      }
      prev_it = it;
    }
    complete_ = complete_session;
  }
}
```
下面则是对于decodable_ 的判定，如果decode_error_mode 为kWithErrors模式，则有一个包decodable_ 即可以为ture。如果decode_error_mode 为kSelectiveErrors，则根据rtt,帧类型，已经有的包数等条件来综合得出decodable_ 。
具体条件则为：rtt<100，或者为关键帧，或者已经收到的包数在0.2\*rolling_average_packets_per_frame和0.8\*rolling_average_packets_per_frame之间时，则decodable_ 不去改变，否则为true。rolling_average_packets_per_frame为平均每帧所含的包数。
```
void VCMSessionInfo::UpdateDecodableSession(const FrameData& frame_data) {
  // Irrelevant if session is already complete or decodable
  if (complete_ || decodable_)
    return;
  // TODO(agalusza): Account for bursty loss.
  // TODO(agalusza): Refine these values to better approximate optimal ones.
  // Do not decode frames if the RTT is lower than this.
  const int64_t kRttThreshold = 100;
  // Do not decode frames if the number of packets is between these two
  // thresholds.
  const float kLowPacketPercentageThreshold = 0.2f;
  const float kHighPacketPercentageThreshold = 0.8f;
  if (frame_data.rtt_ms < kRttThreshold
      || frame_type_ == kVideoFrameKey
      || !HaveFirstPacket()
      || (NumPackets() <= kHighPacketPercentageThreshold
                          * frame_data.rolling_average_packets_per_frame
          && NumPackets() > kLowPacketPercentageThreshold
                            * frame_data.rolling_average_packets_per_frame))
    return;

  decodable_ = true;
}
```
```
PS：
1、如果rolling_average_packets_per_frame>5,
kLowPacketPercentageThreshold * frame_data.rolling_average_packets_per_frame>1
那么来一个包就判定  decodable_ = true;这个是否合理？
2、 rtt很小的时候，不判定为true，是希望能够complete_ ，至于kRttThreshold = 100是否合理，有待验证。
3、可见，关键帧都是完整的，不完整，不会 设置decodable_ = true;则一直为kIncomplete。
```

对于VCMFrameBuffer中的kCompleteSession和kDecodableSession分别对应VCMSessionInfo中的complete_ 和decodable_ 。若既不是complete_ ，也不是decodable_，则对应kIncomplete。
```
    if (_sessionInfo.complete()) {
      SetState(kStateComplete);
      return kCompleteSession;
    } else if (_sessionInfo.decodable()) {
      SetState(kStateDecodable);
      return kDecodableSession;
    }
    return kIncomplete;
```

对于上面提到的decode_error_mode，通过如下流程进行设置。
```
int Conductor::VideoCreateStream
->int ViEBaseImpl::CreateChannel
->int ViEBaseImpl::CreateChannel
->int ViEChannelManager::CreateChannel
->bool ChannelGroup::CreateSendChannel
->bool ChannelGroup::CreateChannel
->int32_t ViEChannel::Init
->int32_t SetVideoProtection
->int32_t VideoReceiver::SetVideoProtection
->void VCMReceiver::SetDecodeErrorMode
->void VCMJitterBuffer::SetDecodeErrorMode
```
这里根据NACK和FEC的使用情况，来设置decode_error_mode。
```
// Enable or disable a video protection method.
// Note: This API should be deprecated, as it does not offer a distinction
// between the protection method and decoding with or without errors. If such a
// behavior is desired, use the following API: SetReceiverRobustnessMode.
int32_t VideoReceiver::SetVideoProtection(VCMVideoProtection videoProtection,
                                          bool enable) {
  // By default, do not decode with errors.
  _receiver.SetDecodeErrorMode(kNoErrors);
  switch (videoProtection) {
    case kProtectionNack:
    case kProtectionNackReceiver: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        // Enable NACK and always wait for retransmits.
        _receiver.SetNackMode(kNack, -1, -1);
      } else {
        _receiver.SetNackMode(kNoNack, -1, -1);
      }
      break;
    }

    case kProtectionKeyOnLoss: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        _keyRequestMode = kKeyOnLoss;
        _receiver.SetDecodeErrorMode(kWithErrors);
      } else if (_keyRequestMode == kKeyOnLoss) {
        _keyRequestMode = kKeyOnError;  // default mode
      } else {
        return VCM_PARAMETER_ERROR;
      }
      break;
    }

    case kProtectionKeyOnKeyLoss: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        _keyRequestMode = kKeyOnKeyLoss;
      } else if (_keyRequestMode == kKeyOnKeyLoss) {
        _keyRequestMode = kKeyOnError;  // default mode
      } else {
        return VCM_PARAMETER_ERROR;
      }
      break;
    }

    case kProtectionNackFEC: {
      CriticalSectionScoped cs(_receiveCritSect);
      if (enable) {
        // Enable hybrid NACK/FEC. Always wait for retransmissions
        // and don't add extra delay when RTT is above
        // kLowRttNackMs.
        _receiver.SetNackMode(kNack, media_optimization::kLowRttNackMs, -1);
        _receiver.SetDecodeErrorMode(kNoErrors);
        _receiver.SetDecodeErrorMode(kNoErrors);
      } else {
        _receiver.SetNackMode(kNoNack, -1, -1);
      }
      break;
    }
    case kProtectionNackSender:
    case kProtectionFEC:
    case kProtectionPeriodicKeyFrames:
      // Ignore encoder modes.
      return VCM_OK;
  }
  return VCM_OK;
}
```


下面则介绍本文的核心，decodable_frames_，incomplete_frames_，free_frames_的处理。
```
  // Is the frame already in the decodable list?
  bool continuous = IsContinuous(*frame);
  switch (buffer_state) {
    case kGeneralError:
    case kTimeStampError:
    case kSizeError: {
      free_frames_.push_back(frame);
      break;
    }
    case kCompleteSession: {
      if (previous_state != kStateDecodable &&
          previous_state != kStateComplete) {
        CountFrame(*frame);
        if (continuous) {
          // Signal that we have a complete session.
          frame_event_->Set();
        }
      }
      FALLTHROUGH();
    }
    // Note: There is no break here - continuing to kDecodableSession.
    case kDecodableSession: {
      *retransmitted = (frame->GetNackCount() > 0);
      if (continuous) {
        decodable_frames_.InsertFrame(frame);
        FindAndInsertContinuousFrames(*frame);
      } else {
        incomplete_frames_.InsertFrame(frame);
      }
      break;
    }
    case kIncomplete: {
      if (frame->GetState() == kStateEmpty &&
          last_decoded_state_.UpdateEmptyFrame(frame)) {
        free_frames_.push_back(frame);
        return kNoError;
      } else {
        incomplete_frames_.InsertFrame(frame);
      }
      break;
    }
    case kNoError:
    case kOutOfBoundsPacket:
    case kDuplicatePacket: {
      // Put back the frame where it came from.
      if (frame_list != NULL) {
        frame_list->InsertFrame(frame);
      } else {
        free_frames_.push_back(frame);
      }
      ++num_duplicated_packets_;
      break;
    }
    case kFlushIndicator:
      free_frames_.push_back(frame);
      return kFlushIndicator;
    default: assert(false);
  }
```
其中第一句 bool continuous = IsContinuous(*frame);主要是判断当前收到帧和上一个解码帧是不是连续的。其中还有一些特殊情况，如decode_error_mode_ == kWithErrors，或者frame->FrameType() == kVideoFrameKey等均判断为连续的。由于不加FEC和NACK时，decode_error_mode_ = kWithErrors，所以，一直continuous 为true。
只有kCompleteSession时，才触发事件frame_event_->Set();等待事件在
bool VCMJitterBuffer::NextCompleteTimestamp中，即取帧的函数中。
注意：kCompleteSession情况，后面没有break,则将完成的帧也插入decodable_frames_队列。
所以，对于VCMSessionInfo中的complete_ 和decodable_ ，都将插入decodable_frames_队列。

##再回头看从jitterbuffer取出frame
```
// Returns immediately or a |max_wait_time_ms| ms event hang waiting for a
// complete frame, |max_wait_time_ms| decided by caller.
bool VCMJitterBuffer::NextCompleteTimestamp(
    uint32_t max_wait_time_ms, uint32_t* timestamp) {
  crit_sect_->Enter();
  if (!running_) {
    crit_sect_->Leave();
    return false;
  }
  CleanUpOldOrEmptyFrames();

  if (decodable_frames_.empty() ||
      decodable_frames_.Front()->GetState() != kStateComplete) 
  {
    const int64_t end_wait_time_ms = clock_->TimeInMilliseconds() +
        max_wait_time_ms;
    int64_t wait_time_ms = max_wait_time_ms;
    while (wait_time_ms > 0) {
      crit_sect_->Leave();
      const EventTypeWrapper ret =
        frame_event_->Wait(static_cast<uint32_t>(wait_time_ms));
      crit_sect_->Enter();
      if (ret == kEventSignaled) {
        // Are we shutting down the jitter buffer?
        if (!running_) {
          crit_sect_->Leave();
          return false;
        }
        // Finding oldest frame ready for decoder.
        CleanUpOldOrEmptyFrames();
        if (decodable_frames_.empty() ||
            decodable_frames_.Front()->GetState() != kStateComplete) {
          wait_time_ms = end_wait_time_ms - clock_->TimeInMilliseconds();
        } else {
          break;
        }
      } else {
        break;
      }
    }
  }
  if (decodable_frames_.empty() ||
      decodable_frames_.Front()->GetState() != kStateComplete) {
    crit_sect_->Leave();
    return false;
  }
  *timestamp = decodable_frames_.Front()->TimeStamp();
  crit_sect_->Leave();
  return true;
}
```
1、其中decodable_frames_.Front()->GetState() 取得的_state，有下列情况赋值：
```
        if (packet.frameType != kFrameEmpty) {
            // first media packet
            SetState(kStateIncomplete);
        }
```
```
    if (_sessionInfo.complete()) {
      SetState(kStateComplete);
      return kCompleteSession;
    } else if (_sessionInfo.decodable()) {
      SetState(kStateDecodable);
      return kDecodableSession;
    }
```
可见，_state还是可以标记这一帧数据的完成情况的，即完成时，为kStateComplete；未完成时，为kStateDecodable或kStateIncomplete，只有一个包时，为kStateIncomplete。凡是没有从kStateIncomplete升为kStateDecodable，则依然为kStateIncomplete。

2、其中条件decodable_frames_.Front()->GetState() != kStateComplete
可见，必须是完整的帧才能够从NextCompleteTimestamp函数中取出来。

再看取出不完整的帧：

```
bool VCMJitterBuffer::NextMaybeIncompleteTimestamp(uint32_t* timestamp) {
  CriticalSectionScoped cs(crit_sect_);
  if (!running_) {
    return false;
  }
  if (decode_error_mode_ == kNoErrors) {
    // No point to continue, as we are not decoding with errors.
    return false;
  }

  CleanUpOldOrEmptyFrames();

  VCMFrameBuffer* oldest_frame;
  if (decodable_frames_.empty()) {
    if (nack_mode_ != kNoNack || incomplete_frames_.size() <= 1) {
      return false;
    }
    oldest_frame = incomplete_frames_.Front();
    // Frame will only be removed from buffer if it is complete (or decodable).
    if (oldest_frame->GetState() < kStateComplete) {
      return false;
    }
  } else {
    oldest_frame = decodable_frames_.Front();
    // If we have exactly one frame in the buffer, release it only if it is
    // complete. We know decodable_frames_ is  not empty due to the previous
    // check.
    if (decodable_frames_.size() == 1 && incomplete_frames_.empty() &&
        oldest_frame->GetState() != kStateComplete) {
      return false;
    }
  }

  *timestamp = oldest_frame->TimeStamp();
  return true;
}
```
从条件中可以看出：
1、decodable_frames_为空时
incomplete_frames_.Front()->GetState()为  kStateEmpty或者kStateIncomplete，则取不出帧，否则可以取出。
2、decodable_frames_不空时
decodable_frames_.size() == 1 && incomplete_frames_.empty() &&
        oldest_frame->GetState() != kStateComplete时，取不出帧。否则取出decodable_frames_中不完整的数据帧。

##总结：
至此，jitterbuffer对于包、帧的处理，已经比较清晰。
所有组包的帧都存在于decodable_frames_，incomplete_frames_队列中，而decodable_frames_又根据状态分为完整的帧和不完整的帧，incomplete_frames_主要保存状态为kIncomplete的帧，也是不完整的，但有包数据。
而在取帧数据的时候先取完整的帧，取不到，则取一定条件下不完整的帧数据，不是有不完整的帧数据就去取。
所以，如果你不想取出丢包的帧，则只调用NextCompleteTimestamp去取完整的帧即可。因为每次来的包，放入帧中之后，都是插入到decodable_frames_或者incomplete_frames_的最前面，所以不存在一直取不出来帧数据的情况。另外一个方法，就是设置decode_error_mode_ 为kNoErrors。
但是，不取出丢包的帧 ，不等于不存在马赛克。因为丢帧，所以解码的时候，可能会存在参考帧的问题。

##追加--如何杜绝马赛克
由上述对于马赛克存在的讨论中可知，丢包是主要原因，如果只取出完整的帧，也会因为参考帧问题，导致马赛克。
如果，丢包以后，就把整个GOP都删除，直到下一个关键帧，同时，发现丢包就去请求关键帧，这样就完全可以杜绝马赛克了。但是这样会导致在丢包时，流畅度更加不好。