WebRTC 之 RED
2022-09-01 本文已影响0人
老瓦在霸都
| Abstract | WebRTC RED 笔记 |
|---|---|
| Authors | Walter Fan |
| Category | learning note |
| Status | v1.0 |
| Updated | 2020-08-28 |
| License | CC-BY-NC-ND 4.0 |
什么是 RED
RED 即 REDundant coding 冗余编码，它是一种 RTP 荷载格式的规范，用来编码冗余的音频和视频媒体数据
，详情参见 RFC 2198.
为什么需要 RED
网络总是会有各种各样的问题，丢包是常事，有一点点包丢了没关系，只要接收方能够容忍并有手段进行丢包恢复，不影响音视频质量就好。
丢包恢复最常见的手段就是发送方多发送一些冗余的数据，这样丢了一份，还有冗余的一份能用，或者用 FEC 恢复出来。
RED 使得我们可以把一个乃至多个 RTP payload 以不同的时间戳存入一个 RTP 包中， 这样比 Opus 自带的 inband FEC 对于丢包有更大的弹性，当然这是以增加带宽为代价的。
怎么实现 RED
SDP 扩展
a=rtpmap:121 red/8000/1
a=fmtp:121 0/5
Packet 数据包格式
在 RTP header 之后的 RTP payload 中，一开始就附上 RED 的 header, 格式如下
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- F：(1 bits），之后是否还有其他 RED Header
- 1: 之后还有其他RED Header，
- 0: 最后一个RED Header
- block PT：(7 bits)，表示该RED Header对应的冗余编码类型
- timeoffset： (14 bits)，表示无符号长度时间戳偏移，该偏移是相对于RTP Header的时间戳。用无符号长度做偏移意味着冗余编码数据必须在发送完原始数据后才能发送
- block length: (10 bits)，表示该RED Header对应编码数据块长度，该长度包含RED Header字段
对于RTP包中最后一个RED Header，可以忽略block length以及timestamp。因为它们可以从RTP Fixed Header以及整个RTP包长度计算得到。最后一个RED Header结构如下：
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|0| Block PT |
+-+-+-+-+-+-+-+-+
- RTP 包示例
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC=0 |M| PT | sequence number of primary |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp of primary encoding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| block PT=7 | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| block PT=5 | |
+-+-+-+-+-+-+-+-+ +
| |
+ LPC encoded redundant data (PT=7) +
| (14 bytes) |
+ +---------------+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+ +
| |
+ +
| |
+ +
| DVI4 encoded primary data (PT=5) |
+ (84 bytes, not to scale) +
/ /
+ +
| |
+ +
| |
+ +---------------+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
WebRTC 相关代码
WebRTC 中不仅在音频 out-band fec 中应用了 RED , 在视频 ULPFEC 中也使用了 RED 来传送冗余数据
ULPFEC 就使用了 RED 来生成冗余的 FEC 包, 代码如下
- 发送方将 FEC 包头和包体都包装在 RED 包中
参见 ulpfec_generator.cc
std::vector<std::unique_ptr<RtpPacketToSend>> UlpfecGenerator::GetFecPackets() {
RTC_DCHECK_RUNS_SERIALIZED(&race_checker_);
if (generated_fec_packets_.empty()) {
return std::vector<std::unique_ptr<RtpPacketToSend>>();
}
// Wrap FEC packet (including FEC headers) in a RED packet. Since the
// FEC packets in `generated_fec_packets_` don't have RTP headers, we
// reuse the header from the last media packet.
RTC_CHECK(last_media_packet_.has_value());
last_media_packet_->SetPayloadSize(0);
std::vector<std::unique_ptr<RtpPacketToSend>> fec_packets;
fec_packets.reserve(generated_fec_packets_.size());
size_t total_fec_size_bytes = 0;
for (const auto* fec_packet : generated_fec_packets_) {
std::unique_ptr<RtpPacketToSend> red_packet =
std::make_unique<RtpPacketToSend>(*last_media_packet_);
red_packet->SetPayloadType(red_payload_type_);
red_packet->SetMarker(false);
uint8_t* payload_buffer = red_packet->SetPayloadSize(
kRedForFecHeaderLength + fec_packet->data.size());
// Primary RED header with F bit unset.
// See https://tools.ietf.org/html/rfc2198#section-3
payload_buffer[0] = ulpfec_payload_type_; // RED header.
memcpy(&payload_buffer[1], fec_packet->data.data(),
fec_packet->data.size());
total_fec_size_bytes += red_packet->size();
red_packet->set_packet_type(RtpPacketMediaType::kForwardErrorCorrection);
red_packet->set_allow_retransmission(false);
red_packet->set_is_red(true);
red_packet->set_fec_protect_packet(false);
fec_packets.push_back(std::move(red_packet));
}
ResetState();
MutexLock lock(&mutex_);
fec_bitrate_.Update(total_fec_size_bytes, clock_->TimeInMilliseconds());
return fec_packets;
}
- 接收方将 RED 包中的 FEC 包剥离出来
参见 ulpfec_receiver.cc
// 0 1 2 3
// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |F| block PT | timestamp offset | block length |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
//
//
// RFC 2198 RTP Payload for Redundant Audio Data September 1997
//
// The bits in the header are specified as follows:
//
// F: 1 bit First bit in header indicates whether another header block
// follows. If 1 further header blocks follow, if 0 this is the
// last header block.
// If 0 there is only 1 byte RED header
//
// block PT: 7 bits RTP payload type for this block.
//
// timestamp offset: 14 bits Unsigned offset of timestamp of this block
// relative to timestamp given in RTP header. The use of an unsigned
// offset implies that redundant data must be sent after the primary
// data, and is hence a time to be subtracted from the current
// timestamp to determine the timestamp of the data for which this
// block is the redundancy.
//
// block length: 10 bits Length in bytes of the corresponding data
// block excluding header.
bool UlpfecReceiver::AddReceivedRedPacket(const RtpPacketReceived& rtp_packet) {
RTC_DCHECK_RUN_ON(&sequence_checker_);
// TODO(bugs.webrtc.org/11993): We get here via Call::DeliverRtp, so should be
// moved to the network thread.
if (rtp_packet.Ssrc() != ssrc_) {
RTC_LOG(LS_WARNING)
<< "Received RED packet with different SSRC than expected; dropping.";
return false;
}
if (rtp_packet.size() > IP_PACKET_SIZE) {
RTC_LOG(LS_WARNING) << "Received RED packet with length exceeds maximum IP "
"packet size; dropping.";
return false;
}
static constexpr uint8_t kRedHeaderLength = 1;
if (rtp_packet.payload_size() == 0) {
RTC_LOG(LS_WARNING) << "Corrupt/truncated FEC packet.";
return false;
}
// Remove RED header of incoming packet and store as a virtual RTP packet.
auto received_packet =
std::make_unique<ForwardErrorCorrection::ReceivedPacket>();
received_packet->pkt = new ForwardErrorCorrection::Packet();
// Get payload type from RED header and sequence number from RTP header.
uint8_t payload_type = rtp_packet.payload()[0] & 0x7f;
received_packet->is_fec = payload_type == ulpfec_payload_type_;
received_packet->is_recovered = rtp_packet.recovered();
received_packet->ssrc = rtp_packet.Ssrc();
received_packet->seq_num = rtp_packet.SequenceNumber();
if (rtp_packet.payload()[0] & 0x80) {
// f bit set in RED header, i.e. there are more than one RED header blocks.
// WebRTC never generates multiple blocks in a RED packet for FEC.
RTC_LOG(LS_WARNING) << "More than 1 block in RED packet is not supported.";
return false;
}
++packet_counter_.num_packets;
packet_counter_.num_bytes += rtp_packet.size();
if (packet_counter_.first_packet_time == Timestamp::MinusInfinity()) {
packet_counter_.first_packet_time = clock_->CurrentTime();
}
if (received_packet->is_fec) {
++packet_counter_.num_fec_packets;
// everything behind the RED header
received_packet->pkt->data =
rtp_packet.Buffer().Slice(rtp_packet.headers_size() + kRedHeaderLength,
rtp_packet.payload_size() - kRedHeaderLength);
} else {
received_packet->pkt->data.EnsureCapacity(rtp_packet.size() -
kRedHeaderLength);
// Copy RTP header.
received_packet->pkt->data.SetData(rtp_packet.data(),
rtp_packet.headers_size());
// Set payload type.
uint8_t& payload_type_byte = received_packet->pkt->data.MutableData()[1];
payload_type_byte &= 0x80; // Reset RED payload type.
payload_type_byte += payload_type; // Set media payload type.
// Copy payload and padding data, after the RED header.
received_packet->pkt->data.AppendData(
rtp_packet.data() + rtp_packet.headers_size() + kRedHeaderLength,
rtp_packet.size() - rtp_packet.headers_size() - kRedHeaderLength);
}
if (received_packet->pkt->data.size() > 0) {
received_packets_.push_back(std::move(received_packet));
}
return true;
}
参考资料
- https://datatracker.ietf.org/doc/html/rfc2198
- https://blog.jianchihu.net/webrtc-research-redundant-rtp-payload-fec.html
- https://www.meetecho.com/blog/opus-red/
本作品采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可
