显示框架之VirtualDisplay 的数据源
2022-04-01 本文已影响0人
Android图形显示之路
Android 支持多个屏幕，主屏(主显的物理屏)，虚拟屏(非物理屏)，外部显示屏(折叠屏)，其中主屏和外部显示屏是实实在在的硬件物理屏，这两者在SurfaceFlinger侧的显示流程相差不大，而VirtualDisplay虽然也是走的SurfaceFlinger流程，但数据源的方式有较大的不同，本文就分析下VirtualDisplay的数据源。
对VirtualDisplay框架层的分析可以看下这篇文章：https://www.jianshu.com/p/c4ea60bc73d2
这里主要探索一下VirtualDisplay的数据源。
CreateDisplay
首先框架层会通过DMS来创建虚拟屏，通过jni调到SurfaceCompoerClient:: createDisplay，再通过binder调到SurfaceFlinger，看下SurfaceFlinger侧：
文件：frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp
sp<IBinder> SurfaceFlinger::createDisplay(const String8& displayName, bool secure) {
class DisplayToken : public BBinder {
sp<SurfaceFlinger> flinger;
virtual ~DisplayToken() {
// no more references, this display must be terminated
Mutex::Autolock _l(flinger->mStateLock);
flinger->mCurrentState.displays.removeItem(this);
flinger->setTransactionFlags(eDisplayTransactionNeeded);
}
public:
explicit DisplayToken(const sp<SurfaceFlinger>& flinger)
: flinger(flinger) {
}
};
// new了一个token ，这个是可以跨进程传递的对象
sp<BBinder> token = new DisplayToken(this);
Mutex::Autolock _l(mStateLock);
// Display ID is assigned when virtual display is allocated by HWC.
DisplayDeviceState state;
state.isSecure = secure;
state.displayName = displayName;
// 把display的状态存放在mCurrentState
mCurrentState.displays.add(token, state);
mInterceptor->saveDisplayCreation(state);
return token;
}
CreateDisplay 的作用是创建了一个token返回给框架层，框架层通过这个token就能识别到这个display，然后存放到mCurrentState.displays 里面。
setDisplaySurface
WMS会通过这个接口给SurfaceFlinger传一个Surface，这个Surface是创建VirtualDisplay的进程用来显示内容的。要注意这个Surface与SurfaceFlinger不在同一个进程。
setDisplaySurface的callback.png
文件：frameworks/native/libs/gui/SurfaceComposerClient.cpp
status_t SurfaceComposerClient::Transaction::setDisplaySurface(const sp<IBinder>& token,
const sp<IGraphicBufferProducer>& bufferProducer) {
...
DisplayState& s(getDisplayState(token));
// 设置DisplayState的surface
s.surface = bufferProducer;
s.what |= DisplayState::eSurfaceChanged;
return NO_ERROR;
}
这个接口的主要作用就是设置surface给SurfaceFlinger，这个surface是个BufferQufferProducer对象，由vds所在的进程创建而成。
processDisplayAdded
有display发生变化时，transactionFlags 就会被置上eDisplayTransactionNeeded 这个flag，有新增加的display时，就会走processDisplayAdded这个逻辑，这个函数承载着主要的逻辑。
文件：frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp
void SurfaceFlinger::processDisplayAdded(const wp<IBinder>& displayToken,
const DisplayDeviceState& state) {
...
// 物理屏的逻辑，虚拟屏不走
if (state.physical) {
const auto& activeConfig =
getCompositionEngine().getHwComposer().getActiveConfig(state.physical->id);
width = activeConfig->getWidth();
height = activeConfig->getHeight();
pixelFormat = static_cast<ui::PixelFormat>(PIXEL_FORMAT_RGBA_8888);
} else if (state.surface != nullptr) {
// 虚拟屏逻辑走这里，查询传进来的Surface的宽，高，格式
int status = state.surface->query(NATIVE_WINDOW_WIDTH, &width);
ALOGE_IF(status != NO_ERROR, "Unable to query width (%d)", status);
status = state.surface->query(NATIVE_WINDOW_HEIGHT, &height);
ALOGE_IF(status != NO_ERROR, "Unable to query height (%d)", status);
int intPixelFormat;
status = state.surface->query(NATIVE_WINDOW_FORMAT, &intPixelFormat);
ALOGE_IF(status != NO_ERROR, "Unable to query format (%d)", status);
pixelFormat = static_cast<ui::PixelFormat>(intPixelFormat);
...
} else {
// Virtual displays without a surface are dormant:
// they have external state (layer stack, projection,
// etc.) but no internal state (i.e. a DisplayDevice).
return;
}
compositionengine::DisplayCreationArgsBuilder builder;
if (const auto& physical = state.physical) {
// 如果是主屏则设置display id
builder.setPhysical({physical->id, physical->type});
}
// 设置display的属性参数
builder.setPixels(ui::Size(width, height));
builder.setPixelFormat(pixelFormat);
builder.setIsSecure(state.isSecure);
builder.setLayerStackId(state.layerStack);
builder.setPowerAdvisor(&mPowerAdvisor);
// 设置是否支持使用HWC合成 VDS
builder.setUseHwcVirtualDisplays((mUseHwcVirtualDisplays && canAllocateHwcForVDS) ||
getHwComposer().isUsingVrComposer());
builder.setName(state.displayName);
// 创建compositionDisplay，这个函数的作用是创建对应的Output和Display
const auto compositionDisplay = getCompositionEngine().createDisplay(builder.build());
sp<compositionengine::DisplaySurface> displaySurface;
sp<IGraphicBufferProducer> producer;
sp<IGraphicBufferProducer> bqProducer;
sp<IGraphicBufferConsumer> bqConsumer;
// 创建一个BufferQueue，拿到对应的BufferQueueProducer和BufferQueueConsumer
getFactory().createBufferQueue(&bqProducer, &bqConsumer, /*consumerIsSurfaceFlinger =*/false);
std::optional<DisplayId> displayId = compositionDisplay->getId();
if (state.isVirtual()) {
// 创建VirtualDisplaySurface
sp<VirtualDisplaySurface> vds =
new VirtualDisplaySurface(getHwComposer(), displayId, state.surface,
bqProducer, bqConsumer, state.displayName,
state.isSecure);
// 将vds设置给displaySurface 和 producer
displaySurface = vds;
producer = vds;
} else {
// 主屏会创建FrameBufferSurface
...
}
// 创建nativeWindowSurface和displaydevice
const auto display = setupNewDisplayDeviceInternal(displayToken, compositionDisplay, state,
displaySurface, producer);
mDisplays.emplace(displayToken, display);
...
}
(1) getCompositionEngine().createDisplay(builder.build()) 会创建相应的Output和 Display对象，对应的类序图如下：
display的类序图.png
（2）new VirtualDisplaySurface，创建VDS，将surface作为 mSource[SOURCE_SINK]
文件：frameworks/native/services/surfaceflinger/DisplayHardware/VirtualDisplaySurface.cpp
VirtualDisplaySurface::VirtualDisplaySurface(HWComposer& hwc,
const std::optional<DisplayId>& displayId,
const sp<IGraphicBufferProducer>& sink,
const sp<IGraphicBufferProducer>& bqProducer,
const sp<IGraphicBufferConsumer>& bqConsumer,
const std::string& name, bool secure)
: ConsumerBase(bqConsumer),
... {
// 将surface作为mSource[SOURCE_SINK]， BufferQueueProducer作为mSource[SOURCE_SCRATCH]
mSource[SOURCE_SINK] = sink;
mSource[SOURCE_SCRATCH] = bqProducer;
resetPerFrameState();
int sinkWidth, sinkHeight;
// 查询surface的宽高
sink->query(NATIVE_WINDOW_WIDTH, &sinkWidth);
sink->query(NATIVE_WINDOW_HEIGHT, &sinkHeight);
mSinkBufferWidth = sinkWidth;
mSinkBufferHeight = sinkHeight;
// 查询和设置usage,format
int sinkUsage;
sink->query(NATIVE_WINDOW_CONSUMER_USAGE_BITS, &sinkUsage);
mSinkUsage |= (GRALLOC_USAGE_HW_COMPOSER | sinkUsage);
setOutputUsage(mSinkUsage);
if (sinkUsage & (GRALLOC_USAGE_SW_READ_MASK | GRALLOC_USAGE_SW_WRITE_MASK)) {
int sinkFormat;
sink->query(NATIVE_WINDOW_FORMAT, &sinkFormat);
mDefaultOutputFormat = sinkFormat;
} else {
mDefaultOutputFormat = HAL_PIXEL_FORMAT_IMPLEMENTATION_DEFINED;
}
mOutputFormat = mDefaultOutputFormat;
// 设置BufferQueueConsumer的name,usage,BufferSize
ConsumerBase::mName = String8::format("VDS: %s", mDisplayName.c_str());
mConsumer->setConsumerName(ConsumerBase::mName);
mConsumer->setConsumerUsageBits(GRALLOC_USAGE_HW_COMPOSER);
mConsumer->setDefaultBufferSize(sinkWidth, sinkHeight);
sink->setAsyncMode(true);
IGraphicBufferProducer::QueueBufferOutput output;
mSource[SOURCE_SCRATCH]->connect(nullptr, NATIVE_WINDOW_API_EGL, false, &output);
}
这一步最重要的就是将surface设置成了mSource[SOURCE_SINK]。
（3） setupNewDisplayDeviceInternal 这个函数创建了nativeWindowSurface和displaydevice
文件：frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp
sp<DisplayDevice> SurfaceFlinger::setupNewDisplayDeviceInternal(
const wp<IBinder>& displayToken,
std::shared_ptr<compositionengine::Display> compositionDisplay,
const DisplayDeviceState& state,
const sp<compositionengine::DisplaySurface>& displaySurface,
const sp<IGraphicBufferProducer>& producer) {
...
// 将producer作为参数创建一个NativeWindowSurface，这个producer就是vds
auto nativeWindowSurface = getFactory().createNativeWindowSurface(producer);
auto nativeWindow = nativeWindowSurface->getNativeWindow();
creationArgs.nativeWindow = nativeWindow;
// Make sure that composition can never be stalled by a virtual display
// consumer that isn't processing buffers fast enough. We have to do this
// here, in case the display is composed entirely by HWC.
if (state.isVirtual()) {
nativeWindow->setSwapInterval(nativeWindow.get(), 0);
}
...
这里重要的是将vds作为producer创建了Surface
vds类序图.png
文件：frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp
sp<DisplayDevice> SurfaceFlinger::setupNewDisplayDeviceInternal(
const wp<IBinder>& displayToken,
std::shared_ptr<compositionengine::Display> compositionDisplay,
const DisplayDeviceState& state,
const sp<compositionengine::DisplaySurface>& displaySurface,
const sp<IGraphicBufferProducer>& producer) {
...
// 虚拟屏一直设为power on
creationArgs.initialPowerMode = state.isVirtual() ? hal::PowerMode::ON : hal::PowerMode::OFF;
// 创建DisplayDevice
sp<DisplayDevice> display = getFactory().createDisplayDevice(creationArgs);
...
// 设置vds displaydevice的参数
display->setLayerStack(state.layerStack);
display->setProjection(state.orientation, state.viewport, state.frame);
display->setDisplayName(state.displayName);
}
文件：frameworks/native/services/surfaceflinger/DisplayDevice.cpp
DisplayDevice::DisplayDevice(DisplayDeviceCreationArgs& args)
: mFlinger(args.flinger),
mDisplayToken(args.displayToken),
mSequenceId(args.sequenceId),
mConnectionType(args.connectionType),
mCompositionDisplay{args.compositionDisplay},
mPhysicalOrientation(args.physicalOrientation),
mIsPrimary(args.isPrimary) {
mCompositionDisplay->editState().isSecure = args.isSecure;
// 创建RenderSurface，将vds 和 windowSurface 作为参数传进来
mCompositionDisplay->createRenderSurface(
compositionengine::RenderSurfaceCreationArgs{ANativeWindow_getWidth(
args.nativeWindow.get()),
ANativeWindow_getHeight(
args.nativeWindow.get()),
args.nativeWindow, args.displaySurface});
...
}
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/RenderSurface.cpp
RenderSurface::RenderSurface(const CompositionEngine& compositionEngine, Display& display,
const RenderSurfaceCreationArgs& args)
: mCompositionEngine(compositionEngine),
mDisplay(display),
// mNativeWindow 为 surface对象 mDisplaySurface 为vds对象
mNativeWindow(args.nativeWindow),
mDisplaySurface(args.displaySurface),
mSize(args.displayWidth, args.displayHeight) {
LOG_ALWAYS_FATAL_IF(!mNativeWindow);
}
这一步最重要的就是创建了nativeWindowSurface和displaydevice对象，到这里初始化的流程就走完了。
数据源
vds创建后，跟着SurfaceFlinger主线程进行刷新，SurfaceFlinger refresh的几个接口在“显示框架之SurfaceFlinger Refresh流程”里面有分析，这个流程对于VDS没差，但有些函数调用有差别，来看下：
Refresh主要执行的几个函数：
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/Output.cpp
void Output::present(const compositionengine::CompositionRefreshArgs& refreshArgs) {
ATRACE_CALL();
ALOGV(__FUNCTION__);
updateColorProfile(refreshArgs);
updateAndWriteCompositionState(refreshArgs);
setColorTransform(refreshArgs);
beginFrame();
prepareFrame();
devOptRepaintFlash(refreshArgs);
finishFrame(refreshArgs);
postFramebuffer();
}
beginFrame
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/Output.cpp
void Output::beginFrame() {
...
mRenderSurface->beginFrame(mustRecompose);
if (mustRecompose) {
outputState.lastCompositionHadVisibleLayers = !empty;
}
}
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/RenderSurface.cpp
status_t RenderSurface::beginFrame(bool mustRecompose) {
return mDisplaySurface->beginFrame(mustRecompose);
}
文件：frameworks/native/services/surfaceflinger/DisplayHardware/VirtualDisplaySurface.cpp
status_t VirtualDisplaySurface::beginFrame(bool mustRecompose) {
// 因为hwc暂不支持vds功能，故这里displayid为null，直接return
if (!mDisplayId) {
return NO_ERROR;
}
mMustRecompose = mustRecompose;
//For WFD use cases we must always set the recompose flag in order
//to support pause/resume functionality
if (mOutputUsage & GRALLOC_USAGE_HW_VIDEO_ENCODER) {
mMustRecompose = true;
}
VDS_LOGW_IF(mDbgState != DBG_STATE_IDLE,
"Unexpected beginFrame() in %s state", dbgStateStr());
mDbgState = DBG_STATE_BEGUN;
return refreshOutputBuffer();
}
注意这里因为hwc不支持vds，故displayid 为null，直接return，没做什么事情。
prepareFrame
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/Output.cpp
void Output::prepareFrame() {
ATRACE_CALL();
ALOGV(__FUNCTION__);
const auto& outputState = getState();
if (!outputState.isEnabled) {
return;
}
// 没有hwcid，故直接走GPU合成
chooseCompositionStrategy();
mRenderSurface->prepareFrame(outputState.usesClientComposition,
outputState.usesDeviceComposition);
}
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/RenderSurface.cpp
void RenderSurface::prepareFrame(bool usesClientComposition, bool usesDeviceComposition) {
DisplaySurface::CompositionType compositionType;
...
} else if (usesClientComposition) {
// 这里直接走GPU合成
compositionType = DisplaySurface::COMPOSITION_GPU;
...
if (status_t result = mDisplaySurface->prepareFrame(compositionType); result != NO_ERROR) {
ALOGE("updateCompositionType failed for %s: %d (%s)", mDisplay.getName().c_str(), result,
strerror(-result));
}
}
文件：frameworks/native/services/surfaceflinger/DisplayHardware/VirtualDisplaySurface.cpp
status_t VirtualDisplaySurface::prepareFrame(CompositionType compositionType) {
// 没有Displayid 直接return
if (!mDisplayId) {
return NO_ERROR;
}
...
}
这个函数的作用就是判断了vds的合成类型，因为hwc不支持的原因，所以目前走GPU合成。
finishFrame
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/Output.cpp
void Output::finishFrame(const compositionengine::CompositionRefreshArgs& refreshArgs) {
...
// dequeueBuffer
auto optReadyFence = composeSurfaces(Region::INVALID_REGION, refreshArgs);
if (!optReadyFence) {
return;
}
// swap buffers (presentation)
mRenderSurface->queueBuffer(std::move(*optReadyFence));
}
std::optional<base::unique_fd> Output::composeSurfaces(
const Region& debugRegion, const compositionengine::CompositionRefreshArgs& refreshArgs) {
...
base::unique_fd fd;
sp<GraphicBuffer> buf;
// If we aren't doing client composition on this output, but do have a
// flipClientTarget request for this frame on this output, we still need to
// dequeue a buffer.
if (hasClientComposition || outputState.flipClientTarget) {
buf = mRenderSurface->dequeueBuffer(&fd);
if (buf == nullptr) {
ALOGW("Dequeuing buffer for display [%s] failed, bailing out of "
"client composition for this frame",
mName.c_str());
return {};
}
}
// GPU合成的逻辑
...
}
文件：frameworks/native/services/surfaceflinger/CompositionEngine/src/RenderSurface.cpp
sp<GraphicBuffer> RenderSurface::dequeueBuffer(base::unique_fd* bufferFence) {
ATRACE_CALL();
int fd = -1;
ANativeWindowBuffer* buffer = nullptr;
// 主要的区别在这里，mNativeWindow 为Surface对象，调到Surface.cpp里面的dequeueBuffer，然后再调到VirtualDisplay的deququBuffer
status_t result = mNativeWindow->dequeueBuffer(mNativeWindow.get(), &buffer, &fd);
if (result != NO_ERROR) {
ALOGE("ANativeWindow::dequeueBuffer failed for display [%s] with error: %d",
mDisplay.getName().c_str(), result);
// Return fast here as we can't do much more - any rendering we do
// now will just be wrong.
return mGraphicBuffer;
}
ALOGW_IF(mGraphicBuffer != nullptr, "Clobbering a non-null pointer to a buffer [%p].",
mGraphicBuffer->getNativeBuffer()->handle);
mGraphicBuffer = GraphicBuffer::from(buffer);
*bufferFence = base::unique_fd(fd);
return mGraphicBuffer;
}
文件：frameworks/native/libs/gui/Surface.cpp
int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {
...
// 这里mGraphicBufferProducer 对象为VirtualDisplaySurface
status_t result = mGraphicBufferProducer->dequeueBuffer(&buf, &fence, reqWidth, reqHeight,
reqFormat, reqUsage, &mBufferAge,
enableFrameTimestamps ? &frameTimestamps
: nullptr);
...
}
文件：frameworks/native/services/surfaceflinger/DisplayHardware/VirtualDisplaySurface.cpp
status_t VirtualDisplaySurface::dequeueBuffer(int* pslot, sp<Fence>* fence, uint32_t w, uint32_t h,
PixelFormat format, uint64_t usage,
uint64_t* outBufferAge,
FrameEventHistoryDelta* outTimestamps) {
if (!mDisplayId) {
// 这里执行的是mSource[SOURCE_SINK] 的dequeueBuffer，mSource[SOURCE_SINK] 实质上就是应用传进来的surface
return mSource[SOURCE_SINK]->dequeueBuffer(pslot, fence, w, h, format, usage, outBufferAge,
outTimestamps);
}
这里最重要的就是理解dequeueBuffer的执行对象是谁，RenderSurface::dequeueBuffer -> NativeWindow:: dequeueBuffer -> Surface::dequeueBuffer ->VirtualDisplaySurface:: dequeueBuffer-> sf对端进程的Surface::dequeueBuffer
同理，queueBuffer的执行对象跟dequeueBuffer一样，RenderSurface::queueBuffer -> NativeWindow:: queueBuffer -> Surface::queueBuffer ->VirtualDisplaySurface:: queueBuffer-> sf对端进程的Surface::queueBuffer
可以看出来dequeueBuffer和queueBuffer都是在sf对端进程实现，从systrace也可以看到，这里SurfaceFlinger作为Client端，media.codec为Server端。
SurfaceFlinger Client端.png
media.codec server端.png
在 ”显示框架之SurfaceFlinger GPU合成 “ 分析到dequeueBuffer出来的Buffer作为输出的Buffer，输入为当前layer的Buffer，可以理解为GPU将输入的n块Buffer合成输出到1块Buffer，具体流程可以看“显示框架之SurfaceFlinger GPU合成” 的分析，可以看到其实数据源就是GPU合成的这块Buffer, 交给media.codec去消费，这块Buffer的acquire进程也是media.codec。
media.codec acquireBuffer.png
之后Buffer就给到media去处理了
mediaserver处理buffer.png
总结：对于VDS，SurfaceFlinger是作为Client端，GPU合成的结果是数据源，所以虚拟屏显示的内容和主屏是一样的。
