[Camera]RK_Camera_HAL
2020-02-15 本文已影响0人
Letcos
platform: RK3399
OS: Android 7.1
kernel：4.4
参考：
1. KrisFei https://blog.csdn.net/kris_fei/article/details/52451409
2. Kiazhu https://blog.csdn.net/kiazhu/article/details/84652749
3. https://source.android.google.cn/devices/camera?hl=en
4.https://www.cnblogs.com/blogs-of-lxl/p/10981303.html
概述
Android Camera Hal 是android framework和kernel联系的重要通道。该部分一般由厂商实现并封装自己的算法或者业务逻辑，进一步抽象出内核设备驱动的各项功能，并通过google实现的统一接口供android framework进行调用。
下面是旧版的Camera框架图(适用于android 8.0以下,android 8.0以上请使用新的camera hal组件及框架)
在这里插入图片描述
应用框架
应用代码位于应用框架级别，它利用 android.hardware.Camera API 与相机硬件进行交互。在内部，此代码会调用相应的 JNI 粘合类，以访问与相机互动的原生代码。
JNI
与 android.hardware.Camera 关联的 JNI 代码位于 frameworks/base/core/jni/android_hardware_Camera.cpp 中。此代码会调用较低级别的原生代码以获取对实体相机的访问权限，并返回用于在框架级别创建 android.hardware.Camera 对象的数据。
原生框架
在 frameworks/av/camera/Camera.cpp 中定义的原生框架可提供相当于 android.hardware.Camera 类的原生类。此类会调用 IPC binder 代理，以获取对相机服务的访问权限。
Binder IPC 代理
IPC binder 代理用于促进跨越进程边界的通信。调用相机服务的 3 个相机 binder 类位于 frameworks/av/camera 目录中。 ICameraService 是相机服务的接口；ICamera 是已打开的特定相机设备的接口；ICameraClient 是返回到应用框架的设备接口。
相机服务
位于 frameworks/av/services/camera/libcameraservice/CameraService.cpp 下的相机服务是与 HAL 进行互动的实际代码。
HAL
硬件抽象层定义了由相机服务调用、且您必须实现以确保相机硬件正常运行的标准接口。
内核驱动程序
相机的驱动程序可与实际相机硬件以及您的 HAL 实现进行互动。相机和驱动程序必须支持 YV12 和 NV21 图片格式，以便在显示和视频录制时支持预览相机图片。
Android Camera HAL简介
HAL1简介
功能调用简图
在这里插入图片描述
Camera hal1中的camera_module_t 接口
camera_module_t HAL_MODULE_INFO_SYM = {
common: {
tag: HARDWARE_MODULE_TAG,
version_major: ((CONFIG_CAMERAHAL_VERSION&0xffff00)>>16),
version_minor: CONFIG_CAMERAHAL_VERSION&0xff,
id: CAMERA_HARDWARE_MODULE_ID,
name: CAMERA_MODULE_NAME,
author: "RockChip",
methods: &camera_module_methods,
dso: NULL, /* remove compilation warnings */
reserved: {0}, /* remove compilation warnings */
},
get_number_of_cameras: camera_get_number_of_cameras,
get_camera_info: camera_get_camera_info,
set_callbacks:NULL,
get_vendor_tag_ops:NULL,
}
HAL3简介
Camera API2/HAL3架构下使用了全新的CameraMetadata结构取代了之前的SetParameter/Paramters等操作，实现了Java到native到HAL3的参数传递。引入了管道的概念将安卓设备和摄像头之间联系起来，系统向摄像头发送 Capture 请求，而摄像头会返回 CameraMetadata，这一切建立在一个叫作 CameraCaptureSession 的会话中。
功能调用简图
在这里插入图片描述
以最常见的android.control Section为例,下图描述了Camera Metadata对不同section以及相应section下不同tag的布局图
在这里插入图片描述
RK Camera HAL整体框架
在这里插入图片描述
canera server通过与CameraHal_Module的标准接口建立和Hal层的连接后，CmaeraHal会通过SensorListener返回消息给上层。如果上层要预览或者拍照，要通过binder机制向Hal层发送命令，Hal层MessageQueue.cpp的消息队列获取到上层的命令后，会通过消息通知器通知CameraHal，CameraHal有一个CommandThread接收命令，收到命令后，最终会下达命令给对应的Adapter去执行对于的动作。
RK 的Camera Hal还实现了人脸检测等功能以及相关的辅助函数，就没有在上面的主体框架中画出来。
RK Camera HAL 调用流程
开机启动
CameraHal_Module.cpp --> camera_get_number_of_cameras
-->profiles = camera_board_profiles::getInstance(); // 解析/etc/cam_board.xml
-->camera_board_profiles::LoadSensor(profiles); //注册摄像头驱动
-->OpenAndRegistOneSensor(profiles->mDevieVector[profiles->mXmlDevInfo[i].index]); //最多支持双摄
-->RegisterSensorDevice(pCamInfo); //注册i2c设备
-->camsys_fd = open(pSensorInfo->mCamsysDevPath, O_RDWR); //打开dev/camsys_marvin1 设备节点(camsys_marvin1注册见“RK平台摄像头驱动”文章)
-->ioctl(camsys_fd, CAMSYS_VERCHK, &(pCamInfo->mCamsysVersion)); //检查驱动版本
--> ioctl(camsys_fd, CAMSYS_REGISTER_DEVIO, &extdev); //注册摄像头
-->rk_sensor_pwrseq(camsys_fd, pCamInfo, 1); // 启动摄像头
--> ioctl(camsys_fd, CAMSYS_I2CWR, &i2cinfo); //配置I2C信息
--> ioctl(camsys_fd, CAMSYS_QUREYIOMMU, &iommu_enabled); //查询IOMMU使能
--> ioctl(camsys_fd, CAMSYS_I2CRD, &i2cinfo); //查询设备ID
-->rk_sensor_pwrseq(camsys_fd, pCamInfo, 0); //power off
--> if(pSensorInfo->mFacing == RK_CAM_FACING_FRONT){ //配置摄像头方向
camInfoTmp[cam_cnt&0x01].facing_info.facing = CAMERA_FACING_FRONT;
--> open(cam_path, O_RDONLY); //打开videoX
-->ioctl(fd, VIDIOC_QUERYCAP, &capability) //查询capability
-->rk_cam_total_info* pNewCamInfo = new rk_cam_total_info(); //所有sensor信息都放在里面
-->ioctl(fd, VIDIOC_ENUM_FMT, &fmtdesc）
--> ioctl(fd, VIDIOC_ENUM_FRAMESIZES, &fsize)
-->rk_DV_info *pDVResolution = new rk_DV_info();
--> ioctl(fd, VIDIOC_ENUM_FRAMEINTERVALS, &fival)
--->camera_board_profiles::ProduceNewXml(profiles); //生成media_profiles.xml
App打开Camera
CameraHal_Module.cpp -->camera_device_open
//标准实现
-->camera_device = (rk_camera_device_t*)malloc(sizeof(*camera_device));
--> camera_ops = (camera_device_ops_t*)malloc(sizeof(*camera_ops));
.... //初始化camera_device camera_ops
//RK自己实现
//CameraHal类负责与cameraservice联系，实现
//cameraservice要求实现的接口。此类只负责公共资源的申请，以及任务的分发。
-->android::CameraHal(cameraid)
-->commandThreadCommandQ("commandCmdQ") //创建commandCmdQ的message queue
-> new IonMemManager
mPreviewBuf = new PreviewBufferProvider(mCamMemManager); //previewBuffer
mVideoBuf = new BufferProvider(mCamMemManager); //video
mRawBuf = new BufferProvider(mCamMemManager); //raw
mJpegBuf = new BufferProvider(mCamMemManager); //jpeg
mUvcBuf = new BufferProvider(mCamMemManager); //uvc
//根据不同类型的camera new不同类型的adapter
--->mCameraAdapter new CameraIspAdapter(cameraId)
-->mDisplayAdapter = new DisplayAdapter(); //new Display adapter
-->displayThreadCommandQ //创建名为displayCmdQ的message queue.
new DisplayThread //处理显示相关事物
-->mEventNotifier = new AppMsgNotifier(mCameraAdapter);
-->encProcessThreadCommandQ //创建名字为pictureEncThreadQ的message queue.
eventThreadCommandQ //创建名字为eventThreadQ的message queue.
create_vpu_memory_pool_allocator //创建vpu 内存池分配器
new CameraAppMsgThread
new EncProcessThread
new CameraAppFaceDetThread //人脸检测线程
new CameraAppCallbackThread //回调线程
-->mCameraAdapter->initialize()
--->cameraCreate(mCamId) //创建一个camera
--->initDefaultParameters(mCamId); //初始化各个参数默认值
--->mCommandThread = new CommandThread(this); //command Thread loop
---->mSensorListener = new SensorListener(); //注册Listerner
---->new SensorLooperThread
---->mLooper->pollOnce //循环查询是否有event上报，有就会运行sensor_events_listener()获取Orientation。
支持的ops
camera_ops->set_preview_window = camera_set_preview_window;
camera_ops->set_callbacks = camera_set_callbacks;
camera_ops->enable_msg_type = camera_enable_msg_type;
camera_ops->disable_msg_type = camera_disable_msg_type;
camera_ops->msg_type_enabled = camera_msg_type_enabled;
camera_ops->start_preview = camera_start_preview;
camera_ops->stop_preview = camera_stop_preview;
camera_ops->preview_enabled = camera_preview_enabled;
camera_ops->store_meta_data_in_buffers = camera_store_meta_data_in_buffers;
camera_ops->start_recording = camera_start_recording;
camera_ops->stop_recording = camera_stop_recording;
camera_ops->recording_enabled = camera_recording_enabled;
camera_ops->release_recording_frame = camera_release_recording_frame;
camera_ops->auto_focus = camera_auto_focus;
camera_ops->cancel_auto_focus = camera_cancel_auto_focus;
camera_ops->take_picture = camera_take_picture;
camera_ops->cancel_picture = camera_cancel_picture;
camera_ops->set_parameters = camera_set_parameters;
camera_ops->get_parameters = camera_get_parameters;
camera_ops->put_parameters = camera_put_parameters;
camera_ops->send_command = camera_send_command;
camera_ops->release = camera_release;
camera_ops->dump = camera_dump;
command Thread loop支持的cmd
CMD_PREVIEW_START
CMD_PREVIEW_STOP
CMD_SET_PREVIEW_WINDOW
CMD_SET_PARAMETERS
CMD_PREVIEW_CAPTURE_CANCEL
CMD_CONTINUOS_PICTURE
CMD_AF_START
CMD_AF_CANCEL
CMD_START_FACE_DETECTION
CMD_EXIT
设置参数
CameraHal::setParameters //参数是char*
setParameters -> //参数是CameraParameters
commandThreadCommandQ.put //cmd是 CMD_SET_PARAMETERS
commandThread ->
mCameraAdapter->setParameters
CameraSOCAdapter::setParameters -> //假设位soc camera
mRefEventNotifier->setPreviewDataCbRes
cameraConfig
ioctl(mCamFd, VIDIOC_S_CTRL, &control); //设置 白平衡
//设置zoom,color effect,scene,anti banding,white balance
//lock ,exposurelock,focus,flash mode,exposure
ioctl(mCamFd, VIDIOC_S_EXT_CTRLS, &extCtrInfos);
设置显示窗口
CameraHal::setPreviewWindow ->
commandThreadCommandQ.put -> //cmd: CMD_SET_PREVIEW_WINDOW
commandThread ->
mDisplayAdapter->setPreviewWindow
// mDisplayAdapter->startDisplay(app_previw_w, app_preview_h); //启动预览
Start preview
CameraHal::startPreview
---> commandThreadCommandQ.put -> //cmd: CMD_PREVIEW_START
-->commandThread
mParameters.getPreviewSize //获取预览参数
mCameraAdapter->getCurPreviewState //获取当前预览状态
//选择picture size, 如果和preview 分辨率不一样，那需要stop preview再start preview
selectPreferedDrvSize(&prefered_w,&prefered_h,false);
//如果在预览中
mDisplayAdapter->pauseDisplay
displayThreadCommandQ.put(&msg) //CMD_DISPLAY_PAUSE
displayThread
cameraDisplayBufferDestory();
mEventNotifier->stopReceiveFrame
flushPicture(); //EncProcessThread::CMD_ENCPROCESS_PAUSE
pausePreviewCBFrameProcess(); //CameraAppMsgThread::CMD_EVENT_PAUSE
stopFaceDection(); //CameraAppFaceDetThread::CMD_FACEDET_PAUSE
mCameraAdapter->stopPreview
cameraStream(false)
ioctl(mCamFd, cmd, &type)
mCameraPreviewThread->requestExitAndWait(); //quit preview thread
mCameraPreviewThread.clear();
cameraStop()
mPreviewBufferProvider->freeBuffer()
mCameraAdapter->startPreview
mPreviewBufProvider->createBuffer //PREVIEWBUFFER
mCamBuffer->createPreviewBuffer
createIonBuffer
cameraSetSize(w, h, mCamDriverPreviewFmt,is_capture)
ioctl(mCamFd, VIDIOC_S_FMT, &format)
cameraStart()
ioctl(mCamFd, VIDIOC_REQBUFS, &creqbuf)
ioctl(mCamFd, VIDIOC_QUERYBUF, &buffer)
ioctl(mCamFd, VIDIOC_QBUF, &buffer)
mmap
cameraStream(true)
mCameraPreviewThread = new CameraPreviewThread(this) //进入preview thread loop
mEventNotifier->startReceiveFrame
mEventNotifier->startFaceDection //如果配置了人脸检测功能就开始人脸检测
//否则同上，只是不用stopPreview
mDisplayAdapter->startDisplay
displayThreadCommandQ.put(&msg) //CMD_DISPLAY_START
displayThread
cameraDisplayBufferDestory()
//准备display buffer放frame,然后thread进入休眠等待frame的到来
cameraDisplayBufferCreate()
//ioctl stream on后frame就会从kernel上来，接着调用previewthread
CameraAdapter::previewThread
getFrame
ioctl(mCamFd, VIDIOC_DQBUF, &cfilledbuffer1)
//判断以什么形式显示：display,video,picture,datacb
//判断调用那个notify 这里是CMD_PREVIEWBUF_DISPING
mRefDisplayAdapter->notifyNewFrame
displayThreadCommandQ.put(&msg) //CMD_DISPLAY_FRAME
displayThread
mANativeWindow->dequeue_buffer
mANativeWindow->lock_buffer
mANativeWindow->enqueue_buffer
mFrameProvider->returnFrame
adapterReturnFrame //把使用好的buffer还给系统
ioctl(mCamFd, VIDIOC_QBUF, &vb)
Stop preview
camera_stop_preview
gCameraHals[rk_dev->cameraid]->stopPreview()
CameraHal::stopPreview
commandThreadCommandQ.put(&msg) //CMD_PREVIEW_STOP
commandThread:
mDisplayAdapter->pauseDisplay()
mEventNotifier->stopReceiveFrame()
mCameraAdapter->stopPreview()
cameraStream(false)
ioctl(mCamFd, cmd, &type)
mCameraPreviewThread->requestExitAndWait(); //quit preview thread
mCameraPreviewThread.clear();
cameraStop()
mPreviewBufferProvider->freeBuffer()
Take Picture
camera_take_picture
gCameraHals[rk_dev->cameraid]->takePicture()
CameraHal::commandThread //CMD_CONTINUOS_PICTURE
fillPicturInfo(picinfo)
mEventNotifier->takePicture(picinfo)
AppMsgNotifier::takePicture //设置running state为STA_RECEIVE_PIC_FRAME，等着previewThread去get frame
//previewThread接到frame之后
CameraAdapter::previewThread
mRefEventNotifier->isNeedSendToPicture
mRefEventNotifier->notifyNewPicFrame
AppMsgNotifier::notifyNewPicFrame
//cmd: EncProcessThread::CMD_ENCPROCESS_SNAPSHOT
encProcessThreadCommandQ.put()
encProcessThread
captureEncProcessPicture
copyAndSendCompressedImage
callback_compressed_image
CameraAppCallbackThread:: CMD_MSG_COMPRESSED_IMAGE
mDataCb
frame->release(frame)
mFrameProvider->returnFrame
Record Video
CameraAdapter::previewThread
getFrame
mRefEventNotifier->notifyNewVideoFrame
eventThreadCommandQ.put //cmdCameraAppMsgThread::CMD_EVENT_VIDEO_ENCING
AppMsgNotifier::eventThread
processVideoCb
AppMsgNotifier::processVideoCb
callback_video_frame
callbackThreadCommandQ.put //cmd:CameraAppCallbackThread::CMD_MSG_VIDEO_FRAME
AppMsgNotifier::callbackThread ->
mDataCbTimestamp
Preview datacallback
CameraAdapter::previewThread ->
mRefEventNotifier->notifyNewPreviewCbFrame -> //前提是设置了callback回调函数
AppMsgNotifier::notifyNewPreviewCbFrame ->
eventThreadCommandQ.put -> //cmd: CameraAppMsgThread::CMD_EVENT_PREVIEW_DATA_CB
AppMsgNotifier::eventThread ->
processPreviewDataCb ->
AppMsgNotifier::processPreviewDataCb -> //格式转换在这里完成
cameraFormatConvert
callback_preview_frame ->
callbackThreadCommandQ.put -> //cmd: CameraAppCallbackThread::CMD_MSG_PREVIEW_FRAME
AppMsgNotifier::callbackThread ->
mDataCb //调用上层传递下来的callback.
frame->release //release buffer.
mFrameProvider->returnFrame
