[MetalKit]Using ARKit with Metal

本系列文章是对 http://metalkit.org 上面MetalKit内容的全面翻译和学习.

MetalKit系统文章目录

---

正如上次强调的那样,在ARKit应用中共有三个层级:渲染,追踪和场景理解.上次我们详细分析了如何用Metal在自定义view中实现渲染. ARKit使用了视觉惯性里程计来精确追踪周围的世界,并将相机传感器数据和CoreMotion数据结合起来.即使当我们在运动时,也无需额外调校来保持画面稳定.本文中,我们关注场景理解-用平面检测,点击测试和光线估计来描述场景属性的方法.ARKit能分析相机中的场景,并找到类似地板那样的水平面.首先,我们需要启用平面检测功能(默认是off),只需在运行会话配置羊添加一行:

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    let configuration = ARWorldTrackingConfiguration()
    configuration.planeDetection = .horizontal
    session.run(configuration)
}

注意:当前API版本下,只能检测水平面

ARSessionObserver代理方法对处理会话错误,追踪改变和打断非常有用:

func session(_ session: ARSession, didFailWithError error: Error) {}
func session(_ session: ARSession, cameraDidChangeTrackingState camera: ARCamera) {}
func session(_ session: ARSession, didOutputAudioSampleBuffer audioSampleBuffer: CMSampleBuffer) {}
func sessionWasInterrupted(_ session: ARSession) {}
func sessionInterruptionEnded(_ session: ARSession) {}

还有另一些代理方法属于ARSessionDelegate协议(属于ARSessionObserver扩展)能让我们处理锚点.在第一个里面写上print():

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
    print(anchors)
}
func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {}
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {}
func session(_ session: ARSession, didUpdate frame: ARFrame) {}

让我们进入到Renderer.swift里面.首先,创建一些需要的类属性.这些变量将帮助我们创建并在屏幕上显示一个调试用的平面:

var debugUniformBuffer: MTLBuffer!
var debugPipelineState: MTLRenderPipelineState!
var debugDepthState: MTLDepthStencilState!var debugMesh: MTKMesh!
var debugUniformBufferOffset: Int = 0
var debugUniformBufferAddress: UnsafeMutableRawPointer!
var debugInstanceCount: Int = 0

下一步,在setupPipeline()我们创建缓冲器:

debugUniformBuffer = device.makeBuffer(length: anchorUniformBufferSize, options: .storageModeShared)

我们需要给我们的平面创建新的顶点及片段函数,还有新的渲染管线和深度模板状态.在创建命令队列的前面添加几行:

let debugGeometryVertexFunction = defaultLibrary.makeFunction(name: "vertexDebugPlane")!
let debugGeometryFragmentFunction = defaultLibrary.makeFunction(name: "fragmentDebugPlane")!
anchorPipelineStateDescriptor.vertexFunction =  debugGeometryVertexFunction
anchorPipelineStateDescriptor.fragmentFunction = debugGeometryFragmentFunction
do { try debugPipelineState = device.makeRenderPipelineState(descriptor: anchorPipelineStateDescriptor)
} catch let error { print(error) }
debugDepthState = device.makeDepthStencilState(descriptor: anchorDepthStateDescriptor)

下一步,在setupAssets()里我们需要创建一个新的Model I/O平面网格,并用它创建Metal网格.在函数的末尾添加几行:

mdlMesh = MDLMesh(planeWithExtent: vector3(0.1, 0.1, 0.1), segments: vector2(1, 1), geometryType: .triangles, allocator: metalAllocator)
mdlMesh.vertexDescriptor = vertexDescriptor
do { try debugMesh = MTKMesh(mesh: mdlMesh, device: device)
} catch let error { print(error) }

下一步,在updateBufferStates()中我们需要更新平面所在缓冲器的地址.添加下面几行:

debugUniformBufferOffset = alignedInstanceUniformSize * uniformBufferIndex
debugUniformBufferAddress = debugUniformBuffer.contents().advanced(by: debugUniformBufferOffset)

下一步,在updateAnchors()中我们需要更新变换矩阵和锚点数.在循环之前添加下面几行:

let count = frame.anchors.filter{ $0.isKind(of: ARPlaneAnchor.self) }.count
debugInstanceCount = min(count, maxAnchorInstanceCount - (anchorInstanceCount - count))

然后,在循环中用下面几行替换最后的三行:

if anchor.isKind(of: ARPlaneAnchor.self) {
    let transform = anchor.transform * rotationMatrix(rotation: float3(0, 0, Float.pi/2))
    let modelMatrix = simd_mul(transform, coordinateSpaceTransform)
    let debugUniforms = debugUniformBufferAddress.assumingMemoryBound(to: InstanceUniforms.self).advanced(by: index)
    debugUniforms.pointee.modelMatrix = modelMatrix
} else {
    let modelMatrix = simd_mul(anchor.transform, coordinateSpaceTransform)
    let anchorUniforms = anchorUniformBufferAddress.assumingMemoryBound(to: InstanceUniforms.self).advanced(by: index)
    anchorUniforms.pointee.modelMatrix = modelMatrix
}

我们必须将平面绕z轴旋转90度,来让它呈水平状态.注意,我们使用了一个自定义方法,名为rotationMatrix(),让我们来定义它.我们在以前的文章中,当第一次介绍3D矩阵时就见过了这个矩阵:

func rotationMatrix(rotation: float3) -> float4x4 {
    var matrix: float4x4 = matrix_identity_float4x4
    let x = rotation.x
    let y = rotation.y
    let z = rotation.z
    matrix.columns.0.x = cos(y) * cos(z)
    matrix.columns.0.y = cos(z) * sin(x) * sin(y) - cos(x) * sin(z)
    matrix.columns.0.z = cos(x) * cos(z) * sin(y) + sin(x) * sin(z)
    matrix.columns.1.x = cos(y) * sin(z)
    matrix.columns.1.y = cos(x) * cos(z) + sin(x) * sin(y) * sin(z)
    matrix.columns.1.z = -cos(z) * sin(x) + cos(x) * sin(y) * sin(z)
    matrix.columns.2.x = -sin(y)
    matrix.columns.2.y = cos(y) * sin(x)
    matrix.columns.2.z = cos(x) * cos(y)
    matrix.columns.3.w = 1.0
    return matrix
}

下一步,在drawAnchorGeometry()中我们需要确保我们在绘制之前至少有一个锚点.将第一行替换为下面这行:

guard anchorInstanceCount - debugInstanceCount > 0 else { return }

下一步,让我们创建drawDebugGeometry()函数来绘制我们的平面.它非常类似于锚点绘制函数:

func drawDebugGeometry(renderEncoder: MTLRenderCommandEncoder) {
    guard debugInstanceCount > 0 else { return }
    renderEncoder.pushDebugGroup("DrawDebugPlanes")
    renderEncoder.setCullMode(.back)
    renderEncoder.setRenderPipelineState(debugPipelineState)
    renderEncoder.setDepthStencilState(debugDepthState)
    renderEncoder.setVertexBuffer(debugUniformBuffer, offset: debugUniformBufferOffset, index: 2)
    renderEncoder.setVertexBuffer(sharedUniformBuffer, offset: sharedUniformBufferOffset, index: 3)
    renderEncoder.setFragmentBuffer(sharedUniformBuffer, offset: sharedUniformBufferOffset, index: 3)
    for bufferIndex in 0..<debugMesh.vertexBuffers.count {
        let vertexBuffer = debugMesh.vertexBuffers[bufferIndex]
        renderEncoder.setVertexBuffer(vertexBuffer.buffer, offset: vertexBuffer.offset, index:bufferIndex)
    }
    for submesh in debugMesh.submeshes {
        renderEncoder.drawIndexedPrimitives(type: submesh.primitiveType, indexCount: submesh.indexCount, indexType: submesh.indexType, indexBuffer: submesh.indexBuffer.buffer, indexBufferOffset: submesh.indexBuffer.offset, instanceCount: debugInstanceCount)
    }
    renderEncoder.popDebugGroup()
}

在Renderer中,还有一件需要完成,就是-在update()中结束编码前,调用这个函数:

drawDebugGeometry(renderEncoder: renderEncoder)

最后,让我们进入Shaders.metal文件中.我们需要一个新的结构体,只包含从顶点描述符中传递过来的顶点位置:

typedef struct {
    float3 position [[attribute(0)]];
} DebugVertex;

在顶点着色器中我们用模型-视图矩阵来更新顶点位置:

vertex float4 vertexDebugPlane(DebugVertex in [[ stage_in]],
                               constant SharedUniforms &sharedUniforms [[ buffer(3) ]],
                               constant InstanceUniforms *instanceUniforms [[ buffer(2) ]],
                               ushort vid [[vertex_id]],
                               ushort iid [[instance_id]]) {
    float4 position = float4(in.position, 1.0);
    float4x4 modelMatrix = instanceUniforms[iid].modelMatrix;
    float4x4 modelViewMatrix = sharedUniforms.viewMatrix * modelMatrix;
    float4 outPosition = sharedUniforms.projectionMatrix * modelViewMatrix * position;
    return outPosition;
}

最后,在片段着色器中,我们给平面一个显眼在颜色以便于在视图中观察到它:

fragment float4 fragmentDebugPlane() {
    return float4(0.99, 0.42, 0.62, 1.0);
}

如果你运行应用,当检测到平面时,你将看到添加了一个矩形,像这样:

plane.gif

接下来要做的是当我们检测到更多或从先前检测到的平面上移开时,更新/移除平面.别一个代理方法能帮助我们实现这个效果.接下来,我们将研究碰撞和物理效果.只是对以后的思考.

我要感谢Caroline为本文构造了平面检测.

源代码source code已发布在Github上.
下次见!