原课程链接：https://www.cc.gatech.edu/~hays/compvision/
维基百科定义：https://en.wikipedia.org/wiki/Optical_flow#Methods_for_determination
百度百科定义：https://baike.baidu.com/item/%E5%85%89%E6%B5%81/7013666

Video and Motion

A video is a sequence of frames captured over time
Now our image data is a function of space (x, y) and time (t)

video is a sequence of frames.png

Sometimes, motion is the only cue
Even “impoverished” motion data can evoke a strong percept

cue from motion.png

Motion Estimation: Optical Flow

motion of cube.png motion's optical flow.png

Problem Define

How to estimate the motion of pixels from image I(x, y, t) to I(x, y, t+1)

motion of pixels.png

Key Assumption:
• color constancy:
– a point in I(x,y,t) looks the same in I(x,y,t+1)
– For grayscale images, this is brightness constancy
• small motion:
– Points do not move very far

Then we can obtain the optical flow constrains:

motion constrains.png

Brightness Constancy Constraint (equation):

brightness constancy constrain function.png

Small Motion: (u and v are less than 1 pixel, or smooth):
Taylor series expansion of I:

Taylor series expansion.png

Combining the two equations, we have:

equation.png equation.png

In the limit as u and v go to zero, this becomes exact:

image.png

Brightness constancy constraint equation

image.png

How many equations and unknowns per pixel?
there is only one equation, but two unknows(u, v)

How to get more equations for a pixel?
Use spatial coherence constraint.

Spatial Coherence Constraint(equation):

Assum the pixel's neighbors have the same(u, v)
if we use a 5x5 window, then we can get 25 equations per pixel：

25 equations.png

Least squares solution for d given by:

least squares solution.png least squares solution.png

When is this solvable?

condition.png

if ATA is not invertible, will render Aperture Problem

Aperture Problem Does this remind you of anything?

Criteria for Harris Corner Detector！！！

baber pole illusion.png baber pole illusion.png

Errors in Lucas-Kanade

A point does not move like its neighbors
Motion segmentation

Brightness constancy does not hold
Do exhaustive neighborhood search with normalized correlation -tracking features – maybe SIFT – more later….

The motion is large (larger than a pixel)
Not-linear: Iterative refinement
Local minima: coarse-to-fine estimation

Revisiting the small motion assumption

Is this motion small enough?
Probably not—it’s much larger than one pixel

How might we solve this problem?
Coarse-to-fine optical estimation

coarse to fine.png coarse to fine.png result without pyramids.png result with pyramids.png