Optimizing Through Learned Errors for Accurate Sports Field Registration

Category
Camera Calibration / Multi-view Camera

What

• Field localization from a single broadcast image
• Early work using Deep Learning to directly estimate a Homography Matrix

How?

Homography Optimization w/ Template

Given an homography estimate $h^i$ a registration error network is used to estimate the loss of the estimate $h^i$. Since the registration error network is differential, it can be used to optimize $h^i$.

Initial Registration Network

A ResNet-18 Architecture with the classification head swapped out for a fc-layer with an output of 8 (elements of the homography matrix).

Registration Error Network

ResNet-18 + Spectral Normalization +6-channel input (concat of input image& warped target template).

Trained from scratch.

Data Augmentation

• Random crop (light, 90% ~ 100% of original remains)
• Random horizontal flips
• Shadow simulation (random 50% opacity black patches)
• Random Translation (~1/2 full width), rotation (0~45 deg) and scaling(0.5~2x)
• Gaussian blur with kernel size 9

And?

Interesting idea but is probably very compute intensive since a single iteration requires DNN inference. However, the optimization idea can be modularized and used as a post processing step along with ideas.