loading . . . Mirrorless AF Calibration – Part 2: On-Sensor Phase Detect Autofocus ## Overview
Autofocus (AF) technology has revolutionised the world of photography, allowing photographers to capture sharp images quickly and consistently. One of the most popular and widely used autofocus technologies is Phase Detect Autofocus (PDAF). In this blog post, I’ll delve into the inner workings of PDAF so we have a solid foundation for understanding why calibration is an important aspect of this focusing system.
### About this Series
This post is part of a series looking at on-sensor phase-detect autofocus, the whole autofocus system, and why mirrorless autofocus might still need some calibration.
The full list of posts are:
* Part 1: You SHOULD Calibrate your Mirrorless Autofocus!
* Part 2: On-Sensor Phase Detect Autofocus
* Part 3: On-Sensor Phase-Detect Issues
* Part 4: The Lens
* Part 5: The Autofocus Processor
* Part 6: The Evidence
### Setting the Scene
To write this post and explain the operation of the various components of PDAF, I’ve taken a nostalgic trip down memory lane and installed the POV-Ray raytracer which I have fond memories of being fascinated with while learning its capabilities back in the early 1990s.
Thanks to Przemysław Śliwiński of Wrocław University of Science and Technology and the support material for the paper “ _A Simple Model for On-Sensor Phase-Detection Autofocusing Algorithm_ “, I had a starting model to demonstrate the operation of phase detect autofocus.
Without further ado, here’s a render of our world, complete with a 90’s style wooden plane for everything to sit (float) on top of!
The model that I’ll use to explain PDAF operation
On the left is the “camera” which I’ll explain more about shortly, and to the right are 3 focus targets, just coloured squares facing the camera. The blue one is nearest the camera, the yellow (which actually looks a bit green in the image above) is in the middle, and the red one furthest away.
Looking from behind the camera with the tube removed
There are 3 components to the camera – the lens, the “sensor” and the tube used to block out any stray light. Take away the dark tube, and there’s a simple lens at the front and a special grid at the back, and with the tube back on, this is the sort of view you’d have looking from behind the camera:
The tube back in place, this is the view from the back of the camera
Now, let’s take the “sensor” out of the camera and put our eye where it would have been, taking a look through the tube and lens at the front – this is our focus scene.
The view through the camera tube and lens of the focus targets
The 3 coloured targets are all the same size, but they are different distances from the front of the camera. With the simple lens at the front of the tube, we introduce the concept of focus, with the yellow square being the correct distance from the lens to render it in focus and sharp at the sensor plane (where we’re looking from). The red target is further away, and the blue is nearer, so both are showing out-of-focus blurring.
### The Sensor
What I’ve been calling the sensor is actually an array of small lenses with a black mask around them. In a real camera, this array of “microlenses” sits just above the sensor, such that each lens is located directly above a single light-sensitive photodiode. For most cameras, there’s a colour array between the light-sensitive pixel and the microlens, but I’m ignoring that today as it’s an unnecessary detail for a discussion about phase detect autofocus.
Because I can’t stop playing with POV-Ray, here’s a rendering of the microlens array with the black mask between the lenses:
An array of microlenses, along with the inter-lens mask
Let’s put that back at the end of the tube in our camera (as shown in the image above) and then we’ll pretend to be the actual image sensor and take a look through the lenses:
The focus target images through a coarse grid of 9×9 microlenses
It’s not all that impressive, is it? From where we’re looking, you can just see a small view of each of the focus targets in the lenses. What is interesting, however, is that each lens shows a slightly different view of the targets due to the different paths the light takes travelling from the object, through the lens and to the microlens grid.
### More Lenses
Next, we’ll increase the number of lenses and make them smaller, giving more detail to the overall image. To make the result less distracting, we’ll turn off the background light as well:
The focus targets through a fine grid of 33×33 microlenses
Right, now things are getting a bit more interesting.
First, ignore the faint larger yellow/red ring – this is just an artefact of the imperfect system, with small amounts of light catching the edge of some of the lenses.
Now, look at the yellow – in-focus – target. You’re seeing it through a grid of small circles, but the overall shape is a pretty clear, in-focus square.
The blue and red targets, however, are out of focus. The microlenses catch the rays of light over the blurred image of the squares, and the resulting images tend towards circles and take up significantly more area on the sensor.
This is all pretty logical really – if you took a photo of the 3 squares with a wide aperture lens so only the yellow was in focus, the red and blue squares would blur in the resulting photo. If they were far enough away from the plane of focus, the images of the squares would blur enough to end up looking more like circles.
### Behind the Mask
What’s all this got to do with autofocus? Nothing yet, but we’re getting there…
I’ll make a small change to the grid of microlenses by covering one side of each lens with a dark mask, like this:
Half-masked microlenses
It’s not a big change, but the result is quite surprising.
Here’s the image when the left-hand side of each lens is masked:
The image through the left-side masked lenses
Compare this image with the one above with the unmasked lenses, and you’ll notice that the red and blue images appear to have shifted sideways.
What’s actually happened is that, rather than the image actually moving, we’ve just hidden part of it, but the result is the bulk of the light is now on one side of an imaginary centre line.
Let’s take a look at what happens if we mask the other side of each lens:
The image through the right-side masked lenses
The same thing has happened, but the bulk of the red and blue images appear to have shifted to the opposite side. Notice – and this is really important – **the yellow _in focus_ target light appears equally on both sides.**
> Masking the individual microlenses causes _out of focus_ elements to appear to shift sideways.
### Shifting Patterns
To explain this more clearly and show the consequence of the shift, I’ve cut out a single horizontal strip of the left-masked and right-masked lenses in a line through the image of each of the 3 targets.
Although I’ve shown them below as 2 strips, this is a view of **exactly the same part of the sensor** – the top is just through right-hand masked lenses, and the bottom is through left-hand masked lenses.
To start with, here’s a small strip of the in-focus yellow target:
Right (top) and left (bottom) masked images of the in-focus target
It’s in focus, and it’s clear that the image of the target through the lenses is in the same place with either left- or right-masked lenses.
Here’s the view of the blue target, the one that’s nearer to the camera:
Right (top) and left (bottom) masked images of the near (out of focus) target
Now we’re seeing a difference. The image of the blue target isn’t spread across the same lenses for both masks – it seems to be a bit to the right on one, and a bit to the left on the other.
It’s even clearer with the far target, the red one:
Right (top) and left (bottom) masked images of the far (out of focus) target
### Autofocus
We can clearly see an effect that’s related to how in-focus an object is at the sensor plane. But How do we use this in an autofocus system?
The image below shows the pairs of left- and right-masked lenses as above, but I’ve added a green _Reference_ line that runs through the middle of the centre lens for the bottom (left-masked) row in each case. The absolute position of the reference line is unimportant as long as it marks the same point on the sensor for one of each of the pairs (in this case, it passes through the middle of the 17th lens on the bottom strip).
An illustration of the shift amounts between left- and right-masked lenses for each target
If we shift the top row of lenses so that the bulk of the yellow, red and blue areas of the images approximately align, you get two bits of information – shown by the red and blue arrows – for each target:
1. How _far_ the focus position is away from ideal (the magnitude), and
2. which _direction_ the focus is offset.
So, if you capture _one_ image through a set of masked micro-lenses, you can tell not only how out of focus an area is, but also in which direction it lies relative to the plane of focus.
And there you have it – the sensing part of Phase Detect autofocus!
### Practical Implementation
What I have shown above is the principle of operation of on-sensor phase detect autofocus. The practical implementations are a little different.
First off, you can’t have both left- and right- masks over the same pixel, otherwise you have a fully covered pixel which is not much good for anything!
What generally happens if the sensor is manufactured with a number of special phase-detect pixels -close together pairs of left/right masked pixels spread over the sensor. These pixels aren’t used for imaging – purely for autofocus – but they have a negligible effect on the overall image quality as they are very small and the missing information mostly disappears within the demosaicing algorithm used to generate the final colour image.
A few pairs of pixels are masked and can be used to determine focus offset
I said above that you can’t have both left- and right-masked pixels which is true, but you can achieve more-or-less the same effect by chopping each pixel into two smaller, side-by-side photodiodes which behave as the left and right sides of the masked image. This is what Canon has done in their Dual-Pixel Sensors, allowing every single pixel to be used for both autofocus and imaging.
### DSLRs
On-sensor phase-detect arrays are what’s used in most modern mirrorless (and even newer DSLR when in Live View mode) cameras – the focusing is performed with the same sensor as captures the actual image.
In DSLRs when you’re using the viewfinder, phase detect autofocus is separated from the image sensor and relies on a dedicated autofocus module. This module receives light via a secondary mirror hiding underneath the main mirror, which directs a portion of the incoming light away from the main imaging sensor. This design allows you to look through the viewfinder and see out of the lens while the camera can simultaneously calculate focus, despite the mirror (and shutter) covering the actual image sensor until the shot is taken.
While a DSLR has a separate autofocus sensor, the concept within the sensor is very similar to that shown above – light rays are captured from differing directions and an out-of-focus element will render inside the sensor with a measurable offset between the two paths, giving both a magnitude and direction to the plane of focus.
## What’s Next
Hopefully, this post has shown how phase-detect autofocus operates for mirrorless cameras, measuring the focus at the same plane as the image is captured and thus removing one of the big reasons for calibrating your autofocus system.
Now head over to the post on On-Sensor Phase Detect Issues to see how the results you get from the sensor aren’t always perfect.
## Feedback
I’d love to hear feedback on this post. Is it easy enough to understand? Is the operation of phase-detect AF a bit clearer now? If there’s anything that needs a bit more explanation, please let me know in the comments below and I can clarify! https://blog.reikanfocal.com/2023/05/how-it-works-on-sensor-phase-detect-autofocus/
