How a Smartphone Camera Sensor Turns Light Into Colour

A smartphone camera sensor is a chip covered in millions of tiny light detectors. Each one measures only how much light reaches it, not its colour, so the sensor is covered with a pattern of colour filters and software works out the full-colour picture afterwards. This article explains each step.

Pixels that count light

Wikipedia explains that a CMOS image sensor converts light into electrical signals, one pixel at a time, using small amplifiers built into the chip. The light-sensing part of each pixel is a photodiode, which builds up electric charge in proportion to the light it receives. The more light, the larger the signal.

Because each pixel has its own amplifier, some of the surface is not sensitive to light. To make up for it, a tiny lens, called a microlens, sits over each photodiode and bends light that would otherwise miss it. Some CMOS sensors also use back-side illumination, which puts the wiring behind the light-sensitive layer so more light can be captured.

Adding colour with a filter pattern

A photodiode alone gives only brightness. To record colour, a colour filter array sits over the sensor so that each pixel sees only red, green or blue light. The most common layout is the Bayer pattern, a checkerboard that Wikipedia says has two green pixels for every red and every blue one. The darktable manual, which documents a photo-editing program, agrees that the filters are usually arranged in a Bayer mosaic, though other layouts exist.

Sources generally explain the extra green as a match to human vision, which is more sensitive to green light. That detail is common in explanations of the pattern but was not stated in the two pages used here, so treat it as background rather than a checked fact.

Demosaicing: estimating the missing colours

After the sensor reads out, every pixel holds data for one colour only. The darktable manual explains that a step called demosaicing estimates the two missing colours for each pixel by interpolating, which means working them out from the neighbouring pixels. Different programs use different methods, and the manual notes that some preserve fine detail better while others cope better with noisy, high-ISO images.

Estimating can go wrong. The darktable manual lists artefacts such as moiré, which are false patterns, maze-like structures and exaggerated colours at edges. A phone does this step automatically in its image-processing hardware and software, and also applies noise reduction, colour balancing and other adjustments; the details differ by brand and are mostly not published.

Combining pixels in the dark

Wikipedia notes that some sensors with very high pixel counts use pixel binning: groups of 4 or 9 neighbouring pixels of the same colour are combined into one. This improves sensitivity and dynamic range in dim scenes, at the cost of a lower output resolution. Wikipedia reports that in June 2022 Samsung announced a 200-megapixel sensor with 0.56-micrometre pixels, a size that shows why combining pixels is attractive.

Rolling shutter

Most phone sensors read out rows one after another rather than all at once. Wikipedia describes this electronic rolling shutter: each row is exposed just before it is read, so the exposure rolls down the frame, usually top to bottom. Fast-moving subjects can look slanted as a result. Global shutters, which expose every pixel at the same moment, need extra storage circuits to hold the charge until readout, so they are less common.

Why CMOS won

Earlier digital cameras often used CCD sensors. According to Wikipedia, CMOS sensors can use fewer components, less power and faster readout, and are less vulnerable to static discharge. Sales of CMOS sensors passed CCD sales in 2007, and by the 2010s they had largely displaced CCDs in new designs. The same article dates the first commercial digital camera, the Cromemco Cyclops, to 1975, with a 32 by 32 pixel MOS sensor.

Sources

• Image sensor, Wikipedia
• Demosaic, darktable 5.x user manual