Related web pages

Simatest ISP/Camera Simulator — Instructions and Reference

Using Color/Tone Interactive – Interactive analysis of color & grayscale test charts

Using Color/Tone Auto – Fixed (batch-capable) analysis of color & grayscale test charts

Color/Tone & eSFR ISO noise measurements – describes how to measure image sensor noise from raw images

Dynamic Range – a general introduction with links to Imatest modules that calculate it

Noise in Photographic Images – a basic introduction

Color Correction Matrix (CCM) – Calculate a matrix (usually 3×3) for correcting image colors (best from RAW images)

Image information metrics – new measurements, based on Shannon information theory, that are superior predictors of Machine Vision system performance. 

 
 Introduction – Simatest block diagram –  Modeling image sensor noise – Measuring the noise 
Simulating the noise – Modeling and measuring image sharpness – Simulating image sharpness 
Simulating camera JPEGs – Summary

This page illustrates how Simatest can be used for modeling and measuring camera systems and, most importantly, Image Signal Processing (ISP), using a compact digital camera as an example. It’s designed to be more practical and readable than the full instructions and reference in ISP/Camera Simulator (Simatest).

Starting in February 2025, Simatest will be available through the Imatest Pilot program. 

Introduction

Simatest can model and measure the effects of Image Signal Processing (ISP) starting with

  1. Raw camera image — the raw output of a digital camera (demosaiced or not), which includes lens blur and image sensor noise. Test chart images are typically used for this case, but any image can be used, or
  2. Simulated (test chart) image, which needs to be blurred to match the MTF (SFR) of a raw camera image, typically using a simple gaussian or Airy disk filter. Must also include image sensor noise from the model described below. 
  3. Simulated images degraded by a lens design program. This enables the “soft prototyping” of a camera before the hardware is specified and built.  Image sensor noise (described below) is required. A complete ISP chain can be applied to images from any source.

The camera used for the examples on this page is the Panasonic Lumix LX5, an older (2010) 10.1 megapixel compact digital camera with 2.14 μm pixel pitch and a high quality Leica-branded zoom lens. Its image processing is simple by 2025 standards — it has no Computational Imaging or Artificial Intelligence, and most importantly, it has raw output, which can be processed with or without demosaicing. We will show how to

  • Model the image sensor noise using a raw image of a grayscale test chart.
  • Model the approximate camera spatial domain response and MTF (SFR) using a simple gaussian or Airy disk filter.
  • Use the camera  model to show how camera performance varies as a function of Exposure Index (EI; widely known as ISO speed), where increasing the EI is equivalent to decreasing the illumination.

Modeling image sensor noise

A key step — perhaps the most important step — in simulating digital camera performance is modeling the image sensor noise, which consists of dark noise from several sources and photon (or electron) shot noise. The Simatest model is derived from measurements of raw images of grayscale test charts, preferably from transmissive test charts, which have much larger tonal ranges than reflective charts. 

Once an image sensor’s noise has been properly characterized, Simatest can be used to predict camera performance for a wide variety of camera configurations — lenses, image processing, etc.

Image sensor noise is also described in Simatest instructions and reference and in Color/Tone/eSFR ISO Noise measurements.

For linear (non-HDR) image sensors, noise can be derived from the Digital number (pixel level DN ) of the signal + noise at the image sensor.

DN = DNmeas – DNoff  where DNoff  is an offset frequently included in undemosaiced image files.

Estimating DNoff , which is frequently present in undemosaiced raw files, requires some care. It can be estimated from EXIF metadata or from measurements the signal in the darkest regions of the test chart image. 

For dark noise power, Nd2 

Expected Noise Voltage = VN = (Nd2 + kN DN)1/2 = (Nd2 + kN (DNmeas – DNoff ))1/2   — or —    

Expected Noise Power    = PN = VN2 = Nd2 + kN (DNmeas – DNoff )    for linear sensors

This model is a variant of the noise model in the EMVA 1288 standard for linear sensors, section 2.4.

Measuring the image sensor Noise

Nd2 and kN are the key parameters for the image sensor noise model in Simatest. Measurements must be made carefully from raw (undemosaiced and unprocessed) images.

The steps for modeling the noise

  1. Acquire an image of a grayscale test chart, preferably a transmissive chart designed for measuring dynamic range.
  2. Save the image in a raw format.
  3. Read the raw image into Color/Tone Interactive without demosaicing. For commercial raw files, the Bayer RAW 16-bit linear preset is recommended. (Color Tone Auto also works, but is less flexible.) The converted image is saved as a TIFF. Details are given in Color/Tone/eSFR ISO Noise.
  4. Since the undemosaiced image is monochrome, the Standard monochrome or Bayer RAW? box will appear. Select the appropriate Pixel color mapping (Red in R1C2 (GR/BG) for the Lumix LX5). (If the image contains recognizable color, you can open it in Rawview to find the color mapping.) Then select (Analyze image as) Bayer RAW.
  5. Estimate the offset, DNoff , and saturation level, using methods detailed in Color/Tone/eSFR ISO Noise, and enter it into the Color/Tone settings window (part of which is shown on the right). The values can often be found in the EXIF metadata (in the raw image, but not the JPEG), but the names are different for each manufacturer. DNoff , is always less than the lowest patch level, but can be larger than the lowest individual pixel level. If too large a value is entered, the response will be cut off (DN will be 0) for the darker patches.  Plots 10-12, which show noise and SNR as a function of pixel level, are of greatest interest. These plots are only meaningful for raw images.
    Fortunately, we have found that the model is relatively insensitive to the offset estimate. For the Lumix LX5, the offset is 128, and the saturation level is 4095 (both numbers from the EXIF data, shown here).
  6. Select Display 10. Noise analysis or any of the plots (10-12) that display the noise or SNR as a function of input pixel level. This runs a Levenberg-Marquardt optimizer that finds the best values of Nd2 and kN that best match the data. Plot 10 (noise vs. pixel (dB)) is shown below. We didn’t use SNR plots 11 or 12 because they are sensitive to offset + stray light, which can be difficult to quantify. 

Measured and fit noise vs. input pixel level the LX5 (2.14 μm pixels)

The key results to enter into Simatest are shown on the lower-left:  Noise = 0.01143 and Dark noise = 0.00067. The well capacity of 7345 electrons (e-) is interesting to compare with the well capacity of 33882 pixels for the the EOS-6D, discussed in the Simatest instructions page, which has much larger 6.5 μm pixels.

Another result of importance is RGB = 0.549 1.000 0.537, which represents the color of the middle gray patches, normalized to 1. Because of the incandescent light source and sensor sensitivity, the image (thumbnail shown on the left with no color correction) has a greenish tint.

More details of the noise calculations are in Color/Tone/eSFR ISO Noise. Dark noise is actually a combination of several signal-independent noise sources, described in OnSemi Application Note AND9189/D. Electronic (Johnson) noise has a strong temperature dependence. It is proportional to (4kT)1/2. Dari current noise increases with exposure time.

Plot 2 (S/N vs. input density) for the LX5, shown on the right, closely resembles the Photon Transfer Curve in the OnSemi App Note, where S/N ≅ Noise (electrons (e-)) and Well capacity = {S/N)2max = 7345. The spike in the S/N plot is apparently caused by noise, most likely a dirt speck on the chart. The flat areas on the left side — the dark signal and dark SNR — are affected by stray light and possible uncertainties in the offset setting.

We expect to add more detail to the noise model in future releases. We will also move the noise calculations so results are available in the CSV and JSON output. And (a sort of a holy grail of sensor modeling), we expect to simulate multi-region HDR sensors. 

S/N vs. input density: Photon Transfer Curve

 

Simulating the image sensor noise

The measured parameters for photon shot noise noise, dark noise, and RGB balance can be entered into a Simatest model of the image.

  • Open Simatest and read in an ideal test chart image file created by the Imatest Test Charts program. Our examples use two charts.
  • Open the Simatest settings window, and enter the following settings to replicate the (undemosaiced) raw image we just measured. 
         Gamma Targ 1, Filter gaus 0.75, RGB mult 0.55 1 0.53, Bayer grbg, Noise 0.0114 1 1 0.00067.
    These settings are described in the large table in Simatest — instructions and reference. Click OK when the settings are complete.
  • We recommend checking Auto update in the lower-right of the Settings window, next to OK. This causes the calculations to be updated when OK is pressed to close the Settings window. If Auto update hasn’t been checked, press Update calculations in the main window. The resulting image will be a Bayer raw image.
  • To get an image with the same offset and maximum value as the camera (128 and 4095 for the LX5), press Save processed image to save the image in a named file.
  • Open the image in Color/Tone Interactive, with the 36-patch Dynamic Range chart selected. Because the undemosaiced image file is monochrome, you will get the window shown below. Key settings are circled.

Monochrome or Bayer raw? window opens for monochrome files

  • Select the ROIs or confirm the ROI selection, as required. The results will appear.

Simulated noise vs. input pixel level the LX5 (2.14 μm pixels)

The simulated noise is nearly identical to the measured noise (above), but the plots look slightly different because the minimum x-axis value is different: 4×10-3 for the measured value vs. 7×10-4 for the simulation, which goes lower because it has no veiling glare (stray light; flare light). Veiling glare could have been added with the Fog ISP processing block. 

Modeling and measuring image sharpness

The next step is to measure and model the image sharpness (SFR or MTF). Since we don’t have the degraded image from a lens design program, we need to estimate the lens blur by trial-and-error using a gaussian or Airy disk filter. Once we have a good model of the raw image, we ae free to try out any adjustments or enhancements. 

Since a raw image is required only for finding the noise model, we demosaic (convert) the raw image using LibRaw with the Color 24-bit sRGB preset. The resulting curves are typical for unsharpened images.

Measured Edge/MTF plot for the Lumix LX5, demosaiced to 24-bit sRGB. Key parameters are 10-90% rise and MTF50.

Simulating image sharpness

Now we run Simatest, using a demosaiced, minimally processed input image of the 36-patch chart with slanted edges generated by the Imatest Test Charts module.

Simatest window for matching LX5 simulation with measurement: key setting: Filter gaus 0.75

The following sequence were repeated until we got a good match between the measured Edge/MTF results (above) and the simulated Edge/MTF results.

  •                   Move the Filter setting to ISP Blocks, line 3.


    Click Simatest Settings (upper left of Simatest window).
  • Filter settings:  We tried both Gaussian and Airy disk filters. Gaussian was slightly better, but not perfect. We adjusted the slider several times until we settled on a final value of sigma = 0.75 (pixels).

Recall that Simatest settings require two steps:  (1) set the slider (or other adjustment) to the desired value, then (2) press the Set dropdown menu to copy the setting to the selected the line in ISP Blocks, as shown by the red arrow on the right.

  • Click OK to return to the main Simatest window.
  • Click Update Calculations.
  • (optional) You can display the side-by-side view, with the input image on the left and the processed image on the right, shown above. Although this view is occasionally interesting, we typically skip over it.
  • Click Rescharts near the bottom of the Simatest window (on the left of the light green area).
  • Make sure 1. SFR is chosen for chart, then click on Analyze image.
  • Select the region (for the first try) and make a fine adjustment if necessary, or confirm the saved selection (for succeeding tries) by pressing Yes, Express mode.
  • If the Edge/MTF plot is selected, the simulated Edge and MTF are displayed. Many other plots are available in both the Rescharts and Color/Tone modules.

Simulated Edge/MTF plot for the LX5. Results are very close to the measured plot, above.
Key results — 10-90% rise distance, MTF50, Information capacities C4 and Cmax — are all within a few percent.

The match is not perfect. The shape of the MTF is a little off, but it’s good enough for testing ISP. 

Simulating the camera JPEG image

Now that we have simulated image sensor noise and the sharpness of a demosaiced, minimally processed image (from the LX5) we can move on to something more challenging: simulating the JPEG image. 

Camera JPEGs typically have several additional image processing steps beyond the demosaiced raw image. 

  • Tonal response curve — can be simple gamma encoding (close to gamma specified for the color space, but sometimes slightly larger to give the image a more snappy appearance).
  • Color correction (including White Balance) — Often done by a 3×3 Color Correction Matrix (CCM), but we use simple “gray world” white balance, which is much too simple to handle real-world challenges, in this example. 
  • Sharpening — standard sharpening (instead of USM) is used in most cameras because it’s faster. 
  • Noise reduction — typically with a bilateral filter, which keeps edges sharp, but blurs (smooths) the image elsewhere. Bilateral filters are nonuniform and nonlinear. They are nearly universal in consumer camera JPEG images.
  • Note that the high quality JPEG image compression in most consumer cameras has little effect on image quality.  

Here is the measured JPEG for the LX5, acquired at ISO 100 (Exposure Image). 

Measured LX5 JPEG Edge/MTF response

The second plot is the Edge noise, used to calculate image Information metrics. Edge noise is the lower plot of the Edge & Info Capacity noise display.  Measured LX5 JPEG Edge noise

To simulate this noise, we sharpened the image and applied the MATLAB bilateral filter, which required considerable trial-and-error because it’s not well-documented. Here are the settings.

 

Simatest settings for JPEG image

Here is a quick summary of the iterative procedure. 

Simulated LX5 JPEG Edge/MTF response
Results are similar to the measured image, except near fNyq, where MTF is higher.

Some considerations

  • The same input image (from Imatest Test Charts) as the previous simulations was used. The central slanted edge was used to obtain the MTF results.
  • Standard sharpening with  Radius = 2 and Amount = 1.4 was chosen for a good match to the measured JPEG image. There was some interaction between the sharpening and Bilateral filter settings. To maintain the asymmetry between the larger overshoot and smaller undershoot, sharpening was applied after the encoding gamma (≅0.45) was restored.
  • The MATLAB imbilatfilt filter was used. Since Its documentation is not perfectly clear, we had to do a lot of trial-and error to arrive at the settings. We selected Degree of Sharpening = 0.0025, Spatial sigma = 0.7, and Neighborhood area = 5 (must be an odd integer). The noise away from the peak appeared to decrease as Degree of Sharpening was increased. 
These results show how Simatest ISP simulation agrees well (though not perfectly) with measured results. Since we don’t know the properties of the bilateral filter in the LX5, we correctly guessed that it could be approximated by MATLAB imbilatfilt. 

Summary

Using the humble Panasonic Lumix LX5 as an example, we have shown how to measure, model and simulate

  • image sensor noise, which can be measured from a raw (undemosaiced) image,
  • a minimally processed RGB image (converted from raw), and
  • a JPEG image (which includes sharpening and bilateral filtering).

The techniques presented here can be used with

  • minimally-processed images, converted from raw. These images don’t require the sensor noise or lens blur to be simulated since they’re implicitly included in the image.
  • simulated image, especially ideal test chart images. Sensor noise and blur models are required.
  • simulated images blurred by a lens design program. The sensor noise model is required.

These cases are illustrated in the block diagram near the beginning of this page.

A great many image processing blocks can be simulated. We have focused on blocks that affect MTF, SFR, noise, information metrics, and color accuracy. The early 2025 release is just a beginning. We expect to add more image processing blocks that customers find useful.