Config file format

Libmegapixels gets its camera pipeline definitions from config files stored in one of the config directories. For finding config files libmegapixels will scan a few preconfigured path from the “compatible” names of the device.

The first part of this puzzle is the compatible names, these can be found at runtime from the /proc/device-tree/compatible file or otherwise from the top-most “compatible” node in the device tree.

As an example the Xiaomi Mi Note 2 will be used. The compatible list for the device tree for this device is at https://github.com/torvalds/linux/blob/b85ea95d086471afb4ad062012a4d73cd328fa86/arch/arm64/boot/dts/qcom/msm8996pro-xiaomi-scorpio.dts#L17

The relevant line here is this:

compatible = "xiaomi,scorpio", "qcom,msm8996pro", "qcom,msm8996";

The second part is the lookup paths. They are checked in this order:

• config/$model.conf in the current working directory

• /etc/megapixels/config/$model.conf

• /usr/share/megapixels/config/$model.conf

So on this device the file will be looked up in this order:

• config/xiaomi,scorpio.conf

• /etc/megapixels/config/xiaomi,scorpio.conf

• /usr/share/megapixels/config/xiaomi,scorpio.conf

• config/qcom,msm8996pro.conf

• /etc/megapixels/config/qcom,msm8996pro.conf

• /usr/share/megapixels/config/qcom,msm8996pro.conf

• config/qcom,msm8996.conf

• /etc/megapixels/config/qcom,msm8996.conf

• /usr/share/megapixels/config/qcom,msm8996.conf

The first matched file will be used.

The file format

The config files are parsed by libconfig and define all the cameras present on the specific device and the modes that are possible on this hardware.

An example of a minimal config is this:

Version = 1;
Make: "Xiaomi";
Model: "Scorpio";

Rear: {
    SensorDriver: "imx318";
    BridgeDriver: "qcom-camss";

    Modes: (
    {
      Width: 3840;
      Height: 2160;
      Rate: 30;
      Format: "RGGB10";
      Rotate: 90;

      Pipeline: (
          {Type: "Link", From: "imx318", FromPad: 0, To: "msm_csiphy0", ToPad: 0},
          {Type: "Link", From: "msm_csiphy0", FromPad: 1, To: "msm_csid0", ToPad: 0},
          {Type: "Link", From: "msm_csid0", FromPad: 1, To: "msm_ispif0", ToPad: 0},
          {Type: "Link", From: "msm_ispif0", FromPad: 1, To: "msm_vfe0_rdi0", ToPad: 0},
          {Type: "Mode", Entity: "imx318"},
          {Type: "Mode", Entity: "msm_csiphy0"},
          {Type: "Mode", Entity: "msm_csid0"},
          {Type: "Mode", Entity: "msm_ispif0"},
      );
      },
    );
};

The only top-level keys are Version, Make and Model. The Version should always be 1 and is reserved for possible future breaking updates.

The Make and Model keys set the human-readable name for the device which will be written in the EXIF data of pictures as he camera model.

Camera definitions

All the other top-level keys in the config file are definitions for cameras. The name is not very important and is for documentation purposes.

The keys in the camera block are:

• BridgeDriver: The name of the driver that provides the /dev/media and /dev/video node for this camera

• SensorDriver: The name of the sensor entity in the media graph

• FlashPath: optional, defines the path to a LED device to use as the flash for this camera.

• FlashDisplay: optional, defines that there is no LED that can be used as flash, use the display as the light-source instead.

The camera section requires a SensorDriver and BridgeDriver key. The BridgeDriver will identify the bridge device (eg. /dev/video0) and media device (eg. /dev/media0). This combination should uniquely identify a single camera on the device.

These values can be found using the media-ctl utility.

$ media-ctl --device 0 --print-topology
Media controller API version 6.1.14

Media device information
------------------------
driver          qcom-camss
model           Qualcomm Camera Subsystem
serial
bus info        platform:a34000.camss
hw revision     0x0
driver version  6.1.14

Device topology
- entity 1: msm_csiphy0 (2 pads, 5 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
     pad0: Sink
             [fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb]
             <- "imx318 3-001a":0 [ENABLED,IMMUTABLE]
     pad1: Source
             [fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb]
             -> "msm_csid0":0 []
             -> "msm_csid1":0 []
             -> "msm_csid2":0 []
             -> "msm_csid3":0 []

[ Removed A LOT of entities here for brevity ]

- entity 226: imx318 3-001a (1 pad, 1 link)
              type V4L2 subdev subtype Sensor flags 0
              device node name /dev/v4l-subdev19
     pad0: Source
             [fmt:SRGGB10_1X10/5488x4112@1/30 field:none colorspace:raw xfer:none]
             -> "msm_csiphy0":0 [ENABLED,IMMUTABLE]

- entity 228: ak7375 3-000c (0 pad, 0 link)
              type V4L2 subdev subtype Lens flags 0
              device node name /dev/v4l-subdev20

The BridgeDriver config node needs to match the driver listed at the top of the output. In this case qcom-camss.

For the SensorDriver the entity needs to be found that represents the sensor. This is mentioned as subtype Sensor in the list and in the example is entity 226. The entity will be matched by the name listed directly after the number which is imx318 3-001a. The full name doesn’t need to be used but is matched by prefix to avoid the config needing to hardcode the i2c bus number for the sensor which might change on enabling/disabling kernel modules.

Camera modes

Every camera block requires to have at least one mode block but can have multiple. This is required since different usecases of the camera need different modes. Usually the maximum resolution of the sensor will be used for taking pictures but at that resolution the framerate will generally be too low for realtime preview.

For example the OV5640 sensor can run at 2592x1944 resolution for decently sharp pictures, but at that resolution is limited to 15fps which will be quite annoying for the live-preview and not fast enough for smooth video recording. This is why the PinePhone defines a 2592x1944@15fps mode and a 1280x720@30fps mode for the preview. The Megapixels application will default to the highest defined resolution for taking pictures and will take the resolution closest to the display resolution as the preview mode.

The required configuration keys for the mode are:

• Width and Height for the resolution of image _after_ any processing by the ISP.

• Rate the frame interval in frames-per-second for this mode.

• Format is the pixelformat for the data after ISP processing.

The optional keys are:

• Transfer sets the transfer curve of the image data, can be “srgb” or “raw”.

• Rotate defines the rotation for matching up the orientation of the sensor and the display of the device.

• Mirror can be set to true to flip the image for the front-facing sensors.

• FocalLength sets the effective focal length of the camera in this mode. Sensors generally crop the image slightly depending on the resolution which changes the focal length for the mode. This is only used for EXIF metadata and is defined in milimeters.

• FNumber sets the aperture size of the lens. 3.0 will mean F/3 as aperture size, this is only used for EXIF

Mode pipeline

To make the mode actually work the pipeline has to be added. This is a small script that defines the ioctls that need to be sent to the drivers to set up all the hardware in this mode. This is a full scripting system to deal with inconsistencies in the Linux drivers, especially the staging drivers for sensors.

The pipeline script is a list of command objects. The specific command is set using the Type key in this object. All other keys in the command object depend on the specific command.

The possible commands are:

• Link

• Mode

• Rate

• Crop

Cascading values

In order to not repeat the values for resolutions, formats and framerates in every command values are cascaded internally if nothing is specified. These cascading values are initialized to the resolution and mode for the Mode block that contains this pipeline. If any command in the pipeline hardcodes another value then that new default will also be used for the following commands. This means that if the pipeline script is defined in strict source->sink order the minimum amount of repetitions of these values are required.

For example this pipeline:

Modes: (
    {
        Width: 4208;
        Height: 3120;
        Rate: 30;
        Format: "RGGB8";

        Pipeline: (
            {Type: "Mode", Entity: "imx258", Format: "RGGB10P"},
            {Type: "Mode", Entity: "rkisp1_csi"},
            {Type: "Mode", Entity: "rkisp1_isp"},
            {Type: "Mode", Entity: "rkisp1_isp", Pad: 2, Format: "RGGB8"},
            {Type: "Crop", Entity: "rkisp1_isp"},
            {Type: "Crop", Entity: "rkisp1_isp", Pad: 2},
            {Type: "Mode", Entity: "rkisp1_resizer_mainpath"},
            {Type: "Mode", Entity: "rkisp1_resizer_mainpath", Pad: 1}
        );
    },

The resolution and framerate is never defined in any of the pipeline commands and are inherited from the values set in the mode. The pixelformat is specified twice here since the ISP will convert between RGGB10P and RGGB8 formats. Once to set the format of the sensor and once on the ISP to return it to the value of the Mode block.

This pipeline will be interpreted as:

{Type: "Mode", Entity: "imx258", Width: 4208, Height: 3120, Format: "RGGB10P"},
{Type: "Mode", Entity: "rkisp1_csi", Width: 4208, Height: 3120, Format: "RGGB10P"},
{Type: "Mode", Entity: "rkisp1_isp", Width: 4208, Height: 3120, Format: "RGGB10P"},
{Type: "Mode", Entity: "rkisp1_isp", Pad: 2, Width: 4208, Height: 3120, Format: "RGGB8"},
{Type: "Crop", Entity: "rkisp1_isp", Width: 4208, Height: 3120, Top: 0, Left: 0},
{Type: "Crop", Entity: "rkisp1_isp", Pad: 2, , Width: 4208, Height: 3120, Top: 0, Left: 0},
{Type: "Mode", Entity: "rkisp1_resizer_mainpath", Width: 4208, Height: 3120, Format: "RGGB8"},
{Type: "Mode", Entity: "rkisp1_resizer_mainpath", Pad: 1, Width: 4208, Height: 3120, Format: "RGGB8"}

Link

The Link command is the most important one, this defines the media pipeline itself by specifying which links should be created and broken. This is equivalent to the MEDIA_IOC_SETUP_LINK ioctl.

Parameters:

• From source entity name

• To sink entity name

• FromPad pad index on the source entity

• ToPad pad index on the sink entity

Example:

{Type: "Link", From: "ov5640", FromPad: 0, To: "sun6i-csi-bridge", ToPad: 0},

Is equivalent to:

$ media-ctl -l "'ov5640':0 -> 'sun6i-csi-bridge':0 [1]"

Mode

The mode sets the V4L2 format on the entity pad, some drivers in V4L2 will cascade these values down the pipeline and some will need the mode to be set on every entity in the pipeline to make the pipeline start streaming correctly.

The most minimal option is specifying just the entity name here and the rest of the values are used from the resolution and format specified in the mode block this pipeline is in, or the values of the previous Mode command if one hardcodes a value. This is for pipelines where an ISP changes the format or resolution halfway through the pipeline.

This command is equivalent to the VIDIOC_SUBDEV_S_FMT ioctl.

Parameters:

• Entity media entity to set the mode on

• Pad pad index on the entity, defaults to 0

• Width Resolution width, if not specified it will use value of the previous command

• Height Resolution width, if not specified it will use value of the previous command

• Format Pixelformat for the mode, if not specified it will use the value of the previous command

Example:

{Type: "Mode", Entity: "ov5640", Pad: 0, Width: 640, Height: 480, Format: "BGGR8"},
{Type: "Mode", Entity: "sun6i-csi-bridge"},

Is equivalent to:

$ media-ctl --set-v4l2 '"ov5640":0 [fmt:SBBGR8_1X8/640x480]'
$ media-ctl --set-v4l2 '"sun6i-csi-bridge":0 [fmt:SBBGR8_1X8/640x480]'

Rate

The mode sets the framerate on the entity for the drivers that believe in such things, some drivers will require this to be set and some will reject this ioctl altogether.

Just like the Mode command the rate value will cascade from the rate specified in the Mode block that contains this pipeline and previous Rate calls that changes the pipeline framerate.

This command is equivalent to the VIDIOC_SUBDEV_S_FRAME_INTERVAL ioctl.

Parameters:

• Entity media entity to set the mode on

• Rate frame interval, if not specified it will use value of the previous command

Example:

{Type: "Rate", Entity: "ov5640", Rate: 30},

Is equivalent to:

// Only the @30 part here is relevant
$ media-ctl --set-v4l2 '"ov5640":0 [fmt:SBBGR8_1X8/640x480@30]'

Crop

Some drivers will allow cropping out a section of the frame, usually this is to remove dummy pixels from the edge of the sensor. Some drivers will internally set the cropping automatically when setting the capture resolution and some will fail to start until you specify the cropping to be 0 on all edges instead of having that as default. There are also drivers that don’t update the crop values when changing resolution so the crop ioctl needs to be called.

This command is equivalent to the VIDIOC_SUBDEV_S_CROP ioctl.

Parameters:

• Entity media entity to set the crop on

• Pad pad index on the entity, defaults to 0

• Width Crop width, if not specified it will use the mode width

• Height Crop height, if not specified it will use the mode height

• Top Top offset of the cropped region, defaults to 0

• Left Left offset of the cropped region, defaults to 0

Example:

{Type: "Crop", Entity: "rkisp1_isp", Width: 640, Height: 480, Top: 0, Left: 0},

Is equivalent to:

// Only the last crop: part here is relevant
$ media-ctl --set-v4l2 '"rkisp1_isp":0 [fmt:SBBGR8_1X8/640x480 crop:(0,0)/640x480]'