[Outreachy Status Log] Scaler implementation

The implementation of the scaler node (which increases the size of the received image by a multiplier) is done and available in my github account: https://github.com/helen-fornazier/opw-staging

The debayer filter (from the previous Outreachy Status Log post) is also available there.

The scaler still doesn't work if the received image is in bayer format, I'll let this to a future patch.

The next patch will allow changes in the pad format by user space as the frame size and enumerate the supported pixel formats by the ioctls. I already have a sketch of this code but I need to test it a little more.

[Outreachy Status Log] Debayer implementation

What is a bayer format

The sensor of a camera is composed my many color sensors, each of this color sensors just capture a single color. Thus a sensor may be composed with many color sensors in the order Red and Green in the odd lines and Green and Blue in the even lines for example:

|Red Sensor.....| Green Sensor.|Red Sensor.....| Green Sensor.|

|Green Sensor..| Blue Sensor...|Green Sensor..| Blue Sensor...|

|Red Sensor.....| Green Sensor.|Red Sensor.....| Green Sensor.|

|Green Sensor..| Blue Sensor...|Green Sensor..| Blue Sensor...|

We say it's a RGGB bayer format. We could have any combinations of these color sensors as GRBG (for Green and Read in the odd lines and Blue and Green in the even lines), BGGR, GBRG, GRBG, etc.

Usually our eye are more sensible to green, that is why we are te double of green sensors compared to the other color components.

Here is a Bayer format image, 64x64 RGGB:

If you look closely you can notice that in the odd lines we have only Red and Green and in the even lines we have Green and Blue.

Thus, the camera process the results of each color sensor to build the image we are used to. And we will simulate this filter in out VIMC (Virtual media controler, yes, it was rename from VMC to VIMC).

Simple debayer algorithm

To get each color component of each pixel we will calculate the mean of the color components around a given color sensor, i.e. a Blue color sensor is inside a 3x3 square (or mean window):

R G R
G B G
R G R

The Red component will be the mean of all four Red components in this square, the Green component will be the man of all four Green component in this square and the Blue is the value read from the Blue sensor being evaluated.

The mean window can be any SxS size, where S is an odd value to allow the pixel being evaluated to be in the center of the square.

Results

Debayer, windows size 3x3, image size 64x64

Debayer, windows size 5x5, image size 64x64

Debayer, windows size 11x11, image size 64x64

Code structure - the pix map table

In the implementation I have a table which describes the order:

With this table I am able to write a generic code to all the bayer orders.

I am still cleaning up this code, it will be available in my github soon

[Outreachy Status Log] First VMC version to the linux-media mailing list

I just sent the first version of the Virtual Media Controller driver to the linux-media mailing list, now I am waiting more reviews from other developers.

You can check the code in my github account https://github.com/helen-fornazier/opw-staging in branch vmc/review/video-pipe.

Now I'll setup the Test Patter Generator (tpg) from the Vivid driver to be a separated module, as this code will be reused in the Vmc as well to generate the colored bar image instead of the grey one.

[Outreachy Status Log] Generating a colored bar image

It has being a month since the last status log. Here is status of my progress:

- The architecture of the code has changed a lot since the last log. Several patches reviews were made between me and my mentor.

- The code now supports link validation, e.g, you can't turn on the streaming if the formats between subdevs are not compatible.

- It is more modular and easy to adapt to a dynamic topology configuration.

- Several bugs were corrected.

- The test pattern generator was imported from the vivid driver and now the code generates a non-grey image:

The next patch series will contain this test patter generator and the user will be able to configure the format of the image.

The next step is to work in a debayer block.

[Outreachy Status Log] Third week - Basic video pipe with a capture and sensor node

This is a resume in what I've been working on.
There is no much content in this post if you are looking for useful information to your project. I am posting more useful information under the Nerd category.
Check my first Outreachy post to know how they are organized.


This week I worked in a first pipeline skeleton as first proposed here.

This is the output of the topology from my code:

The sensors generate a simple grey image for now from a kthread, and it is being propagated to the Raw capture nodes. The other nodes (debayer, input and scaler) don't have any inteligent yet.
It is possible to view the generated (grey) image from tools such as qv4l2

The sensors and capture device codes still need to improve.

This week I sent this patch series to my mentor. After my mentor's feedback and the appropriate patch corrections, the next step is implementing a basic debayer, scaler and input intelligence.

You can check the development of this in my github Linux Kernel Fork branch vmc/devel/video-pipe

Compiling a single Linux Kernel module inside the Kernel tree

I recently find out a faster way to compile a single module in the Linux Kernel tree thanks to this post:

What I was doing before was taking much more time:

Or a pure make -j5, even if I had already compiled my Kernel once and just modified one module, was taking me ~5 minutes. So each tiny test I made (like changing a the contents of a string) was taking me that long, now it takes some seconds using the make modules SUBDIRS=path/to/module command \o/

To update the module I just copy the .ko to the right place:

It usually works if you made a simple change in the code. Or you can use the SUBDIRS too:

[Outreachy Status Log] First two weeks

This is a resume in what I've been working these two weeks.
There is no much content in this post if you are looking for useful information to your project. I am posting more useful informations under the Nerd category, check my first Outreachy post to know how they are organized.

I read the docs:

*Understanding userspace V4l2 API: http://linuxtv.org/downloads/v4l-dvb-apis/

*Understanding the internal Linux Kernel API to write a v4l2 driver:
In the Linux Kernel Tree:

Core framework: Documentation/video4linux/v4l2-framework.txt

How to implement the control v4l2 API: Documentation/video4linux/v4l2-controls.txt

How to implement Input Output: Documentation/video4linux/videobuf

The new interface to IO: Videobuf2: https://lwn.net/Articles/447435/

*Kthreads: https://lwn.net/Articles/65178/

I played with the code from

http://linuxtv.org/downloads/v4l-dvb-apis/v4l2grab-example.html

I checked the codes from the media_ctrl and other codes from the v4l_utils

Now I understand much better how the API from kernel to userspace works.

I setup CCache, Icecc and a VM to compile faster and debug faster. I decided to setup a VM because my computer GUI was freezing in a Kernel Ops and I needed to reboot my pc for each test. Those setups took me some time, but now I am saving a bunch of time.

I started reading the Vivid's code and to write a capture node based on the Documentation/video4linux/v4l2-pci-skeleton.c.

The code I did was not usable. I check the mandatory functions with the v4l2 API and now I am able to query the capabilities and enum the pixel formats.
I am testing with v4l2-ctl from the v4l-utils project and the yavta tool.

The code must be modular, so I separate the capture entity from the core in another source file. Latter we should be able to instantiate as many capture device nodes in the topology as we want.

This image will be generated from an internal kthread, then I wrote the kthread skeleton.

Now I am integrating the videobuf2 framework to populate the buffers with a hardcoded image inside the kthread.

After that I'll add a subdev to simulate the sensor.

You can check my progress in my github branch here, its not a clean branch with clean patch, I am going to clean them latter.