Como começar a contribuir com Open Source

Collabora is participating in the Linux Developer Conference Brazil and this post is written in Portuguese to serve as a guide for the attendees to learn more about how to start contributing to Open Source Software.

Muita gente pensa que para começar a contribuir com um projeto de FOSS (Free and Open Source Software) tem que saber codar, isso é um mito, as pessoas precisam conhecer como o projeto é estruturado como uma comunidade, e muitas vezes para contribuir nem é necessário saber escrever código, pois reportar bugs, realizar testes, contribuir com o design da interface, revisar e arquivar bug reports obsoletos, traduzir o software ou ajudar a organizar times ou conferência são contribuições muito bem-vindas.

Compilando o software a partir do código-fonte

Geralmente o projeto possui alguma página web com diversas informações, inclusive com instruções de como fazer o download do código-fonte. A maioria dos projetos usam algum sistema de controle de versão, atualmente o Git é mais popular, mas pode ser Svn, Cvs, Mercurial e outros. Entre no site do projeto desejado, verifique qual sistema é usado e familiarize-se com as ferramentas necessárias.

Todo projeto é diferente, mas provavelmente você irá encontrar alguns dos seguintes arquivos na base do projeto:

• README (txt): Contém explicações iniciais do projeto. Comece por aqui, já que usualmente esse arquivo possui informações de como compilar e instalar o software do código-fonte.
• LICENSE ou COPYING (txt): Possui informações sobre a licença na qual o projeto é distribuído.
• MAINTAINERS (txt): Descreve quais pessoas são responsáveis por qual parte do projeto.
• CONTRIBUTING (txt): Descreve mais informações de como participar da comunidade e contribuir.
• Documents ou docs (pasta): Contém diversas documentações sobre o projeto, tanto de usabilidade quanto sobre a parte técnica.

Quando nos referimos ao projeto "mainline" ou "upstream", significa que é o projeto oficial, onde o desenvolvimento de novas funcionalidades está acontecendo e o que contém as modificações mais recentes. Em geral, ao fazer alguma modificação (patch) no código, ela só será considerada oficial depois de entrar na versão mainline. Por exemplo, o browser que vem em uma distribuição de Linux não é a mainline. Apesar de se basear em uma versão específica do projeto mainline do Linux, a comunidade da distribuição usualmente aplica diversas modificações tanto no código-fonte quanto nas configurações do kernel para atender as necessidades específicas daquela comunidade . Portanto, ao encontrar um bug, é importante testar o código mainline para ver se ele já foi corrigido ou se afeta apenas a versão da sua distribuição.

Como buscar ajuda

Ao buscar ajuda, tenha em mente que a maioria das pessoas da comunidade também são voluntárias como você e não são obrigadas a atender às suas demandas, logo seja educado, verifique se o projeto define algum tipo de código de conduta, mas não tenha medo de perguntar, mostre que você fez uma rápida investigação, isso indica que você está correndo atrás e as pessoas em geral gostam de incentivar gente nova e interessada, por exemplo:

"Olá, eu sou novo no projeto, queria entender sobre X, achei o artigo Y mas ele não parece explicar o que eu gostaria de saber, alguém poderia me explicar ou me indicar onde posso ver essa informação?"

"Olá, estou tentando entender como o código X funciona, me parece que faz a tarefa Y mas estou incerta, existe alguma documentação sobre isso? Procurei e não encontrei. Agradeço se alguém me ajudar"

Se você não tiver certeza que está perguntando na lista de email certa, ou no canal de IRC certo, pergunte onde seria mais apropriado postar a sua pergunta. Caso você não obtenha resposta, não assuma que está sendo ignorado por ter começado agora, as pessoas são ocupadas, espere um pouco (algumas horas no IRC ou uma semana no email) e refaça a pergunta.

Onde buscar ajuda

IRC: Muitos projetos possuem um canal de bate-papo no IRC para a comunidade se coordenar e se ajudar. Veja se o seu projeto possui um canal, baixe um cliente de IRC, conecte no servidor e se junte ao canal. Por exemplo, no servidor da Freenode você pode encontrar os canais #freebsd, #ubuntu, #debian, #python, #docker, há também canais mais específicos, por exemplo, o Debian se organiza por times, logo você pode encontrar no servidor da OFTC os canais #debian-cloud, #debian-mirrors, #debian-ftp entre outros. Muitas vezes o projeto possui canais específicos para quem está começando, como o #kernelnewbies na OFTC.

Guia para configurar o seu IRC https://fedoramagazine.org/beginners-guide-irc/

Listas de emails / fóruns: Procure se o projeto possui alguma lista de email ou fórum para discussão, por exemplo, o Kernel possui uma lista para cada subsistema. Procure a lista apropriada e se inscreva, muitas listas disponibilizam os archives dos emails passados, útil quando está procurando sobre algum tópico que já foi discutido. A dica aqui é fazer bottom post (responder emails em baixo ou entre a cópia) utilizado pela maioria dos projetos. Caso não obtenha resposta em uma ou duas semanas, verifique se mandou a sua pergunta para a lista de email mais apropriada ou as vezes as pessoas estão simplesmente ocupadas, eu geralmente respondo a mesma thread the email com a palavra "ping" para relembrar as pessoas de responderem.

Discussões em algum sistema: alguns projetos usam o GitHub diretamente para perguntas e discussões, verifique se o projeto usa algum sistema específico para discussões e participe.

Pull requests

Pull request é quando você requisita que suas mudanças seja incluído na mainline. Cada projeto possui a sua maneira de enviar modificações (patches) de código para o projeto, no Linux Kernel por exemplo, você deve mandar os patches no texto do email no formato do git-format-patch, já no FreeBSD, você deve anexar o patch em formato diff unified no sistema de controle de bugs, alguns outros projetos aceitam pull requests pelo sistema do GitHub, verifique com o seu projeto como você deve enviar os patches para a comunidade.

Estrutura da comunidade

Cada comunidade se organiza de uma forma diferente, podemos encontrar os diferentes papeis dentro da comunidade

• Autor: quem começou o projeto
• Commiter: quem possui o acesso de commit na mainline
• Mantenedor: o responsável por revisar e aplicar patches de alguma subparte do projeto ou no projeto todo
• Colaboradores: que ajudam o projeto em diversos aspectos
• Time: um subgrupo de colaboradores que fazem alguma tarefa específica do projeto, podendo até fazer o papel de um mantenedor
• Usuários

É importante conhecer a estrutura da comunidade para saber pra quem fazer perguntas, pedir revisões ou mandar contribuições para o time ou grupo de pessoas trabalhando na área relacionada. Lista de emails ou canais de IRC com escopo muito genérico será mais difícil encontrar alguém que revise e aplique um patch, ou responda uma pergunta muito específica sobre algum assunto.

Exemplos de como algumas comunidades funcionam

Debian:

A comunidade é organizada de maneira bem democrática, o líder do projeto é eleito por voto anual, os trabalhos são divididos por times (Ex. time de mirrors, time DSA para a infraestrutura, time de release que coordena o lançamento da próxima versão), e cada pacote no Debian pode ter como responsável um mantenedor específico ou um time. Logo ao encontrar um bug em um determinado pacote, verifique quem é o responsável, entre em contato e envie seus patches para a pessoa, time ou lista de email certa.

Linux Kernel:

O projeto é mantido por Git, o único commiter da mainline é o Linus Torvalds, o projeto é dividido em diversos subsistemas, cada subsistema possui um mantenedor em que o Linus Torvalds confia e aceita seus pull requests. A organização do desenvolvimento de cada subsistema é bem variado e cada um tem suas regras, tem subsistemas que possuem co-mantenedores e outros que não, cada subsistema normalmente tem um canal de IRC e uma lista de email e documentação.

Aprofundando no código

A maioria das pessoas começam contribuindo com algo tão simples quanto corrigir um erro ortográfico, essa simples contribuição trará conhecimento do workflow completo de como trabalhar com a comunidade, mas muitas vezes encontrar problemas técnicos à serem resolvidos nem sempre é fácil e exige um conhecimento maior do projeto.

Ao se aprofundar no código, verifique quais são os métodos de debug que o projeto utiliza, essas técnicas vai ajudá-lo a entender melhor o código e à informar com mais detalhes o seu problema para outras pessoas. Pesquise onde você consegue visualizar os logs de erro, como incluir no código alguma mensagem de log, veja se consegue executar o projeto passo à passo com ferramentas como GDB, Python Trace. Alguns projetos já possuem testes inclusos, veja também se a comunidade usa alguma ferramenta externa para teste, aprenda como reproduzir os testes e à depurar o código.

Achar um problema a ser resolvido

Caso você tenha encontrado um mal funcionamento no projeto de interesse, comece por aí, verifique se alguém já reportou o bug em alguma lista de email, fórum ou no próprio sistema de controle de bugs, entre em contato com as pessoas envolvidas e peça mais informações. Caso não saiba por onde começar a olhar no código, pergunte às listas de email ou canais de IRC, normalmente as pessoas te apontarão para onde olhar a grosso modo, e assim comece a sua investigação do problema. Reporte o bug para a comunidade para que saibam que o problema já está sendo investigado e que podem te contactar para trabalhar em conjunto, evitando assim retrabalho.

Caso você não tenha dado a "sorte" de encontrar um bug, muitos projetos já possuem uma lista de bugs conhecidos só esperando alguém para adotá-los, procure onde está lista se encontra, analise algum bug que consiga reproduzir e não tenha medo de fazer perguntas.

Dependendo do projeto, muitas vezes dar os passos acima é muito complicado e exige muito conhecimento prévio para entender um bug, no Linux Kernel por exemplo, essa lista de problemas já conhecidos mal existe, as que existem só possuem problemas difíceis para um iniciante. O que eu sugiro nesse caso é que você mude a abordagem, ao invés de tentar achar um problema, estude o código, quando você estiver familiarizado o suficiente vai provavelmente visualizar que o código não é perfeito e ver vários pontos de melhorias. Uma dica é pegar algum código (alguma função, classe, módulo ou driver do projeto) e tente reescrever esse código do zero, utilizando o código original apenas como referência, fazendo perguntas para a comunidade das partes que não entende. O conhecimento adquirido neste exercício vai proporcionar uma visão melhor do código, das APIs internas, expor possíveis problemas, além de integrá-lo melhor na comunidade.

Estágios pagos com mentoria

Uma ótima forma de começar à contribuir com FOSS é através de um estágio direcionado. Há algumas empresas ou fundações de FOSS que financiam programas de estágio remoto de aproximadamente 3 meses, onde o mentor é normalmente um voluntário que propõe uma determinada tarefa dentro do projeto. Com isso você já tem uma direção no que contribuir, ter alguém que você possa fazer perguntas e assumir que é sim o papel delas te responder, acompanhar o seu progresso periodicamente, além de ser pago por isso.

Google Summer of Code (GSoC): Estágio remoto em algum projeto de FOSS pago pela Google durante 3 meses de Maio à Julho para estudantes, confira quais projetos participam, se interessar por algum, verifique as propostas feitas pelos mentores voluntários, veja o processo de seleção, normalmente há algumas tarefas que você precisa realizar na aplicação.

Outreachy: Organizado pela Software Freedom Conservancy, similar ao GSoC para grupos sub-representados na comunidade, não precisa ser estudante, acontece duas vezes ao ano (Maio à Junho, e Dezembro à Fevereiro).

Endless Vacation of Code (EVoC): A Fundação X.org tem o próprio programa pra universitários que querem começar a contribuir. O EVoC pode começar em qualquer mês do ano.

Conferências

Muitos projetos de FOSS organizam conferências para reunirem a comunidade e discutirem problemas atuais de forma colaborativa. Ir à conferências é uma ótima forma de se familiarizar com o projeto e conhecer pessoalmente as pessoas com quem você interage online. Verifique quais são as conferências que o projeto no qual você se interessa realiza, ou quais as principais conferências que as pessoas que você interage participa.

Ajuda de custo para conferências

O problema é que a maioria dessas conferências são fora do Brasil e a viagem fica muito cara, principalmente para estudantes. Felizmente, muita dessas conferências distribuem bolsas para ajuda de custo, a Linux Foundation por exemplo disponibiliza um formulário para requisitar ajuda de custo com passagem de avião e hotel, também há ajuda para grupos sub-representados para incentivar a diversidade na comunidade, e as vezes o próprio projeto possui algum fundo para bolsa. O Debian por exemplo, pode pagar a sua viagem, principalmente se você é uma pessoa que já está ajudando a comunidade, mas mesmos novatos podem conseguir.

Outra forma de conseguir ajuda de custo é se voluntariar par ajudar na organização da conferência, mande um email para a equipe de organização e pergunte se há essa possibilidade.

Espero que essas dicas ajudem, caso tenha alguma dúvida entre em contato ou deixe um comentário

NVMe: Officially faster for emulated controllers!

The Doorbell Buffer Config command

When I last wrote about NVMe, the feature to improve NVMe performance over emulated environments was just a living discussion and a work in progress patch. However, it has now been officially released in the NVMe Specification Revision 1.3 under the name "Doorbell Buffer Config command", along with an implementation that is already in the mainline Linux Kernel! \o/

You can already feel the difference in performance if you compile Kernel 4.12-rc1 (or later) and run it over a virtual machine hosted on Google Compute Engine. Google actually updated their hypervisor as soon as the feature was ratified by the NVMe working group, even before it was publicly released.

There were very few changes from the original proposal, I.e. opcodes, return values and now fancy names; the buffers (as described in my last post) are now called Shadow Doorbell and EventIdx buffers.

In short, the first one mimics the Doorbell registers in memory, allowing the emulated controller to fetch the Doorbell value when convenient instead of waiting for the Doorbell register to be written. For its part, the EventIdx provides a hint given by the emulated controller to tell the host if the Doorbell register needs to be updated (in case the emulated controller is not fetching the Doorbell value from the Shadow Doorbell buffer). You can check section 7.13 of the specification for an example of usage.

Results

The following test results were obtained in a machine of type n1-standard-4 (4 vCPUs, 15 GB memory) at Google Cloud Engine platform with Kernel 4.12.0-rc5 using the following command:

$ sudo fio --time_based --name=benchmark --runtime=30 \
--filename=/dev/nvme0n1 --nrfiles=1 --ioengine=libaio --iodepth=32 \
--direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=1 \
--rw=randread --blocksize=4k --randrepeat=0

Results (in Input/Ouput Operations per Second):
Without Shadow Doorbell and EventIdx buffers: 43.9K IOPS
With Shadow Doorbell and EventIdx buffers: 184K IOPS
Gain ~= 4 times

Screenshot - Without Shadow Doorbell and EventIdx buffers

Screenshot - With Shadow Doorbell and EventIdx buffers

Enjoy your enhanced numbers of IOPS! :D

Collabora Contributions to Linux Kernel 4.11

Linux Kernel v4.11 was released, and 9 different Collabora developers contributed a total of 44 patches, an increase of 5 patches from version 4.10. The majority of Collabora's work this time was around fixes and clean ups in the DRM. In addition to our contributions as authors, Collabora also added 22 Reviewed-by tags for patches reviewed by our engineers. You can learn more information about the v4.11 merge window in LWN.net's extensive coverage: part 1, part 2 and part 3.

Now here is a look at the specific changes made by Collaborans. To begin with, Enric Balletbo fixed an issue and improved documentation of IIO regarding sensors and also added support for several buses and peripherals for the Toby-Churchill SL50 board. Romain Perier added an ASoC machine driver for Rockchip rk3288-based boards that have an HDMI and analog audio output, and also added support for slave mode in the Everest Semi ES8328 audio codec, while including ES8388 as a compatible device in the ES8328's codec driver.

For his part, Tomeu Vizoso fixed the sink display error in DRM EDID when no deep color is available for Rotel RSX-1058 and also fixed several issues and code cleanups regarding CRC in DRM and integrated the new CRC debugfs API in i915. Gabriel Krisman Bertazi, who wrote a great article recently on tracing the user space and Operating System interactions, made several improvements to DRM by adding documentation, cleaning up the code and fixing several issues, including allowing QXL build when FBDEV_EMULATION is disabled.

Lastly, Daniel Stone, Collabora's Graphics Lead, fixed an important issue in DRM regarding the use of Atomic State in legacy ioctls, while Fabien Lahoudere cleaned up the code for the Epson RTC removing an unnecessary spinlock, and Robert Foss fixed a copy of uninitialized memory in Ethernet qed code.

Here is the complete list of Collabora contributions:

Daniel Stone (1):

• drm: Use atomic state for FB in legacy ioctls

Enric Balletbo i Serra (10):

• iio: Documentation: Add proximity unit
• ARM: dts: am335x-sl50: Remove I2C1 node.
• ARM: dts: am335x-sl50: Enable UART1.
• ARM: dts: am335x-sl50: Enable SPI0 interface and Flash Memory.
• ARM: dts: am335x-sl50: Enable BQ32000 Real Time Clock.
• ARM: dts: am335x-sl50: Enable the MCP23017 GPIO Expander.
• ARM: dts: am335x-sl50: Enable the Ambient Light Sensor.
• ASoC: mt8173-max98090: remove the call to snd_soc_jack_add_pins.
• rtc: bq32000: add support to enable disable the trickle charge FET bypass
• iio: cros_ec_sensors: Fix return value to get raw and calibbias data.

Fabien Lahoudere (1):

• rtc: rx8010: change lock mechanism

Gabriel Krisman Bertazi (19):

• drm: Deduplicate driver initialization message
• drm: Drop unused forward declaration of drm_version
• drm: Export drm_ioctl_permit to kernel-doc
• drm: Update TTM initialization documentation
• drm: Document deprecated load/unload hook
• drm: Change the return type of the unload hook to void
• drm: Document drm_cache interface
• drm: Move drm_clflush prototypes to drm_cache header file
• kernel-doc: properly document array arguments of function
• drm: qxl: Let DRM core handle connector registering
• drm: qxl: Open code probing sequence for qxl
• drm: qxl: Open code teardown function for qxl
• drm: qxl: Drop misleading comment
• drm: qxl: Drop duplicated device pointer attribute
• drm: qxl: Drop duplicated pci_device pointer attribute
• drm: qxl: Embed drm_device into driver structure
• drm: Rely on mode_config data for fb_helper initialization
• drm: Fix build when FBDEV_EMULATION is disabled
• PCI: Remove duplicate check for positive return value from probe() functions

Gustavo Padovan (1):

• MAINTAINERS: add dma-fence* files to Sync File maintainership

Martyn Welch (1):

• docs: Fix htmldocs build failure

Romain Perier (4):

• ASoC: es8328-i2c: Add compatible for ES8388
• ASoC: es8328: Add support for slave mode
• ASoC: Allow to select ES8328_I2C and ES8328_SPI directly
• ASoC: rockchip: Add machine driver for RK3288 boards that use analog/HDMI

Tomeu Vizoso (6):

• drm/i915/debugfs: Move out pipe CRC code
• drm: Move locking into drm_debugfs_crtc_crc_add
• drm: crc: Wait for a frame before returning from open()
• drm/i915: Use new CRC debugfs API
• drm/i915: Put "cooked" vlank counters in frame CRC lines
• drm/edid: Add EDID_QUIRK_FORCE_8BPC quirk for Rotel RSX-1058

Robert Foss (1):

• qed: Fix copy of uninitialized memory

Reviewed-by:

Gustavo Padovan (8):

• drm/exynos: use atomic helper commit
• drm/doc: Clarify connector overview
• drm/gma500: Nuke device_is_agp callback
• drm/core: Use recommened kerneldoc for struct member refs
• drm/gem|prime|mm: Use recommened kerneldoc for struct member refs
• drm/hisilicon/hibmc: Fix wrong pointer passed to PTR_ERR()
• drm/atomic: make release_crtc_commit() static
• dma-buf: Update kerneldoc for sync_file_create

Emil Velikov (5):

• drm/i915: Put "cooked" vlank counters in frame CRC lines
• drm/i915: Use new CRC debugfs API
• drm: Move locking into drm_debugfs_crtc_crc_add
• drm/i915/debugfs: Move out pipe CRC code

Gabriel Krisman Bertazi (3):

• drm/tilcdc: Fix hardcoded fail-return value in tilcdc_crtc_create()
• drm/fsl-dcu: remove unneeded 'ret' assignment
• drm: Update drm_cache.c to pull in the new drm_cache.h

Robert Foss (3):

• drm/i915: Put "cooked" vlank counters in frame CRC lines
• drm/i915: Use new CRC debugfs API
• drm: Move locking into drm_debugfs_crtc_crc_add

Daniel Stone (2):

• drm/fb-helper: Allow var->x/yres(_virtual) < fb->width/height again
• drm: Drop locking cargo-cult from drm_mode_config_init

Tomeu Vizoso (1):

• drm: crc: Call wake_up_interruptible() each time there is a new CRC entry

LinuxCon NA 2016 - Highlights

After visiting FISL this summer, my travels have now taken me to LinuxCon NA 2016 in Toronto.

 

As everyone knows, the hot topic of the moment is containers, and they were everywhere at LinuxCon. Several companies are working in this market, there are even hardware optimized for getting the best performance on containers!

However, besides containers, there were several other different subjects of which I had more contact with:

memory-driven computers, workqueues, bluetooth, graphics, file systems, power saving (check the talk highlights below).

I also met several amazing people working in different fields and contributing with the free software community.

The place:

The infrastructure of the event was great, wifi worked everywhere. There was breakfast for attendees and snacks during the small breaks during the day.

In the main Hall, there were several couches and tables, and the conference rooms were great.

Each morning there were keynotes that were hosted in a big fancy room.  These were also streamed to the main hall so other people could watch.

In the afternoons there were several talks happening in parallel in smaller rooms.

 

 

The women lunch:

On the first day there was a women's only lunch event promoted by Intel, which was populated by 100+ women from the tech field. I've never seen so many of us reunited like that!

It was a great event to socialize and learn where everybody works. Several of them work directly with coding, but not the majority.

It was a pleasure to meet everyone and I am looking forward to see even more women in tech.

 

 

Booths (Hightlights):

 

 HP:

In this booth I met The Machine, which is based in a Memory-driven computer architecture that promises to revolutionize how we know computers today.

The main memory is based in memristors, which can be viewed as a non-volatile RAM, so instead of having our basic model of Caches/Main Memory/Disk we would only have one memory based on memristors, all connected through a photonic fabric instead of a copper bus.

This changes our current programing model. HP have a github available with a framework where you can emulate the hardware, test and start programing for it.

 

 

 

Diamanti:

Diamanti is a company that offers a hardware based solution to optimize containers and virtual environments, as mentioned in my NVMe post, I am working in a patch to optimize performance of shared NVMe device for a guest system in software while Diamanti, instead of sharing a NVMe device by software, make their hardware pretend there are multiple NVMe devices and they attach each of this devices directly to a container or virtual machine, thus from a software point of view, the container controls the device without having the VMM interfering.

They also do the same this for other peripherals beside NVMe as network cards.

 

Ubuntu:

Besides the Linux distribution (Ubuntu), this booth was presenting Juju, which is a tool to manage your services in the cloud, and also LXD, an hypervisor for containers

 

Docker:

As most of you know, the Docker project is a great tool to create containers, which are something in the middle of a virtual machine and a chroot, it uses the kernel from the host.

Docker is also the name of the company (I thought is was only the name of the project) and they use LXC as a base to create containers.

The company provides services for other companies using the Docker project as setting up the infrastructure, setting a private Docker Hub, providing support, etc.

 

Microsoft:

Why was Microsoft was in LinuxCon? To declare its love for Linux! :)

In this booth I obtained many stickers written "Microsoft loves Linux".I guess they decided to stop fighting old battles and be friends with Linux in the server market.

 

CoreOs:

CoreOs is a Linux distribution mostly meant to be a lightweight host system for docker containers.

Kubernetes is a tool for managing containers, automating deployment and scaling. So used in conjunction with CoreOs is a good match.

 

Talks (Hightlights):

 

Btrfs with High Speed Devices - Chris Mason, Facebook:

Currently the maintainer of Btrfs, Chris Mason talked about this file system, tools to debug and how to identify bottlenecks.

One of the bottlenecks was btree locking, where he presented a patch that has a new locking scheme that optimizes the file system.

 

Open Source Bluetooth Device Firmware for IoT and Makers - Marcel Holtmann, Intel:

In this talk, Marcel Hltmann gave a great overview of the Bluetooth stack and mentioned that Bluetooth 5.0 is coming with support for mesh network.

As he is the maintainer of the Bluetooth stack on Linux, he talked about BlueZ and other Bluetooth tools in Linux.

For IoT and Makers who usually use an nRF51/nRF52 Bluetooth chip with the proprietary SoftDevice firmware, Marcel talked about how we could use Zyphis or MyNewt (which are open source) instead of SoftDevice and how he managed to get it working on Arduino 101.

 

Async Execution with Workqueue - Bhaktipriya Shridhar, Linux Kernel

Bhaktipriya Shridhar gave a talk about her Outreachy project on workqueues and how she managed to migrated several drivers from the old API to the new one.

Workqueues is a mechanism in Linux Kernel to execute pieces of code in asynchronous fashion, in short: if you have a function to execute and you don't want to wait for it to return, you can add it in the workqueue.

Internally, the kernel has two API's, the old one, with several issues as proliferation of kernel threads (it could run out of process IDs before even executing user space), deadlocks (if wasn't handled correctly) and unnecessary context switches. And the new API, the Concurrency Managed Workqueue (cmwq), which solves most of these issues.

Check her slides here: http://events.linuxfoundation.org/sites/events/files/slides/Async%20execution%20with%20wqs.pdf

 

Kernel Internship Report and Outreachy Panel - Moderated by Karen Sandler, Software Freedom Conservancy; Helen Fornazier, Rik Van Riel & Bhaktipriya Shridhar

 

Outreachy is 3 month internship meant to promote the presence of minorities in free software community.

If you know what GSoC is, Outreachy is similar with small differences in the projects (not necessary about coding), the selection phase, who can participate, etc.

In the panel we had 2 former mentors Rik Van Riel and Tiffany Antopolski (who is also a former intern), Bhaktipriya Shridhar (current intern in the linux kernel), myself as former intern and Karen Sandler as host and part of the organization of the Outreachy program at Software Freedom Conservancy.

Each one shared their experience as a mentor or as an intern.

 

 

 

CPUfreq and The Scheduler: Revolution in CPU Power Management - Rafael J. Wysocki, Intel OTC

To save power when the system can't go idle, CPUFreq can decrease or increase the clock frequency of the CPU based in the current work load.

Rafael Wysocki (ACPI core maintainer) explained the architecture of the old system that was based on timers, that would sample the load from time to time and update the clock frequency accordingly. The new system provides a much better result by using a Scheduler-driven mechanism instead of timers, using data from the scheduler to make decisions on the next frequency.

Check his slides here: http://events.linuxfoundation.org/sites/events/files/slides/cpufreq_and_scheduler_0.pdf

 

Bringing Android Explicit Fencing to Mainline - Gustavo Padovan, Collabora Ltd.

 

In this talk, Gustavo Padovan explained how graphic fences are exposed to userspace to synchronize buffer sharing and increase performance compared to the implicit fencing where userspace is not aware.

Check his slides here: http://events.linuxfoundation.org/sites/events/files/slides/Explicit-Fencing_Talk.pdf

 

 

The gala party:

In the last day we had a great gala for the 25th anniversary of Linux.

I had the pleasure to have a great conversation with Eduardo Habkost from Red Hat who has worked with virtualization for 10+ years and gave me a great explanation on how Qemu connects with KVM.

Increased performance of emulated NVMe devices

Nowadays, in Google Cloud Engine (GCE), it is possible to attach a local SSD with the NVMe interface to your virtual machine. Unfortunately, you only get a good number of iops (input/output operations per second) if you instantiate a machine with nvme-backports-debian-7-wheezy image; other available distributions on GCE will have a lower number of iops.

It turns out that Google's Virtual Machine Monitor (aka Hypervisor) implements a custom NVMe command that allows it to increase up to 4 times the number of  iops (note: this is from what I've tested so far, but it seems to be possible to get up to 5 times faster according to the original commit message; check the  Technical Details sessions to see how this is possible), however the kernel you use needs to support it and this is not yet the case with the mainline kernel.

This is not exclusive to GCE as Google released a patch not only to the kernel  but also to the qemu and is available here.

Collabora has been helping update, refactor and review the patches to the Linux Kernel to send it upstream, however since this is not yet an official nvme standard, it shouldn't be merged into Kernel mainline, as its specification may still receive changes.

Seeing as it considerably increases performance, the feature is in the process of being discussed and proposed to the NVMe workgroup with Collabora's help.

While the nvmexpress.org seems interested in adding an official extension to stardarize it, as published in the mailing list, nothing has been defined yet, as this is a very recent discussion and it can take up to a year to be ratified by the NVMe workgroup.

So, for the time being, you can get a more recent version of the patch and install the driver yourself here: https://git.collabora.com/cgit/user/koike/linux.git/log/?h=nvme/dev

How it works?

Technical details

 

The NVMe interface basicaly works with command queues. The drive writes a command in a region known to both (driver and device controller) and then updates the tail of the queue, writting to an MMIO register called doorbell.

In an environment with several guest OSes on top of a VMM sharing a resource, communication between the guest OS and the real device is usually trapped by the VMM. As an MMIO is usually a syncronous acces to the device, it means that every MMIO access will cause a trap.

Example of emulated device in the VMM

The main idea here is to decrease the number of traps to the VMM by reducing the number of writtes to the doorbells.

This is achieved in two ways:

    1) Batching; or

    2) Letting the VMM pull the current doorbell value when it is already in execution.

The first one is easy, we can wait X commands to be written in the queue to ring the doorbell.

The second one is a bit more complicated. The guest OS needs to inform the emulated device in the VMM where it can pull the doorbell values, and the emulated NVMe device needs to inform the guest OS that it can restart the counter of X.

This is what this new feature does:

It adds a new command in the NVMe interface where the driver can send to the NVMe device controller two memory buffers:

1) A buffer where the real doorbell values are: Instead of writting to the MMIO  doorbell, the driver writtes the value in this buffer; and

2) Another buffer with a hint from the controller about how many commands the driver can write in the queue without ringing the doorbell.

The exact technical details may still change in the future, especially on how to properly implement the second item above. It is also very likely that Google's patches won't be compliant with the future ratified standard.

For the time being though, you can use the Collabora tree. Please let me know if you have any comments/feedback!

Linux Kernel: memory corruption - debug tricks

Note: This post is meant to help you debug a memory corruption if you are building your own kernel from source.

The symptoms:

If you see one of the following errors in your in your dmesg log:

* BUG: unable to handle kernel paging request at 7f45402d
* invalid opcode: 0000 [#1] SMP
* general protection fault: 0000 [#1] SMP
* BUG: unable to handle kernel NULL pointer dereference at 0000000000000258

Then you probably have a memory corruption somewhere.

In my case, I didn't get these errors at first. I was adding objects to a list by calling list_add_tail(my_list, my_obj) and verifying that list_empty(my_list) was returning false as expected (and it was). But latter in the code when I called list_empty(my_list) again, it was returning true, and in nowhere in my code I was removing objects from the list.

Weird behaviors that doesn't make sense probably are due to memory corruption. When I started simplifying the code to isolate the problem, I start getting the errors above in the dmesg log.

Where it is happening:

When you have an error, you get a log message in dmesg like:

We will be looking at 2 infos there:
1) The IP (instruction pointer) address in the 2nd line
2) The call trace at line 23rd

The IP

In the 2nd line of this log, we have the instruction pointer address, it is the address the computer was executing that generated the error.

In this example, the IP is ffffffff817c69f0, then we can find where in the code this address corresponds to using the addr2line tool:

Where the path_to_your_kernel_tree is the path to the kernel code where you downloaded it (where you do make && make install)
The vmlinux file is the uncompressed version of the Linux image (it is created when you do make)

Note: if it doesn't work, check if your make menuconfig if your kernel is compiled with DEBUG flag

But this technique won't work if the corresponding address is not part of the kernel core code (if it was an error caused by some module, as we will see in the call trace section).

The call trace

In the log above, line 23rd, it means that the function devres_remove was called by devres_destroy, which was called by devm_kfree, which was called by vmc_cap_destroy and so on.

Now suppose that the vmc_cap_destroy function call devm_kfree in many places and you want to know exactly which one has triggered the bug. Lets do this in 2 steps:

1) Get the offset in the code section of the function using nm:
nm is a tool to get the offset of a symbol in a section:

In this example, the offset of the vmc_cap_destroy function is 0x680

2) Get the file and the line using addr2line:
In line 27th we have: vmc_cap_destroy+0x40, where the 0x40 is the offset inside the vmc_cap_destroy function where the code execution would return if the call to devm_kfree hadn't triggered the bug.

So lets add the function offset we found in the previous step with the offset inside the function: 0x680 + 0x40 = 0x6c0

And now we use the addr2line with the compile module.ko to find out where is it:

Then the buggy devm_kfree call is the first one just before the line 396 in file vmc-capture.c

Enable debug prints:

Besides the techniques above, you can enable the debug level prints in the dmesg log:

sudo sh -c "echo 8 > /proc/sys/kernel/printk"

If this doesn't work, check if the DYNAMIC_DEBUG flag is enabled in your menuconfig, if so, then check the next section about Dynamic debug.

In the case of a module that I was testing (the vivid module), I needed to change the vivid_debug parameter:

We can do this when we start the module:

sudo modprobe vivid vivid_debug=8

Or changing the parameter while its already running:

sudo sh -c "echo -n 0 > sys/module/vivid/parameters/vivid_debug"

Dynamic debug:

If your kernel is compiled with DYNAMIC_DEBUG flag, then changing the printk level probably won't enable the debug prints in the dmesg log.

Lets say the module we are working is called media, lets add it into the kernel:

sudo modprobe media

And now we will enable all the debug prints in the media module by sending 'module media +p' to the dynamic_debug/control file:

If you look at the dynamic_debug/control file content (as we did above) you can see the "=p" which means that the prints in those lines are enabled.

To disable the prints, we send 'module media -p' to the dynamic_debug/control file:

If you look at the dynamic_debug/control file content again you can see the "=_" which means that the prints in those lines are disabled.

Enabling a specific print

Instead of enabling all the debug prints of the entire module, we can enable all the prints on a specific file, or we can specify a file and a line:

$ sudo sh -c "echo -n 'file media-entity.c +p' > /sys/kernel/debug/dynamic_debug/control"

$ sudo sh -c "echo -n 'file media-entity.c line 301 +p' > /sys/kernel/debug/dynamic_debug/control"

Read dmesg log continuously

Instead of calling dmesg every time to check the log, you can do:

$ tail -f /var/log/kern.log

and the log will be printed continuously.

Linux Kernel: How to start contributing

Is really not that dificult to start contributing with a small patch to the Linux kernel, my first patch was to correct a missing space between binary operands (i==j) -> (i == j) that I found running a script available in the kernel code, I ran something like:

perl scripts/checkpatch.pl -f path_to_some_driver_folder/ | less

And I found the coding style problem. But first you need to download and compile the kernel.

This is going to be a short post with really useful links.

Things you will need to know to start:
* GIT
* C programing

1) Download, compile and install a kernel compiled by your self: follow these instructions
Note1: I takes a really long time: check my posts about CCache and Icecc to speed up compilation.
Note2: Consider to setup a virtual machine if you plan to develop something that can break your system.

2) Setup more tools, find a coding style problem and prepare a patch and submit it: follow these instructions
Note1: the above link tells you to not send the patch to the Linux list. You need to check who are the maintainers of the file you are modifying by running the script:

perl scripts/get_maintainer.pl <path_to_your_patch_generated_with_git_format.patch>

Or:

perl scripts/get_maintainer.pl -f <file_that_you_modified>

Or:

git show HEAD | perl scripts/get_maintainer.pl

This will get you a list of people and emails, so you can send your patch to all of them in CC.

Note2: Do interleaved/bottom-posting when replying emails. Don't put your reply before quoted original message as Gmail does by default.

3) Joing the #kernelnewbies irc channel on freenode, its a chat so you can make newbie questions.
If you don't know how to do it, download the HexChat irc client (sudo apt-get install hexchat), go to menu->Network List, look for freenode and click on connect. Then wait a window to show up asking you which channels you would like to connect and type #kernelnewbies, or type /join kernelnewbies in the freenode chat window.

How to start working in something more interesting then fixing coding style:

4) Start reading the Linux Device Drivers (LDD3) Book, its free and you can read online.
You will learn the basis in writing a driver. The book comes with some code to test the drivers it implement but most of it is not compatible with the last kernel version. If you are stuck and can not compile a driver provided by the LDD3, leave me a comment and I can help.

5) Be aware of the Documentation folder inside the Linux kernel tree, it has a lot of docs about the internal API and software architecture. Sometimes it is hard to find which file should you be reading (just ask in the #kernelnewbies).

6) Where to go now: Check the http://kernelnewbies.org/ site, it is meant for newbies :)

Check this ilustrated guide about Linux Kernel development from another Outreachy intern

I am working on a video4linux driver. If you are interested in learning how to write a driver for your camera, follow my posts :)

[How to] Speed up compilation time with Icecc

With Icecc (Icecream) you can use other machines in your local network to compile for you.

If you have a single machine, usually you would do (for a quad-core machine) something like:

make -j4

This command will generate four compilation jobs and distribute to your CPU cores, compiling the jobs in parallel.

But if you have another machine in your local network, Icecc let you use the cores of this other machine too. If this other machine is dual core, you could run:

make -j6

How it works?

When you call make -jN, instead of calling the classic GNU Gcc, we will "trick" the make so it will call another "Gcc" binary defined by the Icecc (by changing the PATH).

The make command will generate the jobs and call the Icecc Gcc that will send the source files to the scheduler that will forward the jobs to the remote machines (or to him self or to the machine who started the compilation).

How to setup the network?

Easy on Ubuntu:

* Do the following commands in every computer in the network:

$ sudo apt-get install icecc

$ export PATH=/usr/lib/icecc/bin:$PATH

Check if the gcc in the /usr/lib/icecc is being used:

$ which gcc
/usr/lib/icecc/bin/gcc

Let say that the IP address of the machine you chose to be the scheduler is 192.168.0.34. Edit the file /etc/icecc/icecc.conf and change the follow variables (still in all the machines in the network):

ICECC_NETNAME="icecc_net"

ICECC_ALLOW_REMOTE="yes"

ICECC_SCHEDULER_HOST="192.168.0.34"

Reset the Icecc Deamon

sudo service iceccd restart

* Do the following command in the the scheduler machine 192.168.0.34:

sudo service icecc-scheduler start

How can I know if it works?

Install and Run the monitor:

$ sudo apt-get install icecc icecc-monitor

$ icemon -n icecc_net

You should see all machines and an indicator saying that the network is online:

In this case I have 3 machines, the first two have four cores and the last one just one core.

When I compile something with make -j9 I see the Jobs number growing and the slots being filled.

Done!!!

CCache with Icecc (edited):

To  speed up even more your compilation time, you can setup CCache (explained in the last post).

The general ideia is: check in a local cache first (using CCache) if the source files have been already compiled, if not, then give the job to Icecc.

When using with CCache, you don't need to add Icecc in the PATH, we use CCACHE_PREFIX instead:

$ export CCACHE_PREFIX=icecc

$ echo $PATH

/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

$ which gcc
/usr/lib/ccache/gcc

[How to] Speed up compilation time with CCache

CCache stores the compiled version of your source files into a cache. If you tries to compile the same source files again, you will get a cache hit and will retrieve the compiled objects from the cache instead of compiling them again.

How it works?

 Instead of calling the GNU Gcc compiler directly, you will point your PATH to another Gcc binary provided by CCache that will check the cache first and them call the GNU Gcc if necessary:

How to install it?

Easy on Ubuntu:

$ sudo apt-get install ccache

Then change you PATH to point to the Ccache gcc (not just gcc) version:

$ export PATH="/usr/lib/ccache:$PATH"

Check with:

$ which gcc

/usr/lib/ccache/gcc

Done!!!

How can I know if it works? 

You can re-compile something and check if CCache is working with ccache -s command, you should have some cache hits:

You can increase/decrease your cache size with:

$ ccache --max-size=5G

Troubleshooting: Re-compiling Linux kernel never hit the cache?

Check if the flag CONFIG_LOCALVERSION_AUTO is set in you menuconfig, disable it and try again.

This flag seems to change a core header file as it appends the git version to the version string automatically, forcing CCache to recompile almost it all.

CCache with Icecc (edited):

If you want to use CCache with Icecc (that I'll explain about it in another post) to speed up even more your compilation time, use CCACHE_PREFIX=icecc (thanks Joel Rosdahl who commented about this)

$ export CCACHE_PREFIX=icecc

NOTE: You don't need to add icecc /usr/lib/icecc/bin in your PATH

[Outreachy] The Virtual Media Controller in Linux Kernel

If you don't know what the Outreachy program is, you can check my last post.

About my project

My Outreachy project is about a Virtual Media Controller in the Linux Kernel. Thus, I'll be in contact with the Video4Linux (V4L2) API and the Media API.

My mentor ls Laurent Pinchart, you can find more information about the project here.

I have two main goals:

1) To provide a virtual media driver with a given topology* that simulate a real camera for example, similar to the Vivid driver, generating a fake image internally from the kernel. Thus it can be used by people who needs to implement and test a program which interacts with a media device without the need to posses the real equipment.

*A topology is an abstraction of how the hardware is organized. Depending on the hardware, it can have several internal functions like: a) one or more sensors which capture the image b) a filter c) the zoom
So the kernel can model this as a video pipe:

Sensor->Sepia Filter->2x Zoom->Data Bus

It is not necessarily a linear pipe, it could be like:

                                          ->Filter 1
                                        /                 \
Real life image ->Sensor                    -> Compose -> Zoom 2X -> Data Bus 1 -> Digital image 1
                                        \                 /
                                           ->Filter 2  -> Data Bus 2 -> Digital image 2


In this example we would be able to retrieve the image after the Zoom and the image directly from Filter 2 at the same time.

2) To provide a dynamically configurable topology API.

In some devices, the topology can be highly configurable (we could link the Zoom directly in the Filter 1 for example if we wanted too). But the current V4L2 API just allow a pre-configured link to be enabled or disabled. So the goal is to define this API and make the topology of the driver made in 1) configurable through user space.


You can follow how the development is going in my github kernel tree.

Any questions you can leave a comment or ping me on irc (my nick is koike) at freenode or gnome or oftc servers.

Cheers!