focus for this week: Why don't birds fly backwards ?
>1.1 mio views (popular pages, total: 2,030)
Open Advice/Packaging - Providing a Great Route into Free Software
Packaging - Providing a Great Route into Free Software
- by Thom May
- in: Open Advice
This text is available under the CC-BY-SA license. (see also: Open Advice/Info)
Contents |
Introduction
I started out in Free Software over a decade ago. I had been using Debian for some years through university, and decided that I wanted to give something back. So I started the long journey through the Debian New Maintainer’s process, never having really contributed to Free Software before, and concerned that a lack of experience with C would prove to be a major problem.
As it turned out, this concern was mostly unfounded. By starting out working with packages that I used regularly I was able to contribute effectively. As my experience with the myriad of tools and systems that Debian provides to its maintainers grew, I became more efficient with my time, and was able to take on a wider range of packages.
Taking on more packages increased my exposure to a range of build systems, programming languages and toolkits, and also helped to bring me into the Debian community. Abrasive and opinionated though it is, Debian’s community of skilled and experienced maintainers is one of the main reasons Debian has maintained its technical excellence over such a long period.
At about this time the Apache httpd project was finally closing in on the first beta releases of httpd 2.0, which had been several years in the making and was going to be a massive upgrade. Debian’s Apache team had been fairly inactive for some time – the 1.3 packages were stable and changed infrequently – and had no plans for packaging 2.0. I had a strong interest in ensuring that the httpd packages were well maintained – I was working as a sysadmin in charge of numerous Apache web servers – so it made a lot of sense to take on the challenge of producing packages for the new release.
A friend and I started work on the packages and quickly discovered that while the code was approaching an early beta quality, the tooling around the build and customization of httpd was sadly lacking, which is fairly typical for many complex software projects.
Over the course of the best part of a year – whilst upstream stabilised their code and an increasing number of early adopters began to test and deploy the new release – we worked hard to ensure that the build system was sufficiently flexible and robust to cope with the stringent requirements of Debian’s policy. As well as ensuring that our packages were technically correct, we had to ensure that our relationship with upstream allowed us to get patches back upstream whenever possible, and to get a heads up whenever security issues arose and for early testing of release candidates.
My interactions with Apache in the course of packaging and maintaining httpd 2.0 led me to become an upstream committer on the project, meaning I could contribute code directly. This is generally the final step in moving from packaging software to actively developing it for a wider audience than your distribution. On a personal level, this recognition gave me the confidence to contribute to far more Free Software projects, since I knew that my code was of sufficient quality to be welcomed.
Evolution - from packager to developer
So how did this happen? Packaging in its simplest form ensures that a given software project complies with the policy of the distribution; in my case Debian. Generally, this means configuring the software at build time so that files are placed in the correct directory locations (specified by the File Hierarchy Standard, or FHS), that dependencies on other packages are correctly specified, and that the software runs successfully on the distribution.
More complex packaging can require splitting an upstream project into multiple packages, for example libraries and the header files that allow the user to compile software against that library are shipped in separate packages, and platform dependent files can be shipped separately from platform independent ones. Ensuring that the upstream software correctly deploys in these situations will often require changes to the code. These changes are the first step into active work on a project, rather than the sometimes passive act of packaging.
Once your package is available in the distribution it is exposed to millions of potential users. These users are guaranteed to run your software in ways that neither you, as packager, nor your upstream expected. Unsurprisingly, with many eyes come many bug reports. Debian, in common with most distributions, encourages its users to submit bug reports directly to Debian, rather than to the individual upstream projects. This allows maintainers to triage bug reports and ensure that the changes made during the packaging process are not the cause of the reported problem. Often there can be considerable interaction between the reporter of the problem and the package maintainer before the upstream developers become involved.
As the package maintainer increases their knowledge of the project, they will be able to solve most problems directly. The maintainer will often release bug fixes directly into Debian in parallel with feeding them back upstream, allowing for swift problem resolution and considerable testing of fixes. Once a fix is confirmed the maintainer will then work with the upstream project to ensure that the required changes happen in the upstream, definitive project, so that they are available to other users of the software.
Providing successful bug fixes on distributions such as Debian is often a complex art form. Debian runs on many platforms, from IBM mainframes to smart phones, and the range and breadth of these platform swiftly reveals assumptions in the code. More often than not the packager has easier access to a broader range of platforms than upstream does, and so is the first port of call when a knotty porting problem does come up. One quickly learns to recognise the symptoms of pointer size assumptions, endianness problems, and many other esoteric issues; this experience makes one a more versatile and cautious programmer.
As a package collects bug fixes and improvements, it is essential to feed those changes back upstream. Too often the delta between a package and the definitive, upstream software can grow enormously, with the effect that the two become almost entirely separate code bases. Not only does this increase the maintenance burden on both sides, but it can cause huge frustration and waste large amounts of time for your upstream should a user of your package report a bug related to one of the changes in the packaged version to the upstream. To this end, a close working relationship with upstream and an understanding of the best way for both parties to collaborate is vital.
Collaboration between upstream and packager can take many forms. Whether it be finding the correct way to communicate bug reports, making sure you use the correct coding style, or ensuring that you both use the same version control system in the same way, making sure that your interactions are as friction-free as possible, makes for a far better relationship with upstream and a greatly increased likelihood that your upstream will take the time to help you when you need it.
Once the working relationship between you and your upstream is established, it becomes an easy step to contribute more directly to upstream. This, too, can take many forms. Simple first steps can involve synchronising any upstream bug reports with the ones from your distribution, making sure that duplicate effort is not expended to root cause and fix bugs. More direct involvement entails feature development and changes with a wider scope than would be palatable when made in a packaged version.
Conclusion
I think the two core things I wish I had known when starting out are the sense of community that Free Software engenders, and the fantastic route that packaging of Free Software provides into the wider Free Software world.
Community is critical to the success of Free Software. It comes in many forms, from the legion of users willing to invest time in making your software better, to one’s peers in a distribution or software project who invest their time and energy into honing your skills and ensuring that your contributions are as good as possible.
The route from packaging into development is one often traveled. It provides a learning curve less steep than entering a development community cold, and allows one to develop skills at a more gradual rate than would otherwise be the case.
about the author
Thom May is a Debian Developer, an emeritus Member of the Apache Software Foundation and was one of the first hires for Canonical, Ubuntu’s parent company. He currently lives in London and is Head of DevOps for Macmillan Digital Science.
