focus for this week: Why don't birds fly backwards ?
>1.1 mio views (popular pages, total: 2,030)
Open Advice/Where Upstream and Downstream Meet
Where Upstream and Downstream Meet
- by Vincent Untz
- in: Open Advice
This text is available under the CC-BY-SA license. (see also: Open Advice/Info)
Contents |
A long time ago, in a room at night...
I took a last look at the list of bugs to see if I had not forgotten a patch that should be merged. I made sure to write what I thought was a descriptive NEWS entry about the new version. I typed make distcheck to start the release process and looked at the terminal display hundreds of lines. A tarball got created, and I double-checked that the tarball was building fine. Again and again – I was anxious and somehow did not fully trust the make distcheck command. After checking everything several times, I uploaded the tarball to the server and sent a mail announcement.
I had managed to do it: I had released my first tarball of a software of which I had recently become co-maintainer. And I was certainly thinking: “now users can enjoy some goodness!” But mere seconds after my tarball got uploaded, a few people downloaded it and made my release really accessible to users.
This is something I took for granted, as I thought it was mostly a trivial task. I thought wrong.
Upstream Versus Downstream
As users, we do not necessarily understand the different steps required to ship software to us. It is here, and we can simply enjoy it.
Many people contribute to this process of shipping software, and the effort is usually split between two groups of people, which are central in how Free Software works today:
- upstream: This is the group creating the software. It obviously includes coders, but depending on the project, other categories of contributors also are key participants: designers, translators, documenters, testers, bug triagers, etc. Upstream generally only ships the source code in a compressed archive, a tarball.
- downstream: This is the group responsible for distributing the software to the users. In the very same way as for upstream, contributors have a wide range of profiles, as they work on translations, documentation, testing, bug triage and more. There is however a profile that is, as of now, unique to downstream: the packagers, who prepare the software to make it available in a format suitable for easier use than just source code, a package.
Interestingly, this is a rather intuitive split for users too, although we are unaware of it: we often assume that the software developers are unreachable, and we send feedback and ask for help to the distributors instead.
A concrete analogy to clarify this upstream–downstream split could be the usual model for physical goods, with retail stores (≈downstream) distributing products of manufacturers (≈ upstream), and playing an important role for customers (≈ users).
A Closer Look at Downstream
If I had to summarize in one sentence the role of downstream, this is how I would describe it:
- Downstream is the bridge between users and upstream.
When I released my first upstream tarball, I was assuming that for downstream, the work would mostly be compiling the source and building a package out of it, and nothing else. Building a package is indeed the first step, but this is only the beginning of the journey for downstream: then come several different tasks, some of which are purely technical while others are social. I will only very briefly describe this journey here, in a non-exhaustive way, as this could be a whole part of this book[1].
The building of the package itself can be less trivial than expected: it is not uncommon that the packager hits some issues that were unknown to upstream, like when a new version of the compiler is used (with new errors), or a specific library needs to be updated first (because the tarball is using some new API), or the build system of the tarball is tailored for a specific way of working (which does not follow the guidelines of the targetted distribution). What is even more ignored by many is that all those issues can also occur after the tarball has already been packaged, like when migrating the whole distribution to a new compiler or toolchain. None of those technical issues are extremely difficult to handle per se, and upstream is often happy to help solve them; but without downstream, those issues could go unnoticed by upstream for a while.
What is more important to me than those technical challenges is that downstream is generally in direct contact with more users than upstream. This results in bug reports, support requests, requests to change configuration defaults, and more. This is where the downstream crowd really shines: instead of simply forwarding all of this upstream, downstream will work on this feedback from users to only relay summarized bits that upstream will be able to use. Often, bug reports come without enough information on the issue (in which case downstream will ask for more details); often, the support requests stem from a misunderstanding on the user side (which downstream can then, sometimes, translate to a suggestion to change the software to avoid such misunderstanding); often, new configuration defaults are suggested without a good-enough rationale (and downstream will work with the users to see if there is a valid rationale). Of this huge amount of data, downstream will produce a smaller set of information that upstream will be able to easily consume, which will lead to improvements in the software.
There are generally two rewards for downstream contributors: the indirect and direct contributions to the upstream project thanks to the efforts done downstream are enough for many, but on top of that, the direct contact with more users leads to being exposed to the satisfaction of those users. And such exposure easily makes a day for many people.
As a sidenote, when considering the amount of work involved downstream, I would not be surprised if, at the end of the day, many upstream contributors are glad to have downstream people act as a buffer to them: this significantly lowers the amount of feedback, while at the same time improving the quality of the feedback (by avoiding duplicated comments, undetailed issues, etc.). This enables upstream to stay focused on the development itself, instead of forcing upstream to either triage feedback or ignore it.
Just looking at my own upstream experience, I cannot count the number of patches I received from downstream to fix build issues. I also remember countless discussions about the bugs that were affecting users the most, that helped me organize my priorities. And since I joined the downstream ranks, I started sending similar build-related patches to upstream, and chatting with my downstream hat to relay feedback from users. Such upstream–downstream collaboration contributes to improving the overall quality of our Free Software ecosystem, and I would consider it essential to our good health.
Pushing Downstream Upstream!
I am firmly believing that there must be a strong upstream–downstream collaboration for a project to succeed. I doubt there is much disagreement on this by anyone; however, by “downstream”, people usually think of the work being done in distributions. But, especially, for applications, it is becoming more and more viable to push that downstream work out of distributions and to get benefits from such a move upstream.
Tools like the Open Build Service make it easy to have people build and distribute packages of an application for several distributions. This has benefits for both the users (who can more easily and more quickly enjoy updates of their favorite applications) and for upstream (who can help build a stronger relationship with its user base). The only challenge with such a move is that there still needs to be someone doing the packaging work, but also to manage the larger feedback from users. That is, there still needs to be someone doing the downstream work; except that it would be done as part of upstream.
To me, this sounds like an exciting perspective, and I would even go as far as suggesting that we, the Free Software community, should slowly migrate the downstream work being done in distributions to be based on downstream work being done directly upstream whenever possible – and at least for applications, this is often possible. This obviously requires a mind shift, but it would allow more sharing of the efforts that are most of the time being duplicated in all the different downstreams as of today.
For people willing to start contributing nowadays to applications they like, this packaging work upstream is a whole new approach that could be really successful!
I tried it and I stayed, will you?
Downstream has always been essential to my life as a Free Software user – after all, only a few people are manually building their whole system from scratch and I am not one of them. But it also became an asset to me as an upstream developer, as I started taking more time to discuss with downstream people to get more feedback on bugs, features, general quality and even future directions of the software I was working on.
This is only when I started being a downstream myself that I understood that this position is indeed a privileged one to help advise upstream, because of the direct contact to users and because of the different perspective we get from this different position.
Without downstream, we would not be where we are today. If you want to make a difference, be sure that by joining a downstream effort and talking to upstream, you will succeed.
And you will have fun.
about the author
Vincent Untz is an active Free Software enthusiast, GNOME lover and advocate, as well as an openSUSE booster. He held the position of GNOME Release Manager between 2008 and 2011, until GNOME 3.0 went out, was an active GNOME Foundation director (2006-2010) and is leading the GNOME team in openSUSE. However, he finds it simpler to declare he is a ”touche-à-tout”, working on various (some say random) areas of a the desktop and helping openSUSE stay amazing. Vincent is still pushing French as official language for GNOME, and hopes to succeed really soon now. And he loves ice cream.
notes
- ↑ It is worth mentioning that I do not believe that downstream should significantly modify the software released by upstream; some downstreams do that, however, and this adds to their workload.
