The Future of Packaging Software in Linux
[ Saturday, 17 February 2007, kocio ]
GNU/Linux is known for its diversity and freedom of choice. There are multiple window managers and desktop environments, many competing systems for handling sound, graphics and hardware autodetection. This diversity is the power and weakness of free software. Exactly the same problem concerns installing software in GNU/Linux. There are currently at least 5 popular ways of doing it:
- Installing directly from source code,
- Ports-based installation (where the source packages are held in a repository and can be automatically downloaded, compiled and installed), like BSDs ports of Gentoo’s portage,
- Installing from distribution-specific packages like different versions of RPM, DEB, TGZ, and other packaging formats,
- Installing from distribution-independent binaries (most proprietary software is delivered this way),
- Using another distribution-independent system like autopackage, zero-install or klik — none of them gained a significant market share so far.
This situation is not a problem for experienced users — they can make decisions about choosing the best way of installing software themselves. However, for a newcomer in the GNU/Linux world, this situation is pretty confusing. In this article I am going to sum up some of the recent efforts to fix this problem and examine the possible future of packaging software in GNU/Linux.
Package formats wars
The war of packaging formats in the GNU/Linux world is reaching its peak. The developers of competing formats are doing their best to convert as many distributions as possible to their products. Here is a short summary of the latest activities in this area.
RPM unites
Red Hat Magazine has published an article called The Story of RPM about the history of RPM package manager. It used to be a very strange product. Its ancestors: RPP, PMS and PM were separate projects with similar goals. When RPM arrived, replacing the former standards, it was first being developed under the same name by many GNU/Linux distributors independently, causing a lot of confusion and interoperability problems. Recently, this situation has changed and the major distros (those basing on Red Hat and using RPM) decided to unite and work together on the development of RPM. It’s great news for the distributions like Fedora, PCLinuxOS, Mandriva or openSUSE. Still, a lot of other systems like Ubuntu, Debian, Slackware, Gentoo or Linspire do not use the RPM format and do not plan to incorporate it.
Autopackage slowly vanishes
Autopackage, one of the recent efforts of developers to standardize packaging in GNU/Linux has not been very successful. Despite a promising debut and even providing a stable 1.0 release (the most recent version is 1.2.1, Autopackage is dying. The programmers (actually only one is actively working on the project) believe that the lack of success of the project is caused by the reluctance of Linux vendors who are well accustomed to the current status of distro-dependent package systems and don’t want any change.
It may also be that Mike Hearn — the project founder, now a Google employee — is right. He claims that the project did not gain enough popularity (and none of the distribution-independent packaging efforts did, actually) because of the dominating doctrine which states that it’s the distributor who should be responsible for the whole operating system. It is kind of similar to another popular opinion: that the administrator (root) should be responsible for the whole machine, and all the administrative operations should be preformed from a separate root account. These kinds of centralized management systems fit to the server market, but they don’t really reflect the desktop reality that well. On desktops, the administration is only one of the roles the user plays and waiting for half a year for a new version of an application is often unacceptable. It is often a decade in the free software world!
“I eventually concluded that Linux was never going to make the big improvements to desktop computing that I wanted to see, and, therefore, that it’d never get non-trivial market share. That’s one reason why these days I work on servers for Google instead of client-side stuff for a Linux company.” — Hearn concluded.
Conary approach
Another interesting approach to packaging is presented by the Conary team. Conary is a mixture of a package manager and a version control system. This tool has gained the acceptance of a few vendors and it is feasible that it will be adopted in those systems in one form or another. It is however no revolution since it does not resolve the biggest problem — the incompatibility of the current solutions.
Where does the future of packaging lie?
As far as freeing the packaging from the distributor is concerned, there are at least two directions possible. The first is the wide acceptance of the refreshed CNR (Click-And-Run) system, which is going to support all the major GNU/Linux distributions in the near future (and in time, even more platforms). The plans to make the development of CNR more open creates a chance to gain the community and finally — the vendors. Currently only Linspire and Ubuntu (from version 7.04) support the system, but if few other major distros accept it, it may be enough to spread the new de-facto standard in the Linux world, since most of the distributions are Debian, Fedora, Ubuntu, Mandriva or SUSE-based anyway.
Another option is the fast evolution of LSB’s Linux Standard Desktop Project. Linux Standard Base may help to standardize at least the versions of the core libraries used in the major distros making it much easier to produce general packages in the future. It seems to be a serious option since LSB is already widely adopted in most distributions and the fact that the LSB guys chose to cooperate with the vendors instead of creating the new standard on their own works for the benefit of the standard.
Whatever the future if packaging will be, it is good to see that the Linux vendors (seem to) have finally agreed that the current situation is not satisfactory and the future of GNU/Linux acceptance depends on standards, also the packaging ones. Hopefully, one will be formed in one way or another, soon.
More information
- The home of RPM
- The home of Autopackage
- Autopackage in action [Flash demo]
- Conary on rPath wiki
- Click and Run for all — the new home of CNR
This text is based on a translation of article by Daniel Koć published on linuxnews.pl. I (michuk) added some personal thoughts to it (accepted by the original author).












So many more people use linux but it is so under-rated
Do you think it’s really a good idea to have CNR installers standarized? I think it’s very practical to update the system with a apt-get dist-upgrade, for example. I think it would be practical too, if the user click on the RPM, DEB, whatever format, and it opens a GUI program to install it, as we already have on some distros.
@tzbishop2k: You are perfectly right, but the idea of a standard is a bit different. The native package managers are going to stay anyway, so the best thing LSB can do it to help standarize the core libraries of the popular distros and thus make “package translation” useful.
If all major distros provided a specific version of most core libraries, preparing packages for the specific distros would not be necessary. The packager would only have to create one universal package (either DEB, RPM or something completely new), then import it to CNR and immediately all the users of all distros could then download it and use it (either via CNR or via an native platform integrated with it).
That’s at least my vision of what should be happening in the next few months.
How about just running application and not installing at all?
You can run software from remote installed filesystem - see http://vamosproject.org/rootz
Conary isn’t any more distribution independent than for example portage and deb are. A good example of a distribution independent system is zeroinstall ( http://zero-install.sourceforge.net/ )
just use gentoo
Why there is no mention about Debian packaging? It’s as used and popular as RPM, but much better in handling dependencies. RPM is NOT a standard. It’s one of many, and not the best either.
frankly, I think apt-get/deb is great. I’ve never used Redhat to any appreciable extent, so I can’t comment, but I imagine that rpm has gotten to the same point. I think trying to come up with a new super universal package format is really just going to make yet another format and not solve many problems. Perhaps what is needed is a standard way of making commercial software that links to some open-source compatibility layer, and this layer can then be built for any distribution, and this whole package can be maintained like any completely open-source package, but the strictly binary parts are just pulled from the repository that way.
What I’d also like to see is a standard way for a user to install a binary software package in their home directory if the system administrator is unwilling to install it, but it would also use the system libs that were available, etc.
It would also be nice to be able to have multiple versions of packages installed without having different package names (why not just call the package python rather than python2.4 python2.5 etc, and have the ability for multiple version to reside), but that isn’t the current focus, and I really think this functionality should be morphed over from one of the current mainstream methods. Creating a new format, especially one without a distribution behind it, is going to help nobody.
I think that the best possible solution is to use apt-get, but with RPM packages instead of DEBs. We do this at Rutgers University (for Solaris 9) and it works great. DEBs are much more complicated to build than RPMs and RPMs only require the editing of one file (the spec file) to create. If more people were to use this solution, package managing would be greatly simplified and improved.
I guess I’d like to clarify my above implicit point - vendors should be made to realize that compatibility is at the source API level, not at the binary API level. If they want to be able to release one package, probably the simplest way to do that and to not be a complete pain in the ass is to supply that stub source that can be compiled. Hard coded options such as configuration file paths should be in that source as well, allowing distributions, or whomever finally packages things, to set up the paths that make sense for their system, etc. I’d also like to see licenses for closed source software allow redistribution by third parties, so they could just go in normal respositories. Of course this all only really applies or helps the case of closed-source but free-as-in-keg-beer applications (like Skype).
I’d also like to mention that I ran a slakware -desktop- back in the day (kernel 1.2.13), lest you think I’m some Ubuntu fanboy who hasn’t used anything else (although I’m running Ubuntu now and frankly I’d gladly go home with it after the show - if only I could get a backstage pass)
Has anyone used archlinux? Seriously the best package manager. It’s simple and direct, really good at resolving dependancies, and doesn’t have a convoluted way of operating like apt-get, yum, deb, or rpm.
@drfindley: Actually while arch’s ‘pacman’ was good, I found the dependency handling was far from good. Most of the packages were broken, and often missed out dependencies, which you would later have to install after finding the program would not start due to missing libraries.
apt and dpkg actually check all the dependencies, as most of the build environments for packages built by Debian are done in clean chroots, so a broken package is quickly found as it won’t compile on their compile farm, as they did not explicitly specify a certain dependency.
Seriously, apt-get isn’t hard, if you can manage pacman, then you can handle apt.
yum I have serious issues with as it’s highly inefficient, with 10MB index files PLUS having to download metadata files for the RPMs.
Use Ubuntu and you won’t go wrong … APT APT APT … all the way …
apt-cache search HELP!
What about gobolinux? The solution proposed is to rethink the linux file system structure: one directory per package. In this way, the package management is simplified and can be done without the help of any fancy software.
have a look at http://www.gobolinux.org/
It’s not linux, but pcbsd has download and install from single file. Unlike klik, you can download file while in windows.
http://www.pcbsd.org/
Pardus linux and its “pisi” package system is great. Pardus software repositories contain all necessary software pre-packaged. They have quick and fast download servers.
http://www.pardus.org.tr/eng/index.html
I think that specifying versions of the core libraries is a bad idea. Such things nearly always have a lot of momentum, taking a long time for the standard to be updated to specify a new version. This would lead to stagnation and a considerable slowing of innovation.
I’ve tried Debian and Gentoo, and there’s really very little to choose between apt and portage. Debian packages are precompiled and require separate -dev packages; Gentoo ports download as source and compile on your machine. You can also download Debian packages in source form, though.
The only RPM-based distro I’ve tried was Mandriva, back when it was called Mandrake. I soon learned to forget RPM altogether and compile everything from source
IMHO, compiling the source code on your own machine (or maybe rather, having that automated for you; not that I think there is anything hard about typing “./configure”, “make”, “su” and “make install” in that order) is the way it should be done in future — binary packages belong at the bottom of the dustbin of history along with the problems they cause. If you could arrange it so that *only* programs compiled on your machine could ever be run on your machine, then it would be a lot harder for malware to spread.
Maybe the requirements of server administrators (who want full control of all policy decisions and local copies of everything. Ever heard of internal revision?) and desktop users (who want is easy and working, never mind that somebody else is making sure all my packages are up to date and working together) are too different. Why force a package manager on two seperate groups? May be the solution _is_ two very different systems (I currently would propose RPM and Portage)
By the way, the LSB allready has decided. It requires RPM to be installed. Exactly the reason why there is a rpm package for Debian.
In the server world, installing from source is not a solution, as soon as you have to supply various options and take manual steps to install the software. Not quickly reproducable and documentation on how it is done is allways out of date. The package definitions will document the steps to such a point that a computer can do it.
mfg lutz
RPM and DEB are comparable — APT-GET and whatever-redhat-produced are not. I prefer Debian-style. Gentoo’s way looks great to me, too, but I have not tried it so I can’t speak about it.
There is already a single, standard way of installing software across linux systems - compile it from source. Why people demand that Linux becomes more like Windoze is frustrating. They want it spoon fed, rather than just picking up a book and learning a couple of commands.
I’m a big fan of Ubuntu and the apt system (Debian based package manager). It’s proven itself to me over the last 18 months. I use that for an easy life.
Sounds like a copy of the article in FSM last month:
http://www.freesoftwaremagazine.com/blogs/how_to_hate_free_software_in_3_easy_steps
I may be a little biased here as I use gentoo but, as far as I can tell portage is the only solution that will work.
Standardising versions of core libraries and software is not an option as this means that the compile flags would have to be standardised, leaving bloated systems. Why should I have to install X11 to install php just because the distributer compiled php with xpm support? This always leads to problems with people installing with options like –nodeps, an option that should not exist in my optionion!
The only way is if the packages are compiled on the go. This gives you the ability to customise your flags, and in doing so, the requirements related. Each distro could still create use flag templates for different installation options eg. Desktop install would include x11 flag. They could also have default binaries to quicken the installation time using the same method portage uses for binary packages.
“[…] a lot of other systems like Ubuntu, Debian, Slackware, Gentoo or Linspire do not use the RPM format and do not plan to incorporate it.”
And why should they? Three of these 5 examples are in fact of the same ancestry: Debian, and thus they use the DPKG format. DPKG ist (in my experience) vastly more powerful than RPM, and that is exactly the reason the Debian-based distributions did not and will not adopt RPM, although there is now at least an apt port for RPM, afaik.
Gentoo, on the other hand, never used a binary package format as primary distribution system and probably never will, because the distribution’s philosophy always was, still is and will probably stay “use the source, Luke”. That is like taking a vegetarian as example of people not eating veal: nevermind that he does not eat meat at all, that sill means he is not eating veal, right?
I am also very puzzled at the positive bias towards click’n'run. CNR is a formerly commercial effort by Linspire, and thus it is not suprising it only used to support that distribution. Since Linspire switched to Ubuntu as its basis, naturally CNR will work there without major problems. What I do *not* see, however, is CNR getting wide acceptance. It is an apt-based system, so as an Ubuntu user able to use the “Adept” or “Synaptic” GUI, I gain almost nothing by using it. So why should I?
When you write about such an important topic you should include the current situation of the important foundations around Linux as well:
The LSB launched a new discussion and initiative around that topic in December: they put all important people (RPM, deb, etc.) in a room to share their ideas but also to identify the real problems. Ian Murdock wrote some pretty good articles about that and this article should have included at least links to that statements:
Software installation on Linux: Today, it sucks (part 1)
Software installation on Linux: Tomorrow, it won’t (with some cooperation) (part 2)
You probably won’t but the newbie users will. CNR is easier than Synaptic and Adept and it will be probably integrated with the website as well, so you will only need to click some link on the WWW to install an application. Does it remiond you of Windows? Surely it does and this is the intention.
CNR is planned to support RPM-based distros as well, including openSUSE, Fedora and Mandriva. No idea how they are going to achieve this, though, but if they do, it may be a great thing for the begginners — one unified GUI (system) for installing software in GNU/Linux, whichever distribution they try — seems attractive, doesn’t it?
I’m really surprised that Klik didn’t receive a mention here. I don’t know if anyone has cribbed about it in comments before me, but I think the author should have included at least a line about it as an update.
It actually is mentioned as one of the distribution-independent systems together with zero-install and autopackage.
To help with packaging, until the market changes it’s approach, a meta package process is probably the only thing that would work.
AMD’s driver installer has an embryonic form of that approach. The packaging scripts apply the distribution’s policies, and allow either package or tarball approach to installation.
If the distribution is going to package itself, you may as well have a policy layer that the distribution can feed into to allow the packaging to happen automagically based on a 2-level heuristic.
The application provider registers intent in the high level meta-package, the this is converted to the native packaging format below. Distribution vendors can ‘add value’ through affecting the lower section of the instal mechanism.
“One directory per package” might be a nice alternative to RPM, but it doesn’t solve half of the problems that advanced package managers like APT deal with. It’s not a matter of where stuff is installed — even unzipping an directory on windows, going into the directory, and running the exe on windows fixes that. The big problem is how it all interacts, especially over time — how 50 packages are installed that another one needs, what happens when you ask for two packages that have conflicting needs, how upgrades work without rebooting, how new versions of your OS can be installed, even if they have new architectures or policies, etc. APT handles all of this. It should be the Linux default, at least until something better comes along.
I too wish that linking is not (as far as possible) made to specific versions of libraries.
Linking should be made to library.so which acts as a symlink to the real file, maybe through various levels of link.
eg requires libmy, libmy.so -> libmy-2.so -> libmy-2.20.so
Now one can link to libmy.so, or libmy-2.so (major version level) and only to libmy-2.20.so if that particular patch version is needed.
I’m sick of packages not installing because I have libmy-2.21.so installed and software is linked to earlier version. (I can understand if it needed libmy-1.so but that can be installed independantly of version 2.)
You probably should have included the Python language-specific distribution scheme using “eggs” and easy_install. There are a lot of Python programs released for Linux, and many of them are distributed using eggs (which are compressed Python script modules). The rpm-building features of Python distutils are broken because of broken aspects of RPM.
I see two components that should take us a great leap forward.
1. Design and implement a protopackage infrastructure that will help concentrate as much distroagnostic packaging work in one place and that provides every feature the distros need to ease their packaging of the software.
2. Make packages depend on interfaces not on packages.
Symptom fixing syndrome. Before the Linux community could even think of unifying the package manager or install UI, we’d at least needed to “freeze” some APIs (future & backwards compatibility) and agree on package naming and splitting. RPM and DEB (the formats) are pretty similiar, the package managers are different (dpkg is more user-friendly, btw). It would be trivial to make rpm or dpkg support both RPM and DEB, or TGZ, and with some effort even “ports” or plain source tarballs later on. However, there are more important things to standardize first.
Checkinstall rules.
What about Gentoo’s Portage, and Debians Apt-get?
Those are two very fine package systems.
The solution is easy:
setup.exe (it’s Linux equivalent)
95% of desktop computer users (assuming this really IS Linux’s target market) understand, nay, expect this approach.
If Linux ever wants to move beyond the .5% desktop market share, it needs to think beyond techies and hackers.
Even your grandma understands “setup.exe”…if, like most of the world, she’s been using a Windows computer.
MrUbuntu
Extremely poor article. Adresses non of the issues of software complexity, and other issues like granularity and dependency issues.
There exist a slew of problems with Linux packaging. They are another layer of complexity and dependency on top of Make - this is the reason I prefer the BSD ports. Packaging system dogma brought Debian to a halt.
The fact is, long ago, Unix developers thought out the portability issues. Unix is pretty portable…
Debian developers, for instance, would declare something depending on package x.y.2 when in fact, all the functions used existed in package x.y.1.
A better and more advanced concept is PC-BSD’s PBI. With PBI-packaged software, you just drag-and-drop (like Windows or Macs).
What the packaging people need is a whole theory of packaging.
Believe it or not, some Linux users would actually enjoy to see some proprietary software being ported to Linux. However, software vendors don’t bother porting because the effort required is too big for the relatively small market share Linux has. If there was a unified BINARY way of installing software, then more vendors would do it.
For some people, a computer is about using the best tool for the job, and sometimes, a proprietary application is the best tool.
Bah, why would I want to run a wizard to install ONE single app? The Windows install philosophy plainly sucks. It’s hard to understand why users have to answer any questions at all except “what program do you want?”
Even grandma understands “Programs > Add/Remove…” in the menu of Ubuntu.
And about CNR: That’s hardly a “solution” to the proposed issues, since CNR just is a wrapper around DEB and apt.
The reason Autopackage and other nice propsals haven’t caught any market share is that it doesn’t go along with the rest of the distribution well enough. It could become a support nightmare.
As long as there is no uniform installer for Linux, It’s popularity will never grow! This is the only reason I have
not made the commitment to Linux. You can serf the net and
find lot’s of programs for Linux, and you think great! Then
you try to install your down load and the wailing and gnashing
of teeth begins. And like everybody else you switch back to Windows! If you can’t figure it out, how can you recomend Linux to anybody else?
What about ISVs? Software makers or distributors from six months to a year to get their products to market. By then, GNU/Linux distributions have changed so much that patches are often required just to make them install and function Furthermore, the lifecycle of a software product needs to be 3 to 5 years for profitability. And, the market is already split by only being able to support the top one or two GNU/Linux vendors at one or two versions, in most cases.
This issue has been long close to my heart. Personally, I lean toward Autopackage. The Linux kernel is backward compatible right down to version 1.0 and therefore including libs is all that’s needed for long-term viability. Requirements of a so packaged program, let’s say “Blender” should require only a certain minimum kernel version and an LSB-compliant platform.
Matthew C. Tedder
Anyone who is committed enough to learning how to use Linux will become proficient in the operating system and hopefully appreciate its uses. This article wouldn’t exist if “everybody else” was switching back to windows.
I heard about this package manager call dpkg, and its frontend call apt-get. It heard it works well. It’s just waiting for a “Major” vendor to use it before it make it into articles like this.
Uh Jim … dpkg / apt-get / aptitude etc. are tools used in conjunction with Debian packaging .. quite common with Debian like distro’s. Wonder why they didn’t mention it above as it works quite nicely (just like RPM).
Debian’s dpkg system is used by its derivatives (such as Ubuntu and Knoppix), all of these being major distributions. It can hold more information than an RPM, simplifying conflict-resolution, and is easily generated from the latter.
The diversity of installation methods is a strength rather than a weakness. None are difficult to use and all work effectively. A small proportion of people might find the variety of options confusing, but such people are probably better off with the severely limited functionality of Windows anyway.
“As long as there is no uniform installer for Linux, It’s popularity wi