Feeds:
Posts
Comments

Posts Tagged ‘Debian Packaging’

Earlier in the year, I wrote a similar article discussing the Catalyst Web Framework and the MojoMojo Wiki software. At the beginning of December 2009, I wrote an article which was published in the Catalyst Advent Calendar. I’m re-posting it here for posterity, and because it is still relevant to others today.

Introduction

Because Catalyst is a rapidly evolving project, packages supplied by operating system vendors like Debian, Fedora, Ubuntu, and many others have historically been outdated compared to the stable versions. In effect, this limited users of Debian’s package management system to outdated versions of this software.

In 2009, thanks to the efforts of Matt S Trout and many others, Debian’s Catalyst packages have been improving. The idea that Debian’s Perl packages are outdated is an idea that is itself becoming obsolete. There are many situations where system-wide Debian packages (and similarly, Ubuntu packages) can be preferable to installing software manually via CPAN.

Advantages

Here are some reasons why packages managed by Debian are preferable to installing packages manually:

  • Unattended installation: the majority of our packages require absolutely no user interaction during installation, in contrast to installs via CPAN.
  • Quicker installs for binary packages: since binary packages are pre-built, installing the package is as simple as unpacking the package and installing the files to the appropriate locations. When many modules need to be built (as with Catalyst and MojoMojo), this can result in a significant time savings, especially when one considers rebuilding due to upgrades.
  • No unnecessary updates: if an update only affects the Win32 platform, for example, it does not make sense to waste bandwidth downloading and installing it. Our process separates packages with bugfixes and feature additions from those that have no functional difference to users, saving time, bandwidth, and administrative overhead.
  • Only packages offered by Debian are supported by Debian: if there are bugs in your Debian software, it is our responsibility to help identify and correct them. Often this means coordinating with the upstream software developers (i.e. the Catalyst community) and working toward a solution together – but our team takes care of this on your behalf.
  • Updates occur with the rest of your system: while upgrading your system using aptitude, synaptic, or another package management tool, your Perl packages will be updated as well. This prevents issues where a system administrator forgets to update CPAN packages periodically, leaving your systems vulnerable to potential security issues.
  • Important changes are always indicated during package upgrades: if there are changes to the API of a library which can potentially break applications, a supplied Debian.NEWS file will display a notice (either in a dialog box or on the command line) indicating these changes. You will need to install the “apt-listchanges” utility to see these.

This year has seen greatly improved interaction between the Debian Perl Group and the Catalyst community, which is a trend we’d like to see continue for many years to come. As with any open source project, communicating the needs of both communities and continuing to work together as partners will ultimately yield the greatest benefit for everyone.

Disadvantages

As with all good things, there are naturally some situations where using Debian Perl packages (or, indeed, most operating-system managed packages) is either impossible, impractical, or undesirable.

  • Inadequate granularity: due to some restrictions on the size of packages being uploaded into Debian, there are plenty of module “bundles”, including the main Catalyst module bundle (libcatalyst-modules-perl). Unfortunately, this means you may have more things installed than you need.
  • Not installable as non-root: if you don’t have root on the system, or a friendly system administrator, you simply cannot install Debian packages, let alone our Perl packages. This can add to complexity for shared hosting scenarios where using our packages would require some virtualization.
  • Multiple versions: with a solution like local::lib, it’s possible to install multiple versions of the same package in different locations. This can be important for a number of reasons, including ease of testing and to support your legacy applications. With operating-system based packages, you will always have the most recent version available (and if you are using the stable release, you will always have the most recent serious bug/security fixes installed).
  • Less useful in a non-homogeneous environment: if you use different operating systems, it can be easier to maintain a single internal CPAN mirror (especially a mini-CPAN installation) than a Debian repository, Ubuntu repository, Fedora/RedHat repository, etc.

For my purposes, I use Debian packages for everything because the benefits outweigh the perceived costs. However, this is not the case for everyone in all situations, so it is important to understand that Debian Perl packages are not a panacea.

Quality Assurance

The Debian Perl Group uses several tools to provide quality assurance for our users. Chief among them is the Package Entropy Tracker (PET), a dashboard that shows information like the newest upstream versions of modules. Our bug reports are available in Debian’s open bug reporting system.

If you have any requests for Catalyst-related modules (or other Perl modules) that you’d like packaged for Debian, please either contact me directly (via IRC or email) or file a “Request For Package” (RFP) bug. If you have general questions or would like to chat with us, you’re welcome to visit us at any time – we hang around on irc.debian.org, #debian-perl.

See Also

  • Our IRC channel, irc.debian.org (OFTC), channel #debian-perl
  • Package Entropy Tracker is a dashboard where we can see what needs to be updated. It allows us (and others, if interested!) to easily monitor our workflow, and also contains links to our repository.
  • Our welcome page talks about what we do and how you (yes you!) can join. You don’t need to be a Debian Developer to join the group (actually, I’m not yet a DD and yet I maintain 300+ packages through the group).
  • This guide explains how to file a Request For Package (RFP) bug, so that the modules you use can be added to the Debian archive. Note that Debian is subject to many restrictions, so issues like inadequate copyright information may prevent the package from entering the archive.

Statistics

Here are some statistics of note:

  • We maintain over 1400+ packages as of today. For details, see our Quality Assurance report
  • We have quite a few active members; probably around 10 or 20

Acknowledgments

Thanks to Matt S Trout (mst) for working so closely with the group to help both communities achieve our goal of increasing Catalyst’s profile. Also thanks to Bogdan Lucaciu (zamolxes) for inviting us to contribute this article, and Florian Ragwitz (rafl) for his review and feedback.

Everything that is good in nature comes from cooperation. Neither Catalyst, nor Perl, nor Debian Perl packages could exist without the contributions of countless others. We all stand on the shoulders of giants.

Advertisements

Read Full Post »

I’ve recently been pushing for greater support for Catalyst and MojoMojo on Debian. For the uninitiated, Catalyst is a Model-View-Controller Framework designed for writing web applications. MojoMojo is a Wiki application based on Catalyst that provides a lot of neat features; while it seems less popular than Wikimedia’s MediaWiki software, it’s still got plenty of features other wikis don’t.

Here’s a blurb about it from their homepage:

We also have a bunch of features you won’t find in every wiki, like an attachment system that automatically makes a web gallery of your photos, live AJAX previews as you are editing your text, and a proper full text search engine built straight into the software.

Unfortunately, such a rich feature set comes at a price — this shiny piece of software has a rather large dependency chain. As a result, building the module (after building its prerequisites) from CPAN is both slow and prone to failure, since each module must be individually retrieved, extracted, built, tested and then installed.

To make matters worse, any failure anywhere in the chain (perhaps a new version of a module breaks things) will cause a complete failure to build the module — either Catalyst or MojoMojo — which has some serious implications for production applications.

In Debian, we mitigate this risk by having separate unstable and testing distributions, so if a newer version breaks things in unstable, we will catch it and have a chance to fix it before the package makes it into testing. By packaging these modules for Debian, we get the advantages of a faster installation process (since we’re installing pre-built binaries) combined with better Quality Assurance.

One of the big issues blocking both of these has been missing copyright information for a lot of modules. I’ve worked a lot with Matt S. Trout, one of the primary people behind coordinating the efforts of the Catalyst project, and gathered the necessary information for an upgrade and upload into Debian.

Recently, libcatalyst-modules-perl (version 35) and libcatalyst-modules-extra-perl (version 4) were uploaded to Debian, containing many necessary updates and fixes to improve the Catalyst experience on Debian. The next big push is to get MojoMojo’s dependencies packaged (currently only String::Diff is blocking it, due to missing copyright information).

A bounty of $150 is being offered by one of the MojoMojo developers to the first person who can re-implement the String::Diff functionality in a free/open source way.

Read Full Post »

Okay, so this is a long-awaited follow-up to my first post on the topic of  Debian Perl Packaging. Some of you might note I was pretty extreme in the first post, which is partially because people only really ever respond to extremes when they’re new to things. When you first begin programming, the advice you get is “hey, never use goto statements” — but as your progress in your ability and your understanding of how it works, what it’s actually doing in the compiler — then it might not be so bad after all. In fact, I hear the Linux kernel uses it extensively to provide Exceptions in C. The Wikipedia page on exception handling in various languages shows how to implement exceptions in C using setjmp/longjmp (which is essentially a goto statement). But I digress.

Back to the main point of this writeup. Previously I couldn’t really think of cases where packaging your own modules is really all that useful, especially when packaging them for Debian means that you benefit many communities — Debian, Ubuntu, and all of the distributions that are based on those.

During a discussion with Hans Dieter Piercey after his article providing a nice comparison between dh-make-perl and cpan2dist. (Aside: I feel like he was slightly biased toward cpan2dist in his writeup, but I’m myself biased toward dh-make-perl, so he might be right, even though I won’t admit it.)

I’m really glad for that article and the ensuing dialog, because it really got people talking about what they use Debian Perl packages for, and where it is useful to make your own.

Firstly, if you’ve got an application that depends on some Perl module that isn’t managed by Debian, but you need it yesterday, then you can either install that module via CPAN or roll your own Debian package. The idea here is to make and install the package so you can use it, but also file a Request For Package bug at the same time — see the reportbug command in Debian, or use LaunchPad if you’re on Ubuntu. This way, when the package is officially released and supported, you can move to that instead, and thus get the benefits of automatic upgrades of those packages.

Secondly, if you’ve got an application that depends on some internally-developed modules, then they probably wouldn’t exist on CPAN (some call this Perl code part of the DarkPAN), except in the rare case that a company open sources their work. But corporations will never open source all of their work, even if they consider the implications of providing some of it to the open source community, so at some point or another you’ll need to deal with internal packages. Previously, the best way to handle this was to construct your own CPAN local mirror, and have other machines install and upgrade from it — thus your internal code is easily distributed via the usual mechanism.

One of the advantages of using CPAN to distribute things is that it’s available on most platforms, builds things and runs tests automatically on many platforms. CPANPLUS will even let you remove packages, which was one of the main reasons I am so pro-Debian packages anyway. However, it does mean you’ll need to rebuild the package on other systems, which is prone to failures that cost time and money to track down and fix. CPAN and CPANPLUS are the Perl tradition of distributing packages.

If you are using an environment mostly with Debian systems, however, you may benefit from using a local Debian repository. This way, you only need to upgrade packages in your repository, and they’ll be automatically upgraded along with the rest of your operating system (you do run update and upgrade periodically right?). There is even the fantastic aptcron program to automate this, so there’s really no excuse not to automatically update.

In either case, creating a local package means you will be able to easily remove anything you no longer need via the normal package management tools. You can also distribute the binary packages between machines — though it sometimes depends on the platform (for modules that incorporate C or other platform-specific code that needs to be rebuilt). Generally, most Perl modules are Pure Perl, and thus you can compile and test it once, on one machine, and distribute it to other ones simply by installing the .deb package on other machines. You can copy packages to machines and use dpkg to install them, or better yet, create a local Debian mirror so it’s done automatically and via the usual mechanism (aptitude, etc.)

In conclusion: if you’re going to make your own Debian packages, do so with caution, and be aware of all the consequences (positive and negative) of what you’re doing. As always, a real understanding of everything is necessary.

Read Full Post »

One thing that makes Perl different from many other languages is that it has a rather small collection of core commands. There are only a few hundred commands in Perl itself, so the rest of its functionality comes from its rich collection of modules,  many of which are distributed via the Comprehensive Perl Archive Network (CPAN).

When CPAN first came on the scene, it preceded many modern package management systems, including Debian’s Advanced Packaging Tool (APT) and Ruby’s gem system, among others. As a consequence of its rich history, the CPAN Shell is relatively simplistic by today’s standards, yet still continues to get the job done quite well.

Unfortunately, there are two issues with CPAN:

  1. Packages are distributed as source code which is built on individual machines when installing or upgrading packages.
    • Since packages must be re-built on every machine that installs it, the system is prone to breaking and wastes CPU time and other resources. (The CPAN Testers system is a great way module authors can try to mitigate this risk, though.)
    • Due to wide variation in packages, many packages cause problems with the host operating system in terms of where they install files, or expect them to be installed. This is because CPAN does not (and cannot) know every environment that packages will be installed on.
  2. It does not integrate nicely with package managers
    • The standard CPAN Shell is not designed to remove modules, only install them. Removals need to be done manually, which is prone to human error such as forgetting to clean up certain files, or breaking other installs in the process.
    • It cannot possibly know the policies that govern the various Linux flavours or Unices. This means that packages might be installed where users do not expect, which violates the Principle of Least Surprise.
    • It is a separate ecosystem to maintain. When packages are updated via the normal means (eg, APT), packages installed via CPAN will be left alone (ie, not upgraded).

Here is the real problem: packages installed via CPAN will be left alone. This means that if new releases come out, your system will retain an old copy of packages, until you get into the CPAN Shell and upgrade it manually. If you’re administrating your own system, this isn’t a big problem — but it has significant implications for collections of production systems. If you are managing thousands of servers, then you will need to run the upgrade on each server, and hope that the build doesn’t break (thus requiring your, or somebody else’s, intervention).

One of the biggest reasons to select Debian is because of one of its primary design goal: to be a Universal Operating System. What this means is that the operating system should run on as many different platforms and architectures as possible, while providing the same rich environment to each of them to the greatest extent possible. So, whether I’m using Debian GNU/Linux x86 or Debian GNU/kFreeBSD x64, I have access to the same applications, including the same Perl packages. Debian has automated tools to build and test packages on every architecture we support.

The first thing I’m going to say is: if you are a Debian user, or a user of its derivatives, there is absolutely no need for you to create your own packages. None. Just don’t do it; it’s bad. Avoid it like the goto statement, mmkay?

If you come across a great CPAN package that you’d really like to see packaged for Debian, then contact the Debian Perl Packagers (pkg-perl) team, and let us know that you’d like a package. We currently maintain well over a thousand Perl packages for Debian, though we are by no means the only maintainers of Perl packages in Debian. You can do this easily by filing a Request For Package (RFP) bug using the command: reportbug wnpp.

On-screen prompting will walk you through the rest, and we’ll try to package the module as quickly as possible. When we’re done, you’ll receive a nice e-mail letting you know that your package has been created, thus closing the bug. A few days of waiting, but you will have a package in perfect working condition as soon as we can create it for you. Moreover, you’re helping the next person that seeks such a module, since it will already be available in Debian (and in due time it will propagate to its derivatives, like Ubuntu).

All 25,000+ Debian packages meet the rigorous requirements of Debian Policy. The majority of them meet the Debian Free Software Guidelines (DFSG), too; the ones which are not considered DFSG-free are placed in their own repository, separate from the rest of packages. A current work in progress is machine-parseable copyright control files, which will hopefully provide a way for administrators to quickly review licensing terms of all the software you install. This is especially important for small- and medium-sized businesses without their own intellectual property legal departments to review open source software, which is something that continues to drive many businesses away from using open source.

For the impatient, note this well: packages which are not maintained by Debian are not supported by Debian. This means that if you install something using a packaging tool (we’ll discuss these later) or via CPAN, then your package is necessarily your own responsibility. In the unlikely event that you totally break your system installing a custom package, it’s totally your fault, and it may mean you will have to restore an earlier backup or re-install your system completely. Be very careful if you decide to go this route. A few days waiting to ensure that your package will work on every platform you’re likely to encounter is worth the couple days of waiting for a package to be pushed through the normal channels.

The Debian Perl Packaging group offers its services freely to the public for the benefit of our users. It is much better to ask the volunteers (preferably politely) to get your package in Debian, so that it passes through the normal testing channels. You really should avoid making your own packages in a vacuum; the group is always open to new members, and it means your package will be reviewed (and hopefully uploaded into Debian) by our sponsors.

But the thing about all rules is that there are always exceptions. There are, in fact, some reasons when you might want to produce your own packages. I was discussing this with Hans Dieter Pearcey the other day, and he has written a great follow-up blog post about the primary differences between dh-make-perl and cpan2dist, two packaging tools with a similar purpose but very different design goals. Another article is to follow this one, where I will discuss the differences between the two.

Read Full Post »