Feeds:
Posts
Comments

Posts Tagged ‘Debian’

Last year, I had a great time participating in the Google Summer of Code with the Debian project. I had a neat project with some rather interesting implications for helping developers to package and maintain their work. It’s still a work-in-progress, of course, as many projects in open source are, but I was able to accomplish quite a bit and am proud of my work. I learned quite a bit about coding in C, working with Debian and met some very intelligent people.

My student peers were also very intelligent and great to learn from. I enjoyed meeting them virtually and discussing our various projects on the IRC channel as the summer progressed and the Summer of Code kicked into full swing. The Debian project in particular also helps arrange travel grants for students to attend the Debian Conference (this year, DebConf10 is being held in New York City!). DebConf provides a great venue to learn from other developers (both in official talks but also unofficial hacking sessions). As the social aspect is particularly important to Debian, DebConf helps people meet those with whom they are working with the most, thereby creating lifelong friendships and making open source fun.

I have had several interviews for internships, and the bit of my work experience most asked about is my time doing the Google Summer of Code. I really enjoyed seeing a project go from the proposal stage, setting a reasonable timeline with my mentor, exploring the state of the art, and most importantly, developing the software. I think this is the sort of indispensible industry-type experience we often lack in our undergrad education. We might have an honours thesis or presentation, but much of the work in the Google Summer of Code actually gets used “in the field.”

Developing software for people rather than for marks is significant in a number of ways, but most importantly it means there are real stakeholders that must be considered at all stages. Proposing brilliant new ideas is important, however, without highlighting the benefits they can have for various users, the reality is that it simply will not gain traction. Learning how to write proposals effectively is an important skill and working with my prospective mentor (at the time – he later mentored my project once it was accepted) to develop mine was tremendously useful for my future endeavours.

The way I see the Google Summer of Code is, in many ways, similar to an academic grant (and the stipend is about the same as well). It provides a modest salary (this year it’s US$5000) but more importantly, personal contact with a mentor. Mentors are typically veterans in software development or the Debian project and act in the same role as supervisors for post-graduate work: they help monitor your progress and propose new ideas to keep you on track.

The Debian Project is looking for more students and proposals. We have a list of ideas as well as application instructions available on our Wiki. As I will be going on internship starting May, I have offered to be a mentor this year. I look forward to seeing your submissions (some really interesting ones have already begun to filter in as the deadline approaches).

Read Full Post »

Earlier in the year, I wrote a similar article discussing the Catalyst Web Framework and the MojoMojo Wiki software. At the beginning of December 2009, I wrote an article which was published in the Catalyst Advent Calendar. I’m re-posting it here for posterity, and because it is still relevant to others today.

Introduction

Because Catalyst is a rapidly evolving project, packages supplied by operating system vendors like Debian, Fedora, Ubuntu, and many others have historically been outdated compared to the stable versions. In effect, this limited users of Debian’s package management system to outdated versions of this software.

In 2009, thanks to the efforts of Matt S Trout and many others, Debian’s Catalyst packages have been improving. The idea that Debian’s Perl packages are outdated is an idea that is itself becoming obsolete. There are many situations where system-wide Debian packages (and similarly, Ubuntu packages) can be preferable to installing software manually via CPAN.

Advantages

Here are some reasons why packages managed by Debian are preferable to installing packages manually:

  • Unattended installation: the majority of our packages require absolutely no user interaction during installation, in contrast to installs via CPAN.
  • Quicker installs for binary packages: since binary packages are pre-built, installing the package is as simple as unpacking the package and installing the files to the appropriate locations. When many modules need to be built (as with Catalyst and MojoMojo), this can result in a significant time savings, especially when one considers rebuilding due to upgrades.
  • No unnecessary updates: if an update only affects the Win32 platform, for example, it does not make sense to waste bandwidth downloading and installing it. Our process separates packages with bugfixes and feature additions from those that have no functional difference to users, saving time, bandwidth, and administrative overhead.
  • Only packages offered by Debian are supported by Debian: if there are bugs in your Debian software, it is our responsibility to help identify and correct them. Often this means coordinating with the upstream software developers (i.e. the Catalyst community) and working toward a solution together – but our team takes care of this on your behalf.
  • Updates occur with the rest of your system: while upgrading your system using aptitude, synaptic, or another package management tool, your Perl packages will be updated as well. This prevents issues where a system administrator forgets to update CPAN packages periodically, leaving your systems vulnerable to potential security issues.
  • Important changes are always indicated during package upgrades: if there are changes to the API of a library which can potentially break applications, a supplied Debian.NEWS file will display a notice (either in a dialog box or on the command line) indicating these changes. You will need to install the “apt-listchanges” utility to see these.

This year has seen greatly improved interaction between the Debian Perl Group and the Catalyst community, which is a trend we’d like to see continue for many years to come. As with any open source project, communicating the needs of both communities and continuing to work together as partners will ultimately yield the greatest benefit for everyone.

Disadvantages

As with all good things, there are naturally some situations where using Debian Perl packages (or, indeed, most operating-system managed packages) is either impossible, impractical, or undesirable.

  • Inadequate granularity: due to some restrictions on the size of packages being uploaded into Debian, there are plenty of module “bundles”, including the main Catalyst module bundle (libcatalyst-modules-perl). Unfortunately, this means you may have more things installed than you need.
  • Not installable as non-root: if you don’t have root on the system, or a friendly system administrator, you simply cannot install Debian packages, let alone our Perl packages. This can add to complexity for shared hosting scenarios where using our packages would require some virtualization.
  • Multiple versions: with a solution like local::lib, it’s possible to install multiple versions of the same package in different locations. This can be important for a number of reasons, including ease of testing and to support your legacy applications. With operating-system based packages, you will always have the most recent version available (and if you are using the stable release, you will always have the most recent serious bug/security fixes installed).
  • Less useful in a non-homogeneous environment: if you use different operating systems, it can be easier to maintain a single internal CPAN mirror (especially a mini-CPAN installation) than a Debian repository, Ubuntu repository, Fedora/RedHat repository, etc.

For my purposes, I use Debian packages for everything because the benefits outweigh the perceived costs. However, this is not the case for everyone in all situations, so it is important to understand that Debian Perl packages are not a panacea.

Quality Assurance

The Debian Perl Group uses several tools to provide quality assurance for our users. Chief among them is the Package Entropy Tracker (PET), a dashboard that shows information like the newest upstream versions of modules. Our bug reports are available in Debian’s open bug reporting system.

If you have any requests for Catalyst-related modules (or other Perl modules) that you’d like packaged for Debian, please either contact me directly (via IRC or email) or file a “Request For Package” (RFP) bug. If you have general questions or would like to chat with us, you’re welcome to visit us at any time – we hang around on irc.debian.org, #debian-perl.

See Also

  • Our IRC channel, irc.debian.org (OFTC), channel #debian-perl
  • Package Entropy Tracker is a dashboard where we can see what needs to be updated. It allows us (and others, if interested!) to easily monitor our workflow, and also contains links to our repository.
  • Our welcome page talks about what we do and how you (yes you!) can join. You don’t need to be a Debian Developer to join the group (actually, I’m not yet a DD and yet I maintain 300+ packages through the group).
  • This guide explains how to file a Request For Package (RFP) bug, so that the modules you use can be added to the Debian archive. Note that Debian is subject to many restrictions, so issues like inadequate copyright information may prevent the package from entering the archive.

Statistics

Here are some statistics of note:

  • We maintain over 1400+ packages as of today. For details, see our Quality Assurance report
  • We have quite a few active members; probably around 10 or 20

Acknowledgments

Thanks to Matt S Trout (mst) for working so closely with the group to help both communities achieve our goal of increasing Catalyst’s profile. Also thanks to Bogdan Lucaciu (zamolxes) for inviting us to contribute this article, and Florian Ragwitz (rafl) for his review and feedback.

Everything that is good in nature comes from cooperation. Neither Catalyst, nor Perl, nor Debian Perl packages could exist without the contributions of countless others. We all stand on the shoulders of giants.

Read Full Post »

I’ve recently been pushing for greater support for Catalyst and MojoMojo on Debian. For the uninitiated, Catalyst is a Model-View-Controller Framework designed for writing web applications. MojoMojo is a Wiki application based on Catalyst that provides a lot of neat features; while it seems less popular than Wikimedia’s MediaWiki software, it’s still got plenty of features other wikis don’t.

Here’s a blurb about it from their homepage:

We also have a bunch of features you won’t find in every wiki, like an attachment system that automatically makes a web gallery of your photos, live AJAX previews as you are editing your text, and a proper full text search engine built straight into the software.

Unfortunately, such a rich feature set comes at a price — this shiny piece of software has a rather large dependency chain. As a result, building the module (after building its prerequisites) from CPAN is both slow and prone to failure, since each module must be individually retrieved, extracted, built, tested and then installed.

To make matters worse, any failure anywhere in the chain (perhaps a new version of a module breaks things) will cause a complete failure to build the module — either Catalyst or MojoMojo — which has some serious implications for production applications.

In Debian, we mitigate this risk by having separate unstable and testing distributions, so if a newer version breaks things in unstable, we will catch it and have a chance to fix it before the package makes it into testing. By packaging these modules for Debian, we get the advantages of a faster installation process (since we’re installing pre-built binaries) combined with better Quality Assurance.

One of the big issues blocking both of these has been missing copyright information for a lot of modules. I’ve worked a lot with Matt S. Trout, one of the primary people behind coordinating the efforts of the Catalyst project, and gathered the necessary information for an upgrade and upload into Debian.

Recently, libcatalyst-modules-perl (version 35) and libcatalyst-modules-extra-perl (version 4) were uploaded to Debian, containing many necessary updates and fixes to improve the Catalyst experience on Debian. The next big push is to get MojoMojo’s dependencies packaged (currently only String::Diff is blocking it, due to missing copyright information).

A bounty of $150 is being offered by one of the MojoMojo developers to the first person who can re-implement the String::Diff functionality in a free/open source way.

Read Full Post »

One of the most often overlooked–yet arguably most important–issues in software development is copyright and licensing of works. In particular, I will discuss how this affects the open source software community with relevance to the Debian project.

As with any artistic or creative works, software is protected by copyright and its use is often governed by some sort of license. Please note that I am not a lawyer and I am not qualified to give legal advice, so take my suggestions with a grain of salt and please do leave a comment if you know something that I don’t.

A license is a legal contract that permits end users use of software under agreed-upon guidelines. In the open source community, licenses protect the integrity of free software by ensuring that they continue to remain freely available. For example, the GNU General Public License (GPL) stipulates that any derivative works of GPL-licensed code must distribute source code back to the community, which enables a two-way sharing of information between the originating software developers and the others who benefit from their work. Other licenses, such as the BSD License, are more liberal and do not have this restriction, but do have a disclaimer of warranties which shields authors from unintended legal consequences of their work.

Though licensing is probably the most important document detailing the relationship of the supplier (software developer or team) and other users, it cannot mean anything without copyright. In general, it is most useful to provide a copyright statement somewhere in resulting packages. A copyright statement is what allows authors to assert a particular license in the first place.

Moreover, license terms can only be changed when all copyright holders agree to the change. Unless you are explicit in your copyright conditions in the beginning, this can lock your project in to an undesirable license.

To make matters even more complicated, the Berne Convention for the Protection of Literary and Artistic Works (or simply Berne Convention as it’s most often called) describes a mechanism by which copyright is automatically in force upon creation of a work, even if the author does not explicitly assert it. For software, this effectively means that anyone who contributes any code is automatically the copyright holder on their contribution, which means that things quickly get complicated when there are many authors and contributors involved.

In Debian, we cannot and do not distribute software without knowing copyright information (including years of copyright, names, e-mail addresses where people can be reached, or a web site in the case of an incorporated entity). This is pursuant to the Debian Free Software Guidelines (DFSG), which require that we distribute only “free” software in our main repository–it’s part of our Social Contract.

In this regard, I would make the following recommendations:

  1. When beginning any project (open source or not), include a copyright statement immediately. It will eventually become a force of habit; this is a very good thing, and will pay dividends in the future.
  2. Establish a policy whereby contributors are asked to assign you copyright of their work; make a note of this somewhere in your documentation. Better yet, if you are part of an incorporated entity, assign copyright to that entity.
  3. Be explicit about your licensing terms and make sure to include copies of the license with your software. This helps to resolve ambiguities where there are several derivatives of a license (occasionally, developers license software under the BSD License without specifying which version they mean)
  4. Be wary of the “Public Domain” — this is an even more contentious issue than choosing an appropriate license. It is probably preferable to use a non-restrictive license such as the aforementioned BSD License (and its variants) or the MIT/X11 license, which is even more permissive.

Read Full Post »

Okay, so this is a long-awaited follow-up to my first post on the topic of  Debian Perl Packaging. Some of you might note I was pretty extreme in the first post, which is partially because people only really ever respond to extremes when they’re new to things. When you first begin programming, the advice you get is “hey, never use goto statements” — but as your progress in your ability and your understanding of how it works, what it’s actually doing in the compiler — then it might not be so bad after all. In fact, I hear the Linux kernel uses it extensively to provide Exceptions in C. The Wikipedia page on exception handling in various languages shows how to implement exceptions in C using setjmp/longjmp (which is essentially a goto statement). But I digress.

Back to the main point of this writeup. Previously I couldn’t really think of cases where packaging your own modules is really all that useful, especially when packaging them for Debian means that you benefit many communities — Debian, Ubuntu, and all of the distributions that are based on those.

During a discussion with Hans Dieter Piercey after his article providing a nice comparison between dh-make-perl and cpan2dist. (Aside: I feel like he was slightly biased toward cpan2dist in his writeup, but I’m myself biased toward dh-make-perl, so he might be right, even though I won’t admit it.)

I’m really glad for that article and the ensuing dialog, because it really got people talking about what they use Debian Perl packages for, and where it is useful to make your own.

Firstly, if you’ve got an application that depends on some Perl module that isn’t managed by Debian, but you need it yesterday, then you can either install that module via CPAN or roll your own Debian package. The idea here is to make and install the package so you can use it, but also file a Request For Package bug at the same time — see the reportbug command in Debian, or use LaunchPad if you’re on Ubuntu. This way, when the package is officially released and supported, you can move to that instead, and thus get the benefits of automatic upgrades of those packages.

Secondly, if you’ve got an application that depends on some internally-developed modules, then they probably wouldn’t exist on CPAN (some call this Perl code part of the DarkPAN), except in the rare case that a company open sources their work. But corporations will never open source all of their work, even if they consider the implications of providing some of it to the open source community, so at some point or another you’ll need to deal with internal packages. Previously, the best way to handle this was to construct your own CPAN local mirror, and have other machines install and upgrade from it — thus your internal code is easily distributed via the usual mechanism.

One of the advantages of using CPAN to distribute things is that it’s available on most platforms, builds things and runs tests automatically on many platforms. CPANPLUS will even let you remove packages, which was one of the main reasons I am so pro-Debian packages anyway. However, it does mean you’ll need to rebuild the package on other systems, which is prone to failures that cost time and money to track down and fix. CPAN and CPANPLUS are the Perl tradition of distributing packages.

If you are using an environment mostly with Debian systems, however, you may benefit from using a local Debian repository. This way, you only need to upgrade packages in your repository, and they’ll be automatically upgraded along with the rest of your operating system (you do run update and upgrade periodically right?). There is even the fantastic aptcron program to automate this, so there’s really no excuse not to automatically update.

In either case, creating a local package means you will be able to easily remove anything you no longer need via the normal package management tools. You can also distribute the binary packages between machines — though it sometimes depends on the platform (for modules that incorporate C or other platform-specific code that needs to be rebuilt). Generally, most Perl modules are Pure Perl, and thus you can compile and test it once, on one machine, and distribute it to other ones simply by installing the .deb package on other machines. You can copy packages to machines and use dpkg to install them, or better yet, create a local Debian mirror so it’s done automatically and via the usual mechanism (aptitude, etc.)

In conclusion: if you’re going to make your own Debian packages, do so with caution, and be aware of all the consequences (positive and negative) of what you’re doing. As always, a real understanding of everything is necessary.

Read Full Post »

It’s been some time since I re-installed Debian over my Kubuntu install, so I thought I’d discuss some reasons why I changed back to Debian, what my experience was like, and some learning opportunities.

One reason I made the switch was because there was a utility newly packaged for Debian, Frama-C, which was not available in Kubuntu at the time. It also frustrated me that I was having various frustrations with the installation, not the least of which was an unreliable and quite crashy KDE Plasma.

When I reinstalled this time, I picked the normal install but told it to install a graphical environment, which gave me a GNOME desktop environment. I actually rather like it, the setup didn’t ask too many questions and everything was set up perfectly. There was some minor tweaking, but it was all done by the easily accessible System menu and all the applets therein.

Now, I wanted to be able to use the server both as a virtual machine and as a physical dual-boot. This wasn’t working properly with GRUB-2, so I had to stay with version 1.96, which works rather well. I even spent some time making a pretty splashimage for it, which looks rather nice, even if I don’t see it all that often.

If I boot into the Virtual Machine, all the hardware is detected properly, and there aren’t even complaints about the fact that a bunch of hardware disappeared — certainly very good news if you decide to do something like move your hard drive to a different machine. Likewise, if I boot into the desktop, everything works well there too.

One issue I came across during the installation was having to teach Network-Manager how to configure my network interfaces. In my VMware NAT setup, there is no DHCP server, so the IP address, subnet and gateway information needs to be statically defined. Luckily, Network-Manager was able to do this based on the MAC address of the adapter — inside my virtual machine, it had a VMware-assigned static one. Through this, Network-Manager had an easy way to determine how to configure my network, and it works beautifully for Ethernet and Wireless (when Debian is running as the main operating system) and also for VMware NAT (when inside the virtual machine container).

Anyway, I have now been developing quite happily inside a Debian + GNOME desktop environment. The system runs fine even within a constrained environment, though I miss KDE’s setup with sudo; with GNOME, the option seems to be to have the root password entered every time privilege escalation is necessary. I don’t like using a root password — on my server system I don’t use the root password at all, and do everything I need to do via sudo. It’s okay for me because I log into the server with a private key and have disabled SSH password authentication for security reasons.

One thing that is still weird for me is that my system currently shows a time of 01:53 even though it is 23:57 in my timezone. Presumably the few minutes of difference is because the virtual machine clock and my system hardware clock aren’t synchronized perfectly, but more than that, I think it’s an issue with the Date applet somehow. I haven’t looked into this because the thing is running inside a virtual machine, so it doesn’t bother me much.

I have looked high and low to see where to change the time zone, and to my knowledge the system knows that it’s in the America/Toronto time zone. The house picture next to Timmins (the city I am in right now, though it doesn’t matter since the timezone is the same) seems to indicate to me that it’s set to the appropriate time zone.

I think it’s due to VMware synchronizing the virtual machine clock with my host machine clock. Windows (my host operating system) stores the time in the local format, which I believe Linux thinks is UTC. Still, it doesn’t explain the weird display it’s got going.

Someone noted last time that I didn’t make direct mention of which programs are only offered on Windows and not on Linux/etc, and that do not have reasonable replacement on these systems. Kapil Hari Paranjape noted that I was sounding somewhat like a troll by simply saying that I don’t think Linux is yet ready to replace my environment. Here was my reply:

Far from a troll, I’d really like Debian and Ubuntu, but moreso Linux in general, to improve at the pace it has been doing so. It’s made great progress since the last time I tried it out on my desktop, but I have to acknowledge that there are lots of rough edges right now that should be worked out.

One of the advantages of huge proprietary development organizations like Microsoft is that they have tons of developers and can implement new features at a relatively quick pace, even if they’re half-assed. Developers’ pride in the FOSS community prevents this overly quick pace of development in favour of more secure, more stable platforms. Which is a good thing, I think. But nonetheless it results in a “slower” development pace.

The applications I’m complaining about are things like:
– SolidWorks (a CAD tool for designing parts and assemblies, used in manufacturing and mechanical engineering)
– SpectrumSoft Micro-Cap (a version of software similar to PSpice used by my school)
– AutoCAD (another CAD tool)

Luckily this is changing, but only for the large & most popular distributions:
– MathWorks MATLAB (runs on Linux and Solaris, etc.)
– Wolfram Mathematica (which has versions for Linux and MacOS X)
– FEKO (runs on Linux and Solaris among others)

Anyway, I still consider SolidWorks to be a rather big program not supported on Linux, which is a big issue for those working on Civil Engineering programs. There are most probably others which are very domain-specific that I don’t even know about.

There is a nice matrix comparing cross-platform capabilities of CAD software: http://en.wikipedia.org/wiki/Comparison_of_CAD_software

Oh, one final thought: perhaps that KDE Recommends: should be moved to a Suggests: instead, on account of its heavy dependencies, requiring mysql-server installed on desktop machines.. WTF!

Oh, and on another note, I re-installed Debian using the non-expert Auto Install and it installed Gnome rather flawlessly, much like installing Ubuntu, which was pretty nice. So kudos to those who have been working on the main installer; it seems as though the advanced ones really give you some rope to hang yourself with, though :-)

Oh, and k3ninho told me that there is an initiative from the Ubuntu community called “100 Paper Cuts” to help fix small bugs like those I was complaining about. I hope this leads to an improved user experience, and I’d really like to see some of those changes propagated both upstream to Debian and upstream to the KDE folk.

During my install of Kubuntu + KDE, I felt that plasma was crashing more than Windows Explorer — it felt like the days when I was running Windows ME, and the shell would crash, and the system would restart it. Repeatedly. It’s exactly what seemed to happen with plasma. I’m not sure if it was something I screwed up during configuration (presumably so), but KDE was far too complicated for me to try and debug. It might also have been a result of me running my system within a fairly constrained virtual machine environment – the system only gets 768MB of RAM and no access to the actual graphics processor unit (since it’s virtualized).

Read Full Post »

Okay, so allow me to explain my personal development platform. I use Windows XP Professional as my primary operating system for various reasons, but I develop software for Linux (mostly because I like Linux, but also because I intend for a lot of my software to be used on my own servers, all of which can be considered Unix-like).

Because I prefer Windows for daily use, I’ve built various tools around the operating system and have become quite attached to them. I really like TortoiseSVN and Notepad++, for example. I’ve just gotten used to running Windows.

Anyway, I wanted to see if I could transition into using Linux for everything, and because I work in Debian and have servers running Debian, I installed Debian. I had some trouble at first, prompting me to change to Kubuntu, but later came back to Debian after realizing that the packages in unstable and testing that I’d come to love were under an entirely different process in Ubuntu, which is currently unknown to me.

So long Ubuntu, I’ve re-installed Debian and love it so far. One problem I’ve noticed is that Debian would not boot inside VMware Workstation under Windows, while Kubuntu was able to do so out-of-the-box. Instead, I got an error like this:

WARNING bootdevice may be renamed. Try root=/dev/hda2
Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/sda2 does not exist. Dropping to a shell!

Looking into it further, it turned out I could edit the command line options to use a different device name–the debug output was right, I had to use /dev/hda2 instead of /dev/sda2–and things would work. The system would boot up normally, and everything was good. I guess it had to do with my drives actually being SATA (which I suppose are treated like SCSI drives); whereas VMware Workstation (in my setup) presents them as simple IDE drives.

But then I got to wondering how Ubuntu totally got around this problem, and it turns out that Ubuntu uses UUID identifiers to point to a hard drive. After searching on Google, it turns out that using UUIDs is convenient because it can handle cases where drives are removed; especially with external drives, or even re-arranging the drives inside your computer for whatever reason.

Using the command “blkid” from Debian (do this as root), you can get the UUID of all the partitions. You can either reboot into Linux using your normal dual-boot method (selecting the option from GRUB, etc), or you can edit the boot options from GRUB explictly to use /dev/hda2 temporarily. Once you boot into your Debian installation, you can edit /boot/grub/menu.lst to use drive UUIDs instead of /dev names.

Instead of:

# kopt=root=/dev/hda1 ro

or similar, you can change this to:

# kopt=root=UUID=<info from blkid for /dev/hda1> ro

Then you can use ‘update grub’ to update grub’s automagic menu.lst information, so that the new root is recognized. This way, however your disk is installed, the UUID will be detected and the appropriate drive mounted/booted. I haven’t tried this, but I suppose this would help if you have an operating system on a bootable USB drive or something, to help you select those partitions (though I don’t know if GRUB will even detect them on a boot. I’m hoping so since most BIOSes these days support booting from USB Mass Storage Media).

Good luck, I hope this helps somebody out there :-)

Read Full Post »

Older Posts »