Thursday, December 23, 2010

explanation of a systemd.recipe

I have been wanting to write something about packaging for conary. So here is an line-by-line explanation of the systemd recipe that I made for Foresight Linux.

systemd is still under heavy development and my package is far from perfection, so this article is just to show you a typical conary recipe. It's not meant to be a packaging guide for systemd and the recipe is subject to change.

------------------

First let's see how it looks like.
  1 class Systemd(AutoPackageRecipe):
  2     name = 'systemd'
  3     version = '15'
  4  
  5     buildRequires = [
  6         # for autoreconf
  7         'autoconf:runtime',
  8         'automake:runtime',
  9 
 10         'audit:devel',
 11         'dbus:devel',
 12         'libcap:devel',
 13         'libxslt:runtime',
 14         'm4:runtime',
 15         'pam:devel',
 16         'pkgconfig:devel',
 17         'tcp_wrappers:devel',
 18         'udev:devel',
 19         'vala:runtime',
 20 
 21         # for gtk support
 22         'dbus-glib:devel',
 23         'gtk:devel',
 24         'libnotify:devel',
 25         ]
 26 
 27     # XXX our cryptsetup may be too old and doesn't provide libcryptsetup.pc
 28  
 29     extraConfig = (
 30         '--with-distro=foresight '
 31         '--with-rootdir= '
 32         )
 33 
 34     patches = [
 35         # backported from upstream, for vala and libnotify version check
 36         '0001-gnome-ask-password-agent-also-support-libnotify-0.7-.patch',
 37         '0002-Ensure-LIBNOTIFY07-conditional-is-always-set.patch',
 38         # patches for porting to foresight
 39         'foresight-linux-port.patch',
 40         'foresight-units.patch',
 41         # need to update util-linux (>2.18) to have 'fsck -l'
 42         'foresight-workaround-old-fsck.patch',
 43         ]
 44 
 45     extraSources = [
 46         'console.conf',
 47         ]
 48 
 49     def unpack(r):
 50         r.addArchive('http://www.freedesktop.org/software/systemd/')
 51         for patch in r.patches:
 52             r.addPatch(patch)
 53 
 54         for source in r.extraSources:
 55             r.addSource(source)
 56 
 57     def configure(r):
 58         r.Run('autoreconf') # XXX is autoreconf right?
 59         r.Configure(r.extraConfig)
 60 
 61     def policy(r):
 62         r.Install('console.conf', '%(sysconfdir)s/tmpfiles.d/console.conf')
 63 
 64         # /etc/xdg/systemd/user points to /etc/systemd/user
 65         r.ExcludeDirectories(exceptions='%(sysconfdir)s/systemd/user')
 66 
 67         # The unit files (plain-text) are in /lib/systemd/system, so no need to fix
 68         r.FixupMultilibPaths(exceptions='/lib/systemd/system/')
 69 
 70         # warning: DanglingSymlinks: symlink /usr/share/systemd/user/bluetooth.target
 71         # points from package systemd:data to systemd:lib
 72         r.ComponentSpec('lib', '%(datadir)s/systemd/user/.*.target')
 73 
 74         # Create SysV compatibility symlinks. systemctl/systemd are smart
 75         # enough to detect in which way they are called.
 76         r.Symlink('../bin/systemd', '/sbin/init')
 77         r.Symlink('../bin/systemctl', '/sbin/reboot')
 78         r.Symlink('../bin/systemctl', '/sbin/halt')
 79         r.Symlink('../bin/systemctl', '/sbin/poweroff')
 80         r.Symlink('../bin/systemctl', '/sbin/shutdown')
 81         r.Symlink('../bin/systemctl', '/sbin/telinit')
 82         r.Symlink('../bin/systemctl', '/sbin/runlevel')
 83 
 84         # Use graphical.target unconditionaly as default for now
 85         r.Symlink('/lib/systemd/system/graphical.target', '/etc/systemd/system/default.target')
 86 
 87         cmds = (
 88             '/sbin/halt',
 89             '/sbin/init',
 90             '/sbin/poweroff',
 91             '/sbin/reboot',
 92             '/sbin/runlevel',
 93             '/sbin/shutdown',
 94             '/sbin/telinit',
 95             )
 96 
 97         mans = (
 98             '%(datadir)s/man/man1/init.1.*',
 99             '%(datadir)s/man/man8/halt.8.*',
100             '%(datadir)s/man/man8/poweroff.8.*',
101             '%(datadir)s/man/man8/reboot.8.*',
102             '%(datadir)s/man/man8/runlevel.8.*',
103             '%(datadir)s/man/man8/shutdown.8.*',
104             '%(datadir)s/man/man8/telinit.8.*',
105             )
106 
107         for f in cmds:
108             r.DanglingSymlinks(exceptions=f)
109 
110         for f in cmds + mans:
111             r.PackageSpec('systemd-sysvinit', f)
------------------

Now let's break the recipe down into pieces.

1. First of all, the recipe is called systemd.recipe and named after the package.

2. Line 1
class Systemd(AutoPackageRecipe):
Most recipes get started like this. Remember that conary recipes are pure Python code. So here it's a Python "class" that's being defined. Its name is "Systemd" and it is inheriting from one super-class called "AutoPackageRecipe".

The class name is in CamelCase and derived from the package name. Conary will do the transformation for you if the recipe is created using a template:
cvc newpkg systemd --template foresight
The "foresight" template (/etc/conary/recipeTemplates/foresight) is defined as:
class %(className)s(AutoPackageRecipe):
"className" will become "Systemd" for the systemd package.

As for the "%()s" surrounding className, it's (the first argument to) Python's "string interpolation operator". Read the official documentation (http://docs.python.org/library/stdtypes.html#string-formatting-operations) for details.

But there is something strange. The % operator is defined as "format % values" and requires two operands. We have got the format "%()s", but where is the values?

Values will be provided by conary automatically. That's how "macros" work in recipes. As in C, macros will come in handy sometimes. See http://docs.rpath.com/conary/Conaryopedia/sect-macros.html for details.

3. Line 2 to 25.
name = 'systemd'
version = '15'
buildRequires = ['autoconf:runtime', 'automake:runtime', 'audit:devel', ...]
These are the variables that will appear in most recipes and they are built into conary (for example conary has commands to query a package's name, version and buildRequires). Their meanings are obvious: package name, package version, and what you need to build this package. It's almost the same as in RPM .spec files.

4. Line 29 to 47.
extraConfig = ()
patches = []
extraSources = []
These are the variables that I define for my own convenience. Later part of the recipe will make use of them.

For now I only want to point out one thing. extraConfig seems to be a tuple, but it's actually a string. Not sure what's Python's official description but I use it as a trick to make the code look better.

5. methods
def unpack(r):
def configure(r):
def policy(r):
Now we get to some Python functions. Each of them controls a part of the packaging process (extract the tarball, call "./configure", call "make", call "make install", etc).

But where is the function that controls the whole process? It would be the job of setup() and it's being defined in the super-class AutoPackageRecipe. The class Systemd got it through inheritance.

The class AutoPackageRecipe (defined in the package "autopackage") has such methods:
 7     def setup(r):
 8         r.unpack()
 9         r.configure()
10         r.make()
11         r.makeinstall()
12         r.policy()
13 
14     def unpack(r):
15         pass
16     def configure(r):
17         r.Configure()
18     def make(r):
19         r.Make()
20     def makeinstall(r):
21         r.MakeInstall()
22     def policy(r):
23         pass
setup() is the only function that's required by conary to be defined in a recipe and determines how the packaging is done. But thanks for the recipe being Python code, you don't actually have to do it. It's sufficient to just override a few portions of setup(), where special processing is truly required. The other subroutines, namely make() and makeinstall(), can be used as is.

6. "r"

You might have noticed, the first parameter of all methods is an "r". And when methods are invoked, they have "r." before them. Suffice it to say, "r" in conary recipes is what "self" is in usual Python code. Refer to Python documentation for details.

7. unpack(r)

This is the first of our three methods. Here the source code of systemd is prepared for later compiling. A tarball, some patches and one extra configuration file are added to the project using built-in functions,addArchive(), addPatch() and addSource().

The tarball will be extracted to the building directory (e.g. ~/conary/builds/systemd/). The patches will be applied on the source code. Then the extra sources will be copied into the building directory.

The "patches" and "extraSources" defined above are used here, in a for-loop. It's only a engineering habit of me to list patches in the beginning of recipes. You could of course put them one by one in the for-loop, though I prefer otherwise.

Again, "patches" is referenced as 'r.patches' and "r" is just like "self" here.

8. configure()

AutoPackageRecipe provides a straitforward configure() method which is just a thin wrapper of r.Configure() (which is simply "./configure" under the hood, with some default options such as "--prefix=/usr").

But systemd needs more. So I override it with my own configure(), which invokes "autoreconf" and passes some additional options. autoreconf is needed since I have patches against configure.ac and must get configure regenerated.

9. policy()

Now we come to the last part of systemd.recipe. In my recipes policy() is usually used to finalize and clean up the packaging process.

For every package, conary will run a bunch of "policies" against it, checking buildRequires, correcting obvious errors, normalizing against established customs, etc. For example, conary checks for non-executable files in directories that are "usually" for binary files (e.g. /usr/bin). But if you explicitly want such files, now is a good time to set "exceptions" to the policies.

Most of the actions in systemd.recipe's policy() are conary policies, such as FixupMultilibPaths (Fix up and warn about files installed in directories that do not allow side-by-side installation of multilib-capable libraries) and DanglingSymlinks (Disallow dangling symbolic links).

But Install() and Symlink() are not; they are build time functions just like Make() (Runs make with system defaults) and MakeInstall (Runs make utility with install target). In vim, build funtions and policy funtions will have different colors, thanks to the syntax file conaryrecipe.vim.

For details of these conary API, reference to the manuals using "cvc explain <funtion name>".

------------------

That's the overview of systemd.recipe. I hope you get a taste of conary recipes (at least in my style :) Though it's not without its own shortcomings, conary recipes are the most elegant packaging scripts, compared to rpm, deb, bitbake and all the others that I've ever seen (most of them seem to be shell-based).

Wednesday, December 22, 2010

2010 yearly summary

It's already the last month of this year. Time to summarize and look forward into the future.

2010 is surely an important period of time during my life. This year I graduated from university and got a job and finally have a life of my own. Looking back, there is actually more that has happened than I realized.

So what happened during my 2010:

1. I graduated from Beihang University. It's 2006 that I came to Beijing and now four years have passed, it's surely a great tour.

2. Beginning in the later part of 2009, I have an 8-months internship at Sun Microsystems. Even though it's also the last few months of the Sun company, I've a great time there and I always missed the colleagues from the first company that I worked for. Thank you, Lingbo, Aaron, Edward, Edgar, Miao, Emily. And there are many more whom I can't all list here, but I always feel grateful for them when I think of the help that they gave me.

3. Then I got a job at Wind River. It began in May, so it's already seven months! In this position I have been learning more and more about Linux, and although I don't much coding, there are plenty of chances to get knowledge about what I want.

4. I managed to keep up with our Open Source Club. In the first half of the year, it became quite silent since people are busy graduating. But we got over it and things are much better now.

5. About my participation in open source projects, Foresight Linux is the only one in which I'm still active. I gradually dropped out of the others and stopped appearing in local open source meetings. That's kind of sad, which I hope will be changed in the future. On the other hand, I'm happy that we are keep sticking together around Foresight Linux, even it's still so small. 2009 was such a hard year for many people, but we are seeing better in 2010 and hopefully even better in 2011.

6. Until now I wrote 17 blogs during 2010 (20 in 2009 and 31 in 2008). You can't judge everything simply through numbers, but it still means something. I may still post one or two more articles, so the final number will be 20 and on average, it's 18 days per post.

7. The books I read in this year include Lord of the Rings, 1984, Lincoln the Unknown, More Joel on Software (in Chinese), 程序员羊皮卷 (a book on programmer career), Linux Application Development (not quite finished), Linux Programming by Example (in Chinese; not quite finished), Brain Rules (in progress), 程序员的自我修养 (similar topic as in Linkers and Loaders; in progress) and Apprenticeship Patterns (in progress). There might be some that I didn't write down but yeah, I should read more.

8. I have been memorizing English words for about two months, and it feels very good.

That's it. Year 2010.

What do I have in mine about the new year?

1. Write more blogs. I don't have goals but 20 to 30 should be good.

2. Read more books. Again I don't have goals, but I will read more on various topics. And since no one is giving me homework now, I will have more liberty as for what to read.

3. Participation in open source projects and meetings. I'll go to meetings and try to get into some projects. For now I have Foresight Linux and systemd in mind.

4. Keep active in Beihang Open Source Club. It feels great to be among students. Though I was a student not long ago, I'm not now. I'll help get our club going forward with my friends in the university.

Let's see how it goes.

Saturday, December 18, 2010

get serial port output from qemu

So I was testing systemd in qemu, and wanted to see its debug output. Here is how I managed to do it.

1. Start qemu with the serial port redirected somewhere. I chose to redirect to a file since that would be searchable.

qemu-kvm FL-devel-dev.img -serial file:/tmp/systemd-console

-serial will redirect the serial port output to the file.

(The other options I use are "-vga vmware -m 512M -redir tcp:5555::22", but it's irrelevant here.)

2. Tell the kernel that the console is on the serial port.

Edit the bootloader config and add console=ttyS0 to the kernel arguments.

You may also need to remove "splashy", "quiet" or others.

Now you can inspect the log file on the host.