LJ Archive

UpFront

diff -u: What's New in Kernel Development

Zack Brown

Issue #203, March 2011

John Stultz submitted code to enhance the real-time clock. He didn't like the fact that if a process wanted to set an alarm, it would have to take a lock on the RTC and poll it, waiting for that alarm to go off. His new code created an intermediate layer that would keep track of alarms set by any user code and feed the alarms back to the programs that had set them, as they occurred. If his patch is accepted, it could be good news for user programs that no longer will have to deal with the complexities of holding and freeing an RTC lock.

John also noticed that the Android RTC alarm driver had a new approach to the RTC that he felt deserved to be included in the main kernel tree. But, because the Android driver depended on other parts of the Android system that were not included in the official kernel, John rewrote the driver to use native kernel interfaces and submitted it that way. The basic value of the Android driver, as he saw it, was that it gave the user a simpler interface, at the cost of losing some subtle abilities that didn't seem very useful anyway. So as far as John could see, it was a pure win. It'll be interesting to see whether the Android system adapts to use John's version of the driver as well, or whether it will continue to maintain its own version independently.

Vernon Mauery submitted code to support IBM Premium Real-Time Mode (PRTM) on supporting x86 hardware. This powerful operating mode can give user programs a wholesale control over the system's hardware, which normally is tightly regulated and secured by the kernel. PRTM has the potential to be a devastating security hole, but it also can provide very low latency action to applications that require real-time performance. And, it can provide very useful features to debuggers and other meta-applications. Vernon's code provides a SysFS interface to enter and leave PRTM at will. I'm guessing there also will be security features to prevent applications from doing this unless they are absolutely trusted.

Documentation is always nice to see, and Andres Salomon recently submitted some for the staging tree. The staging tree exists in the main kernel source, under drivers/staging, and it acts as a repository for incomplete drivers that aren't ready to be relied on completely. In the old days, people wanting to test these drivers would have to download the driver's patch independently, apply it to their own copy of the kernel, and compile and install that kernel. This barrier to entry was sufficient to keep all but the most enthusiastic testers from ever testing a driver before it was included in the official tree. The staging tree allows any user to test a driver very easily, just by enabling it during the configuration process prior to compiling the kernel. This opens up testing to an immense and seething horde of potential users. The result is that drivers receive far better testing before being incorporated into the main source tree. Users are protected from potentially damaging code in these drivers, because they simply cannot enable drivers in the staging tree when they compile the kernel.

Of course, as with all aspects of kernel development, the rules constantly are being revised, so that the kernel development process can really be seen as a set of algorithms developed in an evolutionary way that has the ultimate goal of making the best possible behaviors also the most natural-seeming for any given kernel contributor. As with all evolutionary development, any given snapshot presents only the imperfect balance of that moment, but it's so great to watch.

Non-Linux FOSS

Shawn Powers

Issue #203, March 2011

Almost every operating system has an FTP client, but they often are command line only or kludgy at best. If you're looking for a solid FTP (or SSH) client, FileZilla is a cross-platform program that does a few things and does them very well. Whether you need to transfer files over FTP, FTP/TLS or even SSH, it has a simple interface and a reliable code base.

The folks at the FileZilla Project went one step further, however, and provided a free FTP server program for Windows as well. FileZilla's FTP server fills a much-needed gap in the desktop Windows operating system and provides extensive customization. It runs in the background, so there's no need to have a window open all the time in order to have an FTP server running. If you need a reliable FTP client for Windows, OS X or Linux, check out FileZilla's client program. If you need an FTP server for Windows, be sure to check out the server program: filezilla-project.org.

Hulu Hoop around the Desktop

Shawn Powers

Issue #203, March 2011

I've been complaining about Netflix's lack of Linux desktop support for more than a year now. All my whining seems to be making little difference, so it seems only fair to look at alternatives. Granted, there aren't any direct competitors to the Netflix streaming service, but there is Hulu. Hulu has a history of not playing well with others and blocking developers like the Boxee team, but I must give credit where credit is due. It has a fully working Linux desktop program. Plus, the Web-based features work as well. If you add the $7.99 monthly subscription for Hulu Plus, you happily can watch many seasons (sometimes the full catalog) of television programs directly on your Linux desktop. With Hulu's desktop application, you can send the whole experience full screen and have your own little home-entertainment system wherever you have Internet access. Sure, there still are ads. Sure, the catalog is different and less robust than the one from Netflix, but heck, at least Linux users are invited to the party!

Linux on a Fingernail

Joey Bernard

Issue #203, March 2011

This issue of Linux Journal is all about how to get Linux in your pocket. In this article, I go one better and tell you how to get Linux on your fingernail. Now, before you get too excited, I won't be discussing some new nano-computer being used by James Bond, unfortunately. Instead, I discuss how to put Linux on a micro-SD card (or any other USB drive, for that matter). Using this, you can run Linux on any machine that can boot off a USB device.

One of the first utilities to receive widespread attention and use is UNetbootin (unetbootin.sourceforge.net). This application is available under both Linux and Windows. It has built-in support for downloading and installing several Linux distributions, including Ubuntu, Fedora, Debian, PCLinuxOS, Linux Mint, Sabayon Linux, openSUSE, Gentoo, Arch Linux, MEPIS and many others.

Figure 1. UNetbootin

UNetbootin also has the ability to load several different system utilities, such as the following:

  • Parted Magic: a partition manager that can resize, repair, back up and restore partitions.

  • SystemRescueCD: a system repair, backup and recovery tool.

  • Super Grub Disk: a boot utility to restore and repair GRUB installations.

  • Backtrack: a utility for network analysis and penetration testing.

  • Ophcrack: a utility to recover Windows passwords.

  • Smart Boot Manager: boots off of CD-ROM and floppy drives on machines with a faulty BIOS.

  • FreeDOS: an open-source DOS to allow you to run BIOS flash utilities or just plain-old DOS.

UNetbootin can download the correct ISO image for all of these distributions automatically, or you can download (or create) your own ISO image and hand the filename to UNetbootin. In either case, the next step is to extract all of the files from the image so they can be copied to the USB drive. The USB drive you want to use needs to be formatted with a filesystem already on it. Once the files all have been extracted, UNetbootin uses some heuristics to figure out where the kernel and initrd files are hiding and places them into /ubnkern and /ubninit. It then goes through the boot configuration files from the ISO to try to figure out what boot options need to be set on the USB drive. Once it figures that out, it puts those options into the configuration file /syslinux.cfg. Then, UNetbootin uses Syslinux to make the USB drive bootable, and you should be good to go.

Another utility that has started garnering a lot of attention is usb-creator (https://launchpad.net/usb-creator). This program is the official tool to create live USB versions of Ubuntu live CDs. This utility started with version 8.04 of Ubuntu. A KDE front end showed up in version 8.10, and a Windows version currently is in the works. This utility can do the same sort of work as UNetbootin. It has the built-in ability to create an Ubuntu live USB, but that's not all. Like UNetbootin, usb-creator can take an arbitrary ISO image and copy that onto your USB drive. It even can take a CD-ROM from your CD drive and copy its contents over to your USB drive. It's a simple matter of selecting the source and the destination, and then running.

Figure 2. usb-creator

One big advantage of usb-creator is its ability to create a persistent live USB for you very easily. Any extra space on your USB drive, above and beyond what is required for the OS files, can be used as writable space for persistent files. This means any changes you make to your system will be written to the USB drive. You even have the option of clearing the persistent space on shutdown. That way, you can use your USB drive as a complete operating system, exactly as if it were on your hard drive. And, there you go, Linux on a thumbnail.

All of these techniques require you to have either a live CD or an ISO image of a live CD. But what if you want to go a bit more low-level, a bit more from scratch? One of the tools available to you is Syslinux (syslinux.zytor.com/wiki/index.php/The_Syslinux_Project). Syslinux actually is a suite of different programs that provide for booting from many different media, including:

  • syslinux: booting from FAT filesystems.

  • pxelinux: network booting.

  • isolinux: bootable “El Torito” CD-ROMs.

  • extlinux: booting from ext2/ext3/ext4 or btrfs filesystems.

  • memdisk: a tool to boot legacy OSes from nontraditional media.

Syslinux installs into the boot sector of your device and puts a copy of the file LDLINUX.SYS into the root directory. It then loads the kernel and other OS files from the actual filesystem. Because the filesystem is just a simple FAT filesystem, all the files, including the kernel, can be manipulated using standard DOS tools. By default, Syslinux assumes the kernel is in the file named LINUX on the boot disk. This default can be changed in the config file. If you hold down the Shift or Alt keys during bootup, Syslinux displays a LILO-style “boot:” prompt where you can enter a kernel filename and options.

Syslinux searches for its configuration file in the following order: /boot/syslinux/syslinux.cfg, /syslinux/syslinux.cfg, /syslinux.cfg.

In this file, you can set parameters to change all the default settings. Any filenames in the configuration file are relative to the directory that syslinux.cfg is in, unless it is a full path. A basic example looks like this:

DEFAULT linux
LABEL linux
  SAY Now booting the kernel from SYSLINUX...
  KERNEL vmlinuz.img
  APPEND ro root=/dev/sda1 initrd=initrd.img

The kernel types that Syslinux supports don't need to be a regular Linux kernel image. They can be a PXE bootstrap program, a boot sector or a COMBOOT image.

These techniques and utilities should give you a good start at putting your USB drives to their best use. You now can carry around your whole OS in your pocket. This is essentially what I've done for my old Eee PC. With Linux on USB, I can keep the same system there, and on my MacBook through Parallels. You also can use the same system on basically any machine that you can get to boot off of USB. Have fun and be creative.

Statistics with R

Joey Bernard

Issue #203, March 2011

In the January 2011 issue, I looked at using Scilab to do some basic data analysis. Although you can do a lot with Scilab, sometimes it makes more sense to use a specialized tool. In the case of statistical problems, that specialized tool is R, developed at Bell Labs. R is actually a full programming language, designed for doing statistics tasks. It's very similar to another statistical programming language, S. Because R and S are programming languages, any actual programs are implementations of this language. Commercial implementations are available, like S-Plus. In this article, I discuss the open-source implementation by the R Project. References to R from here on refer to this open-source implementation.

R is broken up into a core engine and a full complement of libraries and functions that provide all its capabilities. Because of this, different downloads contain different prepackaged functions. However, an entire repository of extra functionality is available at CRAN (Comprehensive R Archive Network) where a strong community of people is willing to share the work. Thus, there usually is a function, or several, to handle almost any task you can imagine. If not, you can develop your own. Because R is broken down into a core engine and functions, any graphical interface simply is bolted on top of the core. Several graphical interfaces exist, such as RCommander or Rgui.

Figure 1. R on a Mac

Graphical interfaces also are available for Windows and Mac OS X. Here, I'm looking at actual commands in R, so I'm using the text interface. To start it, simply type R and press Enter.

Figure 2. Starting R

To quit R, type q(); and press Enter. In R, commands can end either with a semicolon or newline.

R has extensive help files for all the commands and functions, which are in the form of man pages. You can access these help pages in two ways:

> help(command)

Or:

> ?command

You can run a whole series of commands with the source command:

> source("commands.R")

where the file “commands.R” contains all the commands you want to run as a single block. Think of it as if it were a shell script. You can send your output to a file with the sink command:

> sink("output.txt")

Once you are done writing the output, you can reset the output back to the console with:

> sink()

R can handle basic statistics out of the box. The available functions can do all the single variable statistics you're familiar with doing. The first step is to load data. Data is stored in lists, which can be arranged as arrays and matrices. The easiest way to create data arrays is to use the concatenation function c. It works well when you have only a little bit of data to work with:

> data1 = c(2,3,4,2,0,1,2)

This creates a single list and dumps these values into the variable data1. Then you can use this variable in other functions—for example, to find the average of these values, use:

> mean(data1)
[1] 2

The second line above is the output from the function mean. Output is labeled with a numeric label: [1]. The output from the first command in a session has the label [1]; the output from the second command is labeled [2] and so on. If you have a larger amount of data to work with, use the read command to read in a file. If that data is in a table format, you can read the table in as a whole with:

> DataToUse = read.table("research.data")

You can access individual data elements by using the [] characters, so printing the second item is as simple as:

> data1[2]
[1] 3

If you want to set a value in a vector, use:

> data1[2] = 5

Let's say you were making some measurements on whales. You would start by saving the data into a variable:

> whale = c(74,122,235,111,292,111,211,133,156,79)

The first thing you would want to do is to run some basic statistical functions to find the mean, variance and standard deviation:

> mean(whale)
[1] 152.4
> var(whale)
[2] 5113.378
>std(whale)
Error: couldn't find function "std"

It looks like there is no standard deviation function. But, because R is a full programming language, you can write your own:

> std = function(x) sqrt(var(x))
> std(whale)
[3] 71.50789

This shows one of the shortcomings of R. Because it is such a large system, it does tend to have a relatively steep learning curve. More research would have shown that there is a built-in standard deviation function; it's just called sd instead of std:

> sd(whale)
[4] 71.50789

Sometimes it's worth taking the extra time to do some research before storming forward and writing all the code yourself.

R has very extensive graphing capabilities as well. It generates very high-quality plots and graphs you can use in publications. The commands can be very simple, if you just want a quick display so you can inspect your data visually. If you want very specific output, with lots of markup and customizations, R provides full control over all your graphs' details. The most basic plot command is simply plot. Let's say you have a series of 2-D data:

> x = c(1,2,3,4)
> y = c(2,3,4,5)
> plot(x,y)

This gives us the plot shown in Figure 3.

Figure 3. Example Plot

This looks like a straight line, so you probably want to do a linear regression on it:

> abline(lm(y~x))

This command does the linear regression of a straight line and plots it on the same plot (Figure 4).

Figure 4. Linear Regression of a Straight Line Plotted on the Example Plot

It seems a pretty close fit, so you probably should look at the coefficients of the linear model:

> lm(y~x)
Call:
lm(formula = y ~ x)

Coefficients:
(Intercept)           x
          1           1

So you end up with a straight line with a slope of 1 and an intercept of 1.

Once you've done all your analysis, you'll probably want to generate some nice plots for your publication. By default, when you call one of the plotting functions, the output is the graphical display. But, you can change this to an output file of a particular format. Let's say you want to generate some PNGs:

> png("filename.png")
> plot(x,y)

This will save your plot to the file “filename.png”. You also can save to several other file formats (JPG, PDF, PostScript and so on). Many other options are available, so check out the help page.

This short crash course should give you a starting point for exploring R. See the R Project's main page for more information (www.r-project.org), and remember to check out the CRAN repository (cran.r-project.org) before you put too much work into it. If you'd like me to cover any scientific packages or computational science techniques, please let me know.

GUI from Afar

Shawn Powers

Issue #203, March 2011

For most configurations, I prefer to use command-line tools. This is especially true on headless servers. There are a few things, however, that I find easier to configure with GUI tools. Webmin does a fine job for some of those things, but sometimes the native GUI tools are really nice. One good example is virt-manager. Although it's possible to run virt-manager on a workstation separate from the server running KVM, sometimes installing virt-manager also installs kernel modules that are incompatible with other virtual systems.

In my particular scenario, I like to run VirtualBox on my workstation, but KVM on my server. Although it is possible to run virt-manager on a workstation with VirtualBox installed, it can get messy with conflicting kernel modules. That's where ssh -X comes into play. Just install virt-manager on the server, run it remotely with a simple ssh -X user@server virt-manager, and the GUI program starts up right on your current desktop!

It's not a new trick, but it's one I find myself using often. If you haven't considered installing GUI tools on your headless server, just because it seemed silly, don't forget the GUI power of SSH. It might make your life a little easier—or at least more colorful.

Linux Journal Archive

The newly updated 1994–2010 Linux Journal Archive disc is here! In easy-to-use HTML format, the fully searchable, space-saving archive offers immediate access to an essential resource for the Linux enthusiast: Linux Journal. The archive includes all 200 issues of Linux Journal, from the premiere issue in March 1994 through December 2010.

  • Regular price: $34.95

  • Sale price: $29.71 (15% off)

  • Coupon code: ACDNEW

Sale ends March 31, 2011.

They Said It

It is the framework which changes with each new technology and not just the picture within the frame.

—Marshall McLuhan

A computer is like an Old Testament god, with a lot of rules and no mercy.

—Joseph Campbell

They have computers, and they may have other weapons of mass destruction.

—Janet Reno

The technologies which have had the most profound effects on human life are usually simple.

—Freeman Dyson

All technology should be assumed guilty until proven innocent.

—David Brower

Is that LinuxJournal.com in your pocket, or are you just happy to see me?

Katherine Druckman

Issue #203, March 2011

I'm sure many of you carry around your monthly Linux Journal with you everywhere. Sure, the glossy paper does tend to get a little beaten up, but having important reference material with you at all times is worth a bit of wear and tear. On the other hand, we know some of you take meticulous care of your issues of Linux Journal, placing them gingerly on your coffee table for all your guests to marvel at. Either is perfectly fine with us, but we have a portability solution that will work for everyone. Visit LinuxJournal.com from your mobile device, and you'll notice you are now viewing a simplified version of the site that is optimized for a mobile experience. We hope this proves useful to you while you are off on all your geeky adventures, or even when you are just sitting around the dinner table and need to show your friend that awesome blog post by Shawn Powers you read last week. Happy surfing!

Silly Programs

Those of us who have been using Linux for a long time all know the joy of silly programs like xeyes. One of my favorites, however, is good old xsnow. Whether you love the cold weather or live in Florida and like to ski on occasion, xsnow will add some winter fun to your desktop. The xsnow program has been around forever and is surely available for your distribution.

If you're absolutely against snow and all its icy accomplices, you might want to check out a couple other oldies but goodies. Back before we had fancy computer games, we used to waste time with programs like xneko (you'll probably find it now as oneko), which was a little cat that chased your “mouse” around the screen. Or, perhaps you still enjoy the ever-staring eyes of xeyes (or a more modern tuxeyes). Finally, if your time-wasting tastes are a bit more on the macabre side, xroach (or groach) might tickle your fancy with bugs that scurry to hide under the windows on your desktop. Whatever your thoughts on silly time-wasting apps, you owe it to yourself to check out a bit of Linux/UNIX history.

LJ Archive