Create a KNOPPIX USB (boot-)stick from a running Linux system

I was recently examining, how to boot a (live) KNOPPIX from an USB stick. It turns out I did not find a manual about how to create a bootable USB stick containing KNOPPIX without burning KNOPPIX to a CD/DVD first?! So this is a short howto without burning anything. I'm showing how to put a KNOPPIX ISO image to an USB stick. I need an ISO image of KNOPPIX, an USB stick and syslinux installed. I've chosen to put the 4GB DVD image KNOPPIX_V7.2.0DVD-2013-06-16-EN.iso onto a 16GB USB stick. The place left on the USB device should be available to the user as NTFS partition. I'll use the term /dev/sdX to refer to the stick (where X is a lowercase character, e.g. /dev/sdb) and completely empty it first (THIS WILL DELETE ALL DATA INCLUDING THE PARTITION TABLE!):

shred -z /dev/sdX

Now after the ISO image has been downloaded, we need to alter it a bit. For this some information about the USB stick is required. The relevant bits heads and sectors/track are highlighted below:

fdisk -l /dev/sdX

Disk /dev/sdX: 16.0 GB, 16008609792 bytes
64 heads, 32 sectors/track, 15267 cylinders, total 31266816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000

   Device Boot      Start         End      Blocks   Id  System

Now we change the ISO image for hybrid mode:

isohybrid -o 1 -h 64 -s 32 -e 1 KNOPPIX_V7.2.0DVD-2013-06-16-EN.iso

and copy the ISO image to the USB stick:

cat KNOPPIX_V7.2.0DVD-2013-06-16-EN.iso > /dev/sdX

After the command has succeeded, the partition table will show a bootable "Hidden HPFS/NTFS" partition. Note that you can adjust the partition type by using the -t, --type switch of isohybrid.

fdisk -l /dev/sdX

Disk /dev/sdX: 16.0 GB, 16008609792 bytes
64 heads, 32 sectors/track, 15267 cylinders, total 31266816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdX   *           1     8032255     4016127+  17  Hidden HPFS/NTFS

Now we can create one or more partitions on the stick for shipping data or $whatever by using fdisk, e.g. create a new primary partition and set the type to HPFS/NTFS/exFAT:

fdisk /dev/sdX

Command (m for help): p

Disk /dev/sdX: 16.0 GB, 16008609792 bytes
64 heads, 32 sectors/track, 15267 cylinders, total 31266816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x12345678

   Device Boot      Start         End      Blocks   Id  System
/dev/sdX   *           1     8032255     4016127+  17  Hidden HPFS/NTFS

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 2): 2
First sector (8032256-31266815, default 8032256): 
Using default value 8032256
Last sector, +sectors or +size{K,M,G} (8032256-31266815, default 31266815): 
Using default value 31266815

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 7
Changed system type of partition 2 to 7 (HPFS/NTFS/exFAT)

Command (m for help): p

Disk /dev/sdX: 16.0 GB, 16008609792 bytes
64 heads, 32 sectors/track, 15267 cylinders, total 31266816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x12345678

   Device Boot      Start         End      Blocks   Id  System
/dev/sdX1   *           1     8032255     4016127+  17  Hidden HPFS/NTFS
/dev/sdX2         8032256    31266815    11617280    7  HPFS/NTFS/exFAT

Command (m for help): w

[..]

Run mkfs.ntfs on /dev/sdX2 to create the NTFS file system there. The USB stick is ready now. Connected to the system where one wants to start a live KNOPPIX and chosen to boot this system from USB will result in a running KNOPPIX system.

The same should also work for creating a Debian live system USB stick. Thanks to the author of this article.

N54L LCD-Mod with U204FR-A3 (LCDModkit) and lcdproc (II)

Here it goes. I was looking for an LCD display for my microservers 5,25'' bay and found this neat post. The manufacturer of the LCD display used there provides different displays for this purpose. I've decided for an U204FR-A2 in black with a red backlight. It left Hongkong around five days later and arrived here another 6 days later. All in all: I got it after 11 days.

I unpacked the LCD device. It comes with an internal USB connector and is driven by an Hitachi HD44780 LCD controller. The connection wasn't a problem at all. I've already put a Silverstone SST-EC04-P PCIe card with two USB 3.0 external ports and an internal 19pin dual port connector into the systems PCIe 1x slot. Now to connect the LCD with this card I've bought an Inline 19pin USB 3.0 header to 8pin double USB 2.0 header adapter and connected the card with the LCD display. Easy, right?

To make the display "attached" to the case - it comes with two steel sheets and two screw holes each, that cannot be attached to anything in the microserver case - I've used a small workaround: double-faced adhesive tape and two halfs of a matchbox - one can also use small scantlings - and created a bonding between the steel sheets and the case.

That's it. I put the cover plate carefully back - the steels sheets of the LCD display and the LED of the server will bump to each other!

There are two programs to output information to the LCD display. These are lcdproc and lcd4linux. I started with the first one which only provides pre-defined screens. Seems with the latter one can create own screens. This is an option for the future.

lcdproc consists of two programs. First there is a daemon called LCDd. It controls the driver, contrast etc.pp. The relevant parts of its configuration file /etc/LCDd.conf look like as shown below. Note that I did not change the default values for contrast or brightness.

[server]
Driver=hd44780
WaitTime=25

[hd44780]
ConnectionType=lcd2usb
Size=20x4

To print something to the screen one can use the lcdproc command, which is configured via /etc/lcdproc.conf. I've enabled the Iface, TimeDate, SMP-CPU, and MiniClock screens. The program is started during startup via cron. The file /etc/cron.d/lcdproc simply contains this:

@reboot root    lcdproc

The following pictures show the resulting screens, which change every 25 seconds. That's it.

*Pics*

Eine punktierte Zartschrecke (leptophyes punctatissima) womöglich. Zitat Sieht aus wie ein Alien. :)

Fundort: Dresden, 25.07.2013

*Pics*

Ich vermute, es handelt sich um eine Libellenlarve von libellula depressa. Die rote Färbung erklärt sich durch den Austritt von eisenhaltigem Grundwasser in dem Bereich. Größe ca. 2,5 cm.

Fundort: Schönfelder Dorfbach, Kreis Meissen, 15.08.2013

HP N54L Microserver - energy efficiency and power management

I recently worked on activating power management functions, reduce energy consumption and noise of my little HP N54L "toy". During this process I tried to avoid the usage of /etc/rc.local and set things by udev, hdparm and friends. Below are my results.

Actual results

With the following steps my system (N54L + 3xWD20EFRX HDD +1xWD5003AZEX HDD + LCD-mod + case fan mod + Debian Wheezy) uses 27W in idle mode. The USB W-LAN card uses another 10W. In active mode, e.g. compiling source code, the system runs (and boots) with around 57W. The highest power consumption observed is during startup phase with 88W.

First things first

For the following steps it might be necessary to have some packages installed, that maybe do not occur in this post. If I missed something, I appreciate a hint. Further the following steps might produce even better results with a custom kernel. I'm using the stock linux-image-3.2.0-4-amd64 kernel image as the time of writing and I have these packages installed: amd64-microcode, firmware-linux, firmware-linux-free, firmware-linux-nonfree and firmware-atheros (the latter for my WLAN card).

ASPM and ACPI

First I enabled PCIE ASPM in my (non-modded) BIOS and forced it together with ACPI via grub by changing GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub so it looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=force pcie_aspm=force nmi_watchdog=0"

ASPM has now been enabled as lspci prooves:

00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780/RS880 PCI to PCI bridge (PCIE port 0) (prog-if 00 [Normal decode])
[..]
                LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep+ BwNot+
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (PCIE port 2) (prog-if 00 [Normal decode])
[..]
                LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep+ BwNot+
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
[..]
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet PCIe (rev 10)
[..]
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]

Even so /sys/module/pcie_aspm/parameters/policy will still show as below:

[default] performance powersave

I'll show how to set the powersave value in /sys/module/pcie_aspm/parameters/policy in the next section.

JFTR: These are my ACPI related packages installed: acpi, acpid, acpi-support and acpi-support-base.

Enable powersaving via UDEV

The following rules file /etc/udev/rules.d/90-local-n54l.rules has been inspired by a blog post. It enables powersaving modes for all PCI, SCSI and USB devices and ASPM. Further the internal RADEON cards power profile is set to the low value. There is no monitor connected usually. The file contains these rules:

SUBSYSTEM=="module", KERNEL=="pcie_aspm", ACTION=="add", TEST=="parameters/policy", ATTR{parameters/policy}="powersave"

SUBSYSTEM=="i2c", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="pci", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/autosuspend", ATTR{power/autosuspend}="2"
SUBSYSTEM=="scsi", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="spi", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"

SUBSYSTEM=="drm", KERNEL=="card*", ACTION=="add", DRIVERS=="radeon", TEST=="power/control", TEST=="device/power_method", ATTR{device/power_method}="profile", ATTR{device/power_profile}="low"

SUBSYSTEM=="scsi_host", KERNEL=="host*", ACTION=="add", TEST=="link_power_management_policy", ATTR{link_power_management_policy}="min_power"

Set harddrives spindown timeout

I decided to sent my system drive to standby after 20 minutes and the RAID drives after 15 minutes. This is usually ok, because the RAID isn't always used. hdparm is the right tool to realize this. Many people use the /dev/disk/by-uuid/... syntax here, to avoid having to touch the configuration file if some system configuration changes. Because I'm running a RAID, I couldn't use this syntax, although it might be possible to use /dev/disk/by-id/... instead. Well for the moment I stay with the configuration below. The relevant part of /etc/hdparm.conf is:

[..]

# system harddrive
/dev/sda {
        spindown_time = 240
}

# below are the WD20EFRX drives
/dev/sdb {
        spindown_time = 180
}

/dev/sdc {
        spindown_time = 180
}

/dev/sdd {
        spindown_time = 180
}

Idle mode

When there is nothing to do for the system, all I hear is the (still a bit noisy) fan of the power supply, which I might replace in the future too. Either by testing a different fan or by replacing the whole power supply unit by the fanless FORTRON FSP150-50TNF or (even better) a picoPSU.

The system currently shows a power consumption of around 37W in idle mode whereas the USB W-LAN card itself needs around 10W. There is a possibility to enable power savings mode for this card too. I could add this entry to /etc/udev/rules.d/90-local-n54l.rules:

SUBSYSTEM=="net", ACTION=="add", KERNEL=="wlan*" RUN+="/usr/bin/iw dev %k set power_save on"

But it turned out that the connection became a bit unstable after it. So I don't use this rule.

More on the road

There are a lot more options one can easily find via $search_engine. The N54L system could be brought to sleep and woken up by LAN via Wake-on-LAN (WOL). This is a feature I don't use. I've also read rumors about enabling different sleep/suspend states of the system, which seems to require to install a modded BIOS. Well, I'll post news and changes if they happen to come;)

Local link networking on Debian/Linux between server and workstation

My network is based on wireless LAN, which is running at 54MBit/s. Transfering files between two systems therefor is not very fast. I could use wired connections, but cables would need to be pretty long to connect to the router. However the file server and the workstation are located near to each other and both have a free Gigabit LAN link. So why not connect these systems directly?

Both systems run network-manager, whereas the server runs without a graphical/desktop environment. The file /etc/network/interfaces contains this entry related to the LAN devices on both systems:

allow-hotplug eth0
auto eth0

To create the connection it is necessary to have the package avahi-autoipd installed. In a graphical environment is is then as easy as open the network(-manager) preferences dialog, create a new wired connection, change to the IPv4 Settings tab in its Options dialog and choose the Link local only method. Done.

On the server I have no desktop environment. Therefor I wrote the necessary file /etc/NetworkManager/system-connections/LocalLink myself. The UUID value can be obtained via uuid -n 1:

[802-3-ethernet]
duplex=full
mac-address=<MAC-address>

[connection]
id=LocalLink
uuid=<value of: uuid -n 1>
type=802-3-ethernet
timestamp=0
autoconnect=true

[ipv6]
method=auto

[ipv4]
method=link-local

I restarted the manager via service network-manager restart on the server and the link came up too. The direct link was ready.

A note: I was able to create the connection by using a patch cable because my cards are able to handle the situation. On older systems it might be necessary to use a cross-over-cable.

I would really like to use bonding on the wired ethernet and the wireless devices. But the wireless connections are done via router whereas the ethernet connections are created via local link. I think, this is impossible. Any ideas/hints?

Setting up a network buildd with pbuilder, ccache, inoticoming and NFS

I recently bought an N54L HP microserver, which shall act as a build-daemon (buildd) for Debian packaging. The idea of my workflow is the following:

1. do the maintenance tasks locally, usually using svn-buildpackage, quilt etc.pp
2. build a source package locally
3. upload the source to an incoming directory on the server
4. watch this incoming directory and start a build process as soon as a package is uploaded
5. put the result into an accessible directory
6. support different vendors (debian, ubuntu)
7. support different releases (experimental, unstable, stable, oldstable)
8. support different architectures (amd64, i386)
9. speed up the build process

Setting up the building environment

My first idea was to use sbuild as described in several places. However I went with pbuilder for a long time having a setup I personally like. So I decided to stay with it. I reduce the time and work to update the configure files by using

• one central file /etc/pbuilderrc.inc defining all the common/shared settings and
• small configure files defining vendor, release, architecture and mirrors

Here are the relevant parts of my central configure snippet:

[..]
EXTRAPACKAGES="apt-utils debconf debconf-utils libfile-fcntllock-perl"
[..]
BASETGZ="/var/cache/pbuilder/base.$DISTRIBUTION.$ARCHITECTURE.tgz"
APTCACHE="/var/cache/pbuilder/aptcache/$DISTRIBUTION/$ARCHITECTURE/"
BUILDRESULT="/var/cache/pbuilder/result/$DISTRIBUTION/$ARCHITECTURE/"
[..]
case "$VENDOR" in
ubuntu)
MIRRORSITE="http://archive.ubuntu.com/ubuntu"
COMPONENTS="main restricted universe multiverse"
OTHERMIRROR="\
deb http://de.archive.ubuntu.com/ubuntu $DISTRIBUTION main restricted universe multiverse|\
deb http://archive.ubuntu.com/ubuntu $DISTRIBUTION-updates main restricted universe multiverse|\
deb http://security.ubuntu.com/ubuntu $DISTRIBUTION-security main restricted universe multiverse|\
"
;;
*)
MIRRORSITE="http://ftp.debian.org/debian"
COMPONENTS="main contrib non-free"
OTHERMIRROR="\
deb http://ftp.de.debian.org/debian $DISTRIBUTION main contrib non-free|\
"
;;
esac
[..]
DEBBUILDOPTS="-us -uc -j2"
AUTO_DEBSIGN=no
USE_PDEBUILD_INTERNAL=yes
[..]

You get the main idea, right? Create own files and directories for each architecture, release and vendor. Further I only want to pull build related packages into the build environment, not into the server environment. So I enabled USE_PDEBUILD_INTERNAL. I don't want to sign anything on the buildd itself and use both cores. That's what DEBBUILDOPTS and AUTO_DEBSIGN are used for. Further I saw some warnings reported during build processes. I therefor install the packages mentioned in EXTRAPACKAGES to every build enviroment. The following are the configure files for the unstable amd64 (default) and the stable i386 build environments:

VENDOR="debian"
DISTRIBUTION="sid"
ARCHITECTURE="amd64"

. /etc/pbuilderrc.inc

VENDOR="debian"
DISTRIBUTION="wheezy"
ARCHITECTURE="i386"

. /etc/pbuilderrc.inc

OTHERMIRROR+="\
deb http://security.debian.org $DISTRIBUTION/updates main contrib non-free|\
"

Now to not having to update all of these environments manually there is a small parallelized script /usr/local/sbin/update-pbuilder-chroots:

#!/bin/sh

t=$(tempfile -p .upc. -s .list) || exit 1

cat > $t << EOF
/etc/pbuilderrc.amd64.sid
/etc/pbuilderrc.i386.sid
/etc/pbuilderrc.amd64.wheezy
/etc/pbuilderrc.i386.wheezy
/etc/pbuilderrc.amd64.squeeze
/etc/pbuilderrc.i386.squeeze
EOF

parallel /usr/sbin/pbuilder update --override-config --configfile :::: $t | tee -a /var/log/pbuilder.log

rm -f $t

exit 0

The script will make sure, that every change to the configuration file(s) will be considered (--override-config) and is run once a week via cron. The default build environment is updated on a daily base without this restriction:

@reboot root    rm -rf /var/cache/pbuilder/build/*/ >> /dev/null 2>&1
@daily  root    test -x /usr/sbin/pbuilder && /usr/sbin/pbuilder update --configfile /etc/pbuilderrc.amd64.sid | tee -a /var/log/pbuilder.log >> /dev/null 2>&1
@weekly root    test -x /usr/local/sbin/update-pbuilder-chroots && /usr/local/sbin/update-pbuilder-chroots >> /dev/null 2>&1

Speeding up the build process

Sometimes it is necessary to recompile a package. The tool to speed up this process is ccache, which can be easily integrated into pbuilder by setting CCACHEDIR in /etc/pbuilderrc.inc. It acts like a cache for compiled files and avoids recompilation in several buildd runs.

CCACHEDIR=/var/cache/pbuilder/ccache

It is also possible to use a TMPFS for BUILDPLACE to speed up the build process itself. In this case APTCACHEHARDLINK must be disabled. Well, yeah, I don't do this.

Watch an incoming directory and start the build process

Not much to say about this except that the user nobody is used to run the pbuilder command. Therefor it is necessary to mention this in /etc/sudoers to allow him to do this via sudo command:

Defaults        env_keep = "DH_OPTIONS DH_VERBOSE DEB_BUILD_OPTIONS DH_OPTIONS MAKEFLAGS CC CXX CPP F77 F90"
[..]
nobody          ALL=NOPASSWD:/usr/sbin/pbuilder

The first line is only necessary, if a pbuilder command is started with one of these environmental variables set. This can be useful when e.g. debugging a build failure with a future gcc version that is not yet the default or when debugging a clang build issue or when creating non-stripped binary packages for creating a backtrace (BTW: When there is a common repository of all debugging symbols created?).

Now to start the build process, I created an incoming directory with mods 777 at /var/cache/pbuilder/incoming/ and use inoticoming to watch it. This is started during boot via cron. The following shows the entry for the default unstable amd64 build environment:

@reboot root    /usr/bin/inoticoming --logfile /var/log/inoticoming.log --pid-file /var/run/inoticoming.pid /var/cache/pbuilder/incoming --chdir /var/cache/pbuilder/incoming --suffix .dsc --stderr-to-log --stdout-to-log /bin/su -c '/usr/bin/sudo /usr/sbin/pbuilder build --autocleanaptcache {}' nobody \;

Done. That's how the basic buildd works.

Getting the results

The pbuilder BUILDRESULT directory with mods 777 is shared via NFS mount to the workstation. After successfully building the package, debsign and dput can be used as usual.

Setting up the workstation

I use svn-buildpackage for most of my packages. Because pbuilder won't download source tarballs on demand by default (maybe via hook?), the tarball must always be included building the source package, which then gets uploaded via dput to the buildd. This is considered in the command listed in svn-builder in ~/.svn-buildpackage.conf (note, below shows my personal layout):

svn-builder=debuild --post-dpkg-buildpackage-hook=\"dput -f buildd /usr/local/src/packages/$PACKAGE/%p_%s_source.changes\" --no-lintian -d -sa -us -uc -S
svn-override=origDir=/usr/local/src/packages/$PACKAGE
svn-override=buildArea=/usr/local/src/packages/$PACKAGE
[..]

The host buildd has been added to ~/.dput.cf:

[buildd]
fqdn = mybuildd
login = mylogin
method = scp
incoming = /var/cache/pbuilder/incoming
allow_unsigned_uploads = 1
run_lintian = 0

Done. Now the svn-buildpackage command will create a source package including the source tarball and upload it to the buildd, where the package building starts right after the upload.

Conclusions

This is the main setup which works nicely for the default build environment so far. On the TODO list is to extend this setup so I can easily deal with all the architectures and releases (point 6..8 at the beginning).

Update 17.02.2015

The original article suggested to use the %v variable in the hook to transfer the _sources.changes file to the build-daemon. This will fail if the source version e.g. is X:Y-Z in which case the file created is package_Y-Z_source.changes but the command to execute is:

dput -f buildd package_X:Y-Z_source.changes

and thus will fail. According to dpkg-buildpackage(1) the hook command also accepts a %s variable, which seems to extract the superflous version characters.