Instant 10-20% boost in disk performance: the “noatime” option

Many people already know about this option, but it is worth mentioning again. However, a description of what atime is is in order.

The atime is one of the three times associated with a UNIX file: the three are: the ctime (or change time - that is, when the inode was last changed); the mtime (or modified time - that is, when the file is changed); and lastly, the atime (or access time).

It is the continual changes to the atime that cause so much grief. Compared to the mtime and the ctime, the atime changes with alarming frequency. Every single time a file is accessed, the atime is updated to match the current time – whether the file is opened, read, written, or accessed in any manner whatsoever.

There was a Linux kernel mailing list discussion thread that gave rise to some interesting quotes on this topic.

The discussion became quite heated when Ingo Molnar suggested that atime should be the kernel default. He had this to say:

Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_.

and:

It’s also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: ‘For every file that is read from the disk, lets do a … write to the disk! And, for every file that is already cached and which we read from the cache … do a write to the disk!’

and later, this:

Measurements show that noatime helps 20-30% on regular desktop
workloads, easily 50% for kernel builds and much more than that (in
excess of 100%) for file-read-intense workloads.

and, this:

Give me a Linux desktop anywhere and i can
tell you whether it has atimes on or off, just by clicking around and
using apps (without looking at the mount options). That’s how i notice
it that i forgot to turn off atime on any newly installed system – the
system has weird desktop lags and unnecessary disk trashing.

Linus had this to say:

yeah, it’s really ugly. But otherwise i’ve got no real complaint about
ext3 – with the obligatory qualification that “noatime,nodiratime” in
/etc/fstab is a must. This speeds up things very visibly – especially
when lots of files are accessed. It’s kind of weird that every Linux
desktop and server is hurt by a noticeable IO performance slowdown due
to the constant atime updates, while there’s just two real users of it:
tmpwatch [which can be configured to use ctime so it's not a big issue]
and some backup tools. (Ok, and mail-notify too i guess.) Out of tens of
thousands of applications. So for most file workloads we give Windows a
20%-30% performance edge, for almost nothing. (for RAM-starved kernel
builds the performance difference between atime and noatime+nodiratime
setups is more on the order of 40%)

Changing a file system to run without atime is simple; use this command on a mounted filesystem:

# mount -o remount,noatime /disk

Don’t forget to change the /etc/fstab to match this.

The Current State of Window Maker

I’ve always liked the Window Maker window manager. However, the current state of Window Maker is in some turmoil.

Development on the original Window Maker window manager has ceased, and new development has been taken up by wmaker-crm (a fork). Nightly builds of wmaker-crm had been available for Debian from a user-created repository, but no new builds have been put up since 29 April 2011 – and the repository information hasn’t been rebuilt since 26 May 2011. Building a Debian package seems to be problematical in any case. According to this mailing list thread, Andreas Metzler and Martin Dietze are responsible for changes therein, but the changelog hasn’t reflected any changes in wmaker-crm.

It was recommended to the Debian Window Maker package maintainer, John H. Robinson IV, to use the wmaker-crm sources for the package; he was receptive but nothing has happened since. As of 6 July 2011, he stepped down as maintainer of the package, leaving the Debian package orphaned. This event did not go unnoticed; a thread was taken up on the wmaker-crm development list.

The person behind wmaker-crm, Carlos R. Mafra, also created a git repository for numerous Window Maker dockapps that are no longer maintained.

The standard Window Maker display manager, the WINGs Display Manager, apparently is much less desired than Window Maker itself: according to the popularity contest statistics, at a peak during January 2011, 2000+ people had Window Maker installed while less than 200 currently have WDM installed.

Statistics about the installed base (and user base) of Window Maker and other packages can be seen over at popcon.debian.org; these statistics come from people who have installed the popularity-contest package.

Over at ArchLinux, their wiki has an excellent write-up on Window Maker (including basic technical details and information on wmaker-crm) which certainly makes one think that Window Maker is vibrant in that community. Both Window Maker and wmaker-crm are packaged (as windowmaker and windowmaker-crm-git respectively) and available to ArchLinux users from the Extra Repository as well.

If you want to see Window Maker in action (more or less – it does show a lot of Tux Commander too…) you can check out this video showing Window Maker on a Duron 850MHz system with 256Mb.

I may post a short video of my own Window Maker desktop; having forced myself to run with Window Maker as my default desktop has made me a complete convert – and helped me to force myself to research and resolve problems with making Window Maker a default desktop. Recently, I wrote about just what it took to make Window Maker fully capable and up-to-date.

If GNOME or KDE are finally just too much – and XFCE isn’t quite what you want – try Window Maker instead.

 

The Wheel Group: Updated

Working with Ubuntu Server (Lucid Lynx) the wheel group has been changed slightly.

Firstly, there doesn’t seem to be any wheel group at all – not by name. The group is now called root by default, and is enabled the same as before: uncomment the appropriate line in /etc/pam.d/su so it looks like this:

auth required pam_wheel.so

The system uses the root group because that is the group name for group 0, and because there is no group named wheel. However, if you want to maintain the original standard – make the entry look like this instead:

auth required pam_wheel.so group=wheel

Then rename (or duplicate) the group in /etc/group with id 0:

root:x:0:root
wheel:x:0:root

This maintains the highest level of compatibility: the group root remains as before, but the name wheel is also available. Having two groups with the same group ID is not typically recommended, but it doesn’t necessarily break anything either as long as the two groups are seen as completely equivalent. The first group in the list will normally be used when names are given for GIDs, but both names will be recognized from the user.

According to the documentation, this is overkill – but it does force the issue and make su work with the actual wheel group rather than a renamed one.

What pam_wheel actually does is search for group wheel first, then if it can’t find that, searches for group 0 (zero) next. It is this configuration that allows the renaming of the wheel group.

Apparently Debian or Ubuntu named the group sudo at one point, now root. The best thing to do – when there is no distinct advantage to change – is to go with the status quo: in doing so, any administrator that comes along will be able to learn and adapt to the system rapidly, leading to quicker completion of administration tasks, simple and complex.

Logging transfers via SFTP

When you install OpenSSH, you also get SFTP for free: a replacement for FTP. (Note that SFTP is related to FTP by name only.)

SFTP is very useful, but as is normally configured, it does not provide any logging of user’s file transfers. For monitoring usage, this can be a problem.

The original setting in sshd_config might look like this:

Subsystem sftp /usr/libexec/openssh/sftp-server

This lack of logging is easily remedied; the details can be found in the man page for sftp-server:

Subsystem sftp /usr/libexec/openssh/sftp-server -l INFO

For best results, check your system’s man pages for the available options.

Why doesn’t my /bin/sh script run under Ubuntu?

This is a very interesting question – and the resolution is simple. In Ubuntu 6.10 (known as Edgy Eft) the decision was made to replace the Bourne Again Shell (bash) with the Debian Almquist Shell (or dash) as /bin/sh in Ubuntu. There was considerable uproar in Ubuntu brainstorm (community ideas) and in Ubuntu bug reports, as using dash instead of the original bash caused numerous scripts to break.

In particular, the entire reasoning given for this change was efficiency: dash is more efficient (i.e., faster) than bash. According to the explanatory document created by the Ubuntu developer team, Debian has required scripts to work on POSIX-compliant shells for some time (even pre-dating the Ubuntu project). Thus, any scripts that broke were, in essence, not “following directions” and deserved what they got.

To undo this change by the Ubuntu team, one can do this:

sudo dpkg-reconfigure dash

When this command executes, specify that you do not want dash to act as /bin/sh. This will make every script that runs /bin/sh run bash as has traditionally been the case.

You can also make your scripts run /bin/bash instead of /bin/sh; this provides all of the bash capabilities without any concern as to whether /bin/sh will change again.

Making the boot process faster is a laudable goal, but like the removal of OSS from the kernel, it caused a lot of problems for users.

In both cases, it appears that the Ubuntu team is more focused on doing the technologically “right” thing rather than providing a stable and reliable platform. Unfortunately, this means that you cannot rely on Ubuntu to stay reliable – at least from one version to the next. The response of Ubuntu to such system failures has always been that they are doing the “right” thing and the problem must be fixed by someone else (i.e., it’s not Ubuntu’s problem).

Users – many of them system administrators – take the brunt of this: they don’t care whose fault it is, nor do they care whether the boot process is faster or whether the Linux sound environment is “cleaner”; they care about the stability of their systems. A system that boots faster doesn’t matter if it crashes during the boot process because of a broken script.

If the focus of Ubuntu were to provide a stable and unchanging environment, then their decisions would be different – and would result in an improved customer experience.

Tips and Tricks for Using the Shell

There are many things that trip one up in using the shell – normally a Bourne shell, POSIX shell, or Korn shell. These tips can help you understand more of what happens during the usage of the shell, and will help you understand why things might go wrong.

One thing to realize is that the shell can be anything you want; it is a personal choice (unless it is the root shell). While commonly used shells include the Bourne Again Shell (bash), the Korn Shell, or the C shell, there are a lot more than just these. Consider these two alternatives for instance:

  • rc – a small shell used by Plan 9
  • scsh – a shell that incorporates a full Scheme48 interpreter

Now – assuming a Bourne-style shell – consider these two possible commands:

$ mybinary a b c
$ mybinary a* b* c* < f

The first command does not require the shell; any program that executes a command line (such as scripting languages) can execute a command line like that one without using the shell.

The second command requires a shell be started. Why? Because of the use of shell meta-characters like filename wildcards, redirection, and pipes. All of these require parsing by a shell before executing.

When using wildcards and other shell metacharacters, remember that the shell manipulates them first. The executable command in the first example gets three arguments: “a”; “b”; and “c”. The program running in the second command may see: “able”; “baker”; “charlie”; and who knows how many others – the command will not see “a*”, “b*”, or “c*” – unless the wildcard cannot expand to any files at all; in that case, the argument is passed directly into the command as is.

This can cause problems if you don’t watch out for it:

vi m*

If you are trying to edit Makefile and you’ve no files that start with m in that directory, then you start editing the file named m*.

This tidbit also comes in handy if you ever find that the command ls is bad or doesn’t work: echo works just as well as ls -m:

$ echo a*

This will cause the shell to expand the file wildcard, then echo prints the results.

This “pre-scanning” done by the shell also explains why a command like this fails when run in a directory that a user has no access to:

$ sudo echo foobar > restricted.file

The shell sets up redirection before sudo runs – so it is the shell that attempts to write to the file restricted.file – and as the original user, too.

To make this work, you have to find a way to defer the opening of the file (for writes) until after you have root access; a classic way is like this:

$ sudo ksh -c "echo foobar > restricted.file"

Thus, it is not the running shell that opens restricted.file but the executed ksh, which interprets the -c option as a command to run. The quotes prevent the active shell from interpreting the shell characters, leaving them for ksh.

This shell interpretation also explains why the first command may fail with a Too many arguments error, while the second will almost certainly work:

$ ls *
$ ls

In the first case, the shell expands the wild card to include all the files in the current directory; if there are too many files, this becomes too many arguments. In the second case, there are no arguments: it is up to the program itself to handle all the files (which ls does well).

Understanding how the shell scans its input is critical and allows you to understand how things should work. Consider a fragment like this one:

$ AB="*"
$ echo $AB
$ echo "$AB"
$ echo '$AB'

The output from this will be something like the following:

$ echo $AB
able baker charlie
$ echo "$AB"
*
$ echo '$AB'
$AB

Update: Fixed error in filename wildcard expansion – thanks to Brett for catching the error.

Configuring a WindowMaker desktop in Ubuntu

Ubuntu Linux comes with a variety of desktops, including GNOME, KDE, and XFCE – all supported in their own distributions. You can install any of these by selecting the appropriate packages: either ubuntu-desktop, kubuntu-desktop, or xubutnu-desktop respectively. There are, however, other desktops not as well supported such as Window Maker (I’ve just got to discuss 9wm sometime…).

Before discussing too much, note that Window Maker is currently not in active development, and a fork called wmaker-crm has taken on active development for Window Maker. However, there are no packages for wmaker-crm in the Ubuntu repositories; to actually use wmaker-crm you’ll have to build it from source.

Creating a complete and working Window Maker desktop is not hard, but it is not straight-forward, nor documented well. There are a number of things to do which will enhance the desktop.

First, make sure that the menu package is installed. If it isn’t, most of the applications you have installed will be missing from the Window Maker application menus. Install it from your choice of package installers, or from the command line:

apt-get install menu

there are no applets running when you first start out. There are a number of good applets in the repositories; I selected these:

  • wmforkplop
  • wmhdplop
  • wmbattery
  • wmdiskmon
  • wmclockmon
  • wmcpuload
  • wmmemload
  • Mixer.app
  • wmwave

Install these with synaptic or with this terminal command:

apt-get install wmforkplop wmhdplop wmbattery wmdiskmon wmclockmon wmcpuload wmmemload mixer.app wmwave

Get the applets started by starting them at the command line with the appropriate options:

wmforkplop &
wmhdplop &
wmbattery &
wmdiskmon -p /dev/sda7 -p /dev/sda1 -p /dev/sda6 &
wmclockmon &
wmcpuload &
wmmemload -b -c &
Mixer.app &
wmwave &

Be sure to adapt the options to your liking; most likely the only change will be the partitions listed in the wmdiskmon options – for me, these partitions are /, /boot, and /home respectively.

Note that Mixer.app (nor almost any other mixer) will not work in Ubuntu 10.04 and above as the sound daemons it depends on have been removed. That sound system is called OSS, and it has been removed from the Ubuntu kernels entirely. To fix this, you have to compile your own kernel. If you are technically savvy, this will not be too difficult. If you follow some good instructions, you’ll be alright.

The Ubuntu repositories have many more dock apps. If that isn’t enough, there are even more at dockapps.org - including, for instance, a Mixer.app that works with ALSA sound drivers instead of OSS.

You’ll probably want to add multiple desktops. Right-click on the desktop, then select from the menu the option WorkSpace, then Workspaces, then New – and create as many as you like.

The Clip – found at the top left of the screen – is quite handy, but doesn’t provide much without some configuration. To have the Clip automatically consume any running application icons, right-click on the Clip and select Clip Options, then Autoattract Icons. You may also like to have the icons disappear automatically, only to reappear when the mouse hovers over the Clip: select Autocollapse in the Clip options menu.

To tweak even more options, use the Window Maker Preferences application – this is found in the panel with a screwdriver on it. Double-click and prepare to spend a lot of time – though not many options need to be changed.

The desktop appearance and theme can be changed; right-click on the desktop, then select WorkSpace, then Appearance. If you are going to try out a lot of themes and backgrounds, click on the Appearances menu title bar or drag the menu using the title bar to a place of your choosing – then right-click on the desktop. The Appearances menu will stay on the desktop as long as you like; close it like any other window by clicking on its close button.

Backgrounds are normally stored in ~/GNUstep/Library/WindowMaker/Backgrounds or in /usr/share/WindowMaker/Backgrounds. You can quickly set a background from the command line by using this command:

wmsetbg -u myfile.jpg

There are many more options for wmsetbg in its man page.

GNOME applications will have a generic GTK display; most likely, you’ll want to change this. This can be fixed by using the gnome-settings-daemon. However, the gnome-settings-daemon will want to manage your background as well; use the gconf-editor program to fix this; go into /apps/gnomesettingsdaemon/plugins/background and change the active key to false by clicking on (and clearing) the checkbox. You can also use gconftool-2 at the command line:

gconftool-2 --set /apps/gnome_settings_daemon/plugins/background/active --type bool False

To use the gnome-settings-daemon, put it into the ~/GNUstep/Library/WindowMaker/autostart file. This file will handle automatic startup of programs that either don’t recognize an X session or that are daemons. Here is the one that I use:

#!/bin/sh

xset mouse 20/10 4
gnome-settings-daemon &
dropbox start &

Window Maker will not finish starting until this script is complete, so long running applications should be backgrounded.

If you like shadows, you might want to use the package xcompmgr. Run it from your autostart file with a command like:

xcompmgr -C -c -r 8 -t -12 -l -12 &

A problem with xcompmgr is that it hasn’t been updated in quite a long time – and there is talk that it has memory leaks and other bugs. An alternative is the Cairo Composite Manager, although it is not in the Ubuntu repositories. One benefit of the Cairo Composite Manager is that it will use the 3D capabilities of your video card if it has them.

If you want transparency, then load the transset-df package. This package will set transparency of specific windows, but may have bugs and other problems – it hasn’t been updated in many years. Use it like this:

transset 0.7

This will set a transparency of 0.7 (70%) on the window that is clicked. You can select windows by using the -n option which will set transparency of windows with an associated name – for instance, -n Terminal will make all Terminal windows transparent by the specified amount. However, in my experience, using transset resulted in some windows not displaying correctly – the colored terminal window became gray and was not transparent; your experience may differ. There may also be bugs in transset; caveat emptor.

One thing missing from the Window Maker desktop is the ability to control networking interfaces. To fix this, you’ll have to stop the network-manager service as it doesn’t work with Window Manager, and replace it with the wicd package:

sudo service network-manager stop
sudo mv /etc/init/network-manager.conf /etc/init/network-manager.conf.noboot
sudo apt-get install wicd wicd-daemon wicd-gtk python-wicd
sudo service start wicd
wicd-client -n

Configure the network options as you desire. You’ll most likely want to keep the icon for this application around for easy access; if it is under the Clip, then right-click on the icon and select Keep Icon.

Unfortunately, there is no decent front-end for VPNs such as OpenVPN. The way to start a VPN remains the command line, at least until wicd supports VPNs.

Now having gone through this, you’ll have yourself a much nicer Window Maker desktop to work with. Have fun!

Update: I forgot to mention how to set up good icons. Window Maker will automatically choose an icon, and often its not a good choice: either the icon is low resolution and jaggy, or too big and only part of it can be seen.

Cached icons are stored in ~/GNUstep/Library/WindowMaker/CachedPixmaps/. You can find better icons by search with the locate utility – in Ubuntu, many icons can be found in /usr/share/icons/hicolor/ – and if there is a 48×48 or 64×64 icon conversion will not be necessary; just copy the file to the CachedPixmaps directory and change the icons location to the new file: right-click on the icon in the Clip (or wherever), select Settings… and change the location listed for the icon.

If you can’t find a 48×48 or a 64×64 you’ll have to convert the image: use the convert utility to do this:

convert orig-icon.png -resize 48x48 new-icon.png

You can also convert formats with convert; check its man page for details.

One more thing: here is my current desktop:

My Current WindowMaker Desktop

My Current WindowMaker Desktop

Just in case anyone asks… Wikipedia describes R.O.D the TV well.

Putting Debian packages on hold

When administering a Debian (or Ubuntu) system, putting packages on hold can be very useful. For example, if a critical part of the system is used by developers, and is continually updated, the developers will want to be aware of updates and will want to check their code in the new environment. Programs like Tomcat, Cocoon, MySQL will be in this category.

Similarly, if a critical portion of the system is to be updated, you wouldn’t want it to be part of the automatic updates – though you really shouldn’t automatically update, since you don’t know what can break until you test it.

To hold a package or packages, you should use dpkg --set-selections. If you run the command dpkg --get-selections you can see what is set already:

# dpkg --get-selections | head
acct                                            install
adduser                                         install
apparmor                                        install
apparmor-utils                                  install
apt                                             install
apt-transport-https                             install
apt-utils                                       install
aptitude                                        install
at                                              install
auditd                                          install

As an example, let’s consider the package dnsutils. Let’s see what would happen before we do anything:

# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  bind9-host dnsutils libbind9-60 libdns64 libisc60 libisccc60 libisccfg60 liblwres60
8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,257kB of archives.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]? n
Abort.

Now let’s change this. We’ll put the package dnsutils on hold using dpkg --set-selections:

# echo dnsutils hold | dpkg --set-selections

Let us check the results:

# dpkg --get-selections | grep dnsutils
dnsutils                                        hold

Now, when we try the system update again, things have changed:

# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
  bind9-host dnsutils libbind9-60 libdns64 libisc60 libisccfg60 liblwres60
The following packages will be upgraded:
  libisccc60
1 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
Need to get 29.9kB of archives.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]? n
Abort.

Now, dnsutils – and its related packages – are being held back, just as we wanted. The other packages are being held back because they are only required by dnsutils; without upgrading dnsutils, they won’t be upgraded either.

OSS Audio support removed in Ubuntu Maverick Meerkat

According to this bug report (or feature request), all OSS support in Maverick Meerkat (10.10) was removed. This means that a lot of programs which are included in the base Ubuntu install do not work out of the box – and may never work if they are not updated. For some odd reason, this removal of OSS did not make it into the release notes for Ubuntu 10.10.

It appears that Debian has not removed OSS.

The removal of OSS by the Ubuntu team was continually refered to as part of “cleaning up the audio mess”; one person created a large graph of the Linux audio subsystems – which shows the complicated layout of all of the Linux audio possibilities.

There are numerous programs (in Ubuntu repositories and outside) that this affects, including:

  • tksnack
  • xawtv
  • tvtime
  • gnomeradio
  • Enemy Territory
  • Quake
  • Unreal Tournament 99
  • Video4Linux
  • mythTV
  • Wine
  • mplayer/mencoder
  • Mixer.app
  • Kino
  • XSquawkBox plugin for X-Plane
  • transcode
  • Audacity (if using OSS)
  • gtkguitune
  • CMU Sphinx (PocketSphinx)
  • gtick
  • Sniper Elite
  • VLC (if using OSS)
  • wxtoimg

These are just the ones that I could track down; there almost certainly dozens of others. Unfortunately, the removal of OSS was done without notification to users or developers, and the general response is that the programs are broken if they use OSS. This means that Ubuntu Maverick Meerkat was shipped with programs that had no audio because of missing OSS support.

This is no way to create a distribution. The only fix is to revert the “fix”: compiling your own kernel with OSS support. If you do this, you’ll also want to pin the version to the one installed so APT doesn’t try to overwrite your version with a new kernel without OSS support. There is a good tutorial that shows you how to compile your own Ubuntu kernel.

Why I use Window Maker

I keep coming back to Window Maker as a desktop environment. Why? Sometimes I ask myself the same question. However, there are a number of reasons.

Simplicity. There’s not a ton of things happening on the Window Maker screen; basically, there is the Dock and the Clip and minimized application windows. That’s it. This simplicity also translates into lower memory usage.

Light-weight. This is a biggie for me. First time I truly used Window Maker in any depth was on OpenBSD installed onto a Macintosh Quadra 800. It took a while to build, but it built and was comfortable – and in that environment, GNOME or KDE was out of the question.

Unique. This doesn’t sound like a positive aspect – but to me, I love learning a new environment. After a while, the other environments can blend together. The dock is also much better than any that have followed it; others like the MacOS X dock are too small and don’t work well in a user interface. Window Maker is very clean and simple and easy to use.

Since it is included as an available package on almost every Linux installation, what does it take to make a good Window Maker desktop? With Ubuntu, KDE can be installed by installing the kubuntu-desktop; XFCE can be installed by installing the xubuntu-desktop. There is no equivalent Window Maker desktop unfortunately.

For Ubuntu, you might want to install the following packages using Synaptic or apt-get:

The menu application makes sure that all of your applications that are installed with Ubuntu are also available in the Window Maker menus.

There are other packages you may like; search for packages that end in “.app” or begin with “wm” for starters.

For wireless management, you’ll want to stop the network-manager and then install wicd instead. This is because network-manager requires a KDE/GNOME style desktop. Do this with the following:

sudo service network-manager stop
sudo apt-get install wicd wicd-gtk
sudo service wicd start
wicd-gtk &

When you do this, you should be able to configure the wireless connections as you see necessary. Note that there is no graphic controls for VPNs at this time, but you can control them from the command line.

So what do I have in my dock? Here is a list:

  • firefox (application)
  • wmbattery
  • wicd (application)
  • wmdiskmon
  • wmclockmon
  • wmcpuload
  • wmwave
  • wmmaiload
  • wmcalclock
  • wmdrawer

All of these are part of the standard Ubuntu repositories; unfortunately, wmbatteries is not. However, you can get that dock applet (or “dockapp”) and more from dockapps.org.

There are a lot of resources for learning Window Maker, although some are dated; these are all good places to go:

Hopefully, this won’t be too many resources; most are quick overviews. Don’t be afraid to try out Window Maker today!

Follow

Get every new post delivered to your Inbox.

Join 114 other followers