JDPFu.com 2025

I’m asking for help again with my Windows7 final installation. See, Microsoft gave the 32-bit version, not the 64-bit version. This puts a wrinkle in my original plan to host Win7 on the laptop because about 0.6GB of RAM cannot be used. On a system with only 4GB, 0.6GB is a bunch, perhaps too much to waste.

The current goal is:

JeOS/Linux-Host
|____Win7-VM (MCE)
|____xubuntu-VM
|____WinXP-VM (Visio / MS-Office / Quicken)

RAM allocation plans

JeOS – 512MB

Win7 – 1GB

WinXP – 1GB

xubuntu – 1.5GB

If Media Center in Win7 doesn’t work well enough in a VM; safe to leave on 24/7 with USB support, this plan will be trashed. The QAM recording is nice. For me, it is about the recording, not the playback or other features.

There are other complications in using Win7 Media Center. The recorded file format, for example. That’s something for another story.

32-bit DVD – ouch.

So I opened the Windows7 Ultimate DVD and uncovered that it only contains the 32-bit version. After swapping the old/Vista drive with the new/Empty drive in the laptop, I elected to install Win7 even though I’d end up unable to use about 0.5GB of RAM. I wanted to give the new OS a fair chance and gain some experience. The setup was fairly easy, but dumbed down too much for my liking. I actually installed the OS to the wrong partition (280GB), wiped it and reinstalled to the other partition (30GB), that was planned for OS and Apps. Then I proceeded to setup WMC – Media Center.

Windows Media Center – ClearQAM Supported!

I’d heard that ClearQAM was supported and looked forward to using it. My cable system switched almost all channels to QAM 2 months ago. I’d hoped there was an automatic translation between QAM channels and normal cable channels so guide data can be used. I haven’t found that, if it exists. I AM recording a movie as I write this. There’s no noticeable performance hit. Nice.

Data Migration

Overnight, I copied the data and virtual machines from the older drive to the new drive, about 150GB. I split the disk into 2 partitions – C: and D: . C is for the OS and programs. D is for data and virtual machines. This config should make data backup much easier.

I dislike the whole Library BullShite that this new OS forces. I also dislike the new Explorer look and feel. Is there a way to default all Explorer views to Detailed?

VirtualBox Migration

So, after getting the new OS installed, the very first program installed was Sun’s VirtualBox. Initial attempts to migrate all the settings and virtual disks didn’t work as well as I’d hoped. However, I did get 1 VM up and running with a small amount of effort. I’ll write up the actual steps and things that didn’t work in another post. There may be another way to migrate the settings, I did retain both XML files for Vbox and for each VM, so seeing the specific differences should be easy.

The next trick is to migrate a vbox image that includes a snapshot image. I’m cautiously hopeful for a good outcome with that.

I need your help deciding how to use the free Windows7 Ultimate license Microsoft gave away yesterday. I want to use it on my laptop but need some considered feedback on how would be best?

Current Laptop Config

1. 4GB of RAM – may put 8GB in later
2. 320GB disk
3. Main OS is Vista-64bit Home Premium
4. VirtualBox 3.0.6 for Virtual Machines
   1. xubuntu
   2. WinXP Pro
   3. Ubuntu
   4. OpenSolaris
   5. FreeBSD

Initial Thoughts

My initial thoughts are to

• replace Vista with Win7-64
• eventually remove my WinXP-Pro VirtualBox
• use the built-in WinXP Compatibility layer

I spend 14 hrs a day in the xubuntu VM and only boot WinXP to run Quicken, a few MS apps and access TrueCrypt data. Perhaps 3 times a week.

Questions?

1. How good is the USB support in the WinXP VM?
2. HDMI output?
3. GigE networking – WiFi networking?
4. How good is the driver compatibility for Win7-64? All-in-One Fax, printer, scanner, old Creative Xen and built-in laptop camera are the only devices I see using, in addition to normal flash and ext USB disk drives.
5. Hauppauge 950Q ClearQAM TV tuner must work.
   • Does Media Center work with this TV tuner and ClearQAM? The current MCE doesn’t.
6. Can I consider Win7-32bit at all. Does it access the full 4GB of RAM? Is an upgrade to Win7 64-bit easy?
7. TrueCrypt, MS-Visio, MS-Office 2007, and VideoRedoPlus are the only uses for Windows that I have. No gaming, er … very little gaming.

Choices

1. Run Win7 in a VM, get used to it. Decide later
2. Backup the data and VMs, repartition the disk for OS, Apps, Data, and install Win7 ??-bit as the main OS
   • 32-bit or
   • 64-bit?

Thoughts and suggestions? Did I miss an option?

Getting Syslog, Pound and Mongrel to work with Awstats

If you run the Ruby on Rails blog, Typo, it is likely you are using Mongrel as a cluster server and not Apache. Mongrel is easy – really easy. If you need 5 backend ruby servers, change 1 entry in the mongrel_cluster.yml file and restart.

Pound is a very simple load balancer written in perl. Many very busy websites use it. Slashdot for example gets 40M pages a day, all going through pound. Scalable? Check.

I’ll assume you already have syslog, awstats, pound, and typo/mongrel installed and simply want better logging. Explaining the nontrivial setup of these, sometimes complex, systems is not something handled in a blog. You’ll need to be root or have root editing via sudo to make this happen. A knowledge of manpages won’t hurt either.

So now you have two non-standard programs handling your web traffic, mogrel and pound. You’d like to get some normal website statistics about your users. By default, pound logs via syslog. We love syslog, but we don’t like that pound doesn’t use a separate file, by default. All your web traffic logs get intermixed with login, attempted hacks, disk failures and other system messages.

Below we’ll

1. describe the syslog setup to trap pound messages and drop them into a new logfile – /var/log/pound.log
2. setup the new logfile to be automatically created, should it disappear
3. setup pound to write the the new pound.log file via syslog
4. automatically perform log file rotation, in the normal way
5. create a custom log_file_format so awstats gets all the data it can from the logs
6. be certain that restarting pound gets syslog to bounce or restart too

Here are the changes specific to the logging changes. You should have other changes for your server/domain already in these files. Before starting, you probably want to run an

awstats.pl -config=domain.com -update

to capture the latest stats before you move them all into a new location.

/etc/awstats/awstats.domain.com.conf:

 LogFile=“/var/log/pound.log”

 LogFormat=“%time3 – - %host_r %host – - %time1 %methodurl %code %bytesd %refererquot %uaquot”

 SkipHosts=“REGEX[^192\.168\.]”

/etc/syslog.conf

 .;auth,authpriv.none,local0.none              -/var/log/syslog

 local0.*                        -/var/log/pound.log

See, local0 – local7 are approved syslog classes. They are meant to be used just like this. Syslog know about them, we need to be certain that pound will use local0 too. If your system is using local0, then select 1, 2, 3 … local7, which ever isn’t already in use.

/etc/pound/pound.cfg

LogFacility local0

 LogLevel        3

You’ll need to have your Service, Redirect, and BackEnd stanzas too.

/etc/init.d/pound

 if [ ! -e “/var/log/pound.log” ] ; then

   log_warning_msg “Creating pound.log …”

   touch /var/log/pound.log

   chmod 0644 /var/log/pound.log

   chown syslog.adm /var/log/pound.log

   /etc/init.d/sysklogd reload > /dev/null

 else

   log_success_msg “pound.log was found”

   /etc/init.d/sysklogd reload > /dev/null

 fi

/etc/logrotate.d/pound

/var/log/pound.log {

        daily

        missingok

        rotate 14

        compress

        delaycompress

        notifempty

        create 640 syslog adm

        sharedscripts

        postrotate

                /etc/init.d/sysklogd reload > /dev/null

        endscript

 }

Good enough? Now just restart pound with

 sudo /etc/init.d/pound restart

The next time your awstats is updated, you’ll see more and better stats. Note that we didn’t touch any of the old rrd data that awstats may have been able to parse.

This worked on an Ubuntu server 8.04 LTS running in a Xen virtual machine. There are other ways to do this and some settings can be changed without impacting whether this continues to work or not.

Obviously, your situation will be a little different and you’ll need to figure out which differences matter and which don’t. Did I miss something important or does anything need clarification? Use the comments or talkback to let me and other readers know, please.

For the last 10 years, I’ve been doing batch jobs on my server the hard way.

That’s a big confession. For 10 years, I’ve been doing it the hard way. You know, you have a bunch of things to get done, but don’t want them to all run at the same time. Hundreds of little jobs, or perhaps 20 BIG jobs, it doesn’t matter. All this time, I’ve been using at as a manual scheduler. Basically, do something in 20 min, or 60 min or 2 hours or next Friday. Whatever, at is fairly powerful, but for batch jobs where the goal is to use the CPU to the fullest, but not overtax it, at is less than ideal. There could be too many jobs running or unused CPU time. Inefficient.

Then I finally followed up on a freshmeat.net annoucement – ts. Task Spooler, ts, is just that, a queue of tasks. It is a queue where you submit batch jobs to be run. By default, there’s no configuration needed. At this point, I’m not using any configuration. Basically, you pre-pend ‘ts’ in front of your normal command and it adds each to the queue. Installation was trivial – make install
Command line options work as you’d expect – they are passed to the batch unmolested. The environment is also properly retained.

$ ts encode_video some_video_1.mpg
$ ts encode_video some_video_2.mpg
     o
     o
     o
$ ts encode_video some_video_30.mpg
$ ts encode_video some_video_40.mpg

is all that is needed. To monitor your jobs, run ‘ts’ alone.

I’ve told my ts server to run 2 jobs, since I have a dual core processor. It will always ensure no more than 2 jobs are running. The output can be captured and logged or stored into files, or whatever.

You do have to clean up the list of jobs occasionally. That’s just ‘ts -C’`.

It understands that you may want more than 1 queue – using environment variables, you can setup multiple queues with different settings. Then you can set your scripts to use whatever job queue you like. To setup different queues, just set the environment variable that controls the FIFO used. Here’s an example.

export TS_SOCKET=/tmp/tlm-stuff

Uses for different queues?

• a download queue
• a backup queue
• a CPU intensive use queue

Oh, source code is provided. I’m using it on Linux, but guess it will work on any POSIX compliant OS. Get it here.

Today, I upgraded my backend search indexer to the latest version of swish++. I’ve been using swish-e and swish++ for YEARS and YEARS. We’re talking about 10 here. I’ve also used htdig and been mostly happy with it.