JDPFu.com 2024

The script below was created as part of an Alfresco upgrade process and meant to be run manually. This is fairly trivial cold backup script for Alfresco 2.9b, which is a dead release tree from our friends at Alfresco. It hasn’t been tested with any other version and only backs up locally, but could easily backup remote with sshfs or nfs mounts or even rdiff-backup commands swapped in.

For nightly backup of our production servers, we actually perform rdiff-backups of shutdown virtual machines, which take about 3 minutes each. That little amount of downtime to have a differential backup of the entire VM is worth it to us.

#!/bin/sh
# ###############################################################
# This script should not be run from cron. It will wait for the mysql
# DB password to be entered.
# 
#  Created by JDPFU 10/2009
# 
# ###############################################################
# Alfresco Backup Script - tested with Alfresco v2.9b
#   Gets the following files
#    - alf_data/
#    - Alfresco MySQL DB
#    - Alf - Extensions
#    - Alf - Global Settings
# ###############################################################
export TOP_DIR=/opt/Alfresco2.9b
DB_NAME=alfresco_2010_8392
export EXT_DIR=$TOP_DIR/tomcat/shared/classes/alfresco/extension
export BACK_DIR=/backup/ALFRESCO
export BACKX_DIR=$BACK_DIR/extension

# Shutdown Alfresco
/etc/init.d/alfresco.sh stop

# Backup the DB and important files.
# dir.root setting will change in the next version
/usr/bin/mkdir  -p $BACK_DIR
cd  $BACK_DIR/; 
/usr/bin/rsync  -vv -u -a --delete --recursive --stats --progress $TOP_DIR/alf_data $BACK_DIR/

echo "
  Reading root MySQL password from file
"
/usr/bin/mysqldump -u root \
    -p`cat ~root/bin/$DB_NAME.passwd.root` $DB_NAME | \
    /bin/gzip > $BACK_DIR/${DB_NAME}_`date +%Y%m%d`.gz
/usr/bin/find  $BACK_DIR -type f -name "$DB_NAME"/* -atime 60 -delete

/usr/bin/cp  $TOP_DIR/*sh $BACK_DIR
/usr/bin/mkdir  -p $BACKX_DIR
/usr/bin/rsync  -vv -u -a --delete --recursive --stats --progress  $EXT_DIR/* $BACKX_DIR/

# Start Alfresco
/etc/init.d/alfresco.sh start

Why a cold backup? Unless you have a really large DB, being down a few minutes isn’t really a big deal. If you can’t afford to be down, you would already be mirroring databases and automatically fail over anyway. Right?

We use a few extensions for Alfresco, that’s why we bother with the extensions/ directory.
There are many ways to make this script better. It was meant as a trivial example or starting point to show simple scripting methods while still being useful.

A quick script to change a capital I (eye) in the middle of a word into a lowercase l (el). If you like Asian films, you understand why I wrote this script. I had an itch. It needed to be scratched. This is useful for .srt files used in movie subtites.

#!/usr/bin/perl
# Perl script to change every 'I' into an 'l' in the middle of a word
# input is stdin and output is to stdout; redirection is your friend
my $line;
while(<>){
 chomp;
 $line=$_;
 $_=$line;
# Match lines with non-whitespace characters leading a capital I 
  if ( m/[\S]I/ ){
     $line =~ tr/I/l/;
  }
  print "$line\n";
}

It is very common for subtitle files, SRT format, to have a capital I in the middle of words since bitmap patterns are used to create the files. For native speakers of English, this is HIGHLY distracting – to the point that the subtitles must be fixed before a movie can be enjoyed.

I tried a few other methods, before determining this simple character translation was needed.

1. ispell – There were too many words that were not in the dictionary and spacing of words often groups them in strange ways.
2. replacement dictionary – I created a hundred word dictionary replacement sed script. There were always new words that needed to be added for every SRT file.
3. Manual editing – yep, I spent a few hours manually editing files. This wasn’t very efficient and ruined the movie plot since I’d already read it before viewing it.

Some combination of methods will probably be necessary. I intend to merge them into a single perl script and perform them in the most efficient order. It will begin with the I—>l translation.

Part of me has wished I had a fancy IDE like visual studio provides, but without the commercial license and heavy system resources. I usually find vim to be enough syntax highlighting and it is definitely light weight. Occasionally, jEdit or Notepad++ seems like the best compromise between small, fast and features.

This morning I found Geany. It feels like emacs without the memory requirements. It has tabbed files and function indexes. The view for each function or code block can be compressed/hidden easily.

Best of all, it doesn’t depend on either Gnome or KDE toolkits, so it doesn’t force specific desktop environment libraries to be loaded, reducing RAM requirements for half of us that use the othe OE.

I hope my fellow Linux users find this code editor – mini-IDE useful.

Below is a script that will playback a group of audio files in order, grouped by day. Suppose you have files named like this

File-1×01.mp3

   .

File-1×12.mp3

File-2×01.mp3

   .

File-2×10.mp3

File-3×01.mp3

   .

File-3×10.mp3

   .

File-7×10.mp3

File-8×01.mp3

   .

File-8×10.mp3

and want to play group Yx1-6 followed by group Yx6-12 daily. If you just wanted to do this for 1 set of files, it would be easier to just use `at` to play them. But you might have 10-50 files like this and only want to worry about setting up playback once a month or so.

Here’s the bonehead shell script that I’m using to accomplish this.

#!/bin/sh


	
Program to playback  audio tapes in order
	
based on day of the month – best to start on 1st.

#
	
It isn’t pretty, but it works assuming you want to cover
	
half a lesson each day. The filenames look like this:
	

	
File-8×10.mp3

MP=/usr/bin/mplayer

DIR=/Data/Audio/Session1

FILE_ROOT=File

DATE=`date “+%d”` # Returns the day of the month

ODD=`expr $DATE % 2`

START_GRP=`expr 1 + $DATE / 2` # pick a start date

ODD_START_NO=“01 02 03 04 05 06”

EVEN_START_NO=“06 07 08 09 10 11 12”

FILE_EXT=mp3

	
Changing the 0 to a 1 will toggle which group of files to begin

if [ $ODD = “0” ] ; then

  START_NO=$ODD_START_NO

else

  START_NO=$EVEN_START_NO

fi

for count in $START_NO; do

   afile=“$DIR/$FILE_ROOT-${START_GRP}x$count.$FILE_EXT”

   if [ -f “$afile” ] ; then

      $MP “$afile”

   else

      echo " File missing: $afile"

 fi

done

exit;

So, it isn’t very pretty and it is dependent on starting the script on the first of the month. Since today happens to be Oct 31 and I just finished the first group, I tweaked the EVEN/ODD and date modulus to jump 1 day ahead tomorrow – Nov 1. It will fail when a month roles over to the next month.

A fix to that problem would be to convert the date into a Julian day of the year, DOTY, and subtract off the current DOTY from the starting date. Check out date "+%j" for more on Julian dates. Of course, then it will break at the new year, so perhaps getting the number of seconds since epoch and performing calculations based on that would be even better? Even that method will break in 2038. At some point, the complexity outweighs the difficulty to implement.

Lastly, we need to setup crontab to run the script, playing the file.

1 6 * * * /home/jp/bin/daily_audio.sh

Enjoy.

You’ve heard it over and over. Backup, Backup, Backup. You are magically supposed to know how to do it and make it happen. This time, I’ll show exactly how I backup my HOME directory and manage those backups with rdiff-backup.

I’ve gone into why rdiff-backup was selected previously, but the main reasons were:

1. Incremental
2. Versioned
3. Compressed
4. Latest backup is available as a mirror so restore 1 file at a time or all at once

Install rdiff-backup

Run synaptic package manager or whatever package manager you prefer and install the rdiff-backup package.

The Nokia N800/N810/N900 has limited CPU. It is a portable device with fairly long battery life, so this is understandable. However, playback of DVDs or other videos can use the battery beyond what is needed for the screen size. According to Nokia, the optimal playback for video is 400 pixels.

Below is a small script to convert a list of input videos into the “best” quality for our Nokia Internet Tablets. The output does not playback with the built-in Media Player, but plays nicely with mplayer – or gmplayer if you want a GUI.

This script was updated 7/2010 to reduced FPS so when multitasking, the N800 does not become over committed for CPU. I chose 14.985 fps because it seemed to have acceptable playback and acceptable visuals. The fact that it is exactly half the original source frames makes the transcode happen quickly too. If you find the 15 FPS is too much, 20 or 25 FPS will work pretty well too.

#!/bin/sh
# This is for really simple XVID conversion to 400 x whatever, retaining aspect
# Input filenames with spaces are not supported due to the ability to have multiple input files. 
#     Remove the loop to support a single input file with spaces.
# This is a 1-pass solution, so quality could be improved using a 2-pass method
SCALE=",scale=400:-3"
XVIDENCOPTS="fixed_quant=4:max_key_interval=250:trellis:max_bframes=1:vhq=3"
FRAMES="-ofps 15000/1001"

for filename in $@ ; do
   IN=$filename
   nice /usr/bin/mencoder "$IN" $FRAMES -oac mp3lame -lameopts preset=128 -ovc xvid \
                -vf lavcdeint${SCALE} -noodml -forceidx -ffourcc XVID \
                -xvidencopts ${XVIDENCOPTS} -of avi -o "${IN}-n800.avi"
done

For me, this script works quickly and with about 90% of the input files. Basically, anything that mplayer can play (which is just about any non-DRM video files), then you can transcode. I bet other portable media devices like the iPhone, iTouch, and Android-based devices will like this format too.

I’ve used this with FLV, MPEG2 1280i HD, and everything in between to bring it to my N800 so I don’t get bored during workouts or airplane flights. Enjoy.

Some of us have older media playback devices, not PCs, but dedicated devices like a MediaGate or MASSCOOL. These devices let you playback TV and Movie files in the popular VOB, MPEG, Divx, xvid formats. Some allow network playback too from SMB shared folders. Nice.

They have issues playing HiDef content. Here’s a solution to that problem.

Sometimes you find a new podcast that you’d like to catch up on and the RSS feed only has the most recent releases. Then you locate a mirror, but the file locations aren’t easily programmed. What do you do?

Steps to clone/backup a VirtualBox domain

that can be restored to the same or a different machine. Start by shutting down the VM to be backed up/cloned. Then …

$ VBoxManage clonevdi Master.vdi Clone.vdi

and the (undocumented) function

$ cp Master.vdi Clone.vdi
$ VBoxManage internalcommands setvdiuuid Clone.vdi

The 2nd command stuffs the clone with a new SID.

Warning – this doesn’t backup any SNAPSHOTs

you may have made below the {root} VDI file. Here’s a directory view

 C:.

 ├───Machines

 │   ├───Ubuntu Upgrades

 │   │   └───Logs

 │   └───WinXPPro

 │       ├───Logs

 │       └───Snapshots

 └───VDI

VDI/ contains the base installations. Machines/ contains each XML DomU description file and snapshots where they exist. You need these for your backup too. Copying the .vdi files is not enough to back them up. While it will probably work on the same PC with VirtualBox, taking those files to another machine with a slightly different configuration may not work. Use the VBoxManage command.

_* I’m not responsible if this doesn’t work for you. Always work with test data when you are doing any of this until you prove it works on your machine.

For the last 10 years, I’ve been doing batch jobs on my server the hard way.

That’s a big confession. For 10 years, I’ve been doing it the hard way. You know, you have a bunch of things to get done, but don’t want them to all run at the same time. Hundreds of little jobs, or perhaps 20 BIG jobs, it doesn’t matter. All this time, I’ve been using at as a manual scheduler. Basically, do something in 20 min, or 60 min or 2 hours or next Friday. Whatever, at is fairly powerful, but for batch jobs where the goal is to use the CPU to the fullest, but not overtax it, at is less than ideal. There could be too many jobs running or unused CPU time. Inefficient.

Then I finally followed up on a freshmeat.net annoucement – ts. Task Spooler, ts, is just that, a queue of tasks. It is a queue where you submit batch jobs to be run. By default, there’s no configuration needed. At this point, I’m not using any configuration. Basically, you pre-pend ‘ts’ in front of your normal command and it adds each to the queue. Installation was trivial – make install
Command line options work as you’d expect – they are passed to the batch unmolested. The environment is also properly retained.

$ ts encode_video some_video_1.mpg
$ ts encode_video some_video_2.mpg
     o
     o
     o
$ ts encode_video some_video_30.mpg
$ ts encode_video some_video_40.mpg

is all that is needed. To monitor your jobs, run ‘ts’ alone.

I’ve told my ts server to run 2 jobs, since I have a dual core processor. It will always ensure no more than 2 jobs are running. The output can be captured and logged or stored into files, or whatever.

You do have to clean up the list of jobs occasionally. That’s just ‘ts -C’`.

It understands that you may want more than 1 queue – using environment variables, you can setup multiple queues with different settings. Then you can set your scripts to use whatever job queue you like. To setup different queues, just set the environment variable that controls the FIFO used. Here’s an example.

export TS_SOCKET=/tmp/tlm-stuff

Uses for different queues?

• a download queue
• a backup queue
• a CPU intensive use queue

Oh, source code is provided. I’m using it on Linux, but guess it will work on any POSIX compliant OS. Get it here.