JDPFu.com 2025

For the last 10 years, I’ve been doing batch jobs on my server the hard way.

That’s a big confession. For 10 years, I’ve been doing it the hard way. You know, you have a bunch of things to get done, but don’t want them to all run at the same time. Hundreds of little jobs, or perhaps 20 BIG jobs, it doesn’t matter. All this time, I’ve been using at as a manual scheduler. Basically, do something in 20 min, or 60 min or 2 hours or next Friday. Whatever, at is fairly powerful, but for batch jobs where the goal is to use the CPU to the fullest, but not overtax it, at is less than ideal. There could be too many jobs running or unused CPU time. Inefficient.

Then I finally followed up on a freshmeat.net annoucement – ts. Task Spooler, ts, is just that, a queue of tasks. It is a queue where you submit batch jobs to be run. By default, there’s no configuration needed. At this point, I’m not using any configuration. Basically, you pre-pend ‘ts’ in front of your normal command and it adds each to the queue. Installation was trivial – make install
Command line options work as you’d expect – they are passed to the batch unmolested. The environment is also properly retained.

$ ts encode_video some_video_1.mpg
$ ts encode_video some_video_2.mpg
     o
     o
     o
$ ts encode_video some_video_30.mpg
$ ts encode_video some_video_40.mpg

is all that is needed. To monitor your jobs, run ‘ts’ alone.

I’ve told my ts server to run 2 jobs, since I have a dual core processor. It will always ensure no more than 2 jobs are running. The output can be captured and logged or stored into files, or whatever.

You do have to clean up the list of jobs occasionally. That’s just ‘ts -C’`.

It understands that you may want more than 1 queue – using environment variables, you can setup multiple queues with different settings. Then you can set your scripts to use whatever job queue you like. To setup different queues, just set the environment variable that controls the FIFO used. Here’s an example.

export TS_SOCKET=/tmp/tlm-stuff

Uses for different queues?

• a download queue
• a backup queue
• a CPU intensive use queue

Oh, source code is provided. I’m using it on Linux, but guess it will work on any POSIX compliant OS. Get it here.