Handy One-Liners

  • Show all MySQL Users’ GRANTS:
     Bash | 
     
     copy code |
    ?

    1
    2
    # oIFS="$IFS"; export IFS=$'\n'; for i in $(mysql --skip-column-names --batch -e "SELECT user,host FROM mysql.user"); do USER=$(echo $i|awk '{printf $1}'); HOST=$(echo $i|awk '{printf $2}'); mysql --skip-column-names --batch -e "SHOW GRANTS FOR '$USER'@'$HOST'"; done; export IFS="$oIFS";
    3
    		
Posted in Uncategorized | Leave a comment

YtChannelRss: Generating RSS Feeds from YouTube Channels

My favorite internet video publisher, RoosterTeeth (and their sister Achievement Hunter) recently launched a new YouTube channel, LetsPlay, dedicated to their LetsPlay series of videos. Unfortunately, I quickly got behind on the videos, and wanted to catch up. Having 80 videos published to their channel already, I wished to somehow subscribe to the channel in an RSS Aggregator to manage viewing the videos in order.

I found that YouTube offered a channel RSS feed, formatted as such for LetsPlay:
http://gdata.youtube.com/feeds/api/users/LetsPlay/uploads

However, this only shows the most recent videos. Unfortunately, I was unable to find any offered feed that contained all of the videos.

I decided to make my own feed. Using Google’s API, I was able to generate a list of all of the videos from the channel. Simple enough, now I just had to generate the RSS Feed.

As much as I love Perl, Google provides their API Client Library in Python, and sure enough, an easy-to-use Python module was available to generate RSS XML Files

The result was YtChannelRss:
https://github.com/tonyborries/YtChannelRss

A single execution builds an RSS feed of all current videos. By subscribing to this static file, and YouTube’s RSS feed, I was able to get a list of all videos. As I catch up on the videos from this static list, the YouTube provided feed will keep up with the new videos… success.

 Bash | 
 
 copy code |
?

01
02
# ./YtChannelRss.py -h
03
  Usage: ./YtChannelRss.py [ -h ] [ -v ] -k APIKey -c ChannelName
04
 
05
    -h, --help      Show this usage message and exit.
06
    -v, --verbose   Verbose Output to StdErr
07
    -k, --apikey    YouTube API Key
08
    -c, --channel   YouTube Channel / Username
09
 
10
# ./YtChannelRss.py -v -k "Google-API-Key" -c LetsPlay > /var/www/html/LetsPlay.xml
11
ApiKey:  Google-API-Key
12
Channel: LetsPlay
13
Channel ID: UCkxctb0jr8vwa4Do6c6su0Q
14
Found 81 Videos
15
Found 1 Channels
16
Found 20 Playlists
17
Building RSS with 81 Items
18

Posted in Uncategorized | Leave a comment

Django SyncDB and Perl Expect

I’m writing a database migration script in Perl to upgrade to a new database schema. This script often calls Django’s manage.py ‘syncdb’ function to rebuild from the new models. However, there’s one problem with this. Each time it runs, it asks the question:

You just installed Django’s auth system, which means you don’t have any superusers defined.
Would you like to create one now? (yes/no):

Well, I’m copying the data from the legacy database as soon as the new database is built, so no, I don’t want to create the new user. The problem though, is that this question comes up in the middle of the script, which can take up to an hour to run. This means I have to be attentive to the script. I can’t just start it and head to lunch. ‘Expect’ you say? Well, luckily There’s a Perl Module for that.

 Perl | 
 
 copy code |
?

01
02
use Expect;
03
use Term::ANSIColor;
04
# use Expect to automatically answer 'no' to the create superuser prompt
05
my $exp = Expect->spawn("../../syncdb.sh")
06
        or die ("Can't spawn ../../syncdb.sh");
07
 
08
# patidx = Pattern Index, the index of the last pattern matched, starting at 1
09
# so the 2nd pattern, which should be the last, leaves patidx at 2 if successful
10
my $patidx = $exp->expect(60,
11
        [qr'Would you like to create one now\? \(yes\/no\):',
12
                sub {$exp->send("no\n");exp_continue}],
13
        [qr'No fixtures found',
14
                sub {print color('Green'), "Finished syncdb.sh", color('Reset'), "\n";}],
15
        );
16
my $close_status =  $exp->soft_close();
17
if ($close_status != 0 and $patidx != 2) {
18
        print color('Red'), "Unknown error with syncdb.sh", color('Reset'), "\n";
19
        exit(1);
20
}
21

You may have to adjust the second matching pattern depending on your schema. This is the last line returned from the ‘manage.py syncdb’ script in my example.

Posted in Uncategorized | Leave a comment

jQuery ajaxForm Tip

I recently added a jQuery ajaxForm (which submits a form’s contents and returns the data in an ajax call instead of a page load) and I had problems getting it running. It kept throwing errors, and they didn’t seem to make sense at first:

01
02
  $(document).ready(function()
03
  {
04
      $("#searchForm").ajaxForm({
05
      url:'/tools/searchAudioFileNames',
06
      dataType:'json',
07
      success:searchFormCallback 
08
  });
09
});
10

1
2
<form id="searchForm" name="searchForm" action=""&gt;
3
  <input type="text" name="searchString" value="Search"/>
4
  <input type="submit" name="method" value="Search"/>
5
</form>

1
2
Uncaught TypeError: Object #<HTMLInputElement> has no method 'toUpperCase'  jquery.form.js:111
3

Comparing to some working code, I just couldn’t find the differences. Well, after digging through the jQuery source, I finally found it (okay, it only took about 2 minutes, but I spent about 10 looking for the solution on Google first). ajaxForm() expects the form to be define with a method, so I just had to throw in the GET.

1
<form id="searchForm" name="searchForm" action="" method="GET">

I love Open Source 🙂

Posted in Uncategorized | Leave a comment

Multiple GPUs Not Showing Up in CUDA

We recently upgraded our server, adding 6 Tesla GPUs to our existing 580 GTX. However, once we had them installed, we noticed some issues, namely, they weren’t fully being recognized.

1
2
$ deviceQuery --noprompt | grep "^Device"
3
Device 0: "GeForce GTX 580"
4

However, they were being detected and the devices were being setup:

01
02
$ nvidia-smi -a | grep "^GPU"
03
GPU 0000:08:00.0
04
GPU 0000:0A:00.0
05
GPU 0000:0D:00.0
06
GPU 0000:8B:00.0
07
GPU 0000:8D:00.0
08
GPU 0000:96:00.0
09
GPU 0000:98:00.0
10
 
11
$ sudo lspci | egrep "3D controller|VGA compatible"
12
08:00.0 3D controller: nVidia Corporation GT200 [Tesla C1060] (rev a1)
13
0a:00.0 3D controller: nVidia Corporation GT200 [Tesla C1060] (rev a1)
14
0d:00.0 VGA compatible controller: nVidia Corporation GF110 [GeForce GTX 580] (rev a1)
15
10:04.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)
16
8b:00.0 3D controller: nVidia Corporation GT200 [Tesla C1060] (rev a1)
17
8d:00.0 3D controller: nVidia Corporation GT200 [Tesla C1060] (rev a1)
18
96:00.0 3D controller: nVidia Corporation GT200 [Tesla C1060] (rev a1)
19
98:00.0 3D controller: nVidia Corporation GT200 [Tesla C1060] (rev a1)
20
 
21
$ ls -lha /dev/nv*
22
crw-rw-rw- 1 root root 195, 0 2012-05-17 16:29 /dev/nvidia0
23
crw-rw-rw- 1 root root 195, 1 2012-05-17 16:29 /dev/nvidia1
24
crw-rw-rw- 1 root root 195, 2 2012-05-17 16:29 /dev/nvidia2
25
crw-rw-rw- 1 root root 195, 3 2012-05-17 16:29 /dev/nvidia3
26
crw-rw-rw- 1 root root 195, 4 2012-05-17 16:29 /dev/nvidia4
27
crw-rw-rw- 1 root root 195, 5 2012-05-17 16:29 /dev/nvidia5
28
crw-rw-rw- 1 root root 195, 6 2012-05-17 16:29 /dev/nvidia6
29
crw-rw-rw- 1 root root 195, 255 2012-05-17 16:29 /dev/nvidiactl
30

Many users in various forums had a similar problem which was solved by setting the 666 permissions, which were already correct in our setup. However, we luckily found the resolution on http://ambermd.org/gpus/

01
02
$ echo $CUDA_VISIBLE_DEVICES
03
0
04
 
05
$ export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6"
06
$ deviceQuery --noprompt | grep "^Device"
07
Device 0: "GeForce GTX 580"
08
Device 1: "Tesla T10 Processor"
09
Device 2: "Tesla T10 Processor"
10
Device 3: "Tesla T10 Processor"
11
Device 4: "Tesla T10 Processor"
12
Device 5: "Tesla T10 Processor"
13
Device 6: "Tesla T10 Processor"
14

I never did find where this variable was being set, but now we know to include this in our scripts.

Posted in Uncategorized | 3 Comments

Fibonacci vs. Kilometers

This experiment was based on this Reddit posting: http://www.reddit.com/r/fifthworldproblems/comments/s0217/i_can_only_speak_in_a_recurring_meter_the/c4a4c18

Here’s a brief excerpt that sums up the point of this experiment:

Cool fact:

The Fibonacci sequence fairly accurately converts miles to kilometers. For each number in the sequence that represents miles, the next number is the same distance in kilometers

Wanting to test this, I threw together a quick Perl script to plot this out. The results were, surprisingly, “fairly accurate.”

I’ve included here a few quick plots, the numbers, and the raw source code should anyone wish to tinker further with this. Granted these are unrefined results from a presentation stand-point, but you get the point…

The raw results:

Miles, Calculated from Fibonacci, True
1,1,1.609
1,2,1.609
2,3,3.218
3,5,4.827
5,8,8.045
8,13,12.872
13,21,20.917
21,34,33.789
34,55,54.706
55,89,88.495

Miles, Percent Difference
1,-37.8495960223741
1,24.3008079552517
2,-6.77439403356122
3,3.58400662937643
5,-0.559353635798632
8,0.994406463642015
13,0.396806425395604
21,0.624463582822808
34,0.537418199100641
55,0.570653709249105
89,0.557956997507005
144,0.562806436019613
233,0.560954075386042
377,0.561661608360142
610,0.561391353961833
987,0.561494581832304
1597,0.561455152276278
2584,0.561470213023878
4181,0.561464460329799
6765,0.561466657663354
10946,0.561465818356607
17711,0.561466138943263
28657,0.561466016490064
46368,0.561466063263029
75025,0.561466045397334
121393,0.561466052221422
196418,0.561466049614862
317811,0.561466050610479
514229,0.56146605023018
832040,0.561466050375433

The source code:

 Perl | 
 
 copy code |
?

001
002
#!/usr/bin/perl
003
 
004
use GD::Graph::lines;
005
 
006
##################
007
# Computed values
008
 
009
$last = 1;
010
$cur = 1;
011
my @a;
012
 
013
print "Miles, Calculated from Fibonacci, True\n";
014
 
015
for ($i = 0; $i < 10; $i++) {
016
        $a[0][$i] = $last; #miles
017
        $a[1][$i] = $cur;  # Fibonacci Calculated value
018
        $a[2][$i] = $last * 1.609;  # computed value
019
        print $a[0][$i] . "," . $a[1][$i] . "," . $a[2][$i] . "\n";
020
 
021
        # increment the Fibonacci sequence
022
        $x = $cur; $cur += $last; $last = $x;
023
}
024
 
025
my $mygraph = GD::Graph::lines->new(600, 400);
026
$mygraph->set(
027
    x_label     => 'Miles',
028
    y_label     => 'Kilometers',
029
    title       => 'Miles to Kilometers',
030
) or warn $mygraph->error;
031
 
032
my $myimage = $mygraph->plot(\@a) or die $mygraph->error;
033
open OUTPUT, ">Fib_vs_KM.png";
034
print OUTPUT $myimage->png;
035
close(OUTPUT);
036
 
037
#################
038
# % Error 1 - 55
039
 
040
print "\nMiles, Percent Difference\n";
041
 
042
$last = 1;
043
$cur = 1;
044
my @a;
045
 
046
for ($i = 0; $i < 10; $i++) {
047
    $a[0][$i] = $last; #miles
048
    $a[1][$i] = ($cur - ($last * 1.609)) / ($last * 1.609) * 100;  # percent difference
049
 
050
        print $a[0][$i] . "," . $a[1][$i] . "\n";
051
    # increment the Fibonacci sequence
052
    $x = $cur; $cur += $last; $last = $x;
053
}
054
 
055
my $mygraph = GD::Graph::lines->new(600, 400);
056
$mygraph->set(
057
    x_label     => 'Miles',
058
    y_label     => '% Difference',
059
    title       => 'Percent Difference from Real (1 - 55 miles)',
060
) or warn $mygraph->error;
061
 
062
my $myimage = $mygraph->plot(\@a) or die $mygraph->error;
063
open OUTPUT, ">Fib_vs_KM_diff1.png";
064
print OUTPUT $myimage->png;
065
close(OUTPUT);
066
 
067
############
068
# 89 - 6765
069
 
070
for ($i = 0; $i < 10; $i++) {
071
    $a[0][$i] = $last; #miles
072
    $a[1][$i] = ($cur - ($last * 1.609)) / ($last * 1.609) * 100;  # percent difference
073
        print $a[0][$i] . "," . $a[1][$i] . "\n";
074
 
075
    # increment the Fibonacci sequence
076
    $x = $cur; $cur += $last; $last = $x;
077
}
078
 
079
my $mygraph = GD::Graph::lines->new(600, 400);
080
$mygraph->set(
081
    x_label     => 'Miles',
082
    y_label     => '% Difference',
083
    title       => 'Percent Difference from Real (89 - 6765 miles)',
084
) or warn $mygraph->error;
085
 
086
my $myimage = $mygraph->plot(\@a) or die $mygraph->error;
087
open OUTPUT, ">Fib_vs_KM_diff2.png";
088
print OUTPUT $myimage->png;
089
close(OUTPUT);
090
 
091
 
092
#################
093
# 10946 - 832040
094
 
095
for ($i = 0; $i < 10; $i++) {
096
    $a[0][$i] = $last; #miles
097
    $a[1][$i] = ($cur - ($last * 1.609)) / ($last * 1.609) * 100;  # percent difference
098
        print $a[0][$i] . "," . $a[1][$i] . "\n";
099
 
100
    # increment the Fibonacci sequence
101
    $x = $cur; $cur += $last; $last = $x;
102
}
103
 
104
my $mygraph = GD::Graph::lines->new(600, 400);
105
$mygraph->set(
106
    x_label     => 'Miles',
107
    y_label     => '% Difference',
108
    title       => 'Percent Difference from Real (10946 - 832040 miles)',
109
) or warn $mygraph->error;
110
 
111
my $myimage = $mygraph->plot(\@a) or die $mygraph->error;
112
open OUTPUT, ">Fib_vs_KM_diff3.png";
113
print OUTPUT $myimage->png;
114
close(OUTPUT);
115

Posted in Uncategorized | Leave a comment

cPanel Exim and Perl’s Net::SMTP::TLS

Another update: I have previously filed a bug on CPAN for this, and there are several similar posted there as well. https://rt.cpan.org/Ticket/Display.html?id=81215

UPDATE: This was a bug in Net::SMTP::TLS. I’ve included a patch below and also sending this along to the module author.

I noticed an issue recently with a script I have that sends emails through Perl’s Net::SMTP::TLS library. I’ve had this issue before, and I forgot how I fixed it. In any case, I’ve fixed it again.

Here’s my error from the Perl script:

1
EHLO command failed: 220 We do not authorize the use of this system to transport unsolicited,
2
at /usr/local/share/perl/5.12.4/SVN/Notify.pm line 2374

Here’s the headers I receive back from my SMTP server, a cPanel configured Exim server (real domain replaced with example.com):

1
$ telnet example.com 25
2
Connected to example.com.
3
Escape character is '^]'.
4
220-example.com ESMTP Exim 4.77 #2 Sun, 08 Apr 2012 12:59:07 +0400
5
220-We do not authorize the use of this system to transport unsolicited,
6
220 and/or bulk e-mail.
7

It looks to me like Perl’s not liking these additional lines. So let’s just rip them out. 

I changed the smtp_banner in /etc/exim.conf from 

1
smtp_banner = "${primary_hostname} ESMTP Exim ${version_number} \
2
\#${compile_number} ${tod_full} \n\
3
 We do not authorize the use of this system to transport unsolicited, \n\
4
 and/or bulk e-mail."

to 

1
smtp_banner = "${primary_hostname} ESMTP Exim ${version_number} \
2
\#${compile_number} ${tod_full}"

My subsequent headers then were only:

1
220 example.com ESMTP Exim 4.77 #2 Sun, 08 Apr 2012 16:11:21 +0400

Now Perl is happy to communicate with the server again. 

It looks like cPanel recently updated the Exim config in this case, which explains why this popped back up again. I’m assuming this is the same way I fixed it last time. 

1
# fgrep "exim.conf" /var/cpanel/updatelogs/update.133*
2
/var/cpanel/updatelogs/update.1333586462.log:[20120405.061343]   Retrieving /cpanelsync/11.32.2.15/cpanel/etc/exim/distconfig/exim.conf.dist.bz2
3
/var/cpanel/updatelogs/update.1333586462.log:[20120405.072810]      [28588] 2012-04-05 07:28:09 Exim configuration error in line 1111 of /etc/exim.conf:

I’m investigating if this is a bug in the Perl libraries, or cPanel’s exim config. I suspect in the Perl libraries. Sending system is Ubuntu 11.10; SMTP server is cPanel configured Exim on CentOS 5

1
$ perl -e "require Net::SMTP::TLS; print Net::SMTP::TLS->VERSION;"
2
0.12
3
$ perl -v
4
This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux-gnu-thread-multi (with 45 registered patches, see perl -V for more detail)

Patch:

01
02
$ diff Net-SMTP-TLS-0.12/lib/Net/SMTP/TLS.pm  Net-SMTP-TLS-0.12-fixed/lib/Net/SMTP/TLS.pm 
03
117,121c117,126
04
<  # read the line immediately after connecting
05
<  my ($rsp,$txt) = $me->_response();
06
<  if(not $rsp == 220){
07
<  croak "Could not connect to SMTP server: $host $txt\n";
08
<  }
09
---
10
>  # read the line(s) immediately after connecting
11
>  my $rsp;
12
>  my $txt;
13
>  my $more;
14
>  do {
15
>  ($rsp,$txt,$more) = $me->_response();
16
>  if(not $rsp == 220){
17
>  croak "Could not connect to SMTP server: $host $txt\n";
18
>  }
19
>  } while ($more eq '-');
20

Posted in Uncategorized | 1 Comment

SVN Vimdiff

Having recently learned how to use vimdiff, I was looking for a way to incorporate this into SVN for comparing SVN changes against my working copies using vimdiff. I found this script, originally from http://www.vim.org/scripts/script.php?script_id=1797, that performs this this beautifully.

To all contributors, thanks 🙂

#!/bin/bash
#
# Copyright (C) 2007,
#   Geoff Buchan        <geoffrey.buchan@gmail.com>
# Based on the script cvsvimdiff, written by
#   Stefano Zacchiroli  <zack@cs.unibo.it>
#   Enrico Tassi        <tassi@cs.unibo.it>
#
# This is free software, you can redistribute it and/or modify it under the
# terms of the GNU General Public License version 2 as published by the Free
# Software Foundation.
#

vimdiff=”vimdiff”
suffix=”vimsvndiff”
if [[ $1 == “-g” ]] ; then
vimdiff=”gvimdiff -f”
shift 1
fi
if [[ $# < 0 || $1 == “–help” || $1 == “-h” ]] ; then
echo “svnvimdiff – script to show svn diff in vimdiff format”
echo “”
echo “svnvimdiff [options] file”
echo “”
echo “Option:”
echo “-g    Use gvimdiff (graphical mode) instead of vimdiff”
echo “Other options are passed to svn diff”
echo “”
echo “If file is omitted it will cycle through all changed files in”
echo “the current directory.”
exit 1
fi

# Assume the last argument is the filename.
# Save everything to pass to svn diff
if (( $# > 0 )) ; then
shift_args=$(($# – 1))
else
shift_args=$#
fi
args=$*
shift $shift_args
files=”$1″
patch=`tempfile -p $suffix`
orig=`tempfile -p $suffix`
tempfiles=”$patch $orig”
if [ -z $files ] || ! [ -f $files ] ; then
# No file given, so loop over all files svn st says have changed
files=$(svn st 2> /dev/null | grep -e “^[MU]” | cut -c 3-)
for f in $files; do
if ! [ -f $f ]; then break; fi
filename=`basename $f`
ending=${filename/*./}
orig=$orig”.”$ending
cp “$f” $orig
svn diff $args $f > $patch
if ! [ $? -eq 0 ]; then break; fi
patch -R -p0 $orig $patch
$vimdiff $orig $f
tempfiles+=” $orig”
trap “rm -f $tempfiles” EXIT
done
else
# file given, so just work with that one
filename=`basename $files`
ending=${filename/*./}
orig=$orig”.”$ending
cp $files $orig
svn diff $args > $patch
if ! [ $? -eq 0 ]; then break; fi
patch -R -p0 $orig $patch
$vimdiff $orig $files
tempfiles+=” $orig”
trap “rm -f $tempfiles” EXIT
fi

Posted in Linux Tricks | Leave a comment

C++ and Timezones

Have you ever had to work with date / times in c / c++ and had to deal with that F###ing Daylight Savings Time? Even more, have you spent hours, days, weeks trying to force everything to UTC / GMT ?

It took me a while, but I figured it out. Include this in your main() and it seems to do the trick:

#ifdef _WIN32
if (_putenv((char *) “TZ=GMT”))
#else
if (putenv((char *) “TZ=GMT”))
#endif
wxMessageBox(“Failed setting env for date/time”);  // Somehow notify that the update failed.

I have used this on Win32 (XP) and RedHat derivatives. I can’t guarantee support elsewhere, but I believe this should work on all POSIX systems.

Posted in Linux Tricks | Leave a comment

Using Fetchmail To Backup Email

If you’re like me, you’ll occasionally lose sleep over paranoid thoughts; Did I turn the coffee pot off? Did I lock the door? What would happen if GMail lost all of my emails tonight?

Well, technology has helped solve my first problem, as my coffee pot will turn off on its own, usually before I want it to. Living in a gated community eases my worries about late night break-ins, and I’m fairly protected by other means anyway. However, the email loss is a very real threat.

Google offers a wonderful service with GMail (though there are privacy concerns, I won’t get into that here). However, any company is prone to massive disasters and data loss. Not only problems from Google’s end, there’s other very real threats; my password could be sniffed (possible) and used to access my account, or I may configure a mail client wrong and trash the entire account; I actually did this about 6 months ago, and luckily killed Evolution before it had trashed my all of my emails. I suspect I lost about 100 emails, most of which were probably unimportant, but I’ll never know.

I decided it’s time to create a reliable backup of my email. While I have these messages downloaded to multiple clients, I don’t know how they are storing that data and whether it would be reliably accessible if GMail was not. I decided I wanted a backup of all of my messages in a standard format that other software or services could interact with.

I wanted to use Linux software that I could run on any of my many computers or servers. There are many common programs to choose from, and I settled on Getmail, a python replacement to Fetchmail. This is free software licensed under the GPL v2. More information is available on the Getmail project homepage.

The next step was to choose the format to store the downloaded message as. I decided to use the Maildir format, though to be honest at this point I’m not familiar with the various types. All I know is that it’s in use on one of my servers, so I should have less compatibility problems if I were to ever need this backup.

Since this was for backup purposes, I didn’t want the software to store data in my home directory on the machine, so I wanted everything to be contained within one folder. In fact, after I download the mail, I tar up the files and remove the downloaded messages. I created a folder to store everything in one folder, email_backup. In addition, I downloaded Getmail to this folder so that the entire program was contained within it. I also created a symlink to the current version as getmail. So at this point, I have:

[tonyb@server2 email_backup]$ ls -l
lrwxrwxrwx 1 tonyb tonyb     14 Jun  4 08:38 getmail -> getmail-4.20.3
drwxr-xr-x 4 tonyb tonyb   4096 May 30 16:48 getmail-4.20.3

Next, I created the Fetchmail configuration file. There can be multiple files, each named for the account; for example, tonyb.gmail.getmailrc

[options]
delete = false

[retriever]
type = SimpleIMAPSSLRetriever
port = 993
server = imap.gmail.com
username = Gmail Username
password = Gmail Password
mailboxes = (“INBOX”, )

[destination]
type = Maildir
path = ./Maildir/

For more details on these options, and many more that are available, consult the Getmail documentation. Of note is the mailboxes option, where you will need to specify each folder in your account that you wish to backup.

Now I want to create a bash script to run each of these files. The output files are based on the name or the above configuration file and the date. This script file, run_backup.sh, will also serve to show you how to call Getmail with the appropriate options.

#!/bin/bash

rm -r Maildir &> /dev/null

for f in *.getmailrc
do
echo “— running backup for $f  —”
name=`basename $f .getmailrc`.`date +”%d%b%Y”`
mkdir -p Maildir/cur Maildir/new Maildir/tmp \
&& getmail/getmail -a -g . -r $f &> $name.retrieve.log \
&& cp $name.retrieve.log Maildir/ \
&& tar -czf $name.tar.gz Maildir \
&& rm -r Maildir

done

Note that this removes Maildir before and after running, so be sure you don’t have any data you are wanting to keep here.

Once completed, you will have a tar and log file for the account. I would suggest reviewing the log file to ensure there were no errors.

Future Plans

There’s a long way to go in creating a fully automated backup script, namely proper error reporting, but this is a first step that at least creates the backup. Future plans also include un-tarring the file and downloading only the new or modified messages, so that the entire account isn’t downloaded each time. This will be useful if I decide to make daily automated backups.

Posted in Linux Tricks | Leave a comment