If you’re like me, you’ll occasionally lose sleep over paranoid thoughts; Did I turn the coffee pot off? Did I lock the door? What would happen if GMail lost all of my emails tonight?
Well, technology has helped solve my first problem, as my coffee pot will turn off on its own, usually before I want it to. Living in a gated community eases my worries about late night break-ins, and I’m fairly protected by other means anyway. However, the email loss is a very real threat.
Google offers a wonderful service with GMail (though there are privacy concerns, I won’t get into that here). However, any company is prone to massive disasters and data loss. Not only problems from Google’s end, there’s other very real threats; my password could be sniffed (possible) and used to access my account, or I may configure a mail client wrong and trash the entire account; I actually did this about 6 months ago, and luckily killed Evolution before it had trashed my all of my emails. I suspect I lost about 100 emails, most of which were probably unimportant, but I’ll never know.
I decided it’s time to create a reliable backup of my email. While I have these messages downloaded to multiple clients, I don’t know how they are storing that data and whether it would be reliably accessible if GMail was not. I decided I wanted a backup of all of my messages in a standard format that other software or services could interact with.
I wanted to use Linux software that I could run on any of my many computers or servers. There are many common programs to choose from, and I settled on Getmail, a python replacement to Fetchmail. This is free software licensed under the GPL v2. More information is available on the Getmail project homepage.
The next step was to choose the format to store the downloaded message as. I decided to use the Maildir format, though to be honest at this point I’m not familiar with the various types. All I know is that it’s in use on one of my servers, so I should have less compatibility problems if I were to ever need this backup.
Since this was for backup purposes, I didn’t want the software to store data in my home directory on the machine, so I wanted everything to be contained within one folder. In fact, after I download the mail, I tar up the files and remove the downloaded messages. I created a folder to store everything in one folder, email_backup. In addition, I downloaded Getmail to this folder so that the entire program was contained within it. I also created a symlink to the current version as getmail. So at this point, I have:
[tonyb@server2 email_backup]$ ls -l
lrwxrwxrwx 1 tonyb tonyb 14 Jun 4 08:38 getmail -> getmail-4.20.3
drwxr-xr-x 4 tonyb tonyb 4096 May 30 16:48 getmail-4.20.3
Next, I created the Fetchmail configuration file. There can be multiple files, each named for the account; for example, tonyb.gmail.getmailrc
delete = false
type = SimpleIMAPSSLRetriever
port = 993
server = imap.gmail.com
username = Gmail Username
password = Gmail Password
mailboxes = (“INBOX”, )
type = Maildir
path = ./Maildir/
For more details on these options, and many more that are available, consult the Getmail documentation. Of note is the mailboxes option, where you will need to specify each folder in your account that you wish to backup.
Now I want to create a bash script to run each of these files. The output files are based on the name or the above configuration file and the date. This script file, run_backup.sh, will also serve to show you how to call Getmail with the appropriate options.
rm -r Maildir &> /dev/null
for f in *.getmailrc
echo “— running backup for $f —”
name=`basename $f .getmailrc`.`date +”%d%b%Y”`
mkdir -p Maildir/cur Maildir/new Maildir/tmp \
&& getmail/getmail -a -g . -r $f &> $name.retrieve.log \
&& cp $name.retrieve.log Maildir/ \
&& tar -czf $name.tar.gz Maildir \
&& rm -r Maildir
Note that this removes Maildir before and after running, so be sure you don’t have any data you are wanting to keep here.
Once completed, you will have a tar and log file for the account. I would suggest reviewing the log file to ensure there were no errors.
There’s a long way to go in creating a fully automated backup script, namely proper error reporting, but this is a first step that at least creates the backup. Future plans also include un-tarring the file and downloading only the new or modified messages, so that the entire account isn’t downloaded each time. This will be useful if I decide to make daily automated backups.