Bash |copy code |?
1 2 # oIFS="$IFS"; export IFS=$'\n'; for i in $(mysql --skip-column-names --batch -e "SELECT user,host FROM mysql.user"); do USER=$(echo $i|awk '{printf $1}'); HOST=$(echo $i|awk '{printf $2}'); mysql --skip-column-names --batch -e "SHOW GRANTS FOR '$USER'@'$HOST'"; done; export IFS="$oIFS";3
My favorite internet video publisher, RoosterTeeth (and their sister Achievement Hunter) recently launched a new YouTube channel, LetsPlay, dedicated to their LetsPlay series of videos. Unfortunately, I quickly got behind on the videos, and wanted to catch up. Having 80 videos published to their channel already, I wished to somehow subscribe to the channel in an RSS Aggregator to manage viewing the videos in order.
I found that YouTube offered a channel RSS feed, formatted as such for LetsPlay:
http://gdata.youtube.com/feeds/api/users/LetsPlay/uploads
However, this only shows the most recent videos. Unfortunately, I was unable to find any offered feed that contained all of the videos.
I decided to make my own feed. Using Google’s API, I was able to generate a list of all of the videos from the channel. Simple enough, now I just had to generate the RSS Feed.
As much as I love Perl, Google provides their API Client Library in Python, and sure enough, an easy-to-use Python module was available to generate RSS XML Files
The result was YtChannelRss:
https://github.com/tonyborries/YtChannelRss
A single execution builds an RSS feed of all current videos. By subscribing to this static file, and YouTube’s RSS feed, I was able to get a list of all videos. As I catch up on the videos from this static list, the YouTube provided feed will keep up with the new videos… success.
Bash |copy code |?
01 02 # ./YtChannelRss.py -h03 04 05 -h, --help Show this usage message and exit.06 -v, --verbose Verbose Output to StdErr07 -k, --apikey YouTube API Key08 09 10 # ./YtChannelRss.py -v -k "Google-API-Key" -c LetsPlay > /var/www/html/LetsPlay.xml11 12 13 14 Found 81 Videos15 Found 1 Channels16 Found 20 Playlists17 Building RSS with 81 Items18
I’m writing a database migration script in Perl to upgrade to a new database schema. This script often calls Django’s manage.py ‘syncdb’ function to rebuild from the new models. However, there’s one problem with this. Each time it runs, it asks the question:
You just installed Django’s auth system, which means you don’t have any superusers defined.
Would you like to create one now? (yes/no):
Well, I’m copying the data from the legacy database as soon as the new database is built, so no, I don’t want to create the new user. The problem though, is that this question comes up in the middle of the script, which can take up to an hour to run. This means I have to be attentive to the script. I can’t just start it and head to lunch. ‘Expect’ you say? Well, luckily There’s a Perl Module for that.
Perl |copy code |?
01 02 03 04 # use Expect to automatically answer 'no' to the create superuser prompt05 06 07 08 # patidx = Pattern Index, the index of the last pattern matched, starting at 109 # so the 2nd pattern, which should be the last, leaves patidx at 2 if successful10 11 12 13 14 15 16 17 18 19 20 }21
You may have to adjust the second matching pattern depending on your schema. This is the last line returned from the ‘manage.py syncdb’ script in my example.
I recently added a jQuery ajaxForm (which submits a form’s contents and returns the data in an ajax call instead of a page load) and I had problems getting it running. It kept throwing errors, and they didn’t seem to make sense at first:
01 02 03 {04 05 06 07 success:searchFormCallback08 09 10
1 2 <form id="searchForm" name="searchForm" action="">3 <input type="text" name="searchString" value="Search"/>4 <input type="submit" name="method" value="Search"/>5 </form>
1 2 Uncaught TypeError: Object #<HTMLInputElement> has no method 'toUpperCase' jquery.form.js:1113
Comparing to some working code, I just couldn’t find the differences. Well, after digging through the jQuery source, I finally found it (okay, it only took about 2 minutes, but I spent about 10 looking for the solution on Google first). ajaxForm() expects the form to be define with a method, so I just had to throw in the GET.
1 <form id="searchForm" name="searchForm" action="" method="GET">
I love Open Source 🙂
We recently upgraded our server, adding 6 Tesla GPUs to our existing 580 GTX. However, once we had them installed, we noticed some issues, namely, they weren’t fully being recognized.
1 2 3 4
However, they were being detected and the devices were being setup:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Many users in various forums had a similar problem which was solved by setting the 666 permissions, which were already correct in our setup. However, we luckily found the resolution on http://ambermd.org/gpus/
01 02 03 04 05 06 07 08 09 10 11 12 13 14
I never did find where this variable was being set, but now we know to include this in our scripts.
This experiment was based on this Reddit posting: http://www.reddit.com/r/fifthworldproblems/comments/s0217/i_can_only_speak_in_a_recurring_meter_the/c4a4c18
Here’s a brief excerpt that sums up the point of this experiment:
Cool fact:
The Fibonacci sequence fairly accurately converts miles to kilometers. For each number in the sequence that represents miles, the next number is the same distance in kilometers
Wanting to test this, I threw together a quick Perl script to plot this out. The results were, surprisingly, “fairly accurate.”
I’ve included here a few quick plots, the numbers, and the raw source code should anyone wish to tinker further with this. Granted these are unrefined results from a presentation stand-point, but you get the point…
The raw results:
Miles, Calculated from Fibonacci, True
1,1,1.609
1,2,1.609
2,3,3.218
3,5,4.827
5,8,8.045
8,13,12.872
13,21,20.917
21,34,33.789
34,55,54.706
55,89,88.495
Miles, Percent Difference
1,-37.8495960223741
1,24.3008079552517
2,-6.77439403356122
3,3.58400662937643
5,-0.559353635798632
8,0.994406463642015
13,0.396806425395604
21,0.624463582822808
34,0.537418199100641
55,0.570653709249105
89,0.557956997507005
144,0.562806436019613
233,0.560954075386042
377,0.561661608360142
610,0.561391353961833
987,0.561494581832304
1597,0.561455152276278
2584,0.561470213023878
4181,0.561464460329799
6765,0.561466657663354
10946,0.561465818356607
17711,0.561466138943263
28657,0.561466016490064
46368,0.561466063263029
75025,0.561466045397334
121393,0.561466052221422
196418,0.561466049614862
317811,0.561466050610479
514229,0.56146605023018
832040,0.561466050375433
The source code:
Perl |copy code |?
001 002 #!/usr/bin/perl003 004 005 006 ##################007 # Computed values008 009 010 011 012 013 014 015 016 017 018 019 020 021 # increment the Fibonacci sequence022 023 }024 025 026 027 028 029 030 031 032 033 034 035 036 037 #################038 # % Error 1 - 55039 040 041 042 043 044 045 046 047 048 049 050 051 # increment the Fibonacci sequence052 053 }054 055 056 057 058 059 060 061 062 063 064 065 066 067 ############068 # 89 - 6765069 070 071 072 073 074 075 # increment the Fibonacci sequence076 077 }078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 #################093 # 10946 - 832040094 095 096 097 098 099 100 # increment the Fibonacci sequence101 102 }103 104 105 106 107 108 109 110 111 112 113 114 115
Another update: I have previously filed a bug on CPAN for this, and there are several similar posted there as well. https://rt.cpan.org/Ticket/Display.html?id=81215
UPDATE: This was a bug in Net::SMTP::TLS. I’ve included a patch below and also sending this along to the module author.
I noticed an issue recently with a script I have that sends emails through Perl’s Net::SMTP::TLS library. I’ve had this issue before, and I forgot how I fixed it. In any case, I’ve fixed it again.
Here’s my error from the Perl script:
1 2
Here’s the headers I receive back from my SMTP server, a cPanel configured Exim server (real domain replaced with example.com):
1 2 3 4 5 6 7
It looks to me like Perl’s not liking these additional lines. So let’s just rip them out.
I changed the smtp_banner in /etc/exim.conf from
1 smtp_banner = "${primary_hostname} ESMTP Exim ${version_number} \2 \#${compile_number} ${tod_full} \n\3 4
to
1 smtp_banner = "${primary_hostname} ESMTP Exim ${version_number} \2 \#${compile_number} ${tod_full}"
My subsequent headers then were only:
1
Now Perl is happy to communicate with the server again.
It looks like cPanel recently updated the Exim config in this case, which explains why this popped back up again. I’m assuming this is the same way I fixed it last time.
1 2 3
I’m investigating if this is a bug in the Perl libraries, or cPanel’s exim config. I suspect in the Perl libraries. Sending system is Ubuntu 11.10; SMTP server is cPanel configured Exim on CentOS 5
1 2 0.123 $ perl -v4
Patch:
01 02 03 04 05 06 07 08 09 ---10 11 12 13 14 15 16 17 18 19 20
Having recently learned how to use vimdiff, I was looking for a way to incorporate this into SVN for comparing SVN changes against my working copies using vimdiff. I found this script, originally from http://www.vim.org/scripts/script.php?script_id=1797, that performs this this beautifully.
To all contributors, thanks 🙂
#!/bin/bash
#
# Copyright (C) 2007,
# Geoff Buchan <geoffrey.buchan@gmail.com>
# Based on the script cvsvimdiff, written by
# Stefano Zacchiroli <zack@cs.unibo.it>
# Enrico Tassi <tassi@cs.unibo.it>
#
# This is free software, you can redistribute it and/or modify it under the
# terms of the GNU General Public License version 2 as published by the Free
# Software Foundation.
#
vimdiff=”vimdiff”
suffix=”vimsvndiff”
if [[ $1 == “-g” ]] ; then
vimdiff=”gvimdiff -f”
shift 1
fi
if [[ $# < 0 || $1 == “–help” || $1 == “-h” ]] ; then
echo “svnvimdiff – script to show svn diff in vimdiff format”
echo “”
echo “svnvimdiff [options] file”
echo “”
echo “Option:”
echo “-g Use gvimdiff (graphical mode) instead of vimdiff”
echo “Other options are passed to svn diff”
echo “”
echo “If file is omitted it will cycle through all changed files in”
echo “the current directory.”
exit 1
fi
# Assume the last argument is the filename.
# Save everything to pass to svn diff
if (( $# > 0 )) ; then
shift_args=$(($# – 1))
else
shift_args=$#
fi
args=$*
shift $shift_args
files=”$1″
patch=`tempfile -p $suffix`
orig=`tempfile -p $suffix`
tempfiles=”$patch $orig”
if [ -z $files ] || ! [ -f $files ] ; then
# No file given, so loop over all files svn st says have changed
files=$(svn st 2> /dev/null | grep -e “^[MU]” | cut -c 3-)
for f in $files; do
if ! [ -f $f ]; then break; fi
filename=`basename $f`
ending=${filename/*./}
orig=$orig”.”$ending
cp “$f” $orig
svn diff $args $f > $patch
if ! [ $? -eq 0 ]; then break; fi
patch -R -p0 $orig $patch
$vimdiff $orig $f
tempfiles+=” $orig”
trap “rm -f $tempfiles” EXIT
done
else
# file given, so just work with that one
filename=`basename $files`
ending=${filename/*./}
orig=$orig”.”$ending
cp $files $orig
svn diff $args > $patch
if ! [ $? -eq 0 ]; then break; fi
patch -R -p0 $orig $patch
$vimdiff $orig $files
tempfiles+=” $orig”
trap “rm -f $tempfiles” EXIT
fi
Have you ever had to work with date / times in c / c++ and had to deal with that F###ing Daylight Savings Time? Even more, have you spent hours, days, weeks trying to force everything to UTC / GMT ?
It took me a while, but I figured it out. Include this in your main() and it seems to do the trick:
#ifdef _WIN32
if (_putenv((char *) “TZ=GMT”))
#else
if (putenv((char *) “TZ=GMT”))
#endif
wxMessageBox(“Failed setting env for date/time”); // Somehow notify that the update failed.
I have used this on Win32 (XP) and RedHat derivatives. I can’t guarantee support elsewhere, but I believe this should work on all POSIX systems.
If you’re like me, you’ll occasionally lose sleep over paranoid thoughts; Did I turn the coffee pot off? Did I lock the door? What would happen if GMail lost all of my emails tonight?
Well, technology has helped solve my first problem, as my coffee pot will turn off on its own, usually before I want it to. Living in a gated community eases my worries about late night break-ins, and I’m fairly protected by other means anyway. However, the email loss is a very real threat.
Google offers a wonderful service with GMail (though there are privacy concerns, I won’t get into that here). However, any company is prone to massive disasters and data loss. Not only problems from Google’s end, there’s other very real threats; my password could be sniffed (possible) and used to access my account, or I may configure a mail client wrong and trash the entire account; I actually did this about 6 months ago, and luckily killed Evolution before it had trashed my all of my emails. I suspect I lost about 100 emails, most of which were probably unimportant, but I’ll never know.
I decided it’s time to create a reliable backup of my email. While I have these messages downloaded to multiple clients, I don’t know how they are storing that data and whether it would be reliably accessible if GMail was not. I decided I wanted a backup of all of my messages in a standard format that other software or services could interact with.
I wanted to use Linux software that I could run on any of my many computers or servers. There are many common programs to choose from, and I settled on Getmail, a python replacement to Fetchmail. This is free software licensed under the GPL v2. More information is available on the Getmail project homepage.
The next step was to choose the format to store the downloaded message as. I decided to use the Maildir format, though to be honest at this point I’m not familiar with the various types. All I know is that it’s in use on one of my servers, so I should have less compatibility problems if I were to ever need this backup.
Since this was for backup purposes, I didn’t want the software to store data in my home directory on the machine, so I wanted everything to be contained within one folder. In fact, after I download the mail, I tar up the files and remove the downloaded messages. I created a folder to store everything in one folder, email_backup. In addition, I downloaded Getmail to this folder so that the entire program was contained within it. I also created a symlink to the current version as getmail. So at this point, I have:
[tonyb@server2 email_backup]$ ls -l
lrwxrwxrwx 1 tonyb tonyb 14 Jun 4 08:38 getmail -> getmail-4.20.3
drwxr-xr-x 4 tonyb tonyb 4096 May 30 16:48 getmail-4.20.3
Next, I created the Fetchmail configuration file. There can be multiple files, each named for the account; for example, tonyb.gmail.getmailrc
[options]
delete = false[retriever]
type = SimpleIMAPSSLRetriever
port = 993
server = imap.gmail.com
username = Gmail Username
password = Gmail Password
mailboxes = (“INBOX”, )[destination]
type = Maildir
path = ./Maildir/
For more details on these options, and many more that are available, consult the Getmail documentation. Of note is the mailboxes option, where you will need to specify each folder in your account that you wish to backup.
Now I want to create a bash script to run each of these files. The output files are based on the name or the above configuration file and the date. This script file, run_backup.sh, will also serve to show you how to call Getmail with the appropriate options.
#!/bin/bash
rm -r Maildir &> /dev/null
for f in *.getmailrc
do
echo “— running backup for $f —”
name=`basename $f .getmailrc`.`date +”%d%b%Y”`
mkdir -p Maildir/cur Maildir/new Maildir/tmp \
&& getmail/getmail -a -g . -r $f &> $name.retrieve.log \
&& cp $name.retrieve.log Maildir/ \
&& tar -czf $name.tar.gz Maildir \
&& rm -r Maildirdone
Note that this removes Maildir before and after running, so be sure you don’t have any data you are wanting to keep here.
Once completed, you will have a tar and log file for the account. I would suggest reviewing the log file to ensure there were no errors.
Future Plans
There’s a long way to go in creating a fully automated backup script, namely proper error reporting, but this is a first step that at least creates the backup. Future plans also include un-tarring the file and downloading only the new or modified messages, so that the entire account isn’t downloaded each time. This will be useful if I decide to make daily automated backups.