Extracting a single database from a pg_dumpall PostgreSQL dump

Some data got deleted from one of our databases, and we had to restore it from our backups.

To make sure we get everything from our PostgreSQL cluster’s, we use pg_dumpall to make a SQL dump. It’s in no way the most efficient way to do a backup of a PostgreSQL database cluster, but it is by far the most flexible.

Update 16-12-2013: Rune Kaagaard updated the script and added an installer. Thanks Rune!

Continue reading Extracting a single database from a pg_dumpall PostgreSQL dump

Building postgresql8x and psycopg2 for x86_64 and i386 on Snow Leopard (OS X 10.6)

I’ve recently installed Apple’s new 64 bit OS Snow Leopard, on my work computer. I use postgresql extensivly together with python, and usually use apple’s bundled python2.5 for working with django.

As the daredevil I am, I wanted to recompile all my macports to use the new 64 bit system, and therefore deleted them all, and made a fresh install of macports. After building the postgresql81 port, I was about to build the psycopg2 python postgresql driver for python 2.5, when it gave me a warning about not being able to find some symbols in the postgresql library it had linked to. I quickly realized that this might be an architecture problem, and sure enough, it turns out that python 2.5 is a i386/ppc and python 2.6 is x86_64/i386/ppc binary, as can be seen here:

Continue reading Building postgresql8x and psycopg2 for x86_64 and i386 on Snow Leopard (OS X 10.6)

Managing multiple AWS identities

I’m running multiple different project on AWS which was so much of a pain to use, as I often find myself having to use the identity of project-a together with the official amazon ec2 tools.

To help myself manage the multiple identities, I wote a set of bash functions, called:

  • aws_load <config-name> – loads configuration from config-name
  • ec2ssh <instance-number-in-ec2din-list> – ssh’s into a given instance, with the root key
  • ec2scp – a shorthand for scp -i <keyfile>

    Continue reading Managing multiple AWS identities

Detaching a running process on *nix (or how to make a process continue to run after logging out)

Today, I had to copy 70 GiB of data from a ext3 filesystem to a XFS filesystem. This involved a lot of small files. After a couple of hours of waiting, I thought it’d be best to just leave it running, and resume my activities the day after. But oh nooo, I forgot to run it in a screen.

Continue reading Detaching a running process on *nix (or how to make a process continue to run after logging out)

Poormans cloudfront with EC2 and varnish

Recently (10-20 minutes ago), amazon couldfront (a cdn) stopped sending dns replies in europe:

% dig -t ns cloudfront.net

; <<>> DiG 9.4.3-P1 <<>> -t ns cloudfront.net
;; global options:  printcmd
;; connection timed out; no servers could be reached

I was going to do a guide to set up a varnish to replace cloudfront temporarily (and did actually set up the instance, and software – I might do the guide and ami anyway) when I realized, that I (as well as most other people) can just change the relevant url to point to the S3 bucket. Problem solved. That will, however, not be as fast as either cloudfront itself, or a varnish cached backend.

Should anyone be interested in how varnish is setup to handle failures from cloudfront, I’ll happily do an ami.

Django – sharing a memcached instance

Update: Some Curious User brought to my attention, that a ticket has been opened which, when implemented, will add a setting for a cache prefix. It will also allow other cache key manipulations. Django has implemented KEY_PREFIX in the development version, which currently means, that it will be out in 1.4 iirc. Django 1.3 has implemented KEY_PREFIX which solves the problem once and for all.

Until recently I’ve been using the file:// django cache, but that has a “problem” when multiple users needs to manipulate the cache (think uid 80 writes a key, that uid 1000 wants to delete).

My problem with the memcached:// django cache provider has been, that it cannot handle being used on a shared memcached instance, because of the danger of key collissions.

Continue reading Django – sharing a memcached instance