Home | About Chris Hope | Contact Chris Hope | Search:

Moving application data on *nix with minimal downtime

Posted October 6th, 2011 in Linux/Unix/BSD

There may be times on a *nix box when you start to run out of space on a particular partition for a particular app. This post shows a fix that can be applied by moving some of the data to another partition with a minimum of downtime. The next post shows a real world example I experienced recently with Postgres.

But first, a warning 

The information presented in this post assumes some understanding of what you are doing on the command line on a *nix box. It may not work for all applications; some may have issues with the symlink example.

It may cause data corruption if you don't follow the steps carefully. If you are able, try it out on a non critical / test machine first.

I'm sorry, but I'm not responsible for any lost data from following the information in this post.

Please be sure to read the whole post before actually doing this, and again note the above.

Example application

Let's say we have an example application at /opt/myapp with data at /opt/myapp/data. It's mounted on a partition labelled /opt and there are a couple of other partitions including one with plenty of free space at /var. 

The /opt partition is almost at capacity and our app will fail once it is no longer able to continue writing data out to /opt/mypp/data

Solution 1 - move the data and configuration change

Your application may permit you to change the data directory (or log directory etc) in the config. So copy the data from e.g. /opt/myapp/data to /var/myapp/data as follows:

1. rsync the data from the original location to the new location. This may take some time. e.g. rsync -ra /opt/myapp/data /var/myapp

2. shut down the application e.g. /etc/init.d/myapp stop

3. update the configuration so it knows where the new location is

4. do the rsync again to copy any data that might have changed. This will be much quicker as it only has to copy over changes within the data files

5. start up the application again e.g. /etc/init.d/myapp start

6. optionally delete the old directory

Solution 2 - move the data and symlink

If the application doesn't have individual settings for the various directories it might use, or you need a more fine grained approach (for example relocating just one of the data subdirectories) then you may instead be able to use symbolic links.

1. rsync the data as above

2. shut down the application as above

3. do the rsync again as above

4. rename the old directory / delete it, e.g. mv /opt/myapp/data /opt/myapp/data~

5. symlink to the new location, e.g. ln -s /var/myapp/data /opt/myapp

6. start the application as above

Solution 3 - mounting the directory from a partition 

Instead of copying the data from an old location to a new location and updating the config or symlinking it, you can mount a spare partition at the old data location. Let's say you have a spare, unmounted, formatted partition available at /dev/sda5:

1. mount it somewhere temporarily, e.g. mkdir /mnt/sda5 && mount /dev/sda5 /mnt/sda5

2. rsync the data to the new partition, e.g. rsync /opt/myapp/data/ /mnt/sda5 (note the trailing slash in the first directory is important - this means it will copy everything at /opt/myapp/data to /mnt/sda5; without the slash it will copy /opt/myapp/data to /mnt/sda5/data)

3. shut down the app as above

4. do the rsync again

5. umount the partition, e.g. umount /mnt/sda5

6. rename the old directory / delete it, e.g. mv /opt/myapp/data /opt/myapp/data~

7. mount the partition under the app's directory, e.g. mount /dev/sda5 /opt/myapp/data

8. start the application as above

9. update your /etc/fstab so the partition is automatically mounted on system startup

Why rsync twice?

rsync is a synchronisation program and will recursively copy the files from one location to another when using the -ra flags, including symlinks, file permissions and ownership etc.

Running it the first time while the app is running copies everything over (you could always just use "cp -pr" at this stage and rsync in the second step), and the app can still read and write data. 

Running rsync again after the app has shut down ensures anything that has been written to the files will be copied across to the new location.

Doing an rsync both before and after cuts down the amount of downtime for your app; the first copy (while the app is running) may take some time, but the second one (while the app is not running) should be pretty quick as it only has to sync the changes within the files.

Note that if your application is not critical (e.g. it's an offline app, or is only used during certain hours and your migration is outside normal hours) then you can not bother about the first copy.

Moving the data back again

After moving the data to the other location, you might run some processes that free up data and want to move it back to the original location. It's basically the same process as above but the sync process would be e.g. rsync -ra /var/myapp/data /opt/myapp/data (for a config change) or rsync -ra /var/myapp/data/ /opt/myapp/data~/ (for a symlink or partition mount).