November 4, 2009

EC2 -> S3 Cloud Backup

I have been backing up my EC2 cloud servers to S3 using a simple tar script to get the main data from the EBS mounts on each cloud server.  I use cron to run the backup each night, saving the data first to the EBS mounts in an /apps/backup directory, then I copy the backup to the S3 storage.  After the script runs successfully, I remove any old backups locally on EBS and S3 that are older than 3 days.  This way I always have a 3 day rotation of the saved data.
In order to transfer to S3 storage, you will need the s3cmd, which can be downloaded here.  The s3cmd will need to be stored inside the /root/bin directory unless you alter the cloud_backup script.  Once you have the s3cmd files in place, run the s3cmd –configure command to configure it to your S3 instance.  You will need your S3 access key and your secret key to complete the configuration.
I created the script in /root/bin and called it cloud_backup:
# Create the backup of certain /apps directories with the output as <HOSTNAME_DATE>.tgz
/bin/tar -czvf /apps/backup/`hostname`_`date +%m%d%y`.tgz /apps/apache  /apps/data  /apps/keys  /apps/tomcat  /apps/virtual /root && s3cmd put /apps/backup/`hostname`_`date +%m%d%y`.tgz s3://<bucket.domain.com>/`hostname`_`date +%m%d%y`.tgzfor i in `find /apps/backup -name *.tgz -mtime +3| cut -d ‘/’ -f 4`; do echo s3cmd del s3://<bucket.domain.com>/$i; echo rm -f /apps/backup/$i;  done
# Move the backup file to S3 storage
/bin/tar -czvf /apps/backup/`hostname`_`date +%m%d%y`.tgz /apps/apache  /apps/data  /apps/keys  /apps/tomcat  /apps/virtual /root && s3cmd put /apps/backup/`hostname`_`date +%m%d%y`.tgz s3://<bucket.domain.com>/`hostname`_`date +%m%d%y`.tgz
# Look for backups older than 3 days and delete the files from the /apps/backup dir and from S3
for i in `find /apps/backup -name *.tgz -mtime +3| cut -d ‘/’ -f 4`; do echo s3cmd del s3://<bucket.domain.com>/$i; echo rm -f /apps/backup/$i;  done
After the script is in place, make it executable with chmod +x /root/bin/cloud_backup.  Once I have the script saved, I can add the job to cron so that it runs every night.  Open your cron entries with crontab -e.
# Cloud Daily Backup
0 4 * * * /root/bin/cloud_backup

I have been backing up my EC2 cloud servers to S3 using a simple tar script to get the main data from the EBS mounts on each cloud server.  I use cron to run the backup each night, saving the data first to the EBS mounts in an /apps/backup directory, then I copy the backup to the S3 storage.  After the script runs successfully, I remove any old backups locally on EBS and S3 that are older than 3 days.  This way I always have a 3 day rotation of the saved data.
In order to transfer to S3 storage, you will need the s3cmd, which can be downloaded here.  The s3cmd will need to be stored inside the /root/bin directory unless you alter the cloud_backup script.  Once you have the s3cmd files in place, run the s3cmd –configure command to configure it to your S3 instance.  You will need your S3 access key and your secret key to complete the configuration.
I created the script in /root/bin and called it cloud_backup:
# Create the backup of certain /apps directories with the output as <HOSTNAME_DATE>.tgz
/bin/tar -czvf /apps/backup/`hostname`_`date +%m%d%y`.tgz /apps/apache  /apps/data  /apps/keys  /apps/tomcat  /apps/virtual /root && s3cmd put /apps/backup/`hostname`_`date +%m%d%y`.tgz s3://<bucket.domain.com>/`hostname`_`date +%m%d%y`.tgzfor i in `find /apps/backup -name *.tgz -mtime +3| cut -d ‘/’ -f 4`; do echo s3cmd del s3://<bucket.domain.com>/$i; echo rm -f /apps/backup/$i;  done

# Move the backup file to S3 storage
/bin/tar -czvf /apps/backup/`hostname`_`date +%m%d%y`.tgz /apps/apache  /apps/data  /apps/keys  /apps/tomcat  /apps/virtual /root && s3cmd put /apps/backup/`hostname`_`date +%m%d%y`.tgz s3://<bucket.domain.com>/`hostname`_`date +%m%d%y`.tgz

# Look for backups older than 3 days and delete the files from the /apps/backup dir and from S3
for i in `find /apps/backup -name *.tgz -mtime +3| cut -d ‘/’ -f 4`; do echo s3cmd del s3://<bucket.domain.com>/$i; echo rm -f /apps/backup/$i;  done
After the script is in place, make it executable with chmod +x /root/bin/cloud_backup.  Once I have the script saved, I can add the job to cron so that it runs every night.  Open your cron entries with crontab -e.
# Cloud Daily Backup
0 4 * * * /root/bin/cloud_backup

Posted in Cloud (Tags: , , , , , , , , , , , ) | No Comments

October 30, 2009

Linux CPU Affinity

I just had an opportunity to assist someone setting up processor affinity for a WebSphere application server. Processor affinity is forcing an application to only use a single processor or specific processors in a computer. This can be done for troubleshooting as well as for reducing licensing costs in a multiple CPU or multiple core system.

In RedHat and Debian operating systems, you can install the “schedutils” package to get the taskset command. To use the taskset command, you have to set it per CPU and per PID. For our purposes, WebSphere creates a PID file in <WAS_HOME>/profiles/<PROFILE_NAME>/logs/<SERVER_NAME>. You can add a line to the bottom of the WebSphere start script like:

taskset -c <CPU#> -p `cat <WAS_HOME>/profiles/<PROFILE_NAME>/logs/<SERVER_NAME>/server1.pid

This will pin the WebSphere process to a certain CPU every time is starts.

Posted in UNIX (Tags: , , , , , ) | No Comments

September 28, 2009

BASH IF/ELSE Mail Script

I had a request to track when a leased domain was expiring after we were finished using it.  Instead of manually giving status twice a day I decided to create a script to do the work for me.  The script is pretty well documented already, but the basic process is checking DOMAIN1.COM in a WHOIS search and analyzing the results.  If the DNS names servers point back to DOMAIN2.COM, I know that the domain is still registered to us and is still alive.  To check the WHOIS results, I am running a grep for DOMAIN2.COM, and if found the built-in command success/error shows a “0″.  If the results do not include DOMAIN2.COM, the command with error with a “1″.  Based on the error code value, I send an email telling if the domain is UP or DOWN.

#!/bin/bash

# 9/28/09 MKW

#

# Creating STATUS variable, it will look for the string and then assign a status code.  0 means success, it found the string, 1 means error, it did not find the string.

STATUS=`/bin/grep domain2.COM /tmp/domain_check.log >/dev/null ; echo $?`

# Running a WHOIS search for the domain we are checking status on, then redirecting the results to a file so that we can run grep on it.

/usr/bin/whois domain1.com > /tmp/domain_check.log

# Beginning of IF/ELSE statement, we are going to send an email depending on the results of the STATUS variable.

# If the results are a success (0), then send the email formatted for success.

if [ $STATUS -eq 0 ] ; then

(echo “Subject: domain1.com Domain Status: UP”;echo “From: user1@domain2.com”;echo “To: user2@domain2.com”;echo “Cc: user1@domain2.com”;echo “Reply-To: user1@domain2.com”;echo The domain1.com domain is pointing to our DNS servers still.)|/usr/lib/sendmail -t

# Now the ELSE statement

# If the results are an error (1), then send the email formatted for failure.

else

if [ $STATUS -eq 1 ]; then

(echo “Subject: domain1.com Domain Status: DOWN”;echo “From: user1@domain2.com”;echo “To: user2@domain2.com”;echo “Cc: user1@domain2.com”;echo “Reply-To: user1@domain2.com”;echo The domain1.com domain no longer points to our DNS servers.)|/usr/lib/sendmail -t

# Close out the IF/ELSE statments with FI.

fi

fi

# Exit the script.

exit

#!/bin/bash
# 9/28/09 MKW
#
# Creating STATUS variable, it will look for the string and then assign a status code.  0 means success, it found the string, 1 means error, it did not find the string.
STATUS=`/bin/grep RIPTIDESOFTWARE.COM /tmp/riptide_check.log >/dev/null ; echo $?`
# Running a WHOIS search for the domain we are checking status on, then redirecting the results to a file so that we can run grep on it.
/usr/bin/whois riptide.com > /tmp/riptide_check.log
# Beginning of IF/ELSE statement, we are going to send an email depending on the results of the STATUS variable.
# If the results are a success (0), then send the email formatted for success.
if [ $STATUS -eq 0 ] ; then
(echo “Subject: Riptide.com Domain Status: UP”;echo “From: matt.wygant@riptidesoftware.com”;echo “To: philip.loeffel@riptidesoftware.com”;echo “Cc: matt.wygant@riptidesoftware.com”;echo “Reply-To: matt.wygant@riptidesoftware.com”;echo The Riptide.com domain is pointing to our DNS servers still.)|/usr/lib/sendmail -t
# Now the ELSE statement
# If the results are an error (1), then send the email formatted for failure.
else
if [ $STATUS -eq 1 ]; then
(echo “Subject: Riptide.com Domain Status: DOWN”;echo “From: matt.wygant@riptidesoftware.com”;echo “To: philip.loeffel@riptidesoftware.com”;echo “Cc: matt.wygant@riptidesoftware.com”;echo “Reply-To: matt.wygant@riptidesoftware.com”;echo The Riptide.com domain no longer points to our DNS servers.)|/usr/lib/sendmail -t
# Close out the IF/ELSE statments with FI.
fi
fi
# Exit the script.
exit

Next is to automate the running of the script, I used cron since I am running on Linux.  Use crontab -e to edit your crontab file and input the schedule and script location.  Here is my example:

0 9 * * 1-5 /home/mwygant/bin/domain_check.sh

0 16 * * 1-5 /home/mwygant/bin/domain_check.sh

Posted in UNIX (Tags: , , , , , , , , , ) | No Comments

« Previous Entries