Amazon recently allowed Elastic Block Store to boot persistent images. However, there are two concerns I have with the method.
* The EBS boot volumes must be EBS Snapshots, which cost more than regular EBS volumes. (http://aws.amazon.com/ec2/#pricing)
* The EBS boot volumes currently do not work within the Virtual Private Cloud (VPC) infrastructure. (http://aws.amazon.com/vpc/faqs/#45)
To work around these two issues, it is possible to “pivot” from a normal AMI’s local root volume (/dev/sda1) to an arbitrary EBS volume that contains a full OS installation. An additional benefit is that a very small AMI may be used to launch the EBS backed instance, so that the instance launches much faster than a large AMI.
The goal of this post is to provide instructions on how to build a small AMI that is capable of launching a full-blown persistent Linux installation that is backed on EBS storage. BusyBox, “The Swiss Army Knife of Embedded Systems”, provides a solid foundation for our small AMI. To build a BusyBox AMI and an instance backed by EBS, simply follow these directions:
1. Start an instance from an AMI that is running the desired kernel and software configuration. (You can use an existing public AMI or a custom AMI that you created. Personally, I prefer to create my own AMI’s, so I know what is in them.)
2. Log into the instance.
3. Download the latest version of BusyBox.
wget “http://busybox.net/downloads/busybox-1.15.3.tar.bz2”
4. Create a busyroot directory.
mkdir busyroot
5. Extract BusyBox.
bunzip2 busybox-1.15.3.tar.bz2
tar xvf busybox-1.15.3.tar
6. Configure BusyBox. (You can experiment here to reduce the image size, but the configuration listed below works.)
cd busybox-1.15.3
make config
Select “y” for the STATIC option and the default values for everything else.
Build BusyBox as a static binary (no shared libs) (STATIC) [N/y/?] y
7. Make and install BusyBox.
make CONFIG_PREFIX=$HOME/busyroot install
chmod 4755 $HOME/busyroot/bin/busybox
8. Create required directories.
cd $HOME/busyroot
mkdir dev sys etc proc mnt mnt/new-root
9. Create the necessary devices. (We will use /dev/sdj for the EBS volume, but this could be any block device not used by the normal AMI.)
MAKEDEV -d $HOME/busyroot/dev -x sdj
MAKEDEV -d $HOME/busyroot/dev -x console
MAKEDEV -d $HOME/busyroot/dev -x null
MAKEDEV -d $HOME/busyroot/dev -x zero
10. Create the init file.
mv $HOME/busyroot/sbin/init $HOME/busyroot/sbin/init.orig
cat <<‘EOL’ > $HOME/busyroot/sbin/init
#!/bin/busybox sh
PATH=/bin:/usr/bin:/sbin:/usr/sbin
NEWDEV=”/dev/sdj”
NEWTYP=”ext3″
NEWMNT=”/mnt/new-root”
OLDMNT=”/mnt/old-root”
OPTIONS=”noatime,ro”
SLEEP=10
echo “Remounting writable.”
mount -o remount,rw /
[ ! -d $NEWMNT ] && echo “Creating directory $NEWMNT.” && mkdir -p $NEWMNT
while true ; do
echo “sleeping…”
sleep $SLEEP
echo “Trying to mount $NEWDEV writable.”
mount -t $NEWTYP -o rw $NEWDEV $NEWMNT || continue
echo “Mounted.”
break;
done
[ ! -d $NEWMNT/$OLDMNT ] && echo “Creating directory $NEWMNT/$OLDMNT.” && mkdir -p $NEWMNT/$OLDMNT
echo “Remounting $NEWMNT $OPTIONS.”
mount -o remount,$OPTIONS $NEWMNT
echo “Trying to pivot.”
cd $NEWMNT
pivot_root . ./$OLDMNT
for dir in /dev /proc /sys; do
echo “Moving mounted file system ${OLDMNT}${dir} to $dir.”
mount –move ./${OLDMNT}${dir} ${dir}
done
echo “Trying to chroot.”
exec chroot . /bin/sh -c “umount ./$OLDMNT; exec /sbin/init $*” < /dev/console > /dev/console 2>&1
EOL
chmod 755 $HOME/busyroot/sbin/init
11. Create the fstab file.
cat <<‘EOL’ > $HOME/busyroot/etc/fstab
/dev/sda1 / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
EOL
12. Create a 4MB loopback file.
cd
dd if=/dev/zero of=busybox.fs bs=1M count=4
mkfs.ext3 busybox.fs
13. Mount the loopback file.
mkdir $HOME/busyimg
mount -o loop $HOME/busybox.fs $HOME/busyimg
14. Copy the staged files and directories to the image. (Technically, the BusyBox image could have been built directly in $HOME/busyimg, but we were not sure how big the image was going to be.)
cp -rp $HOME/busyroot/* $HOME/busyimg
15. Un-mount the image.
sync
umount -d $HOME/busyimg
16. Set environment variables.
export EC2_HOME=/opt/ec2-api-tools
export EC2_CERT=/path/to/your/cert.pem
export EC2_PRIVATE_KEY=/path/to/your/pk.pem
export AWS_ACCOUNT_NUMBER=”NNNN-NNNN-NNNN”
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export EC2_BUCKET=”your_bucket”
export JAVA_HOME=/usr/java/default
export ARCH=`uname -i`
export AKI=`curl -s http://169.254.169.254/latest/meta-data/kernel-id`
export ARI=`curl -s http://169.254.169.254/latest/meta-data/ramdisk-id`
export INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
export AVAIL_ZONE=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`
export SEC_GROUP=`curl -s http://169.254.169.254/latest/meta-data/security-groups`
export PUB_KEY=`wget -q -O – “http://169.254.169.254/latest/meta-data/public-keys” | awk -F= ‘{print $2}’`
17. Bundle the image.
ec2-bundle-image -i $HOME/busybox.fs -d /tmp -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_NUMBER -r $ARCH –kernel $AKI –ramdisk $ARI
18. Upload the image.
ec2-upload-bundle -b $EC2_BUCKET -m /tmp/busybox.fs.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY
19. Register the AMI.
BUSYBOX_AMI=`ec2-register “$EC2_BUCKET/busybox.fs.manifest.xml” | awk ‘{print $2}’`
echo “BUSYBOX_AMI: $BUSYBOX_AMI”
20. Create an EBS volume of the desired size (10G or more) in the desired availability zone.
VOLUME_ID=`ec2-create-volume -s 10 -z $AVAIL_ZONE | awk ‘{print $2}’`
echo “VOLUME_ID: $VOLUME_ID”
21. Attach the volume to the current instance as /dev/sdj.
ec2-attach-volume $VOLUME_ID -i $INSTANCE_ID -d /dev/sdj
22. Create an EXT3 file system on /dev/sdj.
mkfs.ext3 /dev/sdj
22. Mount the EBS volume.
mkdir /mnt/ebs_boot
mount /dev/sdj /mnt/ebs_boot
23. Copy the current AMI to the EBS volume.
rsync -avHx / /mnt/ebs_boot
24. Fix the /etc/fstab file.
vi /mnt/ebs_boot/etc/fstab
Remove the local file systems.
/dev/sda1 / ext3 defaults 1 1
/dev/sdb /mnt ext3 defaults 1 2
/dev/sda3 swap swap defaults 0 0
Add the /dev/sdj file system.
/dev/sdj / ext3 defaults 1 1
25. Fix the /etc/inittab file. The cloud AMI’s are normally configured for runlevel 4.
vi /mnt/ebs_boot/etc/inittab
Edit the following line if necessary:
id:4:initdefault:
26. Un-mount the EBS volume.
sync
umount /mnt/ebs_boot
27. Detach the volume.
ec2-detach-volume $VOLUME_ID -i $INSTANCE_ID -d /dev/sdj
28. Create a new instance running the BusyBox AMI.
BUSYBOX_ID=`ec2-run-instances $BUSYBOX_AMI -z $AVAIL_ZONE -k $PUB_KEY -g $SEC_GROUP | awk ‘{print $6}’`
29. Wait until the instance is running…
ec2-describe-instances $BUSYBOX_ID
30. Attach the EBS volume to the BusyBox instance as /dev/sdj.
ec2-attach-volume $VOLUME_ID -i $BUSYBOX_ID -d /dev/sdj
31. Reboot the BusyBox instance to make sure it picks up the new device.
ec2-reboot-instances $BUSYBOX_ID
32. Check the BusyBox instance’s console output to make sure it came up as expected.
ec2-get-console-output $BUSYBOX_ID
33. Log into the new EBS backed instance.
That should be it. You now have a persistent instance that is backed by EBS storage!