Tutorial

How To Create a High Availability Setup with Heartbeat and Reserved IPs on Ubuntu 14.04

How To Create a High Availability Setup with Heartbeat and Reserved IPs on Ubuntu 14.04
Not using Ubuntu 14.04?Choose a different version or distribution.
Ubuntu 14.04

Introduction

Heartbeat is an open source program that provides cluster infrastructure capabilities—cluster membership and messaging—to client servers, which is a critical component in a high availability (HA) server infrastructure. Heartbeat is typically used in conjunction with a cluster resource manager (CRM), such as Pacemaker, to achieve a complete HA setup. However, in this tutorial, we will demonstrate how to create a 2-node HA server setup by simply using Heartbeat and a DigitalOcean Reserved IP.

If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.

Goal

When completed, the HA setup will consist of two Ubuntu 14.04 servers in an active/passive configuration. This will be accomplished by pointing a Reserved IP, which is how your users will access your services or website, to point to the primary, or active, server unless a failure is detected. In the event that the Heartbeat service detects that the primary server is unavailable, the secondary server will automatically run a script to reassign the Reserved IP to itself via the DigitalOcean API. Thus, subsequent network traffic to the Reserved IP will be directed to your secondary server, which will act as the active server until the primary server becomes available again (at which point, the primary server will reassign the Reserved IP to itself).

Active/passive Diagram

Note: This tutorial only covers setting up active/passive high availability at the gateway level. That is, it includes the Reserved IP, and the load balancer servers—Primary and Secondary. Furthermore, for demonstration purposes, instead of configuring reverse-proxy load balancers on each server, we will simply configure them to respond with their respective hostname and public IP address.

To achieve this goal, we will follow these steps:

  • Create 2 Droplets that will receive traffic
  • Create Reserved IP and assign it to one of the Droplets
  • Create DNS A record that points to Reserved IP (optional)
  • Install Heartbeat on Droplets
  • Configure Heartbeat to Run Reserved IP Reassignment Service
  • Create Reserved IP Reassignment Service
  • Test failover

Prerequisites

In order to automate the Reserved IP reassignment, we must use the DigitalOcean API. This means that you need to generate a Personal Access Token (PAT), which is an API token that can be used to authenticate to your DigitalOcean account, with read and write access by following the How To Generate a Personal Access Token section of the API tutorial. Your PAT will be used in a script that will be added to both servers in your cluster, so be sure to keep it somewhere safe—as it allows full access to your DigitalOcean account—for reference.

In addition to the API, this tutorial utilizes the following DigitalOcean features:

Please read the linked tutorials if you want to learn more about them.

Create Droplets

The first step is to create two Ubuntu Droplets in the same datacenter, which will act as the primary and secondary servers described above. In our example setup, we will name them “primary” and “secondary” for easy reference. We will install Nginx on both Droplets and replace their index pages with information that uniquely identifies them. This will allow us a simple way to demonstrate that the HA setup is working. For a real setup, your servers should run the web server or load balancer of your choice.

Create two Ubuntu 14.04 Droplets, primary and secondary, with this bash script as the user data:

Example User Data
#!/bin/bash

apt-get -y update
apt-get -y install nginx
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, IP Address: $PUBLIC_IPV4 > /usr/share/nginx/html/index.html

This will install Nginx and replace the contents of index.html with the droplet’s hostname and IP address (by referencing the Metadata service). Accessing either Droplet via its public IP address will show a basic webpage with the Droplet hostname and IP address, which will be useful for testing which Droplet the Reserved IP is pointing to at any given moment.

Create a Reserved IP

In the DigitalOcean Control Panel, click Networking, in the top menu, then Reserved IPs in the side menu.

No Reserved IPs

Assign a Reserved IP to your primary Droplet, then click the Assign Reserved IP button.

After the Reserved IP has been assigned, check that you can reach the Droplet that it was assigned to by visiting it in a web browser.

http://your_reserved_ip

You should see the index page of your primary Droplet.

Configure DNS (Optional)

If you want to be able to access your HA setup via a domain name, go ahead and create an A record in your DNS that points your domain to your Reserved IP address. If your domain is using DigitalOcean’s nameservers, follow step three of the How To Set Up a Host Name with DigitalOcean tutorial. Once that propagates, you may access your active server via the domain name.

The example domain name we’ll use is example.com. If you don’t have a domain name right now, you should use the Reserved IP address instead.

Install Heartbeat

The next step is to install Heartbeat on both servers. The simplest way to install Heartbeat is to use apt-get:

sudo apt-get update
sudo apt-get install heartbeat

Heartbeat is now installed but it needs to be configured before it will do anything.

Configure Heartbeat

In order to get our desired cluster up and running, we must set up these Heartbeat configuration files in /etc/ha.d, identically on both servers:

  1. ha.cf: Global configuration of the Heartbeat cluster, including its member nodes
  2. authkeys: Contains a security key that provides nodes a way to authenticate to the cluster
  3. haresources: Specifies the services that are managed by the cluster and the node that is the preferred owner of the services. Note that this file is not used in a setup that uses a CRM like Pacemaker

We will also need to provide a script that will perform the Reserved IP reassignment in the event that the primary Droplet’s availability changes.

Gather Node Information

Before configuring ha.cf, we should look up the names of each node. Heartbeat requires that each node name matches their respective uname -n output.

On both servers, run this command to look up the appropriate node names:

  1. uname -n

Note the output of the command. The example node names are “primary” and “secondary”, which matches what we named the Droplets.

We will also need to look up the network interface and IP address that each node will use to communicate with the rest of the cluster, to determine which nodes are available. You may use any network interface, as long as each node can reach the other nodes in the cluster. We’ll use the public interface of our Droplets, which happens to be eth0.

On both servers, use this command to look up the IP address of the eth0 interface (or look it up in the DigitalOcean Control Panel):

  1. ip addr show eth0
ip addr show eth0 output:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 04:01:76:a5:45:01 brd ff:ff:ff:ff:ff:ff inet 104.236.6.11/18 brd 104.236.63.255 scope global eth0 valid_lft forever preferred_lft forever inet 10.17.0.28/16 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::601:76ff:fea5:4501/64 scope link valid_lft forever preferred_lft forever

Note the IP address of the network interface (highlighted in the example). Be sure to get the IP addresses of both servers.

Create ha.cf File

On both servers, open /etc/ha.d/ha.cf in your favorite editor. We’ll use vi:

  1. sudo vi /etc/ha.d/ha.cf

The file should be new and empty. We need to add the network interfaces and names of each node in our cluster.

Copy and paste this configuration into the file, then replace the respective node names and IP addresses with the values that we looked up earlier. In this example, primary’s IP address is 104.236.6.11 and secondary’s IP address is 104.236.6.22:

node primary
ucast eth0 104.236.6.11
node secondary
ucast eth0 104.236.6.22

Save and exit. Next, we’ll set up the cluster’s authorization key.

Create authkeys File

The authorization key is used to allow cluster members to join a cluster. We can simply generate a random key for this purpose.

On the primary node, run these commands to generate a suitable authorization key in an environment variable named AUTH_KEY:

if [ -z "${AUTH_KEY}" ]; then
  export AUTH_KEY="$(command dd if='/dev/urandom' bs=512 count=1 2>'/dev/null' \
      | command openssl sha1 \
      | command cut --delimiter=' ' --fields=2)"
fi

Then write the /etc/ha.d/authkeys file with these commands:

sudo bash -c "{
  echo auth1
  echo 1 sha1 $AUTH_KEY
} > /etc/ha.d/authkeys"

Check the contents of the authkeys file like this:

  1. sudo cat /etc/ha.d/authkeys

It should like something like this (with a different authorization key):

/etc/ha.d/authkeys example:
auth1 1 sha1 d1e6557e2fcb30ff8d4d3ae65b50345fa46a2faa

Ensure that the file is only readable by root:

  1. sudo chmod 600 /etc/ha.d/authkeys

Now copy the /etc/ha.d/authkeys file from your primary node to your secondary node. You can do this manually, or with scp.

On the secondary server, be sure to set the permissions of the authkeys file:

  1. sudo chmod 600 /etc/ha.d/authkeys

Both servers should have an identical /etc/ha.d/authkeys file.

Create haresources File

The haresources file specifies preferred hosts paired with services that the cluster manages. The preferred host is the node that should run the associated service(s) if the node is available. If the preferred host is not available, i.e. it is not reachable by the cluster, one of the other nodes will take over. In other words, the secondary server will take over if the primary server goes down.

On both servers, open the haresources file in your favorite editor. We’ll use vi:

  1. sudo vi /etc/ha.d/haresources

Now add this line to the file, substituting in your primary node’s name:

/etc/ha.d/haresources
  1. primary floatip

Save and exit. This configures the primary server as the preferred host for the floatip service, which is currently undefined. Let’s set up the floatip service next.

Create Reserved IP Reassignment Service

Our Heartbeat cluster is configured to maintain the floatip service, which a node can use to assign the Reserved IP to itself, but we still need to create the service. Before we set up the service itself, however, let’s create a script that will assign the Reserved IP, via the DigitalOcean API, to the node that runs it. Then we will create the floatip service which will run the Reserved IP reassignment script.

Create assign-ip Script

For our example, we’ll download a basic Python script that assigns a Reserved IP to a given Droplet ID, using the DigitalOcean API.

On both servers, download the assign-ip Python script:

  1. sudo curl -L -o /usr/local/bin/assign-ip http://do.co/assign-ip

On both servers, make it executable:

  1. sudo chmod +x /usr/local/bin/assign-ip

Use of the assign-ip script requires the following details:

  • Reserved IP: The first argument to the script, the Reserved IP that is being assigned
  • Droplet ID: The second argument to the script, the Droplet ID that the Reserved IP should be assigned to
  • DigitalOcean PAT (API token): Passed in as the environment variable DO_TOKEN, your read/write DigitalOcean PAT

Feel free to review the contents of the script before continuing.

Now we’re ready to create the floatip service.

Create floatip Service

To create the floatip service, all we need to do is create an init script that invokes the assign-ip script that we created earlier, and responds to start and stop subcommands. This init script will be responsible for looking up the Droplet ID of the server, via the Droplet Metadata service. Also, it will require the Reserved IP that will be reassigned, and the DigitalOcean API token (the Personal Access Token mentioned in the prerequisites section).

On both servers, add open /etc/init.d/floatip in an editor:

  1. sudo vi /etc/init.d/floatip

Then copy and paste in this init script, replacing the highlighted parts with your DigitalOcean API key and the Reserved IP that should be reassigned:

/etc/init.d/floatip
  1. #!/bin/bash
  2. param=$1
  3. export DO_TOKEN='b7d03a6947b217efb6f3ec3bd3504582'
  4. IP='45.55.96.8'
  5. ID=$(curl -s http://169.254.169.254/metadata/v1/id)
  6. if [ "start" == "$param" ] ; then
  7. python /usr/local/bin/assign-ip $IP $ID
  8. exit 0
  9. elif [ "stop" == "$param" ] ; then
  10. exit 0;
  11. elif [ "status" == "$param" ] ; then
  12. exit 0;
  13. else
  14. echo "no such command $param"
  15. exit 1;
  16. fi

Save and exit.

Make the script executable:

  1. sudo chmod u+x /etc/init.d/floatip

When this floatip service is started, it will simply call the assign-ip Python script and assign the specified Reserved IP to the Droplet that executed the script. This is the script that will be called by the secondary server, to reassign the Reserved IP to itself, if the primary server fails. Likewise, the same script will be used by the primary server, to reclaim the Reserved IP, once it rejoins the cluster.

Start Heartbeat

Now that Heartbeat is configured, and all of the scripts it relies on are set up, we’re ready to start the Heartbeat cluster!

On both servers, run this command to start Heartbeat:

  1. sudo service heartbeat start

You should see output like this:

Heartbeat output:
Starting High-Availability services: Done.

Our HA setup is now complete! Before moving on, let’s test that it works as intended.

Test High Availability

It’s important to test that a high availability setup works, so let’s do that now.

Currently, the Reserved IP is assigned to the primary node. Accessing the Reserved IP now, via the IP address or by the domain name that is pointing to it, will simply show the index page of the primary server. If you used the example user data script, it will look something like this:

Reserved IP is pointing to primary server
Droplet: primary, IP Address: 104.236.6.11

This indicates that the Reserved IP is, in fact, assigned to the primary Droplet.

Now, let’s open a terminal and use curl to access the Reserved IP on a 1 second loop. Use this command to do so, but be sure to replace the URL with your domain or Reserved IP address:

  1. while true; do curl http://example.com; sleep 1; done

Currently, this will output the same Droplet name and IP address of the primary server. If we cause the primary server to fail, by powering it off or stopping the Heartbeat service, we will see if the Reserved IP gets reassigned to the secondary server.

Let’s power off the primary server now. Do so via the DigitalOcean Control Panel or by running this command on the primary server:

  1. sudo poweroff

After a few moments, the primary server should become unavailable. Pay attention to the output of the curl loop that is running in the terminal. You should notice output that looks like this:

curl loop output:
Droplet: primary, IP Address: 104.236.6.11 ... curl: (7) Failed to connect to example.com port 80: Connection refused Droplet: secondary, IP Address: 104.236.6.22 Droplet: secondary, IP Address: 104.236.6.22 ...

That is, the Reserved IP address should be reassigned to point to the IP address of the secondary server. That means that your HA setup is working, as a successful automatic failover has occurred.

You may or may not see the Connection refused error, which can occur if you try and access the Reserved IP between the primary server failure and the Reserved IP reassignment completion.

Now, you may power on your primary Droplet, via the DigitalOcean Control Panel. Because Heartbeat is configured with the primary Droplet as the preferred host to run the Reserved IP reassignment script, the Reserved IP will automatically point back to the primary server as soon as it becomes available again.

Conclusion

Congratulations! You now have a basic HA server setup using Heartbeat and a DigitalOcean Reserved IP.

If you are looking to create a more robust HA setup, look into using Corosync and Pacemaker or Keepalived.

If you want to extend your Heartbeat setup, the next step is to replace the example Nginx setup with a reverse-proxy load balancer. You can use Nginx or HAProxy for this purpose. Keep in mind that you will want to bind your load balancer to the anchor IP address, so that your users can only access your servers via the Reserved IP address (and not via the public IP address of each server).

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about us


About the authors

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
10 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Hi,

any strategy to make this work on cross datacenters ?

This comment has been deleted

    Everything works great with this setup. Just one questions, is there a way to be notified when the primary server fails?

    Isn’t there a more automated way ?

    hello ,

    I think i understand it’s me which have to create a script to launch the /etc/init.d/floatip on Second server when the first server is down ? Or i have missed something on the tutorial ?

    Thanks

    Hello ,

    I followed this tutorial but my Loadbaloancer didn’t work. When i launch Heartbeat i had this message: Starting High-Availability services: INFO: Resource is stopped Done.

    When i test my loadbalancer with this command: while true; do curl http://$floatIP; sleep 1; done

    I stopped my primary server and the second server didn’t answer.

    Thanks for your help

    If you use Ubuntu 16.04 don’t think you will save time by following this method. I had all sorts of problems with heartbeat failing on the backup server, not sure if this is why the author mentioned “more robust” setups.

    This worked for me: https://www.digitalocean.com/community/tutorials/how-to-set-up-highly-available-web-servers-with-keepalived-and-floating-ips-on-ubuntu-14-04

    BUT not without this update for Ubuntu 16.04: https://www.digitalocean.com/community/questions/keepalived-for-ubuntu-16

    Hello Need this to work apt-get install python-requests Regards

    I am running Ubuntu 15.10 and the automatic ip switch over WAS NOT working even though heartbeat was taking control over the resource group in the logs, having looked into the /etc/init.d/floatip script etc further it was giving me an error about a requests module not being available etc. You need to do ‘apt-get install python-requests’ command and it started working straight away for me. Hopefully this helps someone.

    This comment has been deleted

      Try DigitalOcean for free

      Click below to sign up and get $200 of credit to try our products over 60 days!

      Sign up

      Join the Tech Talk
      Success! Thank you! Please check your email for further details.

      Please complete your information!

      Get our biweekly newsletter

      Sign up for Infrastructure as a Newsletter.

      Hollie's Hub for Good

      Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

      Become a contributor

      Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

      Welcome to the developer cloud

      DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

      Learn more
      DigitalOcean Cloud Control Panel