How to deploy a High Available application in Pouta

This is a simple High Available web application deployment in Pouta. We have a similar tutorial for High availability in Rahti.

Schema

In the schema above you can see the end result that you will achieve at the end of this tutorial. We will deploy two URLs, one for the application itself (app.example.com), and the other for the monitoring dashboard(s) (grafana.example.com). The application runs on the Frontend VMs, we will create two replicas. The frontend is connected to a Postgres database running on CSC's Pukki database on demand service. The monitoring is provided by Grafana and Prometheus, two very commonly used software solutions for monitoring. Prometheus gathers the metrics exposed by the frontend and Grafana show the data in nice graphs.

We will deploy each part step by step, and describe them in more detail while doing so.

Manual deployment

Create the VMs

We will start by creating 4 VMs: haproxy-1, haproxy-2, frontend-1, frontend-2 and Monitoring. You can follow the create a new VM guide. We are creating all VMs from the start so we get the IPs of each of them, this will make the networking configuration easier. Few notes:

• In order to save quota and resources, please use the smallest flavor available.
• We only need one single floating IP for the whole deployment and it will be initially assigned to haproxy-1.
• Make sure you can SSH into haproxy-1. If in doubt, you can use the Connecting to your virtual machine guide. We will use this machine as a SSH jumphost to connect to the other VMs.

In order to be able to SSH easily to each machine we will create a SSH config file:

mkdir -p ~/.ssh/config.d/
echo 'Include config.d/*' >>~/.ssh/config
cat >~/.ssh/config.d/pouta-ha-tutorial <<EOF
Host haproxy-1
    User ubuntu # Replace if not using an Ubuntu distribution
    Hostname <floating_IP> # Replace by the floating IP

Host haproxy-2
    User ubuntu
    ProxyJump haproxy-1
    Hostname <private_IP> # Replace by the private IP

Host frontend-1
    User ubuntu
    ProxyJump haproxy-1
    Hostname <private_IP> # Replace by the private IP

Host frontend-2
    User ubuntu
    ProxyJump haproxy-1
    Hostname <private_IP> # Replace by the private IP

Host Monitoring
    User ubuntu
    ProxyJump haproxy-1
    Hostname <private_IP> # Replace by the private IP
EOF

After this, you will be able to ssh to any VM by just running ssh <name_of_vm>, like for frontend-2:

ssh frontend-2

Load Balancer

For Load balancing, we will use two HAProxy VMs with Keepalived to achieve high availability at the load balancer layer. Keepalive in haproxy-2 will continually monitor haproxy-1, and if it goes down, haproxy-2 takes over the Floating IP automatically using the OpenStack API. When haproxy-1 is back to running status, haproxy-1 will use again the OpenStack API to regain the use of the floating IP.

Install and configure HAProxy

Do each step in both haproxy-1 and haproxy-2 we want HAProxy to behave the same on both machines.

1. Install HAProxy, Keepalived, and the OpenStack CLI:

   sudo apt update && sudo apt install -y haproxy keepalived python3-openstackclient
   

2. Edit /etc/haproxy/haproxy.cfg:

   global
       log /dev/log local0
       maxconn 4096
   
   defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
       timeout connect 5s
       timeout client  30s
       timeout server  30s
   
   frontend http_front
       bind *:80
       # Route grafana.example.com -> grafana_back, everything else -> http_back
       acl is_grafana hdr(host) -i grafana.example.com
       use_backend grafana_back if is_grafana
       default_backend http_back
   
   backend http_back
       balance roundrobin
       option httpchk GET /
       server frontend1 <FRONTEND_1_IP>:5000 check
       server frontend2 t <FRONTEND_2_IP>:5000 check
   
   backend grafana_back
       balance roundrobin
       option httpchk GET /api/health
       server grafana1 <GRAFANA_IP>:3000 check
   

   You will replace <FRONTEND_1_IP> and <FRONTEND_2_IP> with the internal IP addresses of frontend-1 and frontend-2, and <GRAFANA_IP> with the internal IP address of Monitoring.

3. Enable and start the HAProxy service to apply the configuration:

   sudo systemctl enable --now haproxy
   

Configure Keepalived

1. Create the failover script /etc/keepalived/failover.sh on both HAProxy VMs. This script moves the Floating IP to whichever node becomes MASTER using the OpenStack CLI:

   #!/bin/bash
   STATE=$3
   FLOATING_IP_ID="<FLOATING_IP_ID>"
   
   if [ "$STATE" = "MASTER" ]; then
       MY_PORT=$(openstack port list --server $(hostname) -f value -c ID | head -1)
       openstack floating ip set --port "$MY_PORT" "$FLOATING_IP_ID"
   fi
   

   Application credentials

   The script requires valid OpenStack credentials on each HAProxy VM. Create an application credential and configure it in /etc/openstack/clouds.yaml on both nodes.

   Floating IP ID

   You can get the floating IP ID by running: openstack floating ip list --port "$(openstack port list --server haproxy-1 -c ID -f value)"

2. Make the script executable:

   sudo chmod +x /etc/keepalived/failover.sh
   

3. On haproxy-1, create /etc/keepalived/keepalived.conf:

   vrrp_script chk_haproxy {
       script "systemctl is-active haproxy"
       interval 2
       weight -20
   }
   
   vrrp_instance VI_1 {
       state MASTER
       interface ens3
       virtual_router_id 51
       priority 100
       advert_int 1
       authentication {
           auth_type PASS
           auth_pass changeme
       }
       track_script {
           chk_haproxy
       }
       notify /etc/keepalived/failover.sh
   }
   

4. On haproxy-2, use the same file but set state BACKUP and priority 90, this makes haproxy-1 the main server.

5. Enable and start Keepalived on both nodes:

   sudo systemctl enable --now keepalived
   

Open Security Group ports

In the Pouta web interface, add the following security group rules:

VM	Protocol	Port	Source
haproxy-1, haproxy-2	TCP	80	0.0.0.0/0
haproxy-1, haproxy-2	112 (VRRP)	-	Internal network
frontend-1, frontend-2	TCP	5000	Internal network

For more reference on Security Groups, check out our documentation.

Database

The database is provided by Pukki DBaaS, CSC's managed PostgreSQL service. Using Pukki removes the need to manage database replication and backups yourself.

1. Follow the Pukki getting started guide to create a new PostgreSQL instance. When creating it:

   • Note the public IP
   • Create a database and a user, for example called ha (user) and ha (database) to match the defaults used in this tutorial.
   • Set a good password for the database user

2. Allow the Frontend VMs to connect to Pukki. In the Pukki web interface, add the IP address used as egress IP by frontend-1 and frontend-2 to the allowed hosts list. You can get this IP by login in any of the two Frontend servers and run:

curl ifconfig.me -4

Or via pouta's web interface in Network → Routers , click in the name of the Router. You will find the IP in the External Fixed IPs section under IP address.

Frontend VMs

We will install the following test application:

• https://github.com/CSCfi/rahti-ha-tutorial/

It is the same repository used for the Rahti tutorial mentioned before. It contains all the necessary files to also run it in Pouta. You can clone it in your local machine and check out the code, it is a simple Python application. You need to make these changes in frontend-1 and frontend-2, both servers will run the same software.

1. Make sure that git is installed and then clone the repository mentioned above:

   sudo apt install -y git
   git clone https://github.com/CSCfi/rahti-ha-tutorial /opt/rahti-ha-tutorial
   

2. Install Python 3 and the application dependencies:

   sudo apt update && sudo apt install -y python3 python3-pip netcat-openbsd
   cd /opt/rahti-ha-tutorial
   python3 -m venv venv
   . venv/bin/activate
   pip3 install -r requirements.txt
   

3. The application reads the database connection from environment variables. Create a file to store them:

   sudo cat >/opt/rahti-ha-tutorial/.env <<EOF
   DATABASE_URL=postgresql://ha:<DBPASSWORD>@<PUKKI_DB_HOST>:5432/ha
   EOF
   

   Replace DB_HOST and DB_PASSWORD with the values you got from the Pukki database.

4. Copy the systemd rahti-ha-tutorial.service file to /etc/systemd/system/.

5. Enable and start the service:

   sudo systemctl daemon-reload
   sudo systemctl enable --now rahti-ha-tutorial
   

   This start the application and also assure that it is started every time the VM restarts.

Testing

Point your DNS records (or local /etc/hosts for testing) to the Floating IP of the load balancer:

<FLOATING_IP>  app.example.com
<FLOATING_IP>  grafana.example.com

Verify the application is reachable through the load balancer:

curl http://app.example.com/

Monitoring

For monitoring we will use Prometheus to collect metrics from the Frontend VMs and Grafana to visualize them. Both will run on a dedicated Monitoring VM.

Install the software

1. SSH into the Monitoring VM and install Prometheus:

   sudo apt update && sudo apt install -y prometheus
   

2. For Grafana, add the official repository and install it:

   sudo apt install -y apt-transport-https software-properties-common
   wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
   echo "deb https://packages.grafana.com/oss/deb stable main" | \
       sudo tee /etc/apt/sources.list.d/grafana.list
   sudo apt update && sudo apt install -y grafana
   

Configure Prometheus

Edit /etc/prometheus/prometheus.yml to scrape the Flask application metrics from both Frontend VMs:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'ha-tutorial'
    static_configs:
      - targets:
          - '<FRONTEND_1_IP>:5000'
          - '<FRONTEND_2_IP>:5000'

Restart Prometheus:

sudo systemctl enable --now prometheus

Configure Grafana

1. Start Grafana:

   sudo systemctl enable --now grafana-server
   

2. Access Grafana using the grafana.example.com name you configured earlier:

   The default credentials are admin/admin.

3. Add Prometheus as a data source: go to Connections → Data sources → Add data source, select Prometheus, and set the URL to http://localhost:9090. Click Save & test. This works because Prometheis runs on the same VMas Grafana.

4. Go to Explore, select the Prometheus data source, and enter the query:

   rate(flask_app_request_count_total[5m])
   

   You will see the request rate per second for each Frontend instance, similar to what we saw in the Rahti tutorial.

Automated deployment with ansible

All of the manual steps above can be automated with the Ansible playbooks in the ha-ansible repository. Clone it first:

git clone https://github.com/lvarin/ha-ansible.git
cd ha-ansible

Prerequisites

• Ansible ≥ 2.14 on your local machine

• OpenStack credentials sourced:

  source <project>-openrc.sh
  

• An SSH key pair registered in Pouta, with the private key available locally

• Python openstackclient available locally (for ad-hoc queries)
• A working database, we recommend Pukki DBaaS, but any PostgreSQL database works. You will need: db_host, db_port, db_user, db_password, and db_name.

Install the required Ansible collection:

ansible-galaxy collection install -r requirements.yml

Configure variables

Edit group_vars/all.yml and replace every REPLACE_WITH_* placeholder:

Variable	Description
project_cidr	Your Pouta project network CIDR (e.g. 192.168.1.0/24)
keepalived_auth_pass	Shared VRRP password (choose any strong password)
os_project_id / os_project_name	Your OpenStack project
os_application_credential_id / os_appliccation_credential_secret	OpenStack application credentials for the failover script
db_host / db_password	Pukki DBaaS connection details (see Database)

Note

The floating_ip_id variable is filled in the next step.

Create a local.yml file from the local.yml.example. You need to fill up two values:

• key_name, a key pair name as is registered in Pouta.
• network, your project's network name.

Provision infrastructure

ansible-playbook create_infra.yml

When it finishes it prints a summary like:

inventory.ini
  haproxy1   ansible_host=<FLOATING_IP>
  haproxy2   ansible_host=<HAPROXY_2_PRIVATE_IP>
  frontend1  ansible_host=<FRONTEND_1_PRIVATE_IP>
  frontend2  ansible_host=<FRONTEND_2_PRIVATE_IP>
  monitoring ansible_host=<MONITORING_PRIVATE_IP>

group_vars/all.yml
  frontend_1_ip: <FRONTEND_1_PRIVATE_IP>
  frontend_2_ip: <FRONTEND_2_PRIVATE_IP>
  floating_ip_id: (run: openstack floating ip list)

Get the floating IP UUID and add it to group_vars/all.yml:

openstack floating ip list

Variable	Description
floating_ip_id	UUID of the floating IP from the command above

Create the Pukki database

Follow the Database section above to create the Pukki PostgreSQL instance, then fill in db_host and db_password in group_vars/all.yml.

Configure all VMs

ansible-playbook -i inventory.ini site.yml

This single run configures all five VMs in three plays:

1. HAProxy play, installs HAProxy, Keepalived, and the OpenStack CLI; deploys the load-balancer and the keepalive configs.
2. Frontend play, clones the rahti-ha-tutorial Flask app and runs it as a systemd service.
3. Monitoring play, installs Prometheus and Grafana.

Teardown

To destroy all provisioned resources, set state: absent in group_vars/all.yml and re-run:

ansible-playbook -i inventory.ini create_infra.yml

File reference

.
├── create_infra.yml       # Provision VMs, security groups, and floating IP
├── site.yml               # Configure all VMs
├── inventory.ini          # Host list and SSH settings
├── requirements.yml       # Ansible collection dependencies
├── group_vars/
│   └── all.yml            # All variables (fill in REPLACE_WITH_* values)
└── templates/
    ├── haproxy.cfg.j2          # HAProxy load-balancer config
    ├── keepalived.conf.j2      # Keepalive config (master/backup priority)
    ├── failover.sh.j2          # Keepalived notify script (reassigns floating IP)
    ├── clouds.yaml.j2          # OpenStack credentials for the failover script
    ├── ha-tutorial.env.j2      # Flask app environment variables
    ├── ha-tutorial.service.j2  # systemd unit for the Flask app
    └── prometheus.yml.j2       # Prometheus scrape config

Conclusion

This tutorial shows how to build a highly available web application stack on Pouta using standard open-source tools. Unlike Rahti, where the platform handles load balancing, container orchestration, and health checks automatically, on Pouta you have full control, and full responsibility — over each layer:

• HAProxy + Keepalived: provide load balancing and automatic VIP failover at the network layer.
• Frontend VMs: run the application with automatic restarts via systemd.
• Pukki DBaaS: provides a managed, reliable PostgreSQL backend without the complexity of self-managed replication.
• Prometheus + Grafana: collect and visualize application metrics from all Frontend instances.

There are several ways this deployment can be expanded:

• Add HTTPS termination to HAProxy using Let's Encrypt certificates (e.g. with Certbot).
• Add Grafana alerting to notify you when the error rate spikes or a backend becomes unavailable.
• Replace the local Prometheus storage with a persistent Cinder volume so metrics survive a VM rebuild or deletion.

Do you find this page useful?

Yes

Thank you for your feedback!

No

Thank you! Your feedback is appreciated.

Please let us know how we could improve this page by contacting us or by creating an issue in GitHub. You can also edit this page yourself and contribute your improvements by creating a pull request.