Red Hat OpenShift Installation mit GlusterFS Storage

OpenShift is a PaaS (Platform as a Service) that simplifies the operation of Docker containers in a cluster. A key component of the platform is storage. Every application has different capacity requirements. The idea is that the application simply defines how much storage space it needs and the platform dynamically provisions the required capacity in the form of volumes. These volumes are available to the container regardless of which node the container is running on. What could a solution look like? In simple forms, a SAN could be connected, as could NFS as a shared mount. However, these technologies are „old school“ and not suitable for modern container platforms. In this article, we describe Container Native Storage (CNS) introduced by Red Hat. „Native“ means that all components are first class citizens and run as containers in OpenShift themselves. In contrast, there is Container Ready Storage. This refers to a platform-external storage solution that is then integrated into OpenShift. Both solutions use storage clusters such as GlusterFS or Ceph. These two technologies theoretically promise infinite expandability of storage. We will see below how an installation of GlusterFS in the CNS variant works.

A GlusterFS cluster consists of at least 3 nodes. GlusterFS supports different volume types that determine how data is distributed and made fail-safe. OpenShift only supports the Distributed Replicated variant. For those interested, more about volume types here . In an OpenShift cluster, individual nodes with connected raw devices are defined as Gluster nodes. A Gluster Docker container must run on each of these nodes. To do this, OpenShift generates a DaemonSet that ensures that a container is placed on each node. In our example, we will start a total of 3 nodes: 1 master and 2 worker nodes. Since this is only for demonstration purposes, the master node is also used to deploy a Gluster container. In production, the master node should be configured with 

$ oadm manage-node <node1> <node2> --schedulable=false

to not playable. This means that the cluster management will not deploy any pods on the master. To install OpenShift on AWS, there is a suitable GitHub project here . Simply follow the steps described in the Readme. However, the aws cli must first be installed locally and configured with access data. Here are instructions for installing the aws cli. Once you have cloned the Github project, you can use 

$ make infrastructure

to build an infrastructure consisting of 1 master and 2 nodes on AWS. For this we use terraform, which is ideal for automatically provisioning infrastructures. To access the 3 nodes, a bastion host is also installed as a jump server, a VPC and the corresponding security group. The following picture emerges:

Now you could 

$ make openshift

start the OpenShift installation. The GitHub project uses the official OpenShift Ansible playbooks. It should be noted that the GitHub project does not yet support a GlusterFS installation. This must be installed afterwards, as we will see below. Furthermore, we have created a pull request that completely automates the GlusterFS integration as a CNS. This means that the steps that we will carry out below will be merged into the GitHub project.

However, we still need to make one change before installing OpenShift from the above step. In the file: install-from-bastion.sh

ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ./inventory.cfg ./openshift-ansible/playbooks/byo/config.yml


change to

ANSIBLE_HOST_KEY_CHECKING=False /usr/local/bin/ansible-playbook -i ./inventory.cfg ./openshift-ansible/playbooks/byo/config.yml --extra-vars "openshift_storage_glusterfs_wipe=True" -vvv

change. OpenShift CNS uses Heketi to dynamically provision storage. Heketi offers a REST-based interface to request and release storage from the GlusterFS cluster. Heketi expects that the defined GlusterNodes are equipped with raw devices. For example, hard drives (or EBS) without volumes and without formatting, since Heketi takes care of this itself. If something goes wrong during an installation step and volumes have already been created, you cannot simply repeat the installation. In these cases, you must delete the corresponding devices using the wipefs command. The setting above ensures that this happens automatically when setting up node storage. After setting the settings, you can now install OpenShift as described above. This takes about 15 minutes and you can then use

$ make browse-openshift

Open a browser with the OpenShift URL on AWS. You can log in with the username: admin and the password: 123. Htpasswd is entered as the identity provider, so that only certain users can log in. OpenShift offers various identity providers such as Ldap, DenyAll or AllowAll. The admin user is not the same as the cluster admin. He or she does not yet have the rights for that.

Now we want to take the first steps towards GlusterFS. It should be noted that dynamic storage volumes can already be created with the installation, but as EBS volumes on AWS. A StorageClass with the Kubernetes/aws-ebs provisioner is already included. This means that if an application registers a storage space requirement in the form of a PersistentVolumeClaim, the StorageClass creates an EBS volume on AWS. This is then mounted to the container that defines its volume requirement in the yaml file.

We are using RHEL Enterprise Linux version 7.5 and OpenShift Container Platform 3.6 as the operating system. First of all, we need to set up our Red Hat subscription on all nodes. We use our developer licenses for this. To log in to the respective node, we use

$ make ssh-masteror make ssh-node1ormake ssh-node2

$ sudo subscription-manager register --username <username> --password <password> --auto-attach


Then we add the Red Hat Gluster repo. Run the following commands on the master.

sudo subscription-manager repos --enable=rh-gluster-3-client-for-rhel-7-server-rpms


Now we install the CNS and Heketi Client Tool. This gives us access to cns-deploy, with which we can deploy a topology for our GlusterFS cluster, as well as the heketi-cli for administration.

sudo yum install cns-deploy heketi-client

In order for the Gluster nodes to communicate with each other, we need to add firewall rules on each node:

sudo vi /etc/sysconfig/iptables
Insert the following rules before the last COMMIT and save the file:

-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m multiport --dports 49152:49664 -j ACCEPT

Then re-read the rules with:

$ sudo systemctl reload iptables

GlusterFS and Heketi use LVM to manage storage devices. For this to work smoothly, certain kernel modules must be installed:

dm_thin_pool
dm_multipath
dm_mirror
dm_snapshot

If one of the modules is missing ( $ lsmod | grep dmcheck with ), simply add it with $ modprobe <modul>. Now we can add a new project in OpenShift (run on the master node):

$ oc new-project storage-project

In this project, we deploy the necessary pods for Heketi, the deployer itself, etc. Since Gluster containers need to provide storage across projects, they need to be deployed as privileged containers. This gives them access to the host and other pods.

$ oadm policy add-scc-to-user privileged -z default
$ oadm policy add-scc-to-user privileged -z router
$ oadm policy add-scc-to-user privileged -z default

Now set up the router, which ensures that requests to Heketi can also reach the POS.

$ oadm router storage-project-router --replicas=1

Now we can start creating the topology YAML file for Heketi. In this file we define the individual Gluster nodes and their connected hard drives.

{
    "clusters": [
        {
            "node": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-0-1-66.eu-central-1.compute.internal"
                            ],
                            "storage": [
                                "10.0.1.66"
                            ]
                        },
                        "zones": 1
                    },
                    "devices": [
                        "/dev/docker-vg/testlv"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-0-1-69.eu-central-1.compute.internal"
                            ],
                            "storage": [
                                "10.0.1.69"
                            ]
                        },
                        "zones": 1
                    },
                    "devices": [
                        "/dev/docker-vg/testlv"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-0-1-144.eu-central-1.compute.internal"
                            ],
                            "storage": [
                                "10.0.1.144"
                            ]
                        },
                        "zones": 1
                    },
                    "devices": [
                        "/dev/docker-vg/testlv"
                    ]
                }
            ]
        }
    ]
}

There are a few things to note here. In node.hostnames.manage there must be an FQDN hostname, not an IP. In node.hostname.storage there must be an IP. Every node must be listed here and in devices there can be several comma-separated raw devices.

In addition to the existing 50GB EC2 root device, the GitHub project mentioned also installs an EBS volume which is set up exclusively for Docker storage during installation. I mentioned above that Heketi only expects raw devices or partitions. To make this work, I simply created additional EBS volumes via the AWS Console and attached them to the instances (nodes) Master, Node1 and Node2. At AWS, the EBS volumes are mounted on a defined path without restarting the ec2 instances. 

For the Heketi topology, I chose /dev/xvdg, so a new, fresh EBS volume (for each node). On the master you will find /usr/share/heketi/topology-sample.json in the folder. You can use this as a starting point or the file that I have inserted above. Please note that you will of course also have to change the host names and IPs. Once you have created such a topology file, you can start the deployment with:

sudo cns-deploy -n storage-project -g topology.json

Unfortunately, a lot can go wrong at this point. If that is the case, you can

sudo cns-deploy -n storage-project -g topology.json --abort Cancel the deployment, make changes and deploy again. This is exactly where the setting discussed at the beginning of the OpenShift installation comes into play. If we didn’t have this setting, we would have to manually empty the disk/device on each node using wipefs.

If everything goes well, the Gluster Pods and Heketi are deployed. So that we can call the heketi cli directly from the master node, please execute the following command:

$ sudo export HEKETI_CLI_SERVER=$(oc describe svc/heketi | grep "Endpoints:" | awk '{print "http://"$2}')

Now we can  sudo heketi-cli topology infolook at the GlusterFS topology. It will appear something like:

Now we need to define and deploy a StorageClass. StorageClasses are the interface between OpenShift and storage solutions such as EBS, Gluster, etc. This means that a StorageClass receives the request and knows how to obtain the storage depending on the implementation. In the case of AWS, it knows how to call the AWS API. In the case of GlusterFS, it knows how to address Heketi, etc. For GlusterFS, we create a corresponding Yaml file on the master. In my case, it is still in /usr/share/heketi.

sudo vim /usr/share/heleti/glusterfs-storage-class.yaml

kind: StorageClass
metadata:
  name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://heketi-storage-project.18.197.68.255.xip.io"
comments:
  storageclass.beta.kubernetes.io/is-default-class: "true"
  storageclass.kubernetes.io/is-default-class: "true"

Please note that the resturl should correspond to the URL of your Heketi installation. Also note that we have set an „is-default“ as an annotation. This means that every storage request from OpenShift is provided via GlusterFS by default. Otherwise you have to explicitly define each time which storage class should be used. The AWS EBS version is also included in the GitHub project (you can see this from the „provisioner“ property, by the way.

$ sudo oc get storageclass
NAMETYPE
glusterfs-storage (default) kubernetes.io/glusterfs
gp2 kubernetes.io/aws-ebs

This allows you to see whether the storage class was successfully deployed. Unfortunately, there is a bug in the OpenShift version we are using. Now we have two default entries as StorageClass. To really set our new StorageClass as the default, we have to delete the default annotation for the existing (old) StorageClass. This can be done with the command:

sudo oc edit storageclass gp2

Then simply set „true“ to „false“ and save. 

It’s done. Now we can either create storage directly in OpenShift and assign it to pods. Or request it directly via Yaml in the pod definition. To test this, we used an example project (Django-Psql-Persistent) from the OpenShift Catalog in the Web GUI. This example project deploys a PostgreSql and asks for a volume claim with the name postgresql. Since we have set our StorageClass to GlusterFS as the default, OpenShift uses this StorageClass to provide storage.

It becomes even clearer in the pod’s Yaml file:

Finally, a screenshot of manually setting up storage. In the dropdown you can choose between the different storage classes:

As you can see, the topic is not trivial and a lot can go wrong during installation. But once you have done it, you have a very flexible and automated system as a storage solution. Nodes can simply be added to the cluster and defined as GlusterNode. It is important to note that Red Hat recommends a maximum number of 300 nodes.