Nishadi's Tech Blog

Load balancing in WSO2 app cloud's Kubernetes Cluster is configured via HAProxy load balancer. In order to load balance in the Kubernetes cluster, we need to update the HAProxy configuration file with newly created applications details in real time. Thus, to achieve that what is used in the app cloud is a feature that provides load balancing to using services in Kubernetes Contrib repo.
This blog post will guide you on how to configure a HAProxy load balancer in WSO2 appcloud.

Get the kubernetes contrib repo to your local machine

$go get github.com/kubernetes/contrib.git

Move to the service load balancer

kubernetes/contrib/service-loadbalancer

In order to enable HTTPS via the load balancer, it needs to add proper CA certificates file, worker nodes' certificate file and the key file to the load balancer. Thus, to include these files in the loadbalancing kubernetes pod, we need to include them in the docker image. Follow the below guidelines to add these files to the docker image of HAProxy service load-balancer. For this tutorial, we assume that the CA certificate is ca.crt, SSL certificate of the worker node is ssl.crt and the private key is ssl.key.

Combine the SSL certificate file and the private key of the worker node to a single .pem file

$ cat ssl.crt ssl.key \
           |  tee ssl.pem

Locate the obtained ssl.pem file and ca.crt file in the service loadbalancer folder.
Update the Docker file to add the certificates

##kubernetes/contrib/service-loadbalancer/Dockerfile


FROM gcr.io/google_containers/haproxy:0.2
MAINTAINER Prashanth B <beeps@google.com>

RUN mkdir -p /etc/haproxy/errors /var/state/haproxy
RUN for ERROR_CODE in 400 403 404 408 500 502 503 504;do curl -sSL -o /etc/haproxy/errors/$ERROR_CODE.http \
 https://raw.githubusercontent.com/haproxy/haproxy-1.5/master/examples/errorfiles/$ERROR_CODE.http;done

RUN wget -O /sbin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.0.0/dumb-init_1.0.0_amd64 && \
  chmod +x /sbin/dumb-init

ENTRYPOINT ["dumb-init", "/service_loadbalancer"]

ADD haproxy.cfg /etc/haproxy/haproxy.cfg
ADD service_loadbalancer service_loadbalancer
ADD service_loadbalancer.go service_loadbalancer.go
ADD template.cfg template.cfg
ADD loadbalancer.json loadbalancer.json
ADD haproxy_reload haproxy_reload
ADD README.md README.md
ADD ssl.pem /etc/haproxy/ssl.pem
ADD ca.crt /etc/haproxy/ca.crt

RUN touch /var/run/haproxy.pid

Now that you have added the corresponding certificate files to the loadbalancing docker file and we need to build it with service loadbalncer. Thus run the 'make' command to run the make file located in 'kubernetes/contrib/service-loadbalancer/Make'

When the docker image is successfully built, the following rc.yaml can be used to create the HAProxy loadbalancer in Kubernetes cluster.

apiVersion: v1
kind: ReplicationController
metadata:
  name: service-loadbalancer
  labels:
    app: service-loadbalancer
    version: v1
spec:
  replicas: 1
  selector:
    app: service-loadbalancer
    version: v1
  template:
    metadata:
      labels:
        app: service-loadbalancer
        version: v1
    spec:
      nodeSelector:
        role: loadbalancer
      containers:
      - image: nishadi/lb-wso2-appcloud-prod:0.1
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        name: haproxy
        ports:
        # All http services
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        # nginx https
        - containerPort: 443
          hostPort: 8443
          protocol: TCP
        # mysql
        - containerPort: 3306
          hostPort: 3306
          protocol: TCP
        # haproxy stats
        - containerPort: 1936
          hostPort: 1936
          protocol: TCP
        resources: {}
        args:
        - --tcp-services=mysql:3306,nginxsvc:443
        - --ssl-cert= /etc/haproxy/ssl.pem
        - --ssl-ca-cert=/etc/haproxy/ca.crt

We can deploy the above replication controller using the below command.

$ kubectl create -f ./rc.yaml
replicationcontrollers/service-loadbalancer
$  kubectl get pods -l app=service-loadbalancer
NAME                         READY     STATUS    RESTARTS   AGE
service-loadbalancer-dapxv   0/2       Pending   0          1m
$ kubectl describe pods -l app=service-loadbalancer
Events:
  FirstSeen                                    From            Reason                  Message
  Tue, 21 Jul 2015 11:19:22 -0700              {scheduler }    failedScheduling        Failed for reason MatchNodeSelector and possibly others

The above mentioned error would stop the pod from starting because the scheduler is waiting for you to tell it which nodes to use as a load balancer. Thus we need to label the node with the corresponding node selector defined in the rc.yaml.

$ kubectl label node e2e-test-beeps-minion-c9up role=loadbalancer
NAME                         LABELS                                                                STATUS
e2e-test-beeps-minion-c9up   kubernetes.io/hostname=e2e-test-beeps-minion-c9up,role=loadbalancer   Ready

This will start the HAProxy load balancing pod in the kubernetes cluster.

References:
Kubernetes Service LoadBalancer

As Kubernetes is an open source cluster management system to run and manage containerized applications, the users need a way to expose the services created in Kubernetes cluster to be accessible from outside.

Kubernetes built-in mechanisms to expose services in Kubernetes cluster to external traffic, provide layer 4 load balancing for the Kubernetes cluster.

This post is intended to demonstrate an existing problem in session affinity in kubernetes, when the kubernetes services are load balanced though ingress controllers and to explain the solution provided by kubernetes as a solution.

Existing Methods for exposing Kubernetes Services

Kubernetes provides a couple of methods for exposing services to the external traffics. NodePort and LoadBalancer can be specified via ServiceType attribute in service definition yaml/json file. Another approach introduced by Kubernetes release 1.1 is Ingress API.

NodePort

By defining ServiceType as NodePort and specifying a port number as nodePort will expose the specified service in all nodes with the specified nodePort number. In order to expose that service to internet, the user can expose one or more nodes on that port. When incoming traffic hits a node on that port, the load get load balanced among the pods of the service.

LoadBalancer

In declaring the ServiceType as LoadBalacer, helps to distribute the incoming traffic among the pods of the service through a cloud load balancer. This approach is supported only by certain cloud providers and Google Container Engine.

Ingress API

An ingress is a collection of rules that enables the user to expose services to internet through custom URLs and multiple virtual hosts.

Simple Fanout Ingress Definition

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: s1
          servicePort: 80
      - path: /bar
        backend:
          serviceName: s2
          servicePort: 80

This definition allows the user to access the service s1 at the URL /foo and the service s2 to be accessed at URL /bar.

Name Based Virtual Hosting

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: s1
          servicePort: 80
  - host: bar.foo.com
    http:
      paths:
      - backend:
          serviceName: s2
          servicePort: 80

This definition allows the user to access the service s1 at the foo.bar.com host name and the other service at the bar.foo.com.

In order to use Ingress API to expose services, one approach is to use an Ingress Controller. The main responsibility of an Ingress controller is to watch the API servers /ingresses endpoint for new ingresses. The existing ingress controller uses nginx load balancer and updates the nginx configuration file according to the ingress definition.

For an example if we deploy nginx-alpha ingress controller and create the above mentioned simple fanout example ingress definition, the ingress controller would generate nginix.conf file as follows and reload nginx server.

events {
  worker_connections 1024;
}
http {
  server {
    listen 80;
    server_name foo.bar.com;
    resolver 127.0.0.1;

    location /foo {
      proxy_pass http://s1;
    }
    location /bar {
      proxy_pass http://s2;
    }
  }
}

Problem with session affinity in using an ingress controller

What is session affinity?

Session affinity is a feature that the requests from the same client always get routed back to the same server within a cluster of servers.

How to enable session affinity in a kubernetes service?
To enable the session affinity in kubernetes, we can add the following to the service definition.

service.spec.sessionAffinity to "ClientIP"

The existing problem with session affinity and load balancing

With this approach the nginx ingress controller load balances the external traffic to the service. The requests from all clients come to the service through the load balancer. The enabled session affinity in service takes into consideration the client IP to handle session affinity. Since the service receives all the requests from the load balancer, it sees the same client IP for all the requests. Thus the service directs the requests from all clients to the same pod since it is unable to differentiate between the clients. This scenario will overload a pod and it would problems for the running application.

The Solution...

The solution is to directly load balance to the pods without load balancing the traffic to the service. This functionality is implemeted with service-loadbalancer in kubernetes. This implementation uses haproxy to enable session affinity and directly load balance the external traffic to the pods without going through services.

In order to create this HA Proxy loadbalancer, you can use https://github.com/kubernetes/contrib/tree/master/service-loadbalancer
By creating the replication controller in that repo named as rc.yaml , you can create the HAProxy load balancing pod.

$ kubectl create -f rc.yaml

This command will create the ha proxy load balabncer pod, but it would not come to the running state until the nodes that need to be exposed via ingress resource are tagged with

role=loadbalancer

tag.

In order to tag the targeted nodes run the below command.

$ kubectl label node <node-ip> role=loadbalancer

This will run the load-balancer in tagged nodes. Thus when we create an ingress resource, the load-balancer would distribute the external traffic to the pods according to the ingress definition.

References :

Special thanks for the guidance and support given by Mr. Imesh Gunaratne, Product Lead, WSO2 PPaaS

What is Kubernetes?

Yet another revolutionary discovery of google has come in to play in the name of Kubernetes as a result of Google's 10+ years of experience in containerized apps. In a nutshell, Kubernetes is an open source platform solution provided by Google to manage a cluster of containers.

Kubernetes has the ability of scheduling and running application containers on a cluster of physical or virtual machines. In managing the applications, the concepts of 'labels' and 'pods' are used to group the containers which make up an application. Currently, it supports Docker for containers.

The best part of Kubernetes is its self healing mechanisms such as auto restarting, rescheduling and container replicating used to manage the 'desired state' requested by the user. When the user defines the required state, Kubernetes ensures that the state is maintained at all times on the cluster. For an example if a server goes down, it brings it back again to maintain the state.

Typically, the deployment of a containerized application is a lot of hard work. For an example, think of an application that has a cluster of containers across multiple hosts. Say it has a web tier, application tier and a database tier running on different set of containers. In order the application to function properly, it has to communicate between web tier, application tier and the database tier. That is communication between separate containers. This needs a lot of hard work including scripting, monitoring and management of containers. But, with Kubernetes the problem is solved as it does all the hard work for the user.

Kubernetes Terminology

Pod

A pod is a single schedulable unit in Kubernetes which represents a co-located group of containers. It can not span machines, thus all the containers within a pod should be scheduled on the same host and can be deployed and scaled as a single application. Each pod has a unique IP address and the containers in the pod use a shared network namespace.

Replication Controller

A pod, itself is not able to handle a failure situation. In a situation like a crash in the node, where the pod is running, the pod is removed from the cluster. But in order to provide the desired state to the user, we need to ensure that the state is maintained. The replication controller monitors the cluster and ensures that a given number of pods are running at all the time. If there are more than specified, it will kill some, If there are too few, it will create some. In order to a pod to be monitored by a replication controller, we need to define the pod as a template inside the replication controller.

Service - the basic unit that acts as a load balancer for other containers.

Label - a key/value pair that is assigned to a resource to identify a set of related resources

Selector - an expression that supports to match user defined labels

Kubernetes Architecture

The basic Kubernetes architecture contains nodes and master.

Kubernetes Master

The three important parts of the master are as follows;

API Server - manages the tasks of all components in master and nodes via API calls
Etcd - stores and replicates the current configuration and run state of the cluster
Scheduler and Controller Manager - responsible in scheduling containers

Kubernetes Nodes

A node in Kubernetes has the following important processes:

Kubelet - responds to the commands from the master to manage pods and their containers
Proxy - a simple network proxy, which separates the IP address of a target container from the name of the service

In conclusion, it seems pertinent to remember that Kubernetes is a milestone in container cluster management systems.

Subscribe to: Posts ( Atom )

Navigation

Configuring HAProxy Load Balancer for WSO2 AppCloud

Load Balancing Kubernetes Services and Enabling Session affinity

Existing Methods for exposing Kubernetes Services

NodePort

LoadBalancer

Ingress API

Name Based Virtual Hosting

Problem with session affinity in using an ingress controller

The Solution...

References :

Getting Started with Kubernetes

What is kubectl?

Creating a pod

Viewing a pod

Deleting a pod

Creating a replication controller

Viewing a replication controller

Deleting a replication controller

Deploying a Sample Tomcat Application on Kubernetes

Kubernetes, at a first glance

Kubernetes Terminology

Pod

Replication Controller

Kubernetes Architecture

Kubernetes Master

Kubernetes Nodes

Instagram