Introduction to CRAFT

CRAFT (Custom Resource Abstraction and Fabrication Tool) declares Kubernetes operators in a robust, idempotent, and generic way for any resource.

Creating a Kubernetes operator requires domain knowledge of abstraction and expertise in Kubernetes and Golang. With CRAFT you can create operators without a dependent layer and in the language of your choice!

Declare your custom resource in JSON files and let CRAFT generate all the files needed to deploy your operator into the cluster. CRAFT Wordpress operator generates 571 lines of code for you, a task that otherwise takes a few months to complete.

Reduce your workload by automating resource reconciliation with CRAFT to ensure your resource stays in its desired state. CRAFT also performs schema validations on your custom resource while creating the operator. Through automated reconciliation and schema validations, CRAFT achieves the objectives listed in the Kubernetes Architecture Design Proposal.

Advantages of using Craft

1. Easy onboarding : Create an operator in your language of choice.
2. Segregation of duties : Developers can work in the docker file while the Site Reliability or DevOps engineer can declaratively configure the operator.
3. Control access : Control which users have access to the operator resources.
4. Versioning and API interface : Work on a different version of the operator or resource than your users.
5. Save time : Get schema and input validation feedback before runtime.
6. Automated reconciliation : Automate resource reconciliation to lower your maintenance workload.
7. Dependent teams work independently : Dependent teams can automate independently and create abstraction layers on top of your abstraction.

Built with

CRAFT is built with open source projects Kubebuilder and Operatify:

• Kubebuilder : CRAFT augments the operator skeleton generated by Kubebuilder with custom resource definitions and controller capabilities.
• Operatify : CRAFT leverages Operatify’s automated reconciliation capabilities.

Quick Setup

Prerequisites

• Go version v1.13+
• Docker version 17.03+
• Kubectl version v1.11.3+
• Kustomize version v3.1.0+
• Kubebuilder version v2.0.0+

Installation

# dowload latest craft binary from releases and extract 
curl -L https://github.com/salesforce/craft/releases/download/0.1.7/craft.tar.gz | tar -xz -C /tmp/

# move to a path that you can use for long term
sudo mv /tmp/craft /usr/local/craft
export PATH=$PATH:/usr/local/craft/bin

Instead of having to add to PATH everytime you open a new terminal, we can add our path to PATH permanently.

$ sudo vim /etc/paths

Add the line "/usr/local/craft/bin" at the end of the file and save the file.

Create a CRAFT Application

From the command line, cd into a directory where you'd like to store your CRAFT application and run this command:

craft init

This will initiate a CRAFT application in your current directory and create the following skeleton files:

• controller.json: This file holds Custom Resource Definition (CRD) information like group, domain, operator image, and reconciliation frequency.
• resource.json: This file contains the schema information for validating inputs while creating the CRD.

Next Steps

Follow the Wordpress operator tutorial to understand how to use CRAFT to create and deploy an operator into a cluster. This deep-dive tutorial demonstrates the entire scope and scale of a CRAFT application.

Tutorial : Wordpress Operator

Unlike most tutorials who start with some really contrived setup, or some toy application that gets the basics across, this tutorial will take you through the full extent of the CRAFT application and how it is useful. We start off simple and in the end, build something pretty full-featured and meaningful, namely, an operator for the Wordpress application.

The job of the Wordpress operator is to host the Wordpress application and perform operations given by the user on the cluster. It reconciles regularly, checking for updates in the resource and therefore, can be termed as level-triggered.

We will see how the controller.json and resource.json required to run the wordpress operator have been developed. Then, we'll see how to use CRAFT to create the operator and deploy it onto the cluster.

The config files required to create the operator are already present in example/wordpress-operator

Let's go ahead and see how we have created our files for the wordpress application to understand and generalise this process for any application. First, we start with the controller.json file in the next section.

The following files are involved in creating a Custom Resource:

• controller.json 
• resource.json
• Resource DockerFile

Controller.json

Custom Resource Definition (CRD) information like the domain, group, image, repository, etc. are stored in the controller.json file. This skeleton file for controller.json is created when you create a CRAFT application in quickstart:

"group": "",
"resource": "",
"repo": "",
"domain": "",
"namespace": "",
"version": "",
"operator_image": "",
"image": "",
"imagePullSecrets": "",
"imagePullPolicy": "",
"cpu_limit": "",
"memory_limit": "",
"vault_addr": "",
"runOnce": "",
"reconcileFreq": ""

This table explains the controller.json attributes:

Here’s an example of a controller.json file for the Wordpress operator:

{
 "group": "wordpress",
  "resource": "WordpressAPI",
  "repo": "wordpress",
  "domain": "salesforce.com",
  "namespace": "default",
  "version": "v1",
  "operator_image": "ops0-artifactrepo1-0-prd.data.sfdc.net/cco/wordpress-operator",
  "image": "ops0-artifactrepo1-0-prd.data.sfdc.net/cco/wordpress:latest",
  "imagePullSecrets": "registrycredential",
  "imagePullPolicy": "IfNotPresent",
  "cpu_limit": "500m",
  "memory_limit": "200Mi",
  "vault_addr": "http://10.215.194.253:8200"
}

Resource.json

Resource.json contains the schema information for validating inputs while creating the Custom Resource Definition (CRD). The resource.json has a list of properties and required attributes for a certain operator.

"type": "object"
"properties": {},
"required": [],

The properties field contains the field name, data type and the data patterns. The required field contains the data names which are mandatory for the operator to be created.

Note: Resource.json remains the same for an operator regardless of the developer. controller.json for an operator may look different for each developer.

Populate resource.json by identifying required fields and their properties from the resource code. Here’s an example resource.json file for the Wordpress operator:

  "type": "object",
  "properties": {
    "bootstrap_email": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "bootstrap_password": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "bootstrap_title": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "bootstrap_url": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "bootstrap_user": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "db_password": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "dbVolumeMount": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "host": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "instance": {
      "enum": [
          "prod",
          "dev"
      ],
      "type": "string"
    },
    "name": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "replicas": {
      "format": "int64",
      "type": "integer",
      "minimum": 1,
      "maximum": 5
    },
    "user": {
      "pattern": "^(.*)$",
      "type": "string"
    },
    "wordpressVolumeMount": {
      "pattern": "^(.*)$",
      "type": "string"
    }
  },
  "required": [
    "bootstrap_email",
    "bootstrap_password",
    "bootstrap_title",
    "bootstrap_url",
    "bootstrap_user",
    "db_password",
    "dbVolumeMount",
    "host",
    "instance",
    "name",
    "replicas",
    "user",
    "wordpressVolumeMount"
  ]

Resource DockerFile - How does it help?

For our Wordpress operator, the resource Dockerfile looks like this:

FROM centos/python-36-centos7:latest

USER root

RUN pip install --upgrade pip
RUN python3 -m pip install pyyaml
RUN python3 -m pip install jinja2
RUN python3 -m pip install hvac

ADD kubectl kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

ARG vault_token=dummy
ENV VAULT_TOKEN=${vault_token}

ADD templates templates

ADD initwordpress.sh .
RUN chmod 755 initwordpress.sh
ADD wordpress_manager.py .
RUN chmod 755 wordpress_manager.py

RUN find / -perm /6000 -type f -exec chmod a-s {} \; || true

ENTRYPOINT ["python3", "wordpress_manager.py"]

The above resource Dockerfile for the Wordpress operator has Docker run two files/scripts:

1. initwordpress.sh : This file contains instructions to initialize the wordpress resource and install the required components.
2. wordpress_manager.py : This file contains CRUD operations defined by the Wordpress resource. These operations don’t return the usual output, but return exit codes.

CRUD operations in CRAFT operators

Define CRUD (Create, Read, Update, and Delete) operations for your operator. This diagram illustrates the flow for the 14 possible outputs of a CRUD operation:

Credits: Stephen Zoio & his project operatify

With CRAFT, you can account for all the14 cases by using 14 unique docker exit codes. Use these docker exit codes to check the result of the operation and route the path accordingly. The 14 exit codes that CRAFT provides are:

    201: "Succeeded", // create or update
	202: "AwaitingVerification", // create or update
	203: "Error", // create or update
	211: "Ready", // verify
	212: "InProgress", // verify
	213: "Error", // verify
	214: "Missing", // verify
	215: "UpdateRequired", // verify
	216: "RecreateRequired", // verify
	217: "Deleting", // verify
	221: "Succeeded", // delete
	222: "InProgress", // delete
	223: "Error", // delete
	224: "Missing", // delete

The 14 exit codes are correspondingly mapped to the 14 possibilities that arise from the operations.

While writing our CRUD operation definitions, we use the exit codes to specify output. For example, in the wordpress_manager.py, we can see the CUD operations:

def create_wordpress(spec):
    /*
    ..
    */
    if result.returncode == 0:
        init_wordpress(spec)
        sys.exit(201)
    else:
        sys.exit(203)

def delete_wordpress(spec):
    /*
    ..
    */
    if result.returncode == 0:
        sys.exit(221)
    else:
        sys.exit(223)

def update_wordpress(spec):
    /*
    ..
    */
    if result.returncode == 0:
        sys.exit(201)
    else:
        sys.exit(203)

def verify_wordpress(spec):
    /*
    ..
    */
    if result.returncode == 0:
        result = subprocess.run(['kubectl', 'get', 'deployment', 'wordpress-' + spec['instance'],  '-o', 'yaml'], stdout=subprocess.PIPE)
        deployment_out = yaml.safe_load(result.stdout)
        if deployment_out['spec']['replicas'] != spec['replicas']:
            print("Change in replicas.")
            sys.exit(214)
        sys.exit(211)
    else:
        sys.exit(214)

We map the corresponding output possibility to the exit code.

Now that we have defined our CRUD operations, let's create our operator.

Creating the Operator using CRAFT

Now that we have created controller.json, resource.json and the DockerFile, let's create the Operator using CRAFT.

First, let us check whether CRAFT is working in our machine.

$ craft version

This should display the version and other info regarding CRAFT.

!TIP

If this gives an error saying "$GOPATH is not set", then set GOPATH to the location where you've installed Go.

Now that we have verified that CRAFT is working properly, creating the Operator with CRAFT is a fairly straight forward process:

craft create -c config/controller.json -r config/resource.json \
--podDockerFile resource/DockerFile -p

NOTE

If the execution in the terminal stops at a certain point, do not assume that it has hanged. The command takes a little while to execute, so give it some time.

This will create the Operator template in $GOPATH/src, build operator.yaml for deployment, build and push Docker images for operator and resource. We shall see what these are individually in the next section.

Namespace.yaml

The namespace.yaml file contains information about the namespace of the cluster in which you want to deploy the operator. The namespace.yaml for the Wordpress operator looks like this:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: controller-manager
  name: craft

The operator.yaml file contains all the metadata required to deploy the operator into the cluster. It is the backbone of the operator as it contains the schema validations, the specification properties, the API version rules, etc. This file is automatically populated by CRAFT based on the information provided in the controller.json and the resource.json file. The operator.yaml that CRAFT generates for the Wordpress operator can be found at examples/wordpress-operator/config/deploy/operator.yaml.

Note

Our operator's default user corrently holds the minimum rbac required to run the operator and only the operator itself. If you need any more control of the rbac add those permissions in the operator.yaml.

Deploy operator onto the cluster

In the previous step, we created an operator. Now, we deploy it to the cluster. This involves deploying the namespace and the operator files to the cluster. Create the namespace where you want to deploy the operator with the namespace.yaml file we created in step 2 using this command:

$ kubectl apply -f config/deploy/namespace.yaml

When the command runs successfully, it returns namespace/craft created. You can check the namespace created by running this command:

kubectl get namespace

This should display all the existing namespaces, out of which craft is one. Install the operator onto the cluster with this command:

kubectl apply -f config/deploy/operator.yaml

This will create the required pod in the cluster. We can verify the creation by running:

kubectl get pods

This returns the wordpress pod along with the other pods running on your machine:

NAME                                           READY   STATUS         RESTARTS   AGE
wordpress-controller-manager-8844cf545-gn5rt   1/2     Running           0       11s

Great, your pod is running! You are ready to deploy the resource.

NOTE

If your pod’s status is ContainerCreating, run the command again in a few seconds and the status should change to running.

Deploy the resource onto the cluster using the wordpress-dev-withoutvault YAML file created by CRAFT.

kubectl -n craft apply -f config/deploy/wordpress-dev-withoutvault.yaml

This deploys the wordpress resource onto the cluster. To verify, run:

kubectl -n craft port-forward  svc/wordpress-dev 9090:80

Open http://localhost:9090 on the browser and you’ll see the Wordpress application. You can check the logs to see that reconciliation is running as configured. To see that, we can use stern.

stern -n craft .

Where is the Operator Code?

The source code for the operator is stored in the $GOPATH/src path.

$ cd $GOPATH/src/wordpress
$ ls

The folder contains all the files required to run an operator like the configurations, API files, controllers, reconciliation files, main files, etc. All these files contain information about the operator and it's runtime characteristics, such as the CRUD logic, reconciliation frequency, etc. These files can be classified in four sections:

1. Build infrastructure.

2. Launch Configuration.

3. Entry Point

4. Controllers and Reconciler.

CRAFT creates these files when you create an operator. This saves you a few weeks of effort to write and connect your operator.

Build Infrastructure

These files are used to build the operator:

• go.mod : A Go module for the project that lists all the dependencies.
• Makefile : File makes targets for building and deploying the controller and reconciler.
• PROJECT : Kubebuilder metadata for scaffolding new components.
• DockerFile : File with instructions on running the operator. Specifies the docker entrypoint for the operator.

Launch Configuration

The launch configurations are in the config/ directory. It holds the CustomResourceDefinitions, RBAC configuration, and WebhookConfigurations. Each folder in config/ contains a refactored part of the launch configuration.

• config/default contains a Kustomize base for launching the controller with standard configurations.
• config/manager can be used to launch controllers as pods in the cluster
• config/rbac contains permissions required to run your controllers under their own service account.

Entry point

The basic entry point for the operator is in the main.go file. This file can be used to:

• Set up flags for metrics.
• Initialise all the controller parameters, including the reconciliation frequency and the parameters we received from controller.json and resource.json
• Instantiate a manager to keep track of all the running controllers and clients to the API server.
• Run a manager that runs all of the controllers and keeps track of them until it receives a shutdown signal when it stops all of the controllers.

Controllers and Reconciler

A controller is the core of Kubernetes and operators. A controller ensures that, for any given object, the actual state of the world (both the cluster state, and potentially external state like running containers for Kubelet or loadbalancers for a cloud provider) matches the desired state in the object. Each controller focuses on one root API type but may interact with other API types.

A reconciler tracks changes to the root API type, checks and updates changes in the operator image at controller-runtime. It runs an operation and returns an exit code, and through this process it checks if reconciliation is needed and determines the frequency for reconciliation. Based on the exit code, the next operation is added to the controller queue.

These are the 14 exit codes that a reconciler can return:

## ExitCode to state mapping
  201: "Succeeded", // create or update
  202: "AwaitingVerification", // create or update
  203: "Error", // create or update
  211: "Ready", // verify
  212: "InProgress", // verify
  213: "Error", // verify
  214: "Missing", // verify
  215: "UpdateRequired", // verify
  216: "RecreateRequired", // verify
  217: "Deleting", // verify
  221: "Succeeded", // delete
  222: "InProgress", // delete
  223: "Error", // delete
  224: "Missing", // delete

Commands of CRAFT

craft version

Usage :

craft version

Displays the information about craft, namely version, revision, build user, build date & time, go version.

craft init

Usage :

craft init

Initialises a new project with sample controller.json and resource.json

craft create

Usage :

craft create -c "controller.json" -r "resource.json --podDockerFile "dockerFile" -p

Creates operator source code in $GOPATH/src, builds operator.yaml, builds and pushes operator and resource docker images.

craft build

Has 3 sub commands, code, deploy and image.

build code

Usage:

craft build code -c "controller.json" -r "resource.json

Creates code in $GOPATH/src/operator.

build deploy

Usage:

craft build deploy -c "controller.json" -r "resource.json

Builds operator.yaml for deployment onto cluster.

build image

Usage:

craft build image -b -c "controller.json" --podDockerFile "dockerFile"

Builds operator and resource docker images.

validate

Usage:

craft validate -v "operator.yaml"

Validates operator.yaml to see if everything is in shape