Simplifying Kubernetes for Developers with Hub CLI and Skaffold

Agile Stacks Updated by Agile Stacks

Simplifying Kubernetes for Developers with Hub CLI and Skaffold

Introduction

Until recently, the developer experience with Kubernetes was very complicated. So much, in fact, that it was recommended that smaller shops with only a handful of services should probably not bother with Kubernetes. This was because the overhead of configuration, CI/CD, and, most especially, developer workflows, was prohibitively high. In addition to configuring Kubernetes, there are many questions that don’t have a simple answer: how do I develop locally, how should I test, and how can I debug?

To address these challenges, we at Agile Stacks have created developer workflow automation based on Skaffold, Hub CLI, and VS Code.

Assuming that you have already deployed an application stack, this tutorial should take about 15 minutes.

Note: This tutorial leverages Kubernetes and additional infrastructure application services that that can be easily deployed using Hub CLI and Application Stack.

The goal of this tutorial is to provide automation for developer workflows on Kubernetes. At the core of this approach is Skaffold. Skaffold is a powerful tool for automating dev->test cycles on Kubernetes. In order to unlock some of the powerful features of Skaffold, AgileStacks provides automatic configuration management for Skaffold from Hub CLItool. When used together, these tools offer a combination of power and ease of use.

In this tutorial, you will perform the following steps:

  1. Configure your Application
  2. Deploy to Kubernetes
  3. Realtime Updates and Debugging
TL;DR We will create DevOps automation for Python applications on Kubernetes and deploy the app with the following commands:

$ git clone https://github.com/agilestacks/stack-apps.git
$ cd stack-apps/apps/python-flask
$ cd python-flask
$ hub configure -f hub.yaml
$ source .env
$ skaffold dev --port-forward

Overview

Application developers want the ability to develop and debug applications on remote Kubernetes clusters with the same level of productivity as when deploying locally, without having to commit unfinished changes to Git, and without having to wait for deployments. Your development cycle is just so much quicker when you can run your entire stack locally. By automating the local development workflow, we can significantly shorten the development cycles and provide a quick feedback loop which is crucial for developer productivity. At the same time, we need consistency between how the code behaves locally and in the cloud. For example, we need ability to execute multiple remote dependencies, such as microservices, databases, and messaging systems. Once your application code is ready and committed to Git version control, then it can be deployed to other environments (test, stage, prod) via CI/CD pipeline.

This project allows to automate many steps in the developer workflow for Kubernetes:

  1. Automatically generate Kubernetes manifests for Applications, as well as the Skaffold configuration to automate your workflows.
  2. Deploy applications to local or remote Kubernetes clusters
  3. Monitor source code for local changes and automatically redeploy when needed
  4. Set debugging breakpoints
  5. Stream logs inside the local IDE or terminal
  6. Inspect the API routes and controllers, update configuration files, and iterate without any delays or restarts

Prerequisites

This is a DevOps-focused tutorial, so it assumes that you have the following development and config-management tools installed.

  • vscode: Visual Studio Code IDE for Python. (Installation instructions  are here)
  • hub: Automation Hub CLI to configure and deploy your software stacks (Installation instructions are here)
  • skaffold: to automate the workflow for building and deploying your application
  • jq: JSON parser for SuperHub API routines
  • yq: for JSON to YAML conversion
  • jsonnet: for code generation
  • kubectl: command line tool for controlling Kubernetes clusters
  • make: to control the code generation routines
  • docker: docker client for managing containers

Configure Your Application

  1. Fetch the example application source code.
$ git clone https://github.com/agilestacks/stack-apps.git
$ cd stack-apps/apps/python-flask

Let's review the files downloaded for the application template:

python-flask:
/.hub # SuperHub code generation has been located here
/.vscode # Configuration for vscode
/src # Python Flask application
/test # Container structure texts will be here
Dockerfile # docker image with your application
  1. Validate that your application stack was already installed and your Kubernetes cluster context is set

Use the following example command to configure kubeconfig for an existing EKS cluster.

$ aws eks --region us-east-2 update-kubeconfig --name applications
Added new context arn:aws:eks:us-east-2:10813268X35X:cluster/applications to /Users/dev/.kube/config

$ hub show
{
"components": [
"cert-manager",
"external-dns",
"harbor",
"kube-dashboard",
"postgresql",
"prometheus",
"tiller",
"traefik"
]
}

If hub show does not show any components for harbor and postgresql, you need to create a new Kubernetes cluster with App Stack. You can deploy a new cluster via Hub CLI as described in Application Step Readme.
  1. Configure your application to point to the selected Kubernetes cluster.

The hub configure command is used to specify the cluster name where the application is going to be deployed.

$ hub configure -f hub.yaml
Checking hub files:
* File hub.yaml: exist
Reading .env file:
Stack domain name: acid-shallot-229.bubble.superhub.io
Configuring Kubernetes
* Using context: acid-shallot-229.bubble.superhub.io
* Checking connectivity to cluster: Connected
* Saving kubeconfig to .hub/env/acid-shallot-229.bubble.superhub.io.kubeconfig: Already up to date
Configuring application
* Using name: rubik-e5f2
* Running code generation tasks: generate
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/k8s/deployment.yaml' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/k8s/ingress.yaml' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/k8s/kaniko-secret.yaml' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/k8s/service.yaml' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/./skaffold.yaml' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/.hub/dockerconfig.json' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/.vscode/launch.json' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/.vscode/settings.json' is up to date.
make[1]: `/Users/igor/dev/stack-apps/apps/python-flask/.vscode/tasks.json' is up to date.
Finalizing env files...
* Saving configuration to .hub/env/acid-shallot-229.bubble.superhub.io.env: Already up to date
Done!

$ source .env

By running hub configure , you have generated the necessary configuration files for Kubernetes. Examine the configuration files located in .hub directory:

/python-flask/.hub/env/kubeconfig.cluster_name.superhub.io.kubeconfig
/python-flask/.hub/this/configure
/python-flask/.hub/.vscode/settings.jsonnet
/python-flask/.hub/.vscode/launch.jsonnet

You have also generated a set of configuration files to deploy the application with Skaffold:

  • skaffold.yaml: a configuration file for Skaffold
  • k8s/*.yaml: kubernetes deployment manifests
  • .vscode/*.json: configuration for vscode
  • .hub/dockerconfig.json: a docker auth file. It will be used by Skaffold as docker-cfg secret to push image into docker registry.
  1. Install required plugins

For Python development we will use Visual Studio Code with additional plugins such as Google CloudCode. Alternative setup for IntelliJ IDE can be found here

$ code \
--install-extension "googlecloudtools.cloudcode" \
--install-extension "heptio.jsonnet" \
--install-extension "ms-azuretools.vscode-docker" \
--install-extension "ms-kubernetes-tools.vscode-kubernetes-tools" \
--install-extension "ms-python.python" \
--install-extension "redhat.vscode-yaml" \
--install-extension "xrc-inc.jsonnet-formatter"
  1. Open VS Code from the application directory
$ code -n .

Running VS Code and Skaffold

With the local development environment configured, you’re now ready to launch your application. Typically, you’d have to perform several tedious and error-prone tasks to build Docker containers and create Kubernetes configuration files. Fortunately, Skaffold is the tool that automatically generates required Kubernetes manifests. Skaffold also watches for code changes, and once a change is detected, Skaffold automatically initiates the steps to build, push and deploy the new code to a Kubernetes cluster.

Walk through the application source code

Code

Let's review the example Python Flask application. All application specific source code is available in /src directory

/src
app.py # Flask application goes here
/static # CSS, JavaScript and other media goes here
/templates # Home for HTML templates, managed by Flask
/conf # Flask environment configuration goes here
requirements.txt # Our dependencies (managed by pip)
routes.py # Application routes
/k8s # Generated by the jsonnet kubernetes manifests, managed by Skaffold
/test # Container structure tests (see: https://github.com/GoogleContainerTools/container-structure-test)
Dockerfile # A docker file, managed by Skafold
skaffold.yaml # Skaffold configuration file

Python Flask Application

app.py is a simple web application. It imports routes.py that provides a router for:

  • / - renders the index.html single page template
  • /gimme/<int:howmany> - returns a JSON array with random words (size:(size: howmany variable)
  • /status - for Kubernetes health checks

By default, the application comes with remote debug enabled.

if application.config["PTVSD_PORT"]:
import ptvsd
ptvsd.enable_attach(address=('0.0.0.0', application.config["PTVSD_PORT"]))

For remote debug, the application is integrated with ptvsd, a Python debugger package. It supports integration with vscode (see here) and with Skaffold (see here).

Dockerfile

The Dockerfile is based on Python 3.7 and exposes two ports:

  • 80 for a flask applicaiton (standart HTTP port)
  • 3000 for ptvsd for remote debug.

Note about PRODUCTION use: It is recommended to disable remote debugging for production environments. To disable the debugger, you need to make changes in the following two files: src/app.py and Dockerfile. For production deployments, it is also recommended to define gunicorn (or any other WSGI server of your choice) as entrypoint instead of python + flask.

# CMD ["python3", "-m", "flask", "run", "--no-debugger", "--no-reload"]
ENTRYPOINT ["gunicorn", "-b", "0.0.0.0:80", "app"]

Deploy to Kubernetes

  1. Run skaffold to build and deploy the app to Kubernetes
$ source .env   # skip if you have already done this
$ skaffold dev --port-forward

You should be able to see the following output:

> Executing task: skaffold dev <

Listing files to watch...
- rubik
Generating tags...
- rubik -> cluster1-harbor.app.cluster1.bluesky.superhub.io/library/rubik:20200213-174151
Checking cache...
- rubik: Found. Tagging
Tags used in deployment:
- rubik -> cluster1-harbor.app.cluster1.bluesky.superhub.io/library/rubik:20200213-174151@sha256:07b0c3ccebabfb0337e016e59bae20ee5b6cb3b05d6e7f320db98ea45841fce9
Starting deploy...
- deployment.apps/rubik configured
- ingress.extensions/rubik configured
- secret/rubik-dockerconfig configured
- service/rubik configured
Watching for changes...
[rubik-76f8bf44f8-tjdrx applicaiton] * Serving Flask app "app.py"
[rubik-76f8bf44f8-tjdrx applicaiton] * Environment: docker
[rubik-76f8bf44f8-tjdrx applicaiton] * Debug mode: on
[rubik-76f8bf44f8-tjdrx applicaiton] * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)

Let's review the output of skaffold dev command:

  1. Skaffold has created a Docker image for the application. By default, timestamp is used as Docker image tag (image tag configuration can be customized via skaffold.yaml)
  2. The Docker image was pushed to Harbor docker registry. To build Docker images on the server, it is using Kaniko. If you prefer to build and push Docker images to your local workstation, then you can switch to the local profile
export SKAFFOLD_PROFILE=local
  1. Kubernetes manifests from /k8s directory were applied to the selected cluster (Skaffold also maintains a deployment image)
  2. Skaffold continues to run in the background, waiting for code changes, and ready to automatically rebuild and redeploy the application when local files are changed.

Access the deployed application

Via ingress document - Open the file: k8s/ingress.yaml and find the parameter setting for host.

spec:
rules:
- host: rubik-e5f2.app.acid-shallot-229.bubble.superhub.io

Based on the output shown above, you can access the deployed application using the following URL: https://rubik-e5f2.app.acid-shallot-229.bubble.superhub.io.

Congratulations, you have successfully deployed Python Flask application on Kubernetes!

Realtime Updates and Debugging

When skaffold dev command is running, you should see following log output in the terminal

Watching for changes...
[rubik-7d9655fcdc-dv8fv applicaiton] * Serving Flask app "app.py"
[rubik-7d9655fcdc-dv8fv applicaiton] * Environment: docker
[rubik-7d9655fcdc-dv8fv applicaiton] * Debug mode: on
[rubik-7d9655fcdc-dv8fv applicaiton] * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)

Skaffold is watching for changes in the application source code. Once any application files are changed, skaffold will sync up changes to the running container (additional details here). This operation doesn't require a new image build. You can configure files and directories that skaffold is monitoring for changes. Open skaffold.yaml and you should be able to see the following:

build:
artifacts:
- image: rubik
sync:
manual:
- dest: /app
src: src/**/*.py
strip: src/
- dest: /app
src: src/static/**
strip: src/

Next, you can open file src/app.py and change the following code

WORDS = [
'helm', 'kustomize', 'kubernetes', 'aws', 'gcp', 'azure',
'terraform', 'docker', 'shell', 'vault', 'istio'
]

Let's add some more words into array. For example: "skaffold", "flask", "ptvs", "vscode" so the changed code will look like

WORDS = [
'helm', 'kustomize', 'kubernetes', 'aws', 'gcp', 'azure',
'terraform', 'docker', 'shell', 'vault', 'istio',
"skaffold", "flask", "ptvs", "vscode"
]

Save the file (CMD+S)

You will see the following Skaffold logs updates in the Tasks output window:

Syncing 1 files for ml1-harbor.svc.ml1.demo51.superhub.io/library/rubik:20200216-102754@sha256:84124851bada3019eeda931191d219d5143d1ea686656b0867ecbbda09c23f98
Watching for changes...
[rubik-7fdd6687f5-rgt75 applicaiton] * Detected change in '/app/app.py', reloading
[rubik-7fdd6687f5-rgt75 applicaiton] * Restarting with stat
[rubik-7fdd6687f5-rgt75 applicaiton] * Debugger is active!
[rubik-7fdd6687f5-rgt75 applicaiton] * Debugger PIN: 132-294-265

Once you see the message Detected change in '/app/app.py', reloading, open the internet browser using the following commands: (CMD+Shift+P >> Run Task >> Open in browser). Move your mouse over the Rubik's kube displayed in the browser. Mouse events are routed to /gimme endpoint. These events will update various messages displayed in the browser.

Rubik's cube

Remote debugging

Now let's do some debugging. This is a little tricky, because our Python application has been wrapped into a docker container, which is running inside kubernetes. So, we will use the following tool chain:

  • Visual Studio Code - IDE
  • ptvsd - Python debugger for VS Code
  • Docker should expose port 3000 (can be reconfigured to a different port)
  • Task in launch.json in vscode should point to the same port (3000)
  • Flask application should be running with --no-reload, in our case controlled via environment variable FLASK_RUN_RELOAD=0

Notice that it's not possible to have both automated code sync and the remote debug. The good news, you don't need to redeploy the application to change these settings. Skaffold will do it for you. To enable remote debugging, perform the following steps:

  1. Modify the Dockerfile and save it:
ENV FLASK_RUN_RELOAD 0
  1. Skaffold will rebuild the container automatically. You can see the progress in the terminal window of Task - Skaffold
    Once the application has been rebuilt and the pod is restarted, you will see "Running on http://0.0.0.0:80/" message again:
...
Starting deploy...
- deployment.apps/rubik configured
Watching for changes...

* Serving Flask app "app.py"
* Environment: docker
* Debug mode: on
* Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
  1. To bring up the Debug view, select the Debug icon in the Activity Bar on the side of VS Code. You can also use the keyboard shortcut ⇧⌘D. Select Attach to running container from the dropdown of run configurations:
    Run debugger

Important note: make sure you are running flask in with 'no-reload' option. In addition to the Dockerfile, auto-reload control flag FLASK_RUN_RELOAD can be declared in k8s/deployment.yaml as a pod env.

You can now place a breakpoint in src/app.py. Breakpoints can be toggled by clicking on the editor margin or using F9 on the current line. For example, try setting a debug breakpoint in get_words() function.

In addition to a breakpoint, you can also setup a logpoint. VS Code Logpoint is represented by a "diamond" shaped icon. Log messages are plain text, but can include expressions to be evaluated within curly braces ('{}').

  1. Open your application in a web browser (CMD+Shift+P >> Run Task >> Open in browser) and trigger an event by moving your mouse pointer over the cube.
Rubik's cube

Conclusion

Congratulations! You have created a Flask application, wrapped it in a Docker container, and pushed it to a private Docker registry. Then you scheduled a Kubernetes deployment, enabled live code change reloads with Skaffold, and remote debugging with ptvsd for your applications running on Kubernetes. You have learned how to use Hub CLI, VS Code, and Skaffold to configure, deploy, and debug applications on Kubernetes with a few simple commands. Unless you automate the entire workflow, developers have to manually perform configuration instead of writing application code. By automating the local development workflow, you can shorten the feedback loop, reduce misconfiguration errors, and reduce the effort to maintain local environments. Even if you happen to break your configuration, you can use automation to recreate it.

How did we do?

Creating Components from CloudFormation Templates: SageMaker Pipeline

Developer Workflow: Enable Stateful Applications on Kubernetes (201)

Contact