Table of Contents

Developer Workflow - Accelerated Development and Deployment on Kubernetes with Skaffold (101)

Agile Stacks Updated by Agile Stacks

Developer Workflow: Accelerated Development and Deployment on Kubernetes with Skaffold


Until recently, the developer experience with Kubernetes was very complicated. So much, in fact, that it was recommended that smaller shops with only a handful of services should probably not bother with Kubernetes. This was because the overhead of configuration, CI/CD, and, most especially, developer workflows, was prohibitively high. In addition to configuring Kubernetes, there are many questions that don’t have a simple answer: how do I develop locally, how should I test, and how can I debug?

To address these challenges, we at Agile Stacks have created developer workflow automation based on Skaffold, AgileStacks' Hub CLI, VS Code, and Harbor.  

Assuming that you have the prerequisites already, this tutorial should take about 15 minutes.

Note: For maximum efficiency and the least amount of ceremony, this tutorial leverages Kubernetes clusters from our Demo environment that have been pre-created. Since this is an interactive tutorial with a focus on development workflows, we have removed the initial overhead of creating and configuring a Kubernetes cluster. If you'd like to run this tutorial with your own cluster, you can do so after Introducing AgileStacks to Your Cluster or, if you'd like to create a new one, we can help you with that as well.

The goal of this tutorial is to provide automation for developer workflows on Kubernetes. At the core of this approach is Skaffold. Skaffold is a powerful tool that can assist you in drastically speeding up dev->test cycles in Kubernetes. In order to unlock some of the extra-powerful features of Skaffold, AgileStacks has built automatic configuration management into its CLI tool, called hub. When combined, these tools offer the best in both power, and ease of use.

You will perform the following steps:

  1. Initialize your Application
  2. Running VS Code and Skaffold
  3. Deploy to Kubernetes
  4. Realtime Updates and Debugging


Application developers want the ability to develop and debug applications on remote Kubernetes clusters with the same level of productivity as when deploying locally, without having to commit unfinished changes to Git, and without having to wait for deployments. Your development cycle is just so much quicker when you can run your entire stack locally. By automating the local development workflow, we can significantly shorten the development cycles and provide a quick feedback loop which is crucial for developer productivity. At the same time, we need consistency between how the code behaves locally and in the cloud. For example, we need ability to execute multiple remote dependencies, such as microservices, databases, and messaging systems. Once your application code is ready and committed to Git version control, then it can be deployed to other environments (test, stage, prod) via CI/CD pipeline.

This project allows to automate many steps in the developer workflow for Kubernetes:

  1. Automatically generate Kubernetes manifests for Applications, as well as the Skaffold configuration to automate your workflows.
  2. Deploy applications to local or remote Kubernetes clusters
  3. Monitor source code for local changes and automatically redeploy when needed
  4. Set debugging breakpoints
  5. Stream logs inside the local IDE or terminal
  6. Inspect the API routes and controllers, update configuration files, and iterate without any delays or restarts

TLDR: watch a video version of this tutorial


This is a DevOps-focused tutorial, so it assumes that you have the following development and config-management tools installed.

  • vscode: Visual Studio Code IDE for Python. (Installation instructions are here)
  • hub: SuperHub CLI (installation notes below) to manage and configure your infrastructure
  • skaffold: to automate the workflow for building and deploying your application
  • jq: JSON parser for SuperHub API routines
  • yq: for JSON to YAML conversion
  • jsonnet: for code generation
  • kubectl: command line tool for controlling Kubernetes clusters
  • make: to control the code generation routines
  • docker: docker client for managing containers

On MacOS you can easily install these tools and utilities with Homebrew:
brew install jq yq jsonnet skaffold kubectl make docker
To install the AgileStacks Hub CLI:

On MacOS

curl -Lo hub
chmod +x hub
sudo mv hub /usr/local/bin
hub extensions install

On Linux

curl -Lo hub
chmod +x hub
sudo mv hub /usr/local/bin
hub extensions install

Initialize Your Application

  1. Fetch the example application source code.
$ git clone
$ cd stack-apps/apps/python-flask

Let's review the files downloaded for the application template:

/.hub # SuperHub code generation has been located here
/.vscode # Configuration for vscode
/src # Python Flask application
/test # Container structure texts will be here
Dockerfile # docker image with your application
  1. Retrieve a SuperHub authentication token.

You can retrieve your API token from SuperHub UI or using Hub CLI. In case you deployed Kubernetes and stack-app-eks Agile Stacks certified stack from command-line you should skip this section.

Use the following command to retrieve the authentication token using Hub CLI:

$ hub login
Username: # your SuperHub control plane username
Password: ********** # your SuperHub control plane password

$ export HUB_TOKEN=sergd......kieud

You need to export HUB_TOKEN environment variable to provide authentication for Hub CLI. This step will configure Hub CLI and generate your application configuration for the specified Kubernetes cluster.

If you don't have a SuperHub username and password, you can register for a free account: Register.

Run the following commands:

$ echo $HUB_TOKEN

$ hub ls -p harbor

The hub ls command lets you list and filter Kubernetes clusters in your environment that meet certain criteria. In this case, we're trying to find Kubernetes clusters that provide the Harbor private docker registry.

Note: your environment may use Kubernetes cluster names that are different from this example.

If hub ls does not show any matching clusters, you need to create a new Kubernetes cluster with Harbor registry. You can deploy a new cluster via Hub CLI as described in Deploy Kubernetes with Hub CLI, or via Control Plane UI as described in Creating Stacks tutorial.

  1. Configure your application to point to your chosen k8s cluster.

Run the following commands:

$ hub configure -s
# is the target cluster

$ source .env
$ kubectl cluster-info
Kubernetes master is running at

The hub-configure command is used to specify the cluster name where the application is going to be deployed. Please replace with the name of your cluster. When you execute hub-configure command, SuperHub will save Kubernetes cluster configuration file (Kubeconfig) in the following directory: .hub/env. Also, it will create a symlink pointer to the actual cluster configuration: .hub/current

In case the Agile Stacks certified stack is deployed from command-line, ie. stack-app-eks, then you should additionally specify Kubeconfig context name to hub-configure:

$ hub configure -p -s

Next, you will generate the application configuration files. You need to generate new configuration files each time you change your cluster.

$ make -C ".hub" clean generate
Generated: ../skaffold.yaml
Generated: ../k8s/deployment.yaml
Generated: ../k8s/ingress.yaml
Generated: ../k8s/kaniko-secret.yaml
Generated: ../k8s/service.yaml
Generated: ../.hub/dockerconfig.json
Generated: ../.vscode/launch.json
Generated: ../.vscode/settings.json
Generated: ../.vscode/tasks.json

By running Make, you have generated the necessary configuration files for Kubernetes. Examine the configuration files located in .hub directory:


You have generated a set of configuration files to deploy the application with Skaffold. The most important files are:

  • skaffold.yaml: a configuration file for Skaffold
  • k8s/*.yaml: kubernetes deployment manifests
  • .vscode/*.json: configuration for vscode
  • .hub/dockerconfig.json: a docker auth file. It will be used by Skaffold as docker-cfg secret to push image into docker registry.

  1. Install required plugins

For Python development we will use Visual Studio Code with additional plugins such as Google CloudCode. Alternative setup for IntelliJ IDE can be found here

$ code \
--install-extension "googlecloudtools.cloudcode" \
--install-extension "heptio.jsonnet" \
--install-extension "ms-azuretools.vscode-docker" \
--install-extension "ms-kubernetes-tools.vscode-kubernetes-tools" \
--install-extension "ms-python.python" \
--install-extension "redhat.vscode-yaml" \
--install-extension "xrc-inc.jsonnet-formatter"
  1. Open VS Code from the application directory
$ code -n .

Running VS Code and Skaffold

With the local development environment configured, you’re now ready to launch your application. Typically, you’d have to perform several tedious and error-prone tasks to build Docker containers and create Kubernetes configuration files. Fortunately, Skaffold is the tool that automatically generates required Kubernetes manifests. Skaffold also watches for code changes, and once a change is detected, Skaffold automatically initiates the steps to build, push and deploy the new code to a Kubernetes cluster.

In the previous section you have executed make -C ".hub" generate. This command has generated VS Code configuration files that you can view in /.vscode directory. You have also installed several VS Code extensions. One of the extensions is the Cloud Code, which is an extension that allows to integrate Skaffold with VS Code.

Select the target namespace

On the left side you should be able to see a tab with `<>` icon (Cloud Code). Please select it. Then you should be able to see a kubernetes cluster. Select a target namespace (right click on the desired namespace) where application should be deployed (for example `default` namespace). Skaffold will deploy the application in the selected target namespace.

Active Namespace

In this step you have validated connectivity between VS Code and a Kubernetes cluster. Spend a few minutes exploring the cluster. Next, switch to the source code view: open "Explorer" (Shift + Cmd + E).

Walk through the application source code


Let's review the example Python Flask application. All application specific source code is available in /src directory

/src # Flask application goes here
/static # CSS, JavaScript and other media goes here
/templates # Home for HTML templates, managed by Flask
/conf # Flask environment configuration goes here
requirements.txt # Our dependencies (managed by pip) # Application routes
/k8s # Generated by the jsonnet kubernetes manifests, managed by Skaffold
/test # Container structure tests (see:
Dockerfile # A docker file, managed by Skafold
skaffold.yaml # Skaffold configuration file

Python Flask Application is a simple web application. It imports that provides a router for:

  • / - renders the index.html single page template
  • /gimme/<int:howmany> - returns a JSON array with random words (size:(size: howmany variable)
  • /status - for Kubernetes health checks

By default, the application comes with remote debug enabled.

if application.config["PTVSD_PORT"]:
import ptvsd
ptvsd.enable_attach(address=('', application.config["PTVSD_PORT"]))

For remote debug, the application is integrated with ptvsd, a Python debugger package. It supports integration with vscode (see here) and with Skaffold (see here).


The Dockerfile is based on Python 3.7 and exposes two ports:

  • 80 for a flask applicaiton (standart HTTP port)
  • 3000 for ptvsd for remote debug.

Note about PRODUCTION use: It is recommended to disable remote debugging for production environments. To disable the debugger, you need to make changes in the following two files: src/ and Dockerfile. For production deployments, it is also recommended to define gunicorn (or any other WSGI server of your choice) as entrypoint instead of python + flask.

# CMD ["python3", "-m", "flask", "run", "--no-debugger", "--no-reload"]
ENTRYPOINT ["gunicorn", "-b", "", "app"]

Deploy to Kubernetes

  1. You will deploy the application to Kubernetes with Skaffold. To run a number of predefined tasks in VS Code, press CMD + Shift + P and select Tasks: Run Task. Here you should be able to see a number of predefined tasks. You can customize tasks in .vscode/tasks.json

  1. Select Skaffold More about Skaffold stages can be found here
Active Namespace

  1. Select dev to start the build and deploy process. Note: CMD + Shift + B is a VS Code shortcut for a build task, you can use it instead of steps 1 & 2.
Active Namespace

Alternatively you can run the same command in the terminal

$ source .env   # skip if you have already done this
$ skaffold dev

You should be able to see the following output:

> Executing task: skaffold dev <

Listing files to watch...
- rubik
Generating tags...
- rubik ->
Checking cache...
- rubik: Found. Tagging
Tags used in deployment:
- rubik ->
Starting deploy...
- deployment.apps/rubik configured
- ingress.extensions/rubik configured
- secret/rubik-dockerconfig configured
- service/rubik configured
Watching for changes...
[rubik-76f8bf44f8-tjdrx applicaiton] * Serving Flask app ""
[rubik-76f8bf44f8-tjdrx applicaiton] * Environment: docker
[rubik-76f8bf44f8-tjdrx applicaiton] * Debug mode: on
[rubik-76f8bf44f8-tjdrx applicaiton] * Running on (Press CTRL+C to quit)

Let's review the output of skaffold dev command:

  1. Skaffold has created a Docker image for the application. By default, timestamp is used as Docker image tag (image tag configuration can be customized via skaffold.yaml)
  2. The Docker image was pushed to Harbor docker registry. To build Docker images on the server, it is using Kaniko. If you prefer to build and push Docker images to your local workstation, then you can switch to the local profile
  1. Kubernetes manifests from /k8s directory were applied to the selected cluster (Skaffold also maintains a deployment image)
  2. Skaffold continues to run in the background, waiting for code changes, and ready to automatically rebuild and redeploy the application when local files are changed.

Access the deployed application

There are several ways to access and test the deployed application from the browser.

Option 1: Via vscode Open tasks shortcut (CMD+Shift+P or F1), type Tasks: Run Task and then select: Ingress: Open in browser

Note: On some platforms, the "Ingress: Open in browser" task doesn't seem to be available. If that is your experience, try an approach below.

Option 2 : Via ingress document - Open the file: k8s/ingress.yaml and take note of the host value.

And take note of the spec:
- host:

Based on the output shown above, you can access the deployed application using the following URL: Use get ingress to find your application URL.

Congratulations, you have successfully deployed Python Flask application on Kubernetes!

Realtime Updates and Debugging

When skaffold dev command is running, you should see following log in the terminal

Watching for changes...
[rubik-7d9655fcdc-dv8fv applicaiton] * Serving Flask app ""
[rubik-7d9655fcdc-dv8fv applicaiton] * Environment: docker
[rubik-7d9655fcdc-dv8fv applicaiton] * Debug mode: on
[rubik-7d9655fcdc-dv8fv applicaiton] * Running on (Press CTRL+C to quit)

Skaffold is watching for changes in the application source code. Once any application files are changed, skaffold will sync up changes to the running container (additional details here). This operation doesn't require a new image build. You can configure files and directories that skaffold is monitoring for changes. Open skaffold.yaml and you should be able to see the following:

- image: rubik
- dest: /app
src: src/**/*.py
strip: src/
- dest: /app
src: src/static/**
strip: src/

Next, you can open file src/ and change the following code

'helm', 'kustomize', 'kubernetes', 'aws', 'gcp', 'azure',
'terraform', 'docker', 'shell', 'vault', 'istio'

Let's add some more words into array. For example: "skaffold", "flask", "ptvs", "vscode" so the changed code will look like

'helm', 'kustomize', 'kubernetes', 'aws', 'gcp', 'azure',
'terraform', 'docker', 'shell', 'vault', 'istio',
"skaffold", "flask", "ptvs", "vscode"

Save the file (CMD+S)

You will see the following Skaffold logs updates in the Tasks output window:

Syncing 1 files for
Watching for changes...
[rubik-7fdd6687f5-rgt75 applicaiton] * Detected change in '/app/', reloading
[rubik-7fdd6687f5-rgt75 applicaiton] * Restarting with stat
[rubik-7fdd6687f5-rgt75 applicaiton] * Debugger is active!
[rubik-7fdd6687f5-rgt75 applicaiton] * Debugger PIN: 132-294-265

Once you see the message Detected change in '/app/', reloading, open the internet browser using the following commands: (CMD+Shift+P >> Run Task >> Open in browser). Move your mouse over the Rubik's kube displayed in the browser. Mouse events are routed to /gimme endpoint. These events will update various messages displayed in the browser.

Rubik's cube

Remote debugging

Now let's do some debugging. This is a little tricky, because our Python application has been wrapped into a docker container, which is running inside kubernetes. So, we will use the following tool chain:

  • Visual Studio Code - IDE
  • ptvsd - Python debugger for VS Code
  • Docker should expose port 3000 (can be reconfigured to a different port)
  • Task in launch.json in vscode should point to the same port (3000)
  • Flask application should be running with --no-reload, in our case controlled via environment variable FLASK_RUN_RELOAD=0

Notice that it's not possible to have both automated code sync and the remote debug. Good news, you don't need to redeploy the application to change these settings. Skaffold will do it for you. To enable remote debugging, perform the following steps:

  1. Modify the Dockerfile and save it:
  1. Skaffold will rebuild the container automatically. You can see the progress in the terminal window of Task - Skaffold
    Once the application has been rebuilt and the pod is restarted, you will see "Running on" message again:
Starting deploy...
- deployment.apps/rubik configured
Watching for changes...

* Serving Flask app ""
* Environment: docker
* Debug mode: on
* Running on (Press CTRL+C to quit)
  1. To bring up the Debug view, select the Debug icon in the Activity Bar on the side of VS Code. You can also use the keyboard shortcut ⇧⌘D. Select Attach to running container from the dropdown of run configurations:
    Run debugger

Important note: make sure you are running flask in with 'no-reload' option. In addition to the Dockerfile, auto-reload control flag FLASK_RUN_RELOAD can be declared in k8s/deployment.yaml as a pod env.

You can now place a breakpoint in src/ Breakpoints can be toggled by clicking on the editor margin or using F9 on the current line. For example, try setting a debug breakpoint in get_words() function.

In addition to a breakpoint, you can also setup a logpoint. VS Code Logpoint is represented by a "diamond" shaped icon. Log messages are plain text, but can include expressions to be evaluated within curly braces ('{}').

  1. Open your application in a web browser (CMD+Shift+P >> Run Task >> Open in browser) and trigger an event by moving your mouse pointer over the cube.
Rubik's cube


Congratulations! You have created a Flask application, wrapped it in Docker container, pushed it to a private Docker registry. Then you scheduled a Kubernetes deployment and enabled live reloads with Skaffold and remote debugging with ptvsd for your applications running on Kubernetes. You have learned how to use Hub CLI, VS Code, and Skaffold to configure, deploy, and debug an application on Kubernetes. While it may so und complex, this is the reality of current state of Kubernetes deployments. Unless you automate the entire workflow, developers have to manually perform configuration instead of writing application code. By automating the local development workflow, you can shorten the feedback loop, reduce misconfiguration errors, and reduce the effort to maintain local environments. Even if you happen to break your configuration, you can use automation to regenerate it.

In the next tutorial (D201), you can learn about how to use SuperHub to connect a component, such as a database, with your application.


Q1. Can I change the application name and URL?

Application name and URL are auto-generated. To specify a different application name, set the environment variable HUB_APP_NAME in file .env. After changing HUB_APP_NAME you need run make:

make -C ".hub" clean generate

Q2. Can I use this workflow on Windows?

Currently this developer workflow works only on MacOS and Linux workstations. We are currently working on the Windows version (email for early access). You can generate application deployment files on MacOS, save changes in Git, and share the project with your team. VS Code and Hub CLI work both on MacOS and Windows.

How did we do?

Infrastructure Workflow - Creating Stack Templates

Machine Learning Workflow - Creating an ML Pipeline