Stacks Under the Hood - Manifests

Michael Delzer Updated by Michael Delzer

Hub Manifest is a YAML file named either hub.yaml or hub-component.yaml describing Stack or Component, respectively, for the Hub to obtain dependencies, parameters, source blob locations, and available lifecycle actions (Verbs) from. hub.yaml.elaborate is a Hub assembled read-only Manifest which is self-sufficient, contains multiple YAML documents - fully describing the Stack or Template.

Directory Hierarchy

hub.yaml is placed into top-most directory of the Stack and hub-components.yaml into top-most directory of the Component. We favor Convention over Configuration with Stack resources organized in a well-defined directory structure.

Root of the Stack
hub.yaml
/config
...
/cloud
/terraform
output.tf
...
/os
/etcd
...
/apps
/postgresql
/config
output.json
...
/components
/nginx-frontend
hub-component.yaml
...
/nginx-api
/haproxy

If Stack is a composition of ad-hoc Components with no base inlined, then the only top-level directory is components/:

Root of the Stack
hub.yaml
/components
...

.. or just hub.yaml.

We allow for a single type of Component to participate multiple time in the Stack by adding component instance suffix to the Component name - nginx-* in the example above.

Prologue

There could be multiple YAML documents in a single YAML file delimited by literal --- (YAML syntax for details), so we use this feature to compose hub.yaml.elaborate Manifest by appending hub-component.yaml files to the hub.yaml. Please see Manifest composition below. Thus the first document in the hub.yaml.elaborate is special.

An example of a base Stack hub.yaml with additional and, probably, optional components:

---
version: 1
kind: stack
meta:
name: kubernetes-aws
brief: Kubernetes on AWS
description: ....
fromStack: "k8s:1.2"
source:
git:
remote: git@github.com:agilestacks/stack-kubernetes-aws.git

components:
- instance: frontend
name: nginx
source:
git:
remote: git@github.com:agilestacks/nginx.git
- name: jenkins-customization
optional: true
source:
dir: ./components/jenkins-customization

parameters:
... # Non-default Stack parameters
# or pre-sets for user-level mandatory parameters.

meta.fromStack indicated that the Stack extends another stack - pulling all its components and setup into the current stack. k8s is a well-known name. 1.2 is version, or latest, like Docker. An alternative to well known stack is to specify parent’s source location:

fromStack:
name: k8s-aws
source:
git:
remote: git@github.com:agilestacks/stack-k8s-aws.git

An example of Component hub-component.yaml:

---
version: 1
kind: component
name: kubernetes-monitoring
brief: Kubernetes Monitoring
description: ....
source:
s3: s3://dist.agilestacks.com/blobs/...

An example of a Stack hub.yaml composed from Components with no additional files:

---
version: 1
kind: stack
meta:
name: mail-server
brief: IMAP Mail Server
description: ....
components:
...
parameters:
...

Notice lack of meta.source section in the last example.

Dependencies

A Stack or Component may describe dependencies that must be satisfied - Requires, before (fully parameterized) Stack could be deployed. Some examples of pre-requisites:

  • Type of base Environment - what kind of Layers must be present (to provide operational capabilities and settings for input parameters), such as do we need Kubernetes or plain GCE will do;
  • What other types of components (Provides) must be already deployed in the environment or will be deployed with the Stack.

This is a high-level check which is possible to perform without matching and wiring Parameters to known Output Variables and Environment Facts.

For example:

requires:
- aws
- kubernetes
- "postgresql:9"

Provides

Stack and Components declares Provides - a type of service or capability that is added to the Stack or Environment by deploying the Stack / Component.

provides:
- efk
- logs
- monitoring
- cluster-storage

The Requires and Provides are arbitrary tags or key-value pairs (later in this document - Tags). It could be the name of a product installed or generic capability or function.

Components

Components are building blocks of the Stack. Components may be added, removed, or enable and disabled. There could be multiple instances of the same Component (type). Manifest enumerates all instances of the Components:

components:
- postgresql
- instance: frontend
name: nginx
source:
...
- instance: api
name: nginx

At hub.yaml.elaborate Manifest loading and verification time, Hub must update (reconcile) with components section to update the list of documents in the Manifest. All documents of Components not present must be removed. A single instance of a Component document must be appended with the (instance) name set to Component name.

Component Source Location

hub.yaml is a self-sufficient description of the Stack. The sources are fetched by Hub from internet location set in meta.source section into Workspace. We support the following sources:

  • A pointer to Git repo, optionally with a secret: ssh key or user/password (see Secrets);
  • Cloud storage blob on S3 or GCS for a source archive or an executable file;
  • Local filesystem path.

URL is templated with Parameters.

meta:
source:
s3: s3://dist.agilestacks.com/blobs/customer-inflights/${customer.id}/kubernetes/etcd-setup-b154.tar.bz2
---
meta:
source:
s3:
url: s3://${deployment.bucket.private}/scripts/install.sh
role: arn:iam-role-to-assume

We support flexible - heterogeneous specification as in the example above.

Parameters

For Parameters substitution in configuration files please see Templating. Requested Parameters form a tree-like structure where both nodes and leafs could have values:

parameters:
- name: cloud
brief: Cloud Parameters
kind: user
description: ...
default: aws # use default, if not inferred
value: gcp # set to value provided by user, or inferred from external facts,
# or obtained from output variable in deployment process

parameters:
- name: account
kind: inferred
- name: regions
kind: inferred
choice: [ us-east-1, us-east-2, ... ]
default: [] # use empty default to indicate requested datatype?
- name: component
parameters: [ { name: kubernetes.master.size, choice: [ r3.large, r4.large ], default: r4.large, value: r3.large } ]

- name: deployment
kind: inferred
default: standalone

The values are in most cases plain text or primitive type, yet we support arrays and maps - datatypes native to JSON and YAML. Native YAML syntax could be used to short-circuit nested declarations. (TODO update example)

default is a part of Manifest definition. value is something that exists inside the Hub as part of Template and Instance but could be seen/exported, see State below. Defaulted parameters may have the value set as part of Stack or Template definition.

In the future, we may add auto-detection of parameters. The Hub will scan configuration files and reconcile Manifest-declared tree with the source facts.

Types of Parameters

There are high-level User-provided parameters - the facts our User do care about, like which Cloud and Cloud Account to use, what region to deploy to.

There are also lower-level Technical parameters, which Hub must infer from other places, such as Environment and output variables of Stacks and Components that are deployed prior to the current Stack or Component config-s are templated. Ie. an ElasticSearch Component - deployed first, may provide search cluster connection URL to Kibana log viewer web-application (thus no URL computing on Kibana side, that would introduce high-coupling between Components).

Output Variables

After Components and, later, the whole Stack are deployed the Output Variables are captured from a well known locations in the Workspace. For now, Variables must be advertised in the Manifest - a contract for Component to obey, otherwise it would be impossible to perform parameters connectivity check ahead of actual deployment. In future we may support canary deployments to learn about outputs to reconcile Manifest declarations with actual state.

outputs:
- cloud.deployment.zones
- dns.domain
- dns.domain.apps

Output types are: text, structured - JSON and YAML compatible, or guessed based on content.

Stack Outputs are template strings with references to Component Outputs and Parameters.

Parameters Fact File

An alternative to templating Component configs is to place a fact file named hub-parameters.yaml (or .json) into Component’s root and let the Component figure it out.

We may have a notion of technology-agnostic fact file compatibility between Components deployed subsequently when a Terraform state file, for example, from Component A is transferred to Component B without changes. Then Component B extracts necessary data from that state. This way is prone to leaking abstractions and tight coupling, though.

Parameters via Environment

Another alternative is to supply parameters via OS environment variables. Parameters are set according to Component parameters spec and the environment is reset after Component verb impl is completed.

Lifecycle

Default Verbs are: deploy, destroy, status. Additional verbs could be: check, repair, backup, clone, scale, etc. Verbs are delegated to actions implemented in the Stack / Components source code. Supported actions are: Shell, Terraform, Make, Docker, Chef, Ansible.

lifecycle:
verbs: [deploy, destroy, status, check]
order: [postgresql, api, web-nginx-api, web-nginx-frontend]
readyCondition:
dns: ${dns.domain}
url: https://${dns.domain}/api/v1/
waitSeconds: 1200
ambassadors:
deploy: scripts/deploy.sh
destroy: scripts/destroy.sh

We guess the implementing action by checking for shell scripts, *.tf files, Makefile-s, etc. in the source. Component actions are applied in order defined (or in reverse for destroy). If Ambassador is specified, then it is invoked to perform lifecycle transition automation.

readyCondition specifies additional checks that must complete successfully within waitSeconds duration for component deployment to succeeed.

In future we may support scanning source code to find available Verbs to reconcile Manifest state. We’ll add concurrent deployment of Components based on Variables and Parameters dependency graph, for the cases where Ambassador is not defined.

Stack Status and State

While stack is deployed or is transitioning from state to state, there are two additional entities that are not a part of original Stack design yet crucial to maintain Stack within the Hub:

  1. Status: initial, deployed, destroying, repairing, etc. Status also include in-flight operations, like backing up and cloning (multiple ops at the same time).
  2. State - a native representation: Terraform state files, list of supplied and inferred parameters, output variables, etc. This is the data that is required for native tools (actions) to function.

For external representation, those entities are stored in hub-state.yaml with references to native state files on remote blob storage (it could be Hub internal HTTP server or proxy).

Hub CLI will have an option to pull and push the state and status of a single Stack from/to Hub to perform a limited set of lifecycle actions out-of-the Hub and sync the results back.

Manifest Composition

hub.yaml.elaborate Manifest is composed by concatenating hub.yaml and hub-component.yaml files. It is a task of the Hub or Hub CLI to reconcile with Prologue state.

Well-known Parameters Dictionary

To decrease verbosity of Manifests, brief, description, list of values and defaults of parameters could be off-loaded to hub-well-known-parameters.yaml file which is distributed together with the Hub:

parameters:
- name: cloud.provider
brief: Cloud Parameters
kind: user
description: ...
choice: [AWS, GCP, Azure]
- name: cloud.aws.region
brief: AWS Region
kind: user
description: ...
choice: [us-east-1, ...]
default: us-east-1
- name: ...

Parameter Bundles

Parameter Bundle is a set of parameters that often used together. Such a set could be referenced from Manifest by name.

bundles:
- name: aws
parameters:
- cloud.region
- cloud.account

How did we do?

Performance Monitoring and Alerting

Contact