Stack Design

Components

1. Component's hub-component.yaml must not use fromEnv: to bind kind: user parameters to the environment variables. The parameters are bound to the environment (if required) on Stack level - either in hub.yaml or params*.yaml.
2. Explicitly export required parameters via env: and set defaults via value:.
3. kind:user is for high-level input from user for which no sensible decision can be made automatically - like domain of the stack, number of worker nodes.
4. default: is for kind:user - if no value is specified, then - if stdin is a TTY - user is asked by Hub CLI to accept the default or provide value. If not a TTY, then the default is used.
5. Set defaults in Makefile, e.g. NAME ?= dev, so that the component is usable out of CLI lifecycle (modulo templates), and also self-documented.
6. Do not rely on global environment set prior to stack deployment, e.g. TF_VAR_name (which should go away anyway).
7. Component parameters defined in hub-component.yaml (implicit kind: tech) are scoped to the current component only - they are not visible to subsequent components (anymore). Use outputs: to provide values explicitly, for example:

outputs:

          - name: component.postgresql.namespace

          - name: component.postgresql.instance

            value: ${component.postgresql.name}-rds

          

8. To bind hub.yaml level parameter to a value which is not known in advance (component output) use kind: link:

outputs:

          - name: component.reactor.api

            value: ${component.automation-hub.api}

            kind: link

          

Name

1. <component> is components/<component> of components Git repo.
2. <component-name> is the name of the component in the stack (pg1) - same as ${hub.componentName}:

components:

          - name: pg1

            source:

              dir: components/postgresql-rds

          

3. Finally, component.*.name - if required for particular component implementation, is something user may want to set, and may become TF_VAR_name, Kubernetes ingress name, etc. on the component level. We default it to ${hub.componentName} in hub-component.yaml.

The state is stored as s3://mybucket/<domain>/<component>/<component-name>/terraform.tfstate, see Terraform state in S3.

Stack

1. Bind Instance specific parameters to environment variables via fromEnv: so that Template could be deployed multiple times, and parameterized via environment.
2. Avoid fromEnv: TF_VAR_*.
3. Avoid setting env: - the parameters must be exported by each component in component specific way. Ie. every component may have different idea what TF_VAR_name is for.

Envrc

1. Avoid setting TF_VAR_* variable to influence component directly bypassing hub.yaml / params*.yaml wiring. Hub CLI will filter these out of children process environment by default (--os-environment).
2. Set .envrc vars for hub.yaml / params*.yaml parameters captured via fromEnv.

Parameters ambiguity

Outputs

When more than a single instance of the same component is deployed in the stack - think two PostgreSQL databases, then there is an ambiguity because both components provides same outputs, ie. endpoint and password.

To bind input parameters of the pgweb to specific PostgreSQL, use depends:

components:

            - name: pg1:

              source: components/postgresql

            - name: pg2:

              source: components/postgresql

            - name: pgweb1:

              source: components/pgweb

              depends:

              - pg1

          

The outputs of depends are searched in turn, then the global outputs pool.

Inputs

To set parameters for a specific component instance use component:

parameters:

            - name: component.postgresql

              component: pg1

              parameters:

          

The component <name> first looks for parameters that are qualified with component: <name>, then global parameters space.

Parameters evolution

Elaborate

During elaborate Hub CLI creates a final manifest assembly that consists of:

• a leading document constructed from hub.yaml, params*.yaml, fromStack YAML-s, adding parameters marked kind: user in */hub-component.yaml;
• all component's hub-component.yaml with the component name (meta.name) changed to match hub.yaml specified name.

Let's take AgileStack's infra Dev stack as an example. The stack is built from:

• agilestacks/hub.yaml - core infra;
• agilestacks/params.yaml - parameters for the infra;
• dev/hub.yaml - uses fromStack: agilestacks and adds some components: prometheus, efk, pgweb;
• dev/params.yaml - parameters for additional components;
• dev/params-dev.yaml - Dev stack parameters (Reactor's Cloud Account base DNS domain selection).

Separation between hub.yaml and params*.yaml is a convinience.

The files will be parsed and parameters will be arranged by hub elaborate in following order:

1. hub-component.yaml of all agilestacks components, in order of components:
2. hub-component.yaml of all dev components
3. agilestacks/hub.yaml
4. agilestacks/params.yaml
5. dev/hub.yaml
6. dev/params.yaml
7. dev/params-dev.yaml

hub-component.yaml kind: user parameters (1) are brought to the top level - first document of hub.yaml.elaborate with a component qualifier. If there is a fromEnv and kind is not user then a warning is emited.

Parameters from (1), (2) to (7) are merged in the order specified:

1. kind: user takes priority over kind: tech.
2. value, default, fromEnv, env, and brief are overwritten.
3. In case top-level parameter - ie. from hub.yaml or params*.yaml has no component qualifier, then elaborate additionally checks for all previosly encountered parameters of the same name with a qualifier, then do merge into those. This the reason for the following in hub.yaml.elaborate:

- name: component.auth-service.authApiSecret

            kind: user

            fromEnv: AUTH_API_SECRET

          - name: component.auth-service.authApiSecret

            component: auth-service

            kind: user

            fromEnv: AUTH_API_SECRET

          

Deploy

During deploy following things happens:

1. Top-level parameters are locked:

• kind: user parameters with no values are fetched from fromEnv (if any) or asked on the terminal (if stdin is a TTY);
• default values are substituted;
• expression evaluations are performed;
• cycles become errors;
• empty kind: user parameters become errors until supressed with empty: allow;
• all errors and warning are collected and reported.

2. If there is a state file, then parameters from state are loaded and merged into global state - but only empty (stack) values are replaced (by state values) and warnings are emited.
3. The locked set of top-level parameters are never changed after this step.

Top-level parameters can reference each other via expressions.

On each component deploy:

1. Parameters are expanded:

• parameters with no values are searched in the top-level scope taking in account component and depends qualifiers;
• expression evaluations are performed;
• unknown empty parameters become errors until supressed with empty: allow.

2. Component parameters are never sent to top-level scope, but they are written into state file for informational purposes.
3. env: variables are set, then merged on top of OS environ to create process environment for component invocation.
4. Outputs are collected and put onto global outputs scope - with a qualifier - of the component producing the output. So that when parameters qualifiers are proceed, the name|qualifier is searched first and then just plain name. Outputs and parameters are processed independently although they looks very similar to the user. Outputs have higher precedence.

Component-level parameters cannot reference each other in expressions, only top-level parameters.

Set parameter value: " " or empty: allow to make empty parameter on hub-component.yaml level not an error. During template processing and OS environment setup spaces are trimmed, effectively creating an empty substitution.

Capturing empty parameter from top-level scope does not require empty: allow on component level. But it might be useful to proceed with deployment when optional upstream component failed thus no outputs were produced.

Conditional deployment

When component declares requires - a list of capabilities that must be provided by prior components, platform stack, or environment - it became a list of hard requirements that must be satisfied before component (deploy) implementation is invoked. Failure to satisfy any requirement is hard error.

Component provides capabilities by declaring them in component's manifest provides list. Additionally, dynamic capabilities are collected from deploy output Provides: block, similar to Outputs:.

An aggregation of component's provides became (platform) stack provides.

A list of well known requires are built into Hub CLI. Such requirements could be sourced from environment, such as aws (a working AWS CLI). These requirements cannot be tuned.

Optional and Mandatory

A component might be declared optional on stack level

lifecycle:

            optional:

            - s3-bucket

          

Failure to deploy optional component is soft error. By default all components are mandatory. Those declared optional became optional. If mandatory components are specified, then everything else is optional.

lifecycle:

            mandatory:

            - kubernetes

            - tiller

            - traefik

          

Optional requirements

A requirement could be softened by tuning it via lifecycle in stack manifest:

lifecycle:

            requires:

              optional:

              - vault

              - cdn:control-plane

              - component.ingress.ssl.enabled:acm

          

          

vault and cdn are capabilities, control-plane and acm are component names. vault requirement is relaxed for all components, while cdn just for control-plane. Requirement became optional. acm is deployed only if ingress parameter component.ingress.ssl.enabled requests TLS-enabled stack (evaluates to truth-y value, ie. everything else besides false, "" empty string, 0, (unknown)).

When an optional requirement cannot be satisfied for a particular component, the component deployment (or undeployment) implementation is not invoked.

Requirements tuning of Base stack will be merged with tuning of derived stack. The following rules are used:

• requirement supersedes all prior requirement:component - making requirement optional for everyone;
• requirement:component supersedes prior requirement - making requirement optional for component. There could be multiple declaration for different components;
• : erase all prior tuning;
• requirement: erase all prior tuning for requirement;
• :component erase all prior tuning for component;
• otherwise the entry is appended.

Provides

Hub CLI parameter hub.provides and HUB_PROVIDES env var exports a space-separated list of currently known provides.