Creating Components from CloudFormation Templates: SageMaker Pipeline
Updated
by Igor Mameshin
A custom component is a component that is created and maintained by you, the user. These components can be integrated into any Stack Template in the AgileStacks SuperHub. You can manage, configure, and implement change control for multiple CloudFormation Templates using the SuperHub. In this tutorial, you will create a SageMaker Pipeline component from Amazon SageMaker-Build CloudFormation template:
Automated and continuous deployment of Amazon SageMaker models
Amazon SageMaker is a complete machine learning (ML) workflow service for developing, training, and deploying models. Automating the build and deployment of ML models is an important step in creating production machine learning services. The goal of aws-sagemaker-build is to provide a repository of common and useful pipelines for Amazon SageMaker.
SageMaker Pipeline component uses a CloudFormation template to create SageMaker automation resources in your AWS account. These include an Amazon SageMaker notebook instance, Amazon SageMaker Endpoint, AWS CodeBuild repository, AWS S3 bucket. The cloud resources are billed by the hour, and you can undeploy these resources using instructions at the end of this tutorial. The SageMaker workflow is automated using AWS StepFunctions, AWS Lambda, AWS SNS and other services. Step Functions use Lambdas to do the work of the build. There are functions for starting training, checking on training status, starting CodeBuild, checking on CodeBuild, and so on.
In this tutorial you will deploy SageMaker automation as part of the stack. Creating SageMaker Pipeline as a component on AgileStacks Platform allows you to customize it, make it part of a stack deployment unit, and implement change control.
Component registration on the Control Plane
To create a custom component through the Control Plane follow the steps below:

- Open
Templates > Components > Create
- Enter component
Name
,Title
and choose aCategory
the component belongs to. Users are allowed to create their categories - If this is your first component, a new Git repository must be created for a source code of the component. To avoid issues with permissions, currently we only support Agile Stacks hosted Git Service (repositories from Github or other external sources can't be added). Click
+ Create new
. Git repository name is derived from component's name and can't be changed. NOTE: When a new Git repository is created, Agile Stacks generates and pushes Custom Component skeleton (example) to the repository. It's a simple Kubernetes service with UI, that contains a few pods and an ingress. - If you created Custom Components previously, you can choose one of the existing Git repositories (a single repository can contain multiple custom components). In this case you must specify a directory name in
GIT sub path
field within the existing Git repository that will contain source code of the component. - Provide URL for png or svg file containing component
Logo
:
https://raw.githubusercontent.com/agilestacks/stack-components/master/sagemaker/icon.png - Provide custom component parameters for UI based deployments. Your users will be able to provide these parameters for each stack at deployment time. Component parameters are defined in hub-component.yaml. You can define component parameters in infrastructure-as-code configuration. In addition, a subset of component parameters can be also exposed to the end users via UI component parameters. For SageMaker, you will expose the following two parameters via UI configuration. For each parameter you will setup parameter name, UI label, and (optionally) default value.
Parameter Name
Brief
Value
ConfigFramework
Framework for SageMaker: AMAZON, BYOD, MXNET, TENSORFLOW
BYOD
NoteBookInstanceType
The SageMaker Notebook Instance type: "ml.t2.medium", "ml.m4.xlarge", "ml.p2.xlarge"
ml.t2.medium
- Configure who can access the component in
Permissions
section. Members of the groups who haveAdmin
permissions can modify component meta information, modify access rights to the component, see and choose the component on the Happy Meal, and deploy it. The groups withWrite
permissions can see and choose the component on the Happy Meal, and deploy it. The groups withRead
permissions can see the component on the Happy Meal, however can't deploy it. - Optionally, you can provide a short component description in the
Brief
field and a long description in theDescription
field. In addition,Version
,Maturity
,License
andTags
can be entered on this page. - Click
Save
button.
By default, Custom Component repository contains a source code of an example service. You will need to replace the example with the source code of your own component. See the next section section for more details about editing the source code of custom components.
Create and deploy CloudFormation Component
To deploy an example CloudFormation component for AWS SageMaker:
- Clone the custom component repository.
- Change to the directory where you cloned the component and delete all existing example files.
- The next step is to create a stack component from the CloudFormation template:
https://s3.amazonaws.com/aws-machine-learning-blog/artifacts/sagebuild/v1/template.json - Copy this command to Makefile and parametrize environment specific properties, such as
${component.custom-sagemaker.name}
- For a complete example of component deployment files that are using operators, check out SageMaker Example repository. We provided this repository so you can cut and paste component deployment files instead of writing these files.
- Copy the following files from SageMaker Example to the repository of your custom component:
- hub-component.yaml Automation Hub deployment manifest, that describes a Custom Component. More details are available in the commented lines of the example file or in documentation
- Makefile: Custom component implementation as CloudFormation template. AWS CLI is used to execute the CloudFormation template. Environment parameters can be passed to the CloudFormation template as $(values). See commented lines of the Makefile for more information.
- values.yaml.template SageMaker configuration template. The default template engine is curly for ${}, the others are mustache and commentary, respectively. For more information see templating documentation
- Commit & push the example files to the git repository.
- Create a new overlay stack template, that contains the custom component. For more information, please see
Create an overlay template with Custom Component
section below.
Create and deploy an overlay template with Custom Component
In the previous step we have created a Custom Component. Now the component can be selected from the Catalog and added to a Stack Template:
- Open
Stack Templates > Create
- Enter template name for example
SageMaker
, choose Stack typeOverlay
- The Custom Component created during the previous step should be visible and available for selection. Select the component. NOTE: By default, components with
alpha
maturity are not shown. To seealpha
components in the Templates Catalog, adjust filter settings. You can use search filter to quickly find a component by typing its name. - Click
Save for later
- Click
Deploy
- Select
Standalone
and choose the environment where you are going to deploy the template, for exampleTEST01
- standalone means that the stack does not require any platform stack (such as Kubernetes) to run. - Type overlay stack name, for example
sagemaker
. - Click
Deploy
- View the log of CloudFormation template, and wait until the stack is deployed. If SageMaker automation was deployed successfully, you will see the following output in the stack deployment log:
Stack sagemaker.demo52.superhub.io deployed
- Navigate to Stacks > List > stack name > Details
- Open SageMaker hosted Jupyter notebook by clicking on the
sagemaker
button highlighted above. You may need to enter your AWS console user name and password. You should be able to see the following Jupyter workspace with several pre-loaded notebook examples. - Open
/SageBuild/tensorflow/tensorflow_distributed_mnist.ipynb
notebook to run SageMaker tutorial that demonstrates how to create a convolutional neural network model to train the MNIST dataset using TensorFlow distributed training. Now you are ready to execute notebook steps used for training, evaluation and inference. Step 11 will take several minutes to complete, and you will have a model deployed using SageMaker endpoint. You can view the status of SageMaker training jobs and Model endpoints by going to AWS console. - Finally, you can test the model by invoking the endpoint in step [12]
- Congratulations, you have deployed the SageMaker Pipeline!
Cleanup the Cloud Resources
In the previous step we have deployed several resources in AWS, including Amazon SageMaker notebook instance, Amazon SageMaker Endpoint, and AWS CodeBuild repository. Upon completion of this tutorial you may want to delete these resources to save cloud costs.
- Navigate to Stacks > List > stack name > Details
- Click
Undeploy
to automatically undeploy all cloud resources created by the CloudFormation template - Click
Delete
to delete the stack instance