Pipelines

What are pipelines?

Pipelines are a series of automated steps that run in a specific order to build, test, and deploy your application. Pipelines can define the orchestration required to deploy a release to a specified environment. Additionally, pipelines can be used to automate tasks that do not require a release, such as maintenance or security scans.

Pipelines are created from pipeline templates, which blueprint the stages required to run the pipeline. Pipeline templates are defined using HCL and use semantic versioning. Learn more about pipeline templates.

Stages of a pipeline

A pipeline consists of stages. A stage may contain nested pipelines, tasks, or both. Stages are executed sequentially and the items within a stage are executed in parallel.

A task is a unit of work within a stage. Tasks can define an optional set of actions that use the plugin framework to execute custom logic (see Plugins). A task is executed automatically by default when it's ready; however, a task can also be manual, in which case it would need to be manually started. Learn more about tasks in Pipeline Template Syntax.

Have a question?

Check the FAQ to see if there's already an answer.

Viewing pipelines

To view the pipelines contained within a project, select Pipelines from the project sidebar. The Pipelines table displays all the pipelines in the project in order of most recently created. The table displays the following information:

• Status: current status of the pipeline
• ID: unique identifier for the pipeline
• Type: type of pipeline - DEPLOYMENT, RUNBOOK, NESTED, or RELEASE_LIFECYCLE
• Annotations: optional information about the pipeline in key-value format. In the example below, we see the annotation mr: 1. mr is the key and 1 is the value, which links to a merge request.
• Release: if the pipeline is associated with a release, the release version is displayed
• Trigger: the trigger (user or service account) that created the pipeline and when it was created
• Stages: a status icon is displayed for each stage in the pipeline. Hover over the icon to view the stage name and its status.

To view the details of a pipeline, click on the pipeline status icon or the pipeline ID.

Pipeline details page

The pipeline details page displays the following information:

• Stages: the stages in the pipeline and the tasks contained within each stage. Click on a task to navigate to the task details page.
• Approvals: table displaying any approval rules, approvers, and the status of the approval.
• Input Variables: list of input variables (HCL and environment) that were assigned when the pipeline was created
• Template: the pipeline template that was used to create the pipeline
• Comments: a feed of all the pipeline comments. From here, members can start a conversation and add replies regarding the pipeline.

The example below is a pipeline of type RELEASE_LIFECYCLE, which contains four stages. The dev.pre stage contains a task, and the other stages, dev, pre-prod, and prod, contain nested pipelines. The prod stage contains two nested pipelines.

Pipeline types

Deployment

A DEPLOYMENT pipeline is used to deploy a product to a particular environment. It can be created when a release is created or via the Phobos CLI.

DEPLOYMENT pipelines are always associated with an environment. They can optionally be associated with a release.

Runbook

RUNBOOK pipelines can be used for specific tasks that do not require it to be associated with a particular environment or deployment. For example, it can be used to perform automated maintenance or security scans.

RUNBOOK pipelines are not associated with an environment or a release.

Nested

A NESTED pipeline is a pipeline created by a parent pipeline. It is used to execute a pipeline within another pipeline.

Release lifecycle

A RELEASE_LIFECYCLE pipeline orchestrates the propagation of a release through the various environments defined in the release lifecycle. A RELEASE_LIFECYCLE pipeline is generated when a release is created.

RELEASE_LIFECYCLE pipelines cannot be created manually. They are only created by a release.

More on pipelines

A RELEASE_LIFECYCLE pipeline can be considered as a parent to DEPLOYMENT pipelines and controls how each DEPLOYMENT pipeline is executed.

RELEASE_LIFECYCLE, DEPLOYMENT, and RUNBOOK types can also be considered NESTED pipelines if they are created by a parent pipeline. All three can also be parent pipelines and create pipelines. One restriction is that a DEPLOYMENT pipeline cannot create a DEPLOYMENT pipeline.

Creating a pipeline

Pipelines are created via the following methods:

via the UI

1. Go to the target project and select Pipelines from the project sidebar.

2. Select Create Pipeline.

   

3. On the Create Pipeline form, select a pipeline type: DEPLOYMENT or RUNBOOK.

4. If selecting DEPLOYMENT, you may select an environment from the environment dropdown menu. You may also enter the name of an environment that does not exist yet; when the pipeline runs, the environment will be created within the project. Selecting an environment is optional. RUNBOOK pipelines are not associated with an environment so the environment dropdown will not be displayed.

5. Select a pipeline template. Use the Pipeline Template dropdown to select an available pipeline template. After a pipeline template is selected, select a version of the pipeline template from the Version dropdown. By default, the latest version of the template will be selected, but you can select a different version.

   

6. Set Variables. You can create values to pass data directly to any variables that have been defined in your selected pipeline template. You can add variables individually into your pipeline via the Pipeline Variables table or you can use a project variable set from the Variable Set Revision dropdown menu.

   info

   Adding variables is optional and dependent on the specifications of your pipeline template.

   • Pipeline Variables: select Add Variable to open the variables dialog to enter a new variable.

   

   Adding variables will generate a list of variables on the pipeline form. You can continue to add (Add Variable), edit, or delete (edit and delete icons) your list of variables before creating the pipeline.

   

   • Variable Set Revision: select a project variable set from Variable Set Revision dropdown menu. Select View Revision to open a new window that will show the project variables. See more on project variable sets.

   

7. When the form is complete, click Create Pipeline. You will be navigated to the Pipelines page where you can view its status and go its details page for more information on the pipeline.

   

via a release

When a release is created, a RELEASE_LIFECYCLE pipeline is generated for the release. A DEPLOYMENT pipeline is generated for each deployment defined in the release lifecycle.

In the example below, the following 5 pipelines were generated when the release was created:

• 1 RELEASE_LIFECYCLE pipeline
• 4 DEPLOYMENT pipelines for each release deployment

via the Phobos CLI

Pipelines of type DEPLOYMENT and RUNBOOK can be created via the Phobos CLI using the pipeline create command. You just need a pipeline template and to specify the organization name and project name.

info

The commands below are examples of how to create a pipeline using the Phobos CLI. When using the Phobos CLI, you can run phobos pipeline create -h to view more options, including details on passing in values for variables and submitting annotations. See the Phobos CLI documentation for more information on how to install and use the CLI.

Creating a DEPLOYMENT pipeline

The command below assumes you are in a directory that contains a pipeline template file named pipeline.hcl.

phobos pipeline create --org-name "<organization-name>" --type "deployment" --environment-name "<environment-name>" <project-name>

The command below passes in the directory path to the template file.

phobos pipeline create --org-name "<organization-name>" --type "deployment" --environment-name "<environment-name>" --directory "/path/to/directory" <project-name>

The command below passes in the pipeline template ID instead of using a file from a directory.

phobos pipeline create --org-name "<organization-name>" --type "deployment" --environment-name "<environment-name>" --template-id "<pipeline-template-id>" project-c

Creating a RUNBOOK pipeline

Using the commands above, pass in the --type "runbook" flag to create a RUNBOOK pipeline. Exclude the --environment-name flag.

phobos pipeline create --org-name "<organization-name>" --type "runbook" <project-name>

info

A pipeline of type DEPLOYMENT requires an --environment-name flag; a pipeline of type RUNBOOK should not include this flag.

Managing stages of a pipeline

On the pipeline details page, you can view Stages of the pipeline. A stage consists of either nested pipelines, tasks, or both. The launch icon indicates a nested pipeline, and the file icon indicates a task. Click on a nested pipeline or task to go to the details page for that item.

In the example below, the pipeline has four stages. The first stage contains one task, and the other stages contain nested pipelines. The last stage contains two nested pipelines.

Nested pipelines

Phobos will display the available action for a nested pipeline based on its status.

Starting a nested pipeline

The Run Pipeline button is available for this nested pipeline because it is within the first stage in its parent pipeline, which is a RELEASE_LIFECYCLE pipeline.

Note, the nested pipeline has a status of Not Started. The sidebar on the right displays information for the parent pipeline, which has a status of In Progress.

1. Select Run Pipeline.
2. In the Run Pipeline dialog, click Run.

Retrying a nested pipeline

The Retry Pipeline button is available for this nested pipeline because it has a status of Succeeded. It could also be retried if the status was Failed or Canceled.

1. Select Retry Pipeline.
2. In the Retry Pipeline dialog, click Retry.

Schedule a nested pipeline

Nested pipelines can be scheduled to run at a specific date and time.

1. Click on the upside-down caret next to the Run Pipeline button and select Schedule Nested Pipeline.
2. In the Create Schedule dialog, select the date and time you want the pipeline to run and click OK.
3. Select Set Schedule to schedule the pipeline. Select Cancel to close the dialog without scheduling the pipeline.

Update a scheduled nested pipeline

1. Select Update Schedule.
2. In the Update Schedule dialog, update the date and time you want the pipeline to run and click Update Schedule. Select Cancel to close the dialog without updating the schedule.
3. To remove the schedule, select the Cancel Schedule checkbox and click Update Schedule.

Deferring a nested pipeline

1. Click on the upside-down caret next to the Run Pipeline or Update Schedule button and select Defer Nested Pipeline.
2. In the confirmation dialog, provide a reason for deferring the pipeline and click Defer. Click Cancel to close the dialog without deferring the pipeline.

more on deferring a nested pipeline

Deferring a nested pipeline allows you to skip it while letting the parent pipeline proceed to the next stage. Deferred nested pipelines are marked as deferred in the pipeline activity feed and can be restored at any time. A nested pipeline can be deferred if it is in the Ready or Waiting status, meaning you can defer a nested pipeline before it starts. Any dependents of the deferred nested pipeline will be skipped.

Restoring a deferred nested pipeline

The Restore Pipeline button is available for this nested pipeline because it has a status of Deferred.

1. Select Restore Pipeline.
2. In the confirmation dialog, click Restore to restore the pipeline. Click No to close the dialog without restoring the pipeline.

info

Restoring a deferred nested pipeline reverts it to its original state, making it executable once its dependencies are met. Any dependents of the nested pipeline will be restored as well.

Canceling a nested pipeline

The Cancel Pipeline button is available for this nested pipeline because it has a status of In Progress.

1. Select Cancel Pipeline.
2. In the Cancel Pipeline dialog, click Cancel. Click No to close the dialog without canceling the pipeline.

Tasks

Starting a task

The Run Task button is available for this task as it is the next task in the stage.

1. Select Run Task.
2. In the Run Task dialog, click Run.

Retrying a task

The Retry Task button is available for this task because it has a status of Succeeded. It could also be retried if the status was Failed or Canceled.

1. Select Retry Task.
2. In the Retry Task dialog, click Retry.

Scheduling a task

Like nested pipelines, tasks can be scheduled to run at a specific date and time.

1. Click on the upside-down triangle next to the Run Task button and select Schedule Task.
2. In the Create Schedule dialog, select the date and time you want the task to run and click OK.
3. Select Set Schedule to schedule the task. Select Cancel to close the dialog without scheduling the task.

Updating a scheduled task

1. Select Update Schedule.
2. In the Update Schedule dialog, update the date and time you want the task to run and click Update Schedule. Select Cancel to close the dialog without updating the schedule.
3. To remove the schedule, select the Cancel Schedule checkbox and click Update Schedule.

Deferring a task

1. Click on the upside-down triangle next to the Run Task or Update Schedule button and select Defer Task.
2. In the confirmation dialog, provide a reason for deferring the task and click Defer. Click Cancel to close the dialog without deferring the task.

more on deferring a task

Deferring a task allows you to skip it while letting the parent pipeline proceed to the next stage. Deferred tasks are marked as deferred in the pipeline activity feed and can be restored at any time. A task can be deferred if it is in the Ready or Waiting status, meaning you can defer a task before it starts. Any dependents of the deferred task will be skipped.

Restoring a deferred task

The Restore Task button is available for this task because it has a status of Deferred.

1. Select Restore Task.
2. In the confirmation dialog, click Restore to restore the task. Click No to close the dialog without restoring the task.

info

Restoring a deferred task reverts it to its original state, making it executable once its dependencies are met. Any dependents of the task will be restored as well.

Canceling a task

The Cancel button is available for this task because it has a status of In Progress.

1. Select Cancel.
2. In the Cancel Job dialog, click Cancel. Click No to close the dialog without canceling the task.

Approving a task

The Approve button is displayed because the task requires approval before it executes, and also that you are designated to approve this task. The UI will display the required number of approvals and identify the designated approvers.

Frequently Asked Questions (FAQ)

What's the difference between a pipeline and a pipeline template?

A pipeline template is a blueprint for creating a pipeline. A pipeline is an instance of a pipeline template that has been created and is executable.

Can I create a pipeline without a pipeline template?

No, a pipeline must be created from a pipeline template.

Why don't I see the option to start/cancel/retry my task or pipeline?

The options available to you depend on the type and status of the pipeline. For example, you may not be able to start a task if the task is waiting for a task in a previous stage to complete. Additionally, a pipeline can only be retried if its status is SUCCEEDED, FAILED, or CANCELED.

If the pipeline is of type NESTED, navigate to the parent pipeline to see any available options to manage the pipeline or task.

Can I edit a pipeline?

Pipelines cannot be edited. If you need to make changes to a pipeline, you can create a new version of the pipeline template and create a new pipeline from the new version template.

Can I delete a pipeline?

If a pipeline was generated by a release and the release is deleted, the associated pipelines will also be deleted. For pipelines not created by a release, there currently isn't a single command to delete them. However, we plan to add this feature in the future.

Why don't I see the option to restore my deferred task or pipeline?

The option to restore a deferred task or pipeline is only available if the task or pipeline is in the DEFERRED status. If the task or pipeline is in another status, the option to restore will not be available. Additionally, tasks or pipelines within a deferred task or pipeline cannot be restored directly.