search
Ctrlk
Ops IntelligenceAsset IntelligenceObservabilityRobotic Data

Pipelines

A pipeline is a set of processing elements (bots) connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in sequence or in a time-sliced fashion based on the user-designed logic.

Pipelines are a commonly used concept in everyday life. Car assembly line, CI/CD environment, Big data environments to name a few.

In cfxOIA RDA context, the pipeline mechanism is supported via RDA bots which were explained in the previous section. RDA bots are designed, implemented, and logically connected to solve real-world problems. This section covers few following example pipelines that are part of cfxOIA RDA.

  1. Simple example pipelines

  2. Nagios pipeline

  3. cmdb_file_load example pipeline

Note: The above example pipelines explained in this section are for reference and users are free to modify, extend and/or add new pipelines to solve other use-cases based off-of the above examples.

hashtag
Simple example pipeline

hashtag
Example 1

The below pipeline code snippet explains a very basic and simple pipeline that. In the above example, the pipeline consists of bots that are logically arranged or connected in series to perform the following tasks:

@dm:empty
    --> @dm:addrow d = 'Hello World'
    --> @dm:save name = 'hello-world-pipeline'

    --> @c:new-block
    --> @dm:recall name = 'hello-world-pipeline'
    --> *dm:filter *

Adding a simple row ('Hello World') and saving into a logical dataset (named 'hello-world-pipeline), using new-block (bot), calling back that dataset, and printing to the console using another filter bot. This is similar to reading, saving, and printing the 'Hello World' string using pipelines and bot libraries in RDA.

The above pipeline snippet represented using bots can also be represented using YAML format as shown below:

hashtag
Example 2

The above pipeline code snippet explains another simple basic pipeline of creating three columns ip_address, hostname, id, adding data to those three columns, and adding a simple logic using a bot to create a new column and filling the data. All the above-mentioned steps are put together via bots arranged in a series to solve a logical problem.

The above pipeline snippet represented using bots can also be represented using YAML format as shown below:

From the above two examples, users are free to create pipelines using bots convention and/or using YAML convention.

Note: Standalone RDA AIOps studio can be used to develop, test, and publish new pipelines using the above two methods and reuse those pipelines in cfxOIA RDA for applicable use cases. Few such use cases are explained below. Refer to Standalone RDA AIOps studio documentation for details.

hashtag
Nagios pipeline

The following screenshot provides the 'Nagios' pipeline that is used in the cfxOIA RDA environment for Nagios alert enrichment.

Step 1

Navigating to Nagios alert-enrichment pipeline

Step 2

Select 'View YAML' action button to view the pipeline code snipppet that was used in Nagios alert enrichment.

The following screenshot provides a UI view of Nagios alert enrichment.

In addition to the above screenshot, the following depicts Nagios alert enrichment pipeline code snippet.

The above Nagios alert enrichment pipeline contains various logical blocks (e.g. reading the 'nagios_host_group_members' dataset, and querying the dataset with a logical condition to match for host_name to match 'assetName' via incoming payload, enrich the alert, and saving into another dataset alert_enrichment_definition_nagios' and finally sending to upstream for further alert processing.

The above pipeline code snippet provides a powerful capability of RDA wherein the incoming raw alert payload is enriched by the pipeline (reading, applying logic, enriching alert with additional details) and passed onto the alert module for further processing. This enriched data can be used for ML processing and other RCA analysis.

hashtag
CMDB example pipeline

This section provides an additional CMD example pipeline codebase snippet.

CMDB load file pipeline in YAML format.

In addition to the above screenshot, following provides example pipeline code snippet for users reference.

In addition to adding/viewing pipelines, pipelines UI provides the following actions to end-users to work with pipelines - Details, Execute and, Deactivate.

hashtag
1. Details

The 'Details' action button will navigate users to a new UI that presents various pipeline runs (also called jobs or job run of a pipeline ) and their status as shown in the below screenshot.

Once users click or select the 'Details' action button, UI will navigate users to the status of jobs of that selected pipeline as shown below.

Details view UI that provides detailed status of a pipeline job

The above screen capture provides a very important aspect of pipeline runtime execution (job status of a pipeline execution). Pipeline Jobs table provides important job status detail columns ( Schedule Name, Pipeline Name, Status, Scheduled time, etc). Successful completion of pipeline job shows 'Completed' status, whereas failed job shows 'Error/Failed' and provides more log details.

In addition to the above, each job run provides further actions on each job runs as shown below screenshot.

Three action buttons are provided for status information of pipeilne job

hashtag
Check Status

The 'Check Status' action button provides the status of the pipeline job from end-to-end (as shown below screenshot). In case of any errors, it shows which step of the pipeline errored out and what is the reason.

Check Status UI depicting pipeline job execution flow.

hashtag
Summary

The 'Summary' action button shows the overall summary details of pipeline job execution (results summary) as shown below.

Summary UI depicting status of pipeline job execution summary

hashtag
View Results

The 'View Results' action button shows the end results as part of the current pipeline job execution as shown below.

View Results UI depicting end-results of a pipeline job execution.

hashtag
2. Execute

The 'Execute' action button will let the users execute the pipeline via a job as shown below. Once, users click the 'Execute' button, UI will pop up the pipeline code along with the confirmation button (to let the users confirm the job execution) as shown below.

Execute UI for a pipeline job prompting user to confirm the execution

Once users click 'Execute', the job will be triggered and executes the pipeline in context. Results can be viewed via 'Summary' or 'View Results' buttons as explained earlier.

hashtag
3. Deactivate

The 'Deactivate' action button will let the users disable the pipeline as shown below. In addition to deactivating, it will allow users to edit/change the pipeline as per the user's changes or requirements.

Deactivate UI to disable pipeline job for editing / modifications.
UI depicting inactive state of pipeline

Once the pipeline is in 'Inactive' mode, users will be able to perform the following additional actions on the 'Inactive' pipeline as shown in the following (Edit, Delete, Activate)

Once pipeline is in inactive state, Edit, Delete and Activate buttons are enabled

Users will be able to Edit, Delete and Activate (back after editing) pipeline and perform the operations.

PreviousBotsNextSchedules

Last updated