Pipelines
A pipeline is a set of processing elements (bots) connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in sequence or in a time-sliced fashion based on the user-designed logic.
Pipelines are a commonly used concept in everyday life. Car assembly line, CI/CD environment, Big data environments to name a few.
In cfxOIA RDA context, the pipeline mechanism is supported via RDA bots which were explained in the previous section. RDA bots are designed, implemented, and logically connected to solve real-world problems. This section covers few following example pipelines that are part of cfxOIA RDA.
Simple example pipelines
Nagios pipeline
cmdb_file_load example pipeline
Note: The above example pipelines explained in this section are for reference and users are free to modify, extend and/or add new pipelines to solve other use-cases based off-of the above examples.
Simple example pipeline
Example 1
The below pipeline code snippet explains a very basic and simple pipeline that. In the above example, the pipeline consists of bots that are logically arranged or connected in series to perform the following tasks:
Adding a simple row ('Hello World') and saving into a logical dataset (named 'hello-world-pipeline), using new-block (bot), calling back that dataset, and printing to the console using another filter bot. This is similar to reading, saving, and printing the 'Hello World' string using pipelines and bot libraries in RDA.
The above pipeline snippet represented using bots can also be represented using YAML format as shown below:
Example 2
The above pipeline code snippet explains another simple basic pipeline of creating three columns ip_address, hostname, id, adding data to those three columns, and adding a simple logic using a bot to create a new column and filling the data. All the above-mentioned steps are put together via bots arranged in a series to solve a logical problem.
The above pipeline snippet represented using bots can also be represented using YAML format as shown below:
From the above two examples, users are free to create pipelines using bots convention and/or using YAML convention.
Note: Standalone RDA AIOps studio can be used to develop, test, and publish new pipelines using the above two methods and reuse those pipelines in cfxOIA RDA for applicable use cases. Few such use cases are explained below. Refer to Standalone RDA AIOps studio documentation for details.
Nagios pipeline
The following screenshot provides the 'Nagios' pipeline that is used in the cfxOIA RDA environment for Nagios alert enrichment.
Step 1
Step 2
The following screenshot provides a UI view of Nagios alert enrichment.
In addition to the above screenshot, the following depicts Nagios alert enrichment pipeline code snippet.
The above Nagios alert enrichment pipeline contains various logical blocks (e.g. reading the 'nagios_host_group_members' dataset, and querying the dataset with a logical condition to match for host_name to match 'assetName' via incoming payload, enrich the alert, and saving into another dataset alert_enrichment_definition_nagios' and finally sending to upstream for further alert processing.
The above pipeline code snippet provides a powerful capability of RDA wherein the incoming raw alert payload is enriched by the pipeline (reading, applying logic, enriching alert with additional details) and passed onto the alert module for further processing. This enriched data can be used for ML processing and other RCA analysis.
CMDB example pipeline
This section provides an additional CMD example pipeline codebase snippet.
In addition to the above screenshot, following provides example pipeline code snippet for users reference.
In addition to adding/viewing pipelines, pipelines UI provides the following actions to end-users to work with pipelines - Details, Execute and, Deactivate.
1. Details
The 'Details' action button will navigate users to a new UI that presents various pipeline runs (also called jobs or job run of a pipeline ) and their status as shown in the below screenshot.
Once users click or select the 'Details' action button, UI will navigate users to the status of jobs of that selected pipeline as shown below.
The above screen capture provides a very important aspect of pipeline runtime execution (job status of a pipeline execution). Pipeline Jobs table provides important job status detail columns ( Schedule Name, Pipeline Name, Status, Scheduled time, etc). Successful completion of pipeline job shows 'Completed' status, whereas failed job shows 'Error/Failed' and provides more log details.
In addition to the above, each job run provides further actions on each job runs as shown below screenshot.
Check Status
The 'Check Status' action button provides the status of the pipeline job from end-to-end (as shown below screenshot). In case of any errors, it shows which step of the pipeline errored out and what is the reason.
Summary
The 'Summary' action button shows the overall summary details of pipeline job execution (results summary) as shown below.
View Results
The 'View Results' action button shows the end results as part of the current pipeline job execution as shown below.
2. Execute
The 'Execute' action button will let the users execute the pipeline via a job as shown below. Once, users click the 'Execute' button, UI will pop up the pipeline code along with the confirmation button (to let the users confirm the job execution) as shown below.
Once users click 'Execute', the job will be triggered and executes the pipeline in context. Results can be viewed via 'Summary' or 'View Results' buttons as explained earlier.
3. Deactivate
The 'Deactivate' action button will let the users disable the pipeline as shown below. In addition to deactivating, it will allow users to edit/change the pipeline as per the user's changes or requirements.
Once the pipeline is in 'Inactive' mode, users will be able to perform the following additional actions on the 'Inactive' pipeline as shown in the following (Edit, Delete, Activate)
Users will be able to Edit, Delete and Activate (back after editing) pipeline and perform the operations.
Last updated
