Run and report JOSF tests in Azure DevOps

Running your tests during your pipeline release cycle should be a breeze, and luckily with JOSF, that is the case. For this example we will require:

  • An Azure DevOps git repository, where your JOSF project already resides.
  • A playbook that you’d like to run.
  • Azure DevOps

For this example, we will use Azure DevOps as our pipeline platform. We already have a project in which we’d want to run our tests, so make sure you have one in Azure DevOps for yourself.

Note: If you haven’t already stored your test cases in Azure DevOps, please follow these instructions first and look under the “Import a GIT project from Microsoft Azure DevOps” section.

Add JOSF to Azure DevOps

To use JOSF in your pipelines, we need to add it as an artifact and reference to it in our pipeline script

  1. Install the Azure CLI on your local machine: If you haven’t already, download and install the Azure CLI from the official Microsoft documentation.
  2. Create a new feed: New artifacts require to be put into a Feed. Create a new feed in Azure DevOps by going to Artifacts and then press Create Feed. We are naming it josf-installation and keep the artifact scoped to this project.
  3. Start the Azure CLI and Authenticate: Open a command prompt or terminal and run the following command to sign in to your Azure DevOps account:
    az login 
  4. Create a new artifact source from the JOSF server variant: The new feed can hold our artifact which we can publish via the Azure CLI.
    az artifacts universal publish --organization {YOUR_ORGANZATION} --project="{YOUR_PROJECT_NAME}" --scope project --feed josf-installation --name josf-server --version {JOSF_VERSION} --description "JOSF Test automation" --path {PATH_TO_JOSF_SERVER_DIRECTORY}

    Replace {YOUR_ORGANIZATION} with your Azure DevOps organization URL, and {YOUR_PROJECT_NAME} with the name of your project, {JOSF_VERSION} to the corresponding JOSF version and{PATH_TO_JOSF_SERVER_DIRECTORY} with the JOSF folder that contains a clean JOSF server installation, not-including the projects folder. For an optimized result, download a fresh installation (JOSF server), unzip that folder and point the path to the /JOSF folder.


  5. Verify successful publish: Open your feed in Azure DevOps and verify that JOSF has uploaded correctly.

That’s it! You have successfully added JOSF as a new artifact to Azure DevOps using the Azure CLI.

Use JOSF in your pipeline script

  1. Navigate to Pipelines: Once inside your project, navigate to the “Pipelines” section using the left-hand side menu.
  2. Create a new pipeline: Click on the “New Pipeline” button to start creating a new pipeline.
  3. Select your source control: Choose your source control system where your code is hosted. Azure DevOps supports popular options like GitHub, Azure Repos, Bitbucket, and others. For this tutorial, we will use Azure Repos as our test cases are stored in the Azure repository and select your repository.
  4. Choose your pipeline configuration: Azure DevOps offers different options for configuring your pipeline. Select Starter pipeline and continue.
  5. Configure your pipeline: The pipeline is made up of a combination of tasks and powershell executions. In a birds-eye-view, we’ll be executing the following steps:
    • Install JOSF from the Azure Artifacts on that machine
    • Place the testcases into JOSF
    • Start JOSF and wait for it to be started
    • Execute the playbook
    • Publish the test results

The pipeline script

Your pipeline is executed based on an azure-pipelines.yml file, which contains the steps that should be executed. If you’d like to see the end result: go to the end of this page for the entire YAML file.

1. Switch to windows

By the time of writing this post, Azure generates a pipeline script which has a trigger and a pool of virtual machines, which it uses to execute the scripts on. For now, change the vmImage value to windows-latest. This ensures us that the JOSF instance is executed correctly.

2. Download and install JOSF

The first step is using the correct JOSF installation, which you’ve added as an artifact, by going to the tasks and search for Download package. Select the options, make sure the downloadPath is set to $(System.DefaultWorkingDirectory)\JOSF and press the Add button. This results in the following yaml snippet.

Note that extract: true needs to be present.

- task: DownloadPackage@1
  displayName: 'Unpack the JOSF artifact'
  inputs:
    packageType: 'upack'
    feed: '{FEED-ID}/{PACKAGE-ID]'
    view: '{VIEW-ID}'
    definition: '{DEFINITION-ID}'
    version: '{JOSF-VERSION}'
    downloadPath: '$(System.DefaultWorkingDirectory)\JOSF'
    extract: true

3. Copy the JOSF project to the VM

Next up is to copy the project from your GIT location, into the testcases folder of JOSF. This way, JOSF will automatically add this project upon starting up, and is able to find and execute the tests present.

- task: CopyFiles@2
  enabled: true
  displayName: 'Copy project into testcases folder'
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)'
    Contents: |
      **/*
    TargetFolder: '$(System.DefaultWorkingDirectory)\JOSF\testcases'

4. Start JOSF and wait for it to be online

Next is to start JOSF and wait for it to be online. We do this with the help of powershell, which starts the JOSF start command, and performs a ping on the localhost port where JOSF will respond, once it’s up and running.

Note how we’ve setup a timeoutInMinutes, as this prevents any infinite loops during failures of starting JOSF.

- powershell: |
   start-process '$(System.DefaultWorkingDirectory)\JOSF\api\start.bat'
   Write-Host 'Waiting for JOSF to be online'
   $statusCode = 0;
   Do {
       Write-Host -NoNewline '.'
       try {
         $statusCode = (Invoke-WebRequest 'http://localhost:8090/api/version').StatusCode;
         Start-Sleep -Milliseconds 750
       } catch {
         $statusCode = 0;
       }
   } While (!$statusCode -eq 200)
  displayName: 'Start JOSF and wait for it to be online'
  timeoutInMinutes: 2

5. Execute the playbook

Next is executing the command for running a certain playbook. We are only using the playbook name and the environment name, but other options are documented here or can be found in JOSF under the CLI menu-item (CLI wizard).

Note that we use our playbook named ‘Smoke’ on the environment named ‘local’. Change this to your own playbook and environment names. If you do not use environments, you can omit the entire environment option.

Note how this powershell task also contains the timeoutInMinutes, which you need to set to something which makes sense in your situation. Another important boolean is the continueOnError. If any tests fail, we want the pipeline to continue to its next and last mandatory task, which is publishing the report back to Azure DevOps.

- powershell: |
   $client = '$(System.DefaultWorkingDirectory)\JOSF\cli\josf-cli.jar'
   $arg = "$(System.DefaultWorkingDirectory)\JOSF\api\jre\bin\java.exe -jar '$client' -r playbook `"'Smoke'`" environment `"'local'`""
   Invoke-Expression "cmd /c $arg"
  displayName: 'Execute playbook Smoke on environment local'
  continueOnError: true
  timeoutInMinutes: 10

6. Publish the test results

After JOSF executes a playbook via its command line interface, a JUnit report is generated. The task for this is found via the task section, but comes down to this yaml snippet:

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/*junit-report.xml'
  1. Review and save your pipeline: Once you have finished configuring your pipeline, review the settings and make any necessary changes or adjustments. Once you’re satisfied, click on the “Save” or “Create” button to save your pipeline.
  2. Run your pipeline: After saving your pipeline, you can manually trigger a run or set up triggers based on different events like code commits, pull requests, or scheduled intervals. Once triggered, your pipeline will start running, and you can monitor the progress and view the execution logs.

The entire YAML file

# Pipeline for installing, starting and executing a JOSF playbook
# https://www.josf.nl/?p=3099

trigger:
- master

pool:
  vmImage: windows-latest

steps:
- task: DownloadPackage@1
  displayName: 'Unpack the JOSF artifact'
  inputs:
    packageType: 'upack'
    feed: '74c5de77-45a6-4fbb-8a5a-64036bd59dc2/fca07f91-d944-44f7-a47c-657ca9e6ca81'
    view: '72d6e932-045e-45e9-8c54-219efc00727e'
    definition: 'f983878d-afaa-4e8b-9e27-ae8c2cc915fc'
    version: '3.1.0'
    downloadPath: '$(System.DefaultWorkingDirectory)\JOSF'
    extract: true

- task: CopyFiles@2
  enabled: true
  displayName: 'Copy project into testcases folder'
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)'
    Contents: |
      **/*
    TargetFolder: '$(System.DefaultWorkingDirectory)\JOSF\testcases'

- powershell: |
   start-process '$(System.DefaultWorkingDirectory)\JOSF\api\start.bat'
   Write-Host 'Waiting for JOSF to be online'
   $statusCode = 0;
   Do {
       Write-Host -NoNewline '.'
       try {
         $statusCode = (Invoke-WebRequest 'http://localhost:8090/api/version').StatusCode;
         Start-Sleep -Milliseconds 750
       } catch {
         $statusCode = 0;
       }
   } While (!$statusCode -eq 200)
  displayName: 'Start JOSF and wait for it to be online'
  timeoutInMinutes: 2


- powershell: |
   $client = '$(System.DefaultWorkingDirectory)\JOSF\cli\josf-cli.jar'
   $arg = "$(System.DefaultWorkingDirectory)\JOSF\api\jre\bin\java.exe -jar '$client' -r playbook `"'Smoke'`" environment `"'local'`""
   Invoke-Expression "cmd /c $arg"
  displayName: 'Execute playbook Smoke on environment local'
  continueOnError: true
  timeoutInMinutes: 10

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/*junit-report.xml'