Background

Defence Science and Technology Group (DSTG) is the Australian Government’s lead agency dedicated to providing science and technology support for the country’s defence and security needs.

Problem

Modern software development practices include Agile Software Development, Development Security Operations (DevSecOps) and Lean Practices. These practices enable continuous iterative development and delivery for reduced cycle time, from the conceptual phase to operational phase. DevSecOps is an organisational software engineering culture and practice that aims to unify software development (Dev), security (Sec) and operations (Ops) without a prime contractor to coordinate and manage component suppliers and the integration of their deliverables.

The Department of Defence proposed that a standardised metrics framework be developed based on the DevSecOps approach. This framework would provide guidance to component contractors, as well as ensuring any development effectively contributes to broader system capability goals.

Approach

Shoal was engaged by DSTG to conduct collaborative research to develop a metrics framework to support the development of disaggregated capabilities to deliver integrated software systems.

Shoal, in collaboration with DSTG, conducted research in the state-of-art of large software development processes, metrics definitions and methods and processes for defining and measuring metrics. Contemporary thinking on software development and measurement drawn from professional societies and standards was studied and their relevance to the needs of the client evaluated. The result of the research was the identification of several candidate processes and methods for designing the measurement framework.

The framework was designed to address the following elements of a discipline:

  • Activity Scope: In this case software for Command and Control (C2) including combined human activity;
  • A knowledge base comprising software measurement theory and practices as used in contemporary Continuous Iterative Development (CID) methodologies such as Agile; and
  • A guidance framework covering terminology, worldviews, experiential knowledge, and the sets of metrics themselves.

A key finding is that higher-level metrics should relate to the provision of user value and as such they cannot be generic but will be specific to each software development challenge. This led to studying the theory needed to produce information products from measurements and a process for establishing bespoke software metrics.

In areas where we failed to find an existing solution, we proposed original responses to meet the objectives of the framework.

Results

It was found that the most suitable CID process for the problem was Scaled Agile Framework (SAFe) and a high-level SAFe implementation was mapped to C2 capability elements. Goal-Question-Metrics (GQM) was found to be the appropriate approach in defining the metrics and measures across all levels of SAFe hierarchy. While metrics are readily available for the lower levels of the hierarchy, a unique synthesis of metrics has been designed to address the absence of a prime contractor to monitor and manage integration issues among the component contractors.

It has been shown that software development and integration of capabilities by several contractors without a prime contractor can be monitored and managed using metrics.

The work has been presented at the 31st Annual INCOSE International Symposium, July 2021 as A Metrics Framework to Facilitate Integration of Disaggregated Software Development.