Design of measurement systems and metrics
‘If you can’t measure it then you can’t manage it.’
In order to manage and control the design processes, they have to be monitored and measured. This is true for all aspects of the design processes. Measurements and metrics are covered in detail in the Continual Service Improvement publication. This section covers those aspects that are particularly relevant and appropriate to measuring the quality of the design processes and their deliverables.
Care should be exercised when selecting measurements and metrics and the methods used to produce them. This is because the metrics and measurements chosen will actually affect and change the behaviour of people working within the activities and processes being measured, particularly where this relates to objectives, personal and team performance and performance-related pay schemes. Therefore only measurements that encourage progression towards meeting business objectives or desired behavioural change should be selected.
In all the design activities the requirement should be to:
Measurement methods and metrics should reflect these requirements and be designed to measure the ability of design processes to match these requirements. All of the measurements and metrics used should reflect the quality and success of the design processes from the perspective of the business, customers and users. They need to reflect the ability of the delivered solutions to meet the identified and agreed requirements of the business.
The process measurements selected need to be appropriate for the capability and maturity of the processes being measured. Immature processes are not capable of supporting sophisticated measurements, metrics and measurement methods. There are four types of metrics that can be used to measure the capability and performance of processes:
Measurements and metrics should develop and change as the maturity and capability of a process develops. Initially, with immature processes the first two levels of metrics should be used to measure the progress and compliance of the process as it develops in maturity. As the process maturity develops, greater use should be made of effectiveness and efficiency metrics, but not to the detriment of compromising the progress or compliance of the process.
The selection of the metrics, the point of measurement and the methods of measuring, calculating and reporting on the metrics must be carefully designed and planned. The primary metrics should always focus on determining the effectiveness and the quality of the solutions provided. Secondary metrics can then measure the efficiency of the processes used to produce and manage the solution. The priority should always be to ensure that the processes provide the correct results for the business. Therefore the measurement methods and metrics should always provide this business-focused measurement above all.
The most effective method of measurement is to establish a ‘Metrics Tree’ or ‘KPI tree’. Too many organizations collect measurement in individual areas, but fail to aggregate them together and gain the full benefit of the measurements, and therefore suffer because:
Therefore organizations should attempt to develop automated measurement systems based on a form of ‘Metrics Tree’ such as that illustrated in Figure 3.12.
Figure 3.12 The Metrics Tree
The tree in Figure 3.12 is illustrative of an example of a Metrics Tree based on a typical Balanced Scorecard. Balanced Scorecards represent a management system that enables increasing numbers of organizations to clarify their vision and strategy into action. They provide feedback regarding the internal business processes and external outcomes in order continually to improve strategic performance and results. This enables everybody within the organization to get a picture of the performance of the organization at the appropriate level:
This means that within a hierarchical metrics system, each person in the organization can get access to an appropriate level of information and measurement that suits their particular need. It gives senior management the opportunity to monitor a top-level dashboard to ensure that services continue to be delivered to their agreed levels, and it also provides the capability for technical specialists and processes owners to drill down to the detail to analyse variance from agreed service, component or process performance.
Obviously the collection, analysis and presentation of this data can be a very labour-intensive activity and therefore should be automated wherever possible. This can be achieved using analysis tools based on macros, scripts, spreadsheets, or preferably on specific web-based solutions. The measurements at each of the levels should be specifically defined to meet the needs of the business, customers and users of the information.
More detailed information on measurements, metrics and measurement methods are contained in the Continual Service Improvement publication.