Design of measurement systems and metrics

If you cant measure it then you cant manage it.

In order to manage and control the design processes, they have to be monitored and measured. This is true for all aspects of the design processes. Measurements and metrics are covered in detail in the Continual Service Improvement publication. This section covers those aspects that are particularly relevant and appropriate to measuring the quality of the design processes and their deliverables.

Care should be exercised when selecting measurements and metrics and the methods used to produce them. This is because the metrics and measurements chosen will actually affect and change the behaviour of people working within the activities and processes being measured, particularly where this relates to objectives, personal and team performance and performance-related pay schemes. Therefore only measurements that encourage progression towards meeting business objectives or desired behavioural change should be selected.

In all the design activities the requirement should be to:

  • Design solutions that are fit for purpose
  • Design for the appropriate level of quality not over-engineered or under-engineered
  • Design solutions that are right first time and meet their expected targets
  • Design solutions that minimize the amount of rework or add-ons that have to be rapidly developed after solutions have been deployed
  • Design solutions that are effective and efficient from the perspective of the business and the customers. The emphasis should be on the solutions that are effective above all and that are efficient within the constraint of remaining effective.

Measurement methods and metrics should reflect these requirements and be designed to measure the ability of design processes to match these requirements. All of the measurements and metrics used should reflect the quality and success of the design processes from the perspective of the business, customers and users. They need to reflect the ability of the delivered solutions to meet the identified and agreed requirements of the business.

The process measurements selected need to be appropriate for the capability and maturity of the processes being measured. Immature processes are not capable of supporting sophisticated measurements, metrics and measurement methods. There are four types of metrics that can be used to measure the capability and performance of processes:

  • Progress: milestones and deliverables in the capability of the process
  • Compliance: compliance of the process to governance requirements, regulatory requirements and compliance of people to the use of the process.
  • Effectiveness: the accuracy and correctness of the process and its ability to deliver the right result
  • Efficiency: the productivity of the process, its speed, throughput and resource utilization.

Measurements and metrics should develop and change as the maturity and capability of a process develops. Initially, with immature processes the first two levels of metrics should be used to measure the progress and compliance of the process as it develops in maturity. As the process maturity develops, greater use should be made of effectiveness and efficiency metrics, but not to the detriment of compromising the progress or compliance of the process.

The selection of the metrics, the point of measurement and the methods of measuring, calculating and reporting on the metrics must be carefully designed and planned. The primary metrics should always focus on determining the effectiveness and the quality of the solutions provided. Secondary metrics can then measure the efficiency of the processes used to produce and manage the solution. The priority should always be to ensure that the processes provide the correct results for the business. Therefore the measurement methods and metrics should always provide this business-focused measurement above all.

The most effective method of measurement is to establish a Metrics Tree or KPI tree. Too many organizations collect measurement in individual areas, but fail to aggregate them together and gain the full benefit of the measurements, and therefore suffer because:

  • Measurements are not aligned with business objectives and needs
  • There is no overall visibility of the top-level picture
  • There are gaps in areas where measurements are not recorded
  • Individual areas are well measured and others are poorly measured or are not measured
  • There is no consistency in the method, presentation and calculation of the measurements
  • Decisions and improvement actions are based on incomplete or inaccurate information.

Therefore organizations should attempt to develop automated measurement systems based on a form of Metrics Tree such as that illustrated in Figure 3.12.

Figure 3.12 The Metrics Tree

The tree in Figure 3.12 is illustrative of an example of a Metrics Tree based on a typical Balanced Scorecard. Balanced Scorecards represent a management system that enables increasing numbers of organizations to clarify their vision and strategy into action. They provide feedback regarding the internal business processes and external outcomes in order continually to improve strategic performance and results. This enables everybody within the organization to get a picture of the performance of the organization at the appropriate level:

  • Business managers and customers can get a top-level business dashboard, aligned with business needs and processes
  • Senior IT managers and customers can focus on the top-level IT management dashboard
  • Service Managers and customers can focus on the performance of particular services
  • Process owners and managers can view the performance of their processes
  • Technical specialists can look at the performance of individual components
  • The dashboard also presents an opportunity to view trends over time, rather than static data, so that potential performance degradation can be identified and rectified at an early stage.

This means that within a hierarchical metrics system, each person in the organization can get access to an appropriate level of information and measurement that suits their particular need. It gives senior management the opportunity to monitor a top-level dashboard to ensure that services continue to be delivered to their agreed levels, and it also provides the capability for technical specialists and processes owners to drill down to the detail to analyse variance from agreed service, component or process performance.

Obviously the collection, analysis and presentation of this data can be a very labour-intensive activity and therefore should be automated wherever possible. This can be achieved using analysis tools based on macros, scripts, spreadsheets, or preferably on specific web-based solutions. The measurements at each of the levels should be specifically defined to meet the needs of the business, customers and users of the information.

More detailed information on measurements, metrics and measurement methods are contained in the Continual Service Improvement publication.

sdamzavas.net - 2020 . ! , ...