Investment appraisal in the round: why MCA?

Multi-criteria analysis (‘MCA’) is a set of methods, techniques and tools that considers multiple objectives and criteria to support decision-making. It takes a broader perspective than cost–benefit analysis and cost-effectiveness analysis, which stress monetary considerations. MCA is set to play a greater role in investment appraisal as increasing emphasis is placed on having clear strategic links to policies, ‘levelling up’, and stewardship of the environment. Dr Rupert Booth, Senior Adviser, considers the motivations for the use of MCA and some of its many techniques.

All advanced economies scrutinise major investments to confirm or enhance their benefits, or to reduce their costs. In the UK, the chief document used in investment appraisal is the Treasury’s Green Book,¹ which has primarily focused on cost–benefit analysis (CBA), with monetary considerations at its centre, while cost-effectiveness analysis (CEA) is endorsed under certain conditions.² However, there is also an increasing role for MCA, constituting a vital tool for the structuring and approving of major projects that not only incur immense capital expenditure, but also influence operating costs over their lifecycle.

Green Book changes

In December 2020, a revised version of the UK Treasury’s Green Book was introduced, responding to demands to ‘level up’ the regions. Oxera commented on it at the time,³ noting: ‘(i) applicants will need to place greater effort on creating sound strategic cases and (ii) applicants will potentially need to be less concerned about the precise value of the Benefit Cost Ratio (BCR) when there is a strong strategic case to support the intervention’. Other points highlighted by Oxera included the need to link the strategic objectives of an investment to government policies, and the need to address environmental issues.⁴ This article takes the analysis further, focusing on the UK government’s statement that the Green Book ‘has also been revised to reflect new or updated guidance in relation to Multi-Criteria Decision Analysis’.⁵ This development is timely given the well-established track record of MCA being used in environmental assessments—for instance, of transport megaprojects.⁶

The Green Book has not been without its critics. Coyle and Sensier (2019) assert that it has contributed to the regional imbalance in the UK economy, and that ‘infrastructure investments also need to be based on a strategic view about economic development for the whole of the UK’.⁷ As noted above, the 2020 edition seeks to address this point directly by supporting the levelling-up agenda. Hurst (2018) offers other criticisms of the 2018 edition—namely, that there is:⁸

• an excessive reliance on normal distribution analysis;
• limited coverage of social impact and overemphasis of monetisation;
• limited coverage of project portfolio analysis;
• a tendency towards a reliance on carbon-copy techniques—i.e. the premise that all projects can be subject to the same kinds of appraisal and evaluation techniques.

Whatever their original validity, these points have been at least partially addressed in the 2020 edition of the Green Book. There is explicit reference to low-probability/high-impact events and their treatment in the risk register, and there is no recommendation that normal distributions are to be used in the Monte Carlo simulation. Regarding social impact and monetisation, there are now annexes considering ‘Non-market Valuation and Unmonetisable Values’, ‘Place Based Analysis’, and ‘Distributional Appraisal’. The concept of ‘strategic portfolios’ is introduced: ‘These portfolios consist of the programmes and projects that are required to realise a strategic policy objective’.

The 2020 Green Book also calls for the links between government policies and the proposed project to be made explicit. Finally, regarding the ‘carbon-copy’ comment, the mainstay technique of CBA is now supplemented by an array of supporting methods, including MCA (the supplementary guidance on which covers a wide variety of alternative techniques).⁹

In summary, the 2020 Green Book has addressed prior criticisms, and wider use of MCA is encouraged. The book states that: ‘complex technical trade-offs at the longlist stage, concerning choices of service scope and service solution, may be assisted by the use of expertly facilitated Multi-Criteria Decision Analysis’.¹⁰ Furthermore, the Appraisal Summary Tables at the shortlist stage have an explicit entry for ‘Significant Quantified but unmonetisable benefits’, allowing for a form of multi-criteria selection in addition to the monetary CBA.

International context 

The trends in the UK are consistent with changes in the EU and elsewhere. The ‘CBA guide for Cohesion Policy 2014–20’ is being supplemented by a new ‘Economic Appraisal Vademecum’ for the period 2021–27, ‘offering a guide to simplified CBA, and other appraisal techniques, such as cost-effectiveness tests and multi-criteria analyses’.¹¹ There is no move to jettison CBA, but there is a desire to combine it with MCA, especially in the early stages of the appraisal.

There is one European exception—namely, the manual of the European Investment Bank.¹² This states: ‘In appraising the economic viability of projects, the EIB uses CBA, CEA and MCA as substitutes rather than complements’,¹³ and reserves MCA for education, health, and urban and regional development. This might reflect the difficulty in monetisation of benefits in those sectors. However, the approach of considering the alternative techniques as substitutes rather than complements is unusual, as appraisers usually seek the broadest assessment possible. For example, the Joint Assistance in Supporting Projects in European Regions (JASPERS) approach on the appraisal of EU ICT projects is more typical and suggests the following process:¹⁴

• comparison of strategic project options—normally based on MCA, using, for example, business as usual, different institutional or business models, location, and the like;
• selection of a preferred option and a series of feasible technological alternatives compatible with the identified objectives of the project;
• comparison of the shortlisted technological alternatives, including quantitative methods.

This is a process not dissimilar to that proposed in the UK 2020 Green Book. However, if we go further afield from Europe, we start to encounter different approaches.

The South Korean manual for ‘Pre-Feasibility Studies’,¹⁵ corresponding to the UK’s Strategic Outline Case (SOC) stage,¹⁶ is noteworthy because it recognised the need for regional balance over a decade ago and proposed a solution based on an MCA technique—namely analytic hierarchy processing (discussed below). The approach is illustrated in Figure 1.

Figure 1   Pre-feasibility process

This approach avoids a noted problem with CBA, in that it tends to favour the development of already developed regions, leading to regional imbalances. This is because economically developed regions can offer economies of scope leading to further development (i.e. economic clustering), in turn giving greater benefits per unit of public investment in those regions. The Korean approach also has a ‘grey area’ aspect in the final stage of accept/reject, where dissent between independent reviewers will lead to a cautious conclusion—presumably to permit policymakers greater discretion in making the final decision.

The World Bank has proposed the ‘Infrastructure Prioritisation Framework’, a form of MCA with two variables:¹⁷ the Financial and Economic Index (FEI) and the Social and Environmental Index (SEI). This is a portfolio management technique, to select projects under a set budget constraint, and the World Bank proposes that this could be used where there is limited data to monetise social and environmental performance. Once the FEI and SEI are known and the expenditure is reconciled with the budget constraint, a 2×2 matrix is drawn, as shown in Figure 2, with the attractive projects in the upper-right quadrant.

Figure 2 Infrastructure prioritisation matrix

The approach can be useful for a quick sortation of projects, to eliminate poor candidates and focus scarce analytic effort on those worthy of future consideration.

The MCA process 

At this stage, it would be useful to define in more detail what MCA actually involves. The UK’s supplementary guidance provides a useful overview. It first offers the following generic MCA process.¹⁸

• Establish the decision context: (i) establish the aims of the MCA, and identify decision-makers and other key players; (ii) design the socio-technical system for conducting the MCA; and (iii) consider the context of the appraisal.
• Identify the options to be appraised.
• Identify objectives and criteria: (i) identify criteria for assessing the consequences of each option; and (ii) organise the criteria by clustering them under high- and lower-level objectives in a hierarchy.
• ‘Scoring’: assess the expected performance of each option against the criteria, and then assess the value associated with the consequences of each option for each criterion: (i) describe the consequences of the options; (ii) score the options on the criteria; and (iii) check the consistency of the scores on each criterion.
• ‘Weighting’: assign weights for each of the criteria to reflect their relative importance to the decision.
• Combine the weights and scores for each option to derive an overall value: (i) calculate overall weighted scores at each level in the hierarchy; and (ii) calculate overall weighted scores.
• Examine the results.
• Sensitivity analysis: (i) conduct a sensitivity analysis (do other preferences or weights affect the overall ordering of the options?); (ii) look at the advantages and disadvantages of the selected options, and compare pairs of options; (iii) create possible new options that might be better than those originally considered; and (iv) repeat the above steps until a ‘requisite’ model is obtained.

However, the potential for pitfalls is significant: broad aims may differ between sponsors (e.g. national and regional government), and different stakeholders may have different specific objectives, especially in a politically charged project. Options may be missed and there may be disagreement on the criteria. Disagreements can also arise regarding weights and scoring.

For these reasons, a skilled MCA practitioner is essential. Even this does not guarantee success, if those with the technical expertise needed to assess projects have strong views on particular options, so can influence the outcome to their advantage. A related topic is seniority: in practice, much of the work will be undertaken in workshops and the presence of senior staff can inhibit discussion of alternatives if their preferences are known in advance.

As well as subjectivity, there are the issues of uncertainty and risk. Opinions may differ, and even if there is consensus, opinions may be wrong. A thorough sensitivity analysis is therefore required, as noted above, but it should also be linked to the risk register for the project. The sensitivity analysis may also show that the choice between options is insensitive to certain scores. This can be helpful if it focuses attention on the ‘critical few’. However, if the inputs in question are regarded as ‘crucial’ by stakeholders, there may be a need to re-examine the weights.

Weighing up

The mention of weights prompts a technical discussion on the wide variety of MCA methods that exist. Usually, MCA is concerned with analysing discrete alternatives. If the attractiveness, or utility, of all outcomes is known, the alternative that provides the best utility can be selected through summation—however, in practice, data on utility is not available and, in any case, is likely to differ between stakeholders.

This leads on to an approximation, namely the ‘linear additive models’, where criteria are given weights and scores. The UK Treasury ‘swing weights’ method¹⁹ is based on SMARTS (Simple Multi Attribute Rating Technique with Swings) and SMARTER (SMART Exploiting Ranks), both developed by Edwards and Baron (1994).²⁰ In contrast, analytic hierarchy processing offers a pair-wide comparison approach to identifying the weights, and requires participants to state the relative importance for both of the criteria and the performance of an option for each criterion, which can be time-consuming if the numbers of either are large.

Two further MCA categories are the outranking approaches—e.g. ELECTRE, PROMETHEE—and the compromise-ranking approaches—e.g. TOPSIS and VIKOR. The ‘ranking’ refers to the ranking of the decision alternatives against multiple criteria, either in a search for the option that ‘dominates’ others, or for an option that is a compromise closest to an ideal solution. The literature is vast,²¹ and includes papers that have applied multiple methods to the same problem and drawn conclusions on relative merits.²² A further complication is that some approaches have evolved to deal with ‘fuzzy’ data, offering even more choices. There is now a website that helps select MCA methods: mcda.it. This evolved from the development of a systematic selection approach for methods.²³

For those undertaking an investment appraisal, the choice of method will be influenced by the opinions of the appraising body—for instance, for ‘swing weights’ or analytic hierarchy processing. If the preferred method is problematic, a constructive response could be to comply and then supply an additional alternative analysis, highlighting the added insights that it provides.

Apart from technical innovations, there is also an attempt to make MCA more relevant to stakeholders and policymakers. One initiative is ‘participatory MCA’,²⁴ and the other is ‘policy-led MCA’.²⁵ Both approaches have developed in the field of transport megaprojects and seek to produce a higher quality of appraisal through deeper stakeholder involvement.

It’s all in the process

This article began by identifying shifts in investment appraisal that have been seen in major economies and a consequent growth in the role of MCA, especially given the need to ensure levelling up, or cohesion, and the growing importance of environmental issues. The article has reviewed the different approaches to MCA from around the world and found that MCA for investment appraisal takes a variety of forms. Research into MCA methods continues and new techniques emerge.

However, the most important issue is the process by which these techniques are used: there is always a need to involve stakeholders and policymakers, and to ensure maximum relevance and quality of the investment appraisal, while avoiding a purely desk-based approach.

¹ HM Treasury (2020), ‘The Green Book: central government guidance on appraisal and evaluation’.

² See also the accompanying document, ‘Guide to Developing the Project Business Case’.

³ Oxera (2020), ‘What’s new in the 2020 update to the Green Book?’, November.

⁴ The latest government thinking on incorporating environmental issues into investment appraisal is given in Department for Environment, Food and Rural Affairs (2020), ‘Enabling a Natural Capital Approach: Guidance’, March.

⁵ HM Treasury (2020), ‘Green Book Review 2020: Findings and response’, November. ‘MCA’ and ‘multi-criteria decision analysis’ (MCDA) are treated as synonymous, and the shorter acronym is used here.

⁶ Broniewicz, E. and Ogrodnik, K. (2020), ‘Multi-criteria analysis of transport infrastructure projects’, Transportation Research Part D: Transport and Environment, 83, p.102–351.

⁷ Coyle, D. and Sensier, M. (2019), ‘The imperial treasury: appraisal methodology and regional economic performance in the UK’, Regional Studies, 54:3.

⁸ Hurst, M. (2018), ‘The Green Book: Central Government Guidance on Appraisal and Evaluation’, Journal of Mega Infrastructure & Sustainable Development, 1:1.

⁹ HM Treasury (2013), ‘Green Book supplementary guidance: multi-criteria decision analysis’, 21 April, p. 7.

¹⁰ HM Treasury (2020), ‘The Green Book: central government guidance on appraisal and evaluation’, p. 37.

¹¹ EIPA (2021), ‘Cohesion Policy Project Appraisal standards in 2021-2027; Using CBA; The Connecting Europe Facility’.

¹² European Investment Bank (2013), ‘The Economic Appraisal of Investment Projects at the EIB’, ‘Version March 2013 – Under review’, March.

¹³ Ibid., p. 12.

¹⁴ JASPERS (2020), ‘Good Practices in preparing ICT projects co-financed by EC’, 26 November.

¹⁵ See Public and Private Infrastructure Investment Management Center (2008), ‘General Guidelines for Preliminary Feasibility Studies’, fifth ed.

¹⁶ See HM Treasury (2018), ‘Guide to Developing the Project Business Case’.

¹⁷ Marcelo, D., Mandri-Perrott, C., House, S. and Schwartz, J. (2016), ‘Prioritizing Infrastructure Investment: A Framework for Government Decision Making’, Policy Research Working Paper No. 7674, World Bank, Washington, DC.

¹⁸ HM Treasury (2013), op. cit., p. 37.

¹⁹ The origin of the word ‘swing’ lies in a comparison of differences: how does the swing from 0 to 100 on one preference scale compare to the 0 to 100 swing on another scale?

²⁰ Edwards, W. and Barron, F. H. (1994), ‘SMARTS and SMARTER: Improved simple methods for multiattribute utility measurement’, Organizational behavior and human decision processes, 60:3, pp. 306–25.

²¹ Mardani, A., Jusoh, A., Nor, K., Khalifah, Z., Zakwan, N. and Valipour, A. (2015), ‘Multiple criteria decision-making techniques and their applications—a review of the literature from 2000 to 2014’, Economic research–Ekonomska istraživanja, 28:1, pp. 516–71.

²² See Wu, Z. and Abdul-Nour, G. (2020), ‘Comparison of Multi-Criteria Group Decision-Making Methods for Urban Sewer Network Plan Selection’, CivilEng, 1:1, pp. 26–48. The authors note: ‘As a conclusion, decision-makers identify PROMETHEE II as their favorite method, [analytic hierarchy processing] is more time and energy consuming and results in a number of inconsistencies, while TOPSIS loses information during vector normalization for multi-dimension criteria, and ELECTRE III’s results are inconclusive’.

²³ Wątróbski, J., Jankowski, J., Ziemba, P., Karczmarczyk, A. and Zioło, M. (2019), ‘Generalised framework for multi-criteria method selection’, Omega, 86, pp. 107–24.

²⁴ Dean, M., Hickman, R. and Chen, C. L. (2019), ‘Testing the application of participatory MCA: The case of the South Fylde Line’ Transport Policy, 73, pp. 62–70.

²⁵ Ward, E. J., Dimitriou, H. T. and Dean, M. (2016), ‘Theory and background of multi-criteria analysis: Toward a policy-led approach to mega transport infrastructure project appraisal’, Research in Transportation Economics, 58, pp. 21–45.