In our latest From the Archives piece, RiskBusiness CEO Mike Finlay takes a look at some of the issues surrounding scenario assessment and provides some guidance on building a programme that makes best use of this misunderstood risk-management tool. This article originally appeared in issue 34 of The Risk Universe magazine in October 2014.
The basic concept of scenario assessment is almost as old as mankind itself – asking the questions “what if” and “how bad could it be”. What if we cannot find food or shelter, what if we are attacked by wild animals, how bad could things be? Under the (relatively new but maturing) discipline of operational risk management, these questions and the underlying concept remains at least as relevant as it was to our far-less civilised forefathers – what if a member of staff steals from us, what if someone hacks our systems, what if there is an earthquake, how bad could the consequences and impacts be?
Unfortunately for such a simple concept, its very simplicity generates enormous complexity, usually responded to by developing large and complex policies and supporting procedures, resulting in an unwieldy, misunderstood and often gamed risk management tool. A significant proportion amongst the contributing factors to this view of scenario assessment lies, as is typically the case with operational risk, with the people engaged in performing the assessment. People are, however much we may wish to try and dispute it, influenced by their surroundings and the current “atmosphere” within those surroundings. This generates a lack of consistency in perception, thought patterns and behaviour and, in turn, implies a lack of consistency in the results of scenario assessment. This in turn is something which supervisors and regulators find uncomfortable and unacceptable, leading to anti-scenario assessment attitudes.
However, a scenario is itself simply a story, the outlines of a hypothetical situation that may unexpectedly or adversely affect a firm or the environment within which it operates, by itself meaningless, innocuous and enormously reusable. There are also an almost endless number of scenarios the individual or team, especially if blessed with a vivid imagination, could invent. But it may be more practical to think of scenarios in terms of broad streams rather than a vast array of outlines that only vary by minute detail. What makes the difference is how the scenario and its potential implications for the firm is assessed and how the assessment results are employed within the firm.
Scenarios may be considered for a multitude of reasons, ranging from trying to determine the consequences of a strategic or business decision (strategic or business risk), trying to predict market behaviour and the probable consequences (market or liquidity risk), trying to evaluate exposure for the firm under various conditions (credit or business risk) or trying to determine how staff will react to certain situations (operational risk).
Scenarios may be considered for the firm overall by its senior and executive management (top-down scenarios) or may be incorporated into day-to-day business decision making and in the contemplation of some new business activity (bottom-up scenarios). Yet another use relates to trying to predict the consequences for the firm based on something which has just happened to a market peer (specific exposure scenarios).
With the above as a general background, the following six steps provide an approach to developing a suitable scenario assessment programme for implementation within a firm. Note however, the use of the term “scenario assessment” as opposed to a common use of “scenario analysis”. Firms need to design a programme to select scenarios, select scenario assessors, collect the assessment views of the assessors before they can analyse the resultant implications for the firm. Scenario analysis must be considered to be a step within the overall scenario assessment process.
Determine the objective of scenario assessment
The first and most critical step is to decide on why you are going to perform scenario assessment. This decision is crucial, as it determines the types of approaches and methodologies you can employ and the type of data you wish to obtain. It you are performing scenario assessment to better understand the extreme consequences of an operational risk so as to either mitigate that exposure or manage it better, you need collective buy-in on the deemed consequences, you need engagement in the decisions around how the exposure will be mitigated or managed, whereas if you are performing the assessment to determine how much regulatory capital to set aside, you need consistency in result, avoidance of gaming or bias and a concise set of variables, which can be used as modelling inputs.
Select appropriate scenarios
Once you know why you intend to undertake scenario assessment, the next step is to select appropriate scenarios. If you are assessing some specific exposure that recently affected a peer firm or which a supervisor instructs you to assess, selection becomes simple, informed by the cause of the need to make the assessment. If you are performing scenario assessment for business continuity or general risk assessment purposes, you should review the known risks to the firm, review reported comparable cases affecting the broader industry and brainstorm what other “things” could affect the firm. If your primary purpose is capital estimation, you need to determine how you will co-mingle scenario data with other forms of input data. Then you need to either seek to identify scenarios to cover “blank spots” (that is, areas where there is inadequate data from other sources) and areas with excessive data which may need to be excluded through scenario assessment, or a range of areas where you need to build an impact or severity curve from scenarios which will be supplemented by a loss data driven frequency curve.
Whatever the objectives, ensure that you select appropriate scenarios for each area to be covered, taking care to either employ generic scenarios or else provide alternates that are business specific. For example: a trading system outages for the dealing environment, an online banking application outage for business banking and an ATM network outage for retail banking, all measuring the firm’s exposure to a primary transaction initiation platform outage.
In selecting scenarios, also ensure that the purpose for which the scenarios are intended are represented by the type of scenarios selected. A multi-variable scenario, such as a system outage at peak transaction volume during month-end, combined with a management offsite and the unit supervisor being on vacation, is ideal for exploring what could go wrong, who would respond and what can we fix to prevent unacceptable consequences. But this is virtually impossible to consistently replicate year on year, making multi-variable scenarios inappropriate for capital measurement purposes – rather use single variables for capital purposes.
Document your scenarios, taking care with scenarios intended for capital estimation to minimise the introduction of potential bias. If appropriate, illustrate the scenario with applicable internal historical information and external market information. Be concise in your documentation – try and keep a single scenario outline to less than one page – but ensure sufficient clarity to avoid affecting consistency or allowing too broad a degree of interpretation in response.
How many scenarios should we select? Again, this is dependent on the objectives of the scenario assessment process. Bottom-up or scenario assessment for management purposes can be performed over time and can thus assess an almost unlimited number of scenarios over time. Scenario assessment for capital measurement purposes is usually performed against some specific deadline, making it critical to select a small number of scenarios which can be assessed within those
timeframes. Note that many firms adopt a multi-year rolling process whereby each year a different set of scenarios are assessed, returning to each scenario say once every three years to ensure the assessment results remain reasonably current.
Select assessors
Again, who is asked to assess the scenario is linked to the overall objective behind the scenario assessment. Scenarios for business management usually involve the business unit management, whereas scenarios for capital measurement usually involve senior executives and specific subject matter experts. Once you know what scenarios you wish to assess and who you need to assess them, particularly for capital estimation scenarios, allocate the specific scenario set to each assessor, taking into account knowledge base, scenario topic and applicable product/service/business activity/locational prerequisites, then engage administrative assistance to plan and schedule the actual assessments.
Establish required data attributes
The actual data you require out of the scenario assessment is once again determined by the objectives of the assessment. If you are assessing the scenario for business management purposes, in addition to the probable frequency, typical case and worst case impacts, the more valuable information is around the conditions which would be required to occur for such an event to materialise and what the business believes needs to be done to prevent the envisaged consequences from crystallising. You will probably also be very interested in reputational consequences.
If undertaking the scenario assessment for capital estimation purposes, you have little interest in non-financial consequence and remedial action; you are probably most interested in frequency, probability, time horizon, mean impact, median impact and some measure of worst case impact. Irrespective of objectives, decide on the required data attributes to be collected and ensure that the collection mechanism (form, spreadsheet or application) is preconstructed to ensure all appropriate attributes are collected.
Conduct the assessment
The manner in which you conduct the actual scenario assessment is also affected by the objective of the assessment exercise. If you are assessing scenarios for capital estimation purposes, you need to minimise bias and gaming, hence it may be preferable to conduct one-on-one interviews of each assessor using an experienced facilitator. Whereas if you are performing a business scenario assessment, a workshop attended by all interested parties to debate and discuss every aspect of the scenario is more beneficial. Irrespective of approach, document the responses received in a consistent manner and preferably provide assessors with some form of post-assessment feedback on the outcome. If performing scenario assessment for capital purposes, it may be useful to convene the individuals interviewed post the assessment and allow them to collectively discuss their views and thoughts on the scenarios assessed.
Use the information
Once you have all your assessment data, it is time to move into the scenario analysis phase, to analyse the data, to cross-check it against other internal data and external data sources and to ensure you do not have any data abnormalities. One useful measure is to compare typical case assessments against extreme or worst case assessments, as illustrated below for a capital assessment scenario involving several assessors. Such a comparison allows the identification of outliers and an assessment as to the validity of such outliers and the implications of those data points for your assessment. Overall, be consistent in the use of data and the manner in which it is collected.
