Are the Credit Rating Agencies To Blame ?

Some of the blame for the problems of the present credit crisis has been levelled at the credit rating agencies. In our opinion much of this criticism is justified, but some is not. The surrounding issues deserve some explanation.

What Is It That Credit Rating Agencies Do ? - What Is Done With The Results That They Publish ?

In essence, these agencies assess companies and estimate their creditworthiness. This information is then used by banks, other companies, and investors, to guide them in their investment and/or risk management decisions.

The companies that are assessed are then assigned a 'grade' (such as 'AAA' etc) based on this assessment. Different rating agencies use different schemes of grades, but it is largely possible to map between these schemes where necessary.

These grades provide a ranking of relative creditworthiness, so we can be reasonably confident that a company with a good credit rating will be a safer bet than one with a worse credit rating. It is when we move beyond this level of interpretation that the problems start and we need to take particular note of the 'small print'.

From Credit Grade to Probability of Default

When we calculate simple credit risk using quantitative models, the basic input that is required is not a credit grade, but a number. This number is the probability of default, or 'PD'. A PD of zero would mean that there was no chance of a counterparty defaulting, and a PD of 100% would mean that a counterparty would be certain to default.

To do our risk calculations, therefore we need a means of converting credit grades to PD figures. Unfortunately, this apparently simple step raises a number of issues that can trip up the unwary. There are statistical factors at work which need to be properly taken account of.

The crux of the problem is that we simply do not have the necessary information to perform these conversions exactly, and so we must use statistical techniques, and when this is done, the meaning of the results may be misunderstood.

Note: in the following I am not referring to any particular grading scheme, agency or target company, and the figures used have been chosen purely for the purposes of illustration.

One aspect of this problem is that for 'good' credit grades we are trying to estimate the probabilities of events that are relatively rare; therefore we cannot simply rely on empirical historic studies to calibrate the mappings between grades and default probabilities.

To illustrate this point, let's take an extreme case. Suppose that we use historic data over the past year to estimate the average PD for the companies in our very best credit grade - it is highly likely that there would have been no defaults at all - and our empirical PD would be zero. Obviously this is not a figure we would want to use for risk management purposes.

Even for grades where defaults do occur with a reasonably frequency, it is still not necessarily sensible to just use empirical measures for default probabilities; this is for two reasons.

Firstly, there will still be a 'sampling error' due to limitations on the available data, and this 'error bar' is often substantial relative to the value being estimated.

Secondly, as the small print on investment products points out: "Past behaviour is not necessarily an indicator of future performance". In particular, we must take account of the fact that default probability is sensitive to the prevailing economic conditions, and so we might need to adjust historic estimates to take account of this, and make them relevant for our future risk projections.

In a nutshell, empirical figures are conditional on the environment in which they were observed, and need to be treated as such.

Estimated PDs Should Always Be Treated With Caution

Hopefully it is now clear, that even the process of estimating simple default probabilities requires a degree of statistical analysis and modelling. The results of this type of process always have associated uncertainties or 'error bars'. These are not a result of poor analysis, but are inherent in the statistical nature of the available data. Unfortunately, there are a couple of 'stingers' which make this problem rather more acute:

The first problem is a consequence of the fact that financial institutions usually limit their exposures according to the perceived risks. This mean that exposure limits will largely reflect default probabilities (in an inverse fashion). The second problem is that in relative terms very low PDs are likely to be the ones with the largest 'error bars' and also the greatest sensitivity to economic fluctuations.

For example, if a bank set limits according to their estimated 'Expected Losses' - then, if a bank lent £1 Million to a company with an estimated PD of 1%, they might lend £1 Billion to a company with an estimated PD of 0.001%, believing these to have similar expected losses (of around £10,000). This is obviously simplistic, so what might go wrong ?

Let's suppose that underneath the statistical errors, the 'real' historical PDs were not 1% and 0.001%, but instead 2% and 0.01%. Now lets also suppose that the economy is about to weaken significantly, and taking this into account, the future probabilities will actually be 3% and 0.04%.

In these circumstances, the expected loss from the first loan is now: £30,000; but for the second it is now: £400,000.

The next complication arises because 'Expected Loss' is not really a very good measure of risk, we should also want to have a handle on 'Unexpected Loss'.

To be clear, in this context we are not trying to predict the unknowable, but instead estimate how likely various losses are beyond the E-L value, simply due to statistical fluctuations around the average value - or put another way, due to simple bad luck.

The reason for this should be obvious, suppose that we calculate our Expected Loss value, but then calculate that for one year in ten our losses might be five times this value. How much capital should we have to cover ourselves ?

Economic Capital Risk Models

Fortunately there are sensible ways of doing these calculations, which are often known as 'Economic Capital' models. However, understanding the results of these models requires some expertise, and the small print that should be attached to the results of these calculations is sometimes rather lengthy.

A crucial factor in portfolio management is diversification - this has the effect of reducing the overall risk arising from specific types of exposure. Statistical effects also come into play, significantly reducing the likelihood of large losses that extend well beyond the 'Expected Loss' figure.

However, there are two pieces of 'small print' that need to be 'circled in bold'. Firstly, some of the additional input parameters that are required by these models are very hard to estimate, and are much more uncertain than, for example, the PDs mentioned above. Secondly, the effect of changing these parameters - many of which should be regarded as 'conditional' - can have a dramatic effect on the estimated likelihood of losses that are a significant multiple of the Expected Value.

If these issues are not properly managed, then it is quite possible for these risk measures to increase by an order of magnitude due to a model recalibration. This type of situation can cause significant hand-wringing.

Known Unknowns and Unknown Unknowns - The Wisdom of Rumsfeld

Aside from the issues of interpretation of data, modelling and statistics, one of the main lesson is that:

Risk is really about what we don't know, not about what we do know.

And, of course, we don't know what it is that we don't know.

As Donald Rumsfeld once famously tried to point out: There are known unknowns, and there are unknown unknowns. Unfortunately, the good portfolio risk manager has to try to take account of both.

We should find out in the coming months which financial institutions have had really effective risk management in place, and which do not.

Other Material That We Have Put On-Line

To illustrate some of these points, we put together a simple 'on-line' portfolio credit risk model that performs portfolio risk calculations that run within a browser. This model is obviously only for the purposes of illustration.

• Portfolio Credit Risk Model.

For more information on 'Economic Capital', see our on-line presentations area:

• Our On-line Presentations.