Backtesting operational risk VaR results

M. Cruz[1] has suggested using the Kupiec Test to backtest operational Risk VaR measurement results. However, this test seems to be adapted to market risk and not to operational risk for 2 reasons.  The first reason is that critical values of the test were calculated for the 99% quantile,  the second one - in operational risk we do not have daily operational risk VaR measures but a VaR assessment for the coming year conducted monthly or even quarterly.

In order to overcome this difficulty we can develop 2 approaches. The first one is to modify the Kupiec test; in the second we will use the binomial approach to check for the frequency of exceptions.

Kupiec test[2]

Let us start with the modified Kupiec test.  We need to make 2 adjustments. First, we have to increase the number of observations. As for the VaR calculation:
- we have assessed the parameters of the frequency and severity distributions for each cluster,
- obtained the loss distribution for each cluster, 
- calculated the value of the Pearson’s correlation coefficient (or some other measures of dependency), 
-than calculated daily VaR instead of VaR for one year's period. 

We have observations of losses occurring on consecutive days (from 255 to 365 observations depending if we include or not weekends and other holidays as days of losses occurrence). In this way, we have increased our number of observations. Doing this we have also made the assumption that yearly VaR results can be backtested using daily data. This assumption understates that there is no particular or meaningful importance difference (for the test) between different days of the week/month/year. The second adjustment consists in increasing the VaR confidence level to 99.9%.

The adapted Kupiec Test (proportion of failure test) will allow to measure if the number of failures is consistent with the confidence level.

The null hypothesis:

Where p represents the proportion of failures if we have computed the VaR at required confidence level.

represents the observed proportion of exceptions that is the number (x) of exceptions divided by the number of days (N) in a one year's period.   

 
 The alternative hypothesis is of course:

   

 The test statistics takes the form:  

If the model is correct (null hypothesis), the above test statistics should be asymptotically distributed according to a chi-squared distribution function with one degree of freedom.

The critical values of the above test are for 95% and 90% confidence level respectively 3.84% and 2.71%. As the VaR critical level is extremely high, the power of both test should be low.

If we take the 95% confidence level for the test:

Table 1: Kupiec test with confidence level = 95%

Probability level	VaR confidence level	Non rejection interval for number of exceptions equal to x
		N=255 days	N=365 days	N=510 days
0.001	99.9%	x<2	x<3	x<3
0.01	99%	x<7	x<8	1<x<11

  

If we take the 90% confidence level for the test:

Table 2: Kupiec test with confidence level = 90%

Probability level	VaR confidence level	Non rejection interval  for number of exceptions equal to x
		N=255 days	N=365 days	N=510 days
0.001	99.9%	x<2	x<2	x<3
0.01	99%	x<5	x<7	1<x<9

For VaR 99.9% confidence level, the test is unable to detect too much conservative models, for VaR 99% it is possible provided that the number of observations is sufficiently high (i.e. min 439 for VaR 99%).

  

The binomial approach:

The second approach will be to use the binomial distribution for comparing the associated VaR confidence level with the obtained number of exceptions and confidence level of the test.

To make it clear: suppose that we did not obtained any exception out of 255 observations, for the 95% confidence level for the binomial distribution, the associated VaR confidence level is equal to exp[ln(0.05)/255)] =98.832%. In other words, the test does not reject the hypothesis that the VaR confidence level is 98.832% or more.

If we have found a single exception the confidence level will be the solution of the following equation:

In this case the solution p is equal to 98.153%.

In this way we can obtain the following table:

Table 3: Power of the test

Number of exceptions	Corresponding VaR level
	N=255 days	N=365 days	N=510 Days
X=0	98.832%	99.183%	99.414%
X=1	98.153%	98.706%	99.073%
X=2	97.551%	98.285%	98.77%

The critical value for VaR=99.9% are:                      

Table 4: Critical values of the test

Number of observations:	N=255 days	N=365 days	N=510 days
Maximum number of exceptions	0	1	1

In order to maximize the discriminating power of the test I slightly decreased the confidence level of the test from 95% to 94.77%. This modification reduces the tolerated number of exceptions by 1 in the case of 365 observations. If we accept 1 exception for 255 observations the probability of incorrect rejection of the null hypothesis will be reduced to 2.76%.

All critical values of the test for the 95% confidence level were calculated using the following inequality:

Correction:

We can make an adjustment for the realized value of daily losses.  A smaller than 5% probability of observing a loss above a given value of VaR is shown in table 5. This allows us to check if the loss is compatible with the model.

Table 5: Critical values for the maximum amount of daily losses (alpha=5%)

	N=255 days	N=365 days	N=510 days
The maximum observed  loss should not exceed the following quantile of the calculated daily VaR:	99.8604%	99.9025%	99.9657%

A practical example:

In order to backtest the yearly VaR, we have calculated the amount of losses for each day of the year and compared the outcome with the daily VaR. We obtained 1 exception that is 1 loss exceeding the daily VaR and 364 below this figure.  According to table 4 this result does not allows us to reject the hypothesis that VaR was calculated with 99.9% confidence level.  Using table 3 we can also reject alternative hypothesis stating that VaR was calculated with confidence level not exceeding 98.7%. However, the exception was corresponding to confidence level of daily VaR equal to 99.91%. This figure was above the maximum value given in table 5 and it means that the model should be rejected.

If we have used the adapted to operational risk Kupiec test, the observed number of exceptions (1) will not be sufficient to reject the hypothesis. Consequently we will probably accept the VaR measure result.

Conclusion:

The utilization of the Kupiec test in order to backtest operational risk VaR results is possible but the model dramatically lacks of power. Instead using an approach based on binomial distribution seems more promising and is easier to implement. The proposed test is also easier to put into practice.

Robert M. Korona, PhD

---

[1] Marcelo G. Cruz Modelling (2002), Measuring and Hedging Operational Risk Wiley and Sons,

[2] Kupiec, Paul H. (1995), „Techniques for Verifying the Accuracy of Risk Management Models”, Journal of Derivatives, 7, p. 41-52.

TEXT IN PDF:
https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxsZGFzbWFsbGJhbmtzfGd4OjFhNjczZDI1ZDA2MTk1OWE&pli=1

Another approach to composite indicators

One of the issues with composite indicators is to ensure that it can be as flexible to changes of risk level as ordinary KRIs. Till now proposed composite indicators fulfil this condition when threshold(s) value(s) is (are) exceeded but are unable to give useful information when individual KRI remains in the acceptable (green) zone. I propose a modification of Taylor approach in order to overcome this difficulty.

Original Taylor Model:

Charles Taylor has proposed[1] a way to derive a simple composite indicator.  His approaches consist of 2 steps:

1)    Transforming any KRI into what he calls a T-value,

2)    Combining as many T-values as we want into a composite.

In order to derive the T-value of an KRI realisation (noted as X) he defines 4 threshold  values (L₁, L₀, H₀, H₁) with L_1< L_0<H₀<H_1. 

     T = 1 when L₀< X < H₀                                                                      (1)

T = (X+L₁-2L₀)/ (L₁-L₀) when X < L₀                                        (2)

T = (X+H₁-2H₀) / (H₁-H₀) when X > H₀                                  (3)

The first case (T equal to 1) corresponds to the “green zone” that means acceptable risk and no need to take corrective action

A value of T below L₁ or above H₁ corresponds to the “red zone”_, unacceptable risk and urgent necessity to take corrective action.

The remaining cases L₁< X < L₀   and  H_o< X < H₁ correspond to the “yellow zone” generally considered as the intermediate case[2].

After establishing T value for each indicator, he proposed the following par of equation for composite indicators:

I_rel = max {1 , α[T(1)^w(1)T(2)^w(2)]^β}                              (4)

I_unrel = max {1 , α max[T(1)^w(1);T(2)^w(2] ^β }                  (5)

With w(i) representing the weight of the „i” indicator, α and β adjustment parameters. Equation (4) is used for related indicators and equation (5)   for unrelated indicators.

Taylor approach seems very attractive as it permits to calculate easily composite indicator. However, it flattens all green values of KRI to a single T-value equal to 1.  In this way unfortunately the T-value is equal for a KRI in the middle of the “green” zone (safe) and for a KRI close to the “yellow “zone”. Moreover this T-value does not convey the information which can be obtained by analysing a trend in the green zone (see figure below).

The proposed solution

My suggestion is to keep the Taylor broad framework but to modify the T-value definition.


Let us:

     1)    consider that realisation of KRI variable are always positive numbers,

2)    define 2 critical and 2 threshold values for a KRI determining the “red”, “yellow” and “green” zone[3],

X_c,low<X_t,low<X_t,high<X_c,high

3)    define by X a realization of KRI,

4)    define the t-value for KRI in the following way:

If X≤ X_c,low

T=2+ X_c,low / X                                                      (6a)

If X_c,low≤X≤X_t,low

T=2+ (X_t,low –X)/(X_t,low –X_c,low )                              (6b)

If X_t,low≤X<X_t,high

T=max{1+(X_t,low /X);   1+(X/ X_t,high)}                      (6c)

If X_t,high≤X≤X_c,high

T=2+(X–X_t,high)/(X_c,high –X_t,high)                              (6d)

If X≥X_c,high

T=2+(X/X_c,high)                                                     (6e)

If there is only one (upper) threshold value and one (upper) critical value then  equations (6d) - (6e) will remain and equation (6c) will be replaced by:

 T=1+(X/ X_t,high)  with X≤X_t,high                                                         (6f)  

Please note that now we can distinguish between KRI belonging to the safe “green zone”. 
 All T-values are above one. As a consequence Taylor equation (4) and (5) may be simplified to:

I’_rel= α’[T(1)^w(1)T(2)^w(2)]^β’                                    (7)

I”_unrel= α” max[T(1)^w(1);T(2)^w(2] ^β”}                       (8)

As the above T-values are different from those obtained by Taylor, parameters (α’, β’) and (α”, β”) should be recalibrated by using boundary conditions. Weights w(i) can remain the same.

To clarify, let us suppose that we have 2 related KRI which may take value from 1 onwards. Then the key question is how should be consider a situation when the first of the KRI reaches its critical value and the second remains at its optimal level.

We will have the first equation:

N1 = α’[3^w(1)1^w(2)]^β’  = α’[3^w(1)]^β’                                             (9)

where N1 means the first number .

To obtain the second equation we will make the opposite hypothesis:

N2= α’[3^w(1)1^w(2)]^β’  = α’[3^w(2)]^β’= α’[3^1-w(1)]^β’                       (10)

where N2 means the second number.

To obtain another equation (if needed) we can consider a combination of one parameter reaching the threshold value and the second remaining at its optimal level (equal to 1).

 N3 = α’[2^w(1)1^w(2)]^β’  = α’[2^w(1)]^β’                                            (11)

where N3 represents the third number.

Having 3 equations and 3 unknown parameter values we can solve the equation system. If we add another condition we may face an optimization problem as probably some of the conditions will be only partially met.

With 3 individual KRIs we will need 4 equations, with n KRIs we will need (n+1) equations obtained in a similar way.

A similar approach can be developed for the case of unrelated KRI.

Conclusion

Amending the framework previously developed by Taylor, we can obtain composite indicators based on T-value sensible to the variation of KRI even in the green zone.

Robert M. Korona, PhD

TEXT IN PDF
https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxsZGFzbWFsbGJhbmtzfGd4OjNjMzZkODFkNDQ4ZGYxNjQ

---

[1] Charles Taylor (2006)  Composite Indicators: Reporting KRIs to Senior Management, The RMA Journal , April 2006 , p 16-20

[2] In general in this situation one should consider the necessity to take preventive action before KRI reaches an unacceptable value (reaches the “red zone”).

[3] There is no need to define 2 critical and threshold values. If the KRI is monotonically related to risk one critical and one threshold value are quite enough.