M. Cruz[1]
has suggested using the Kupiec Test to backtest operational Risk VaR
measurement results. However,
this test seems to be adapted to market risk and not to operational risk for 2
reasons. The first reason is that critical
values of the test were calculated for the 99% quantile, the second one - in
operational risk we do not have daily operational risk VaR measures but a VaR
assessment for the coming year conducted monthly or even quarterly.
In order to overcome this difficulty we can
develop 2 approaches. The first one is to modify the Kupiec test; in the second we will use the binomial approach to check for the frequency of exceptions.
Kupiec test[2]
Let us start with the modified Kupiec
test. We need to make 2 adjustments.
First, we have to increase the number of observations. As for the VaR
calculation:
- we have assessed the parameters of the frequency and severity distributions for each cluster,
- obtained the loss distribution for each cluster,
- calculated the value of the Pearson’s correlation coefficient (or some other measures of dependency),
-than calculated daily VaR instead of VaR for one year's period.
We have observations of losses occurring on consecutive days (from 255 to 365 observations depending if we include or not weekends and other holidays as days of losses occurrence). In this way, we have increased our number of observations. Doing this we have also made the assumption that yearly VaR results can be backtested using daily data. This assumption understates that there is no particular or meaningful importance difference (for the test) between different days of the week/month/year. The second adjustment consists in increasing the VaR confidence level to 99.9%.
- we have assessed the parameters of the frequency and severity distributions for each cluster,
- obtained the loss distribution for each cluster,
- calculated the value of the Pearson’s correlation coefficient (or some other measures of dependency),
-than calculated daily VaR instead of VaR for one year's period.
We have observations of losses occurring on consecutive days (from 255 to 365 observations depending if we include or not weekends and other holidays as days of losses occurrence). In this way, we have increased our number of observations. Doing this we have also made the assumption that yearly VaR results can be backtested using daily data. This assumption understates that there is no particular or meaningful importance difference (for the test) between different days of the week/month/year. The second adjustment consists in increasing the VaR confidence level to 99.9%.
The adapted Kupiec Test (proportion of failure
test) will allow to measure if the number of failures is consistent with the
confidence level.
The null
hypothesis:
Where p represents the
proportion of failures if we have computed the VaR at required confidence level.
represents the observed proportion of exceptions that is the number (x) of exceptions divided by the number of days (N) in a one year's period.
The alternative hypothesis is of course:
The test statistics takes the form:
If the
model is correct (null hypothesis), the above test statistics should be
asymptotically distributed according to a chi-squared distribution function with one degree of freedom.
The
critical values of the above test are for 95% and 90% confidence level
respectively 3.84% and 2.71%. As the VaR critical level is extremely high, the
power of both test should be low.
If we take the
95% confidence level for the test:
Table 1: Kupiec test with confidence level =
95%
Probability level
|
VaR confidence level
|
Non rejection interval
for number of exceptions equal to x
|
||
N=255 days
|
N=365 days
|
N=510 days
|
||
0.001
|
99.9%
|
x<2
|
x<3
|
x<3
|
0.01
|
99%
|
x<7
|
x<8
|
1<x<11
|
If we take the
90% confidence level for the test:
Table 2: Kupiec test with confidence level = 90%
Probability level
|
VaR confidence level
|
Non rejection interval
for number of exceptions equal to x
|
||
N=255 days
|
N=365 days
|
N=510 days
|
||
0.001
|
99.9%
|
x<2
|
x<2
|
x<3
|
0.01
|
99%
|
x<5
|
x<7
|
1<x<9
|
For VaR 99.9% confidence level, the test is
unable to detect too much conservative models, for VaR 99% it is possible provided
that the number of observations is sufficiently high (i.e. min 439 for VaR 99%).
The binomial approach:
The second
approach will be to use the binomial distribution for comparing the associated
VaR confidence level with the obtained number of exceptions and confidence level
of the test.
To make it clear: suppose that we did not obtained
any exception out of 255 observations, for the 95% confidence level for the binomial distribution,
the associated VaR confidence level is equal to exp[ln(0.05)/255)] =98.832%. In
other words, the test does not reject the hypothesis that the VaR confidence
level is 98.832% or more.
If we have found a single exception the
confidence level will be the solution of the following equation:
In this way we
can obtain the following table:
Table 3: Power of the test
Number of
exceptions
|
Corresponding
VaR level
|
||
N=255 days
|
N=365 days
|
N=510 Days
|
|
X=0
|
98.832%
|
99.183%
|
99.414%
|
X=1
|
98.153%
|
98.706%
|
99.073%
|
X=2
|
97.551%
|
98.285%
|
98.77%
|
The
critical value for VaR=99.9% are:
Table 4: Critical values of the test
Number of
observations:
|
N=255 days
|
N=365 days
|
N=510 days
|
Maximum number
of exceptions
|
0
|
1
|
1
|
In order to maximize the discriminating power
of the test I slightly decreased the confidence level of the test from 95% to
94.77%. This modification reduces the tolerated number of exceptions by 1 in
the case of 365 observations. If we accept 1 exception for 255 observations the
probability of incorrect rejection of the null hypothesis will be reduced to 2.76%.
All critical values of the test for the 95% confidence level were calculated using the following inequality:
All critical values of the test for the 95% confidence level were calculated using the following inequality:
Correction:
We can make an adjustment for the realized
value of daily losses. A smaller than 5% probability of observing a loss above a given value
of VaR is shown in table 5. This allows us to check if the loss is compatible with the model.
Table 5: Critical values for the maximum amount
of daily losses (alpha=5%)
N=255 days
|
N=365 days
|
N=510 days
|
|
The maximum observed loss should not exceed the following quantile of the calculated daily VaR:
|
99.8604%
|
99.9025%
|
99.9657%
|
A practical example:
In order to backtest the yearly VaR,
we have calculated the amount of losses for each day of the year and compared
the outcome with the daily VaR. We obtained 1 exception that is 1 loss
exceeding the daily VaR and 364 below this figure. According to table 4 this
result does not allows us to reject the hypothesis that VaR was calculated with
99.9% confidence level. Using table 3 we
can also reject alternative hypothesis stating that VaR was calculated with
confidence level not exceeding 98.7%. However, the exception was corresponding
to confidence level of daily VaR equal to 99.91%. This figure was above the
maximum value given in table 5 and it means that the model should be rejected.
If we have used the adapted to operational risk
Kupiec test, the observed number of exceptions (1) will not be sufficient to
reject the hypothesis. Consequently we will probably accept the VaR measure result.
Conclusion:
The utilization of the Kupiec test in order to
backtest operational risk VaR results is possible but the model dramatically
lacks of power. Instead using an approach based on binomial distribution seems
more promising and is easier to implement. The proposed test is also easier to
put into practice.
Robert M. Korona, PhD
[1] Marcelo G. Cruz Modelling (2002), Measuring and Hedging Operational Risk
Wiley and Sons,
[2] Kupiec, Paul H. (1995), „Techniques for Verifying
the Accuracy of Risk Management Models”, Journal
of Derivatives, 7, p. 41-52.
TEXT IN PDF:
https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxsZGFzbWFsbGJhbmtzfGd4OjFhNjczZDI1ZDA2MTk1OWE&pli=1
Brak komentarzy:
Prześlij komentarz