Critique for Thompsons Solicitors of the report produced by Frontier Economics
“Outcomes for legally represented and unrepresented claimants in personal injury compensation a report to the Association of British Insurers”Brian Critchley
Notes on the author of this critique
Background research
Data & Analysis
Annex: calculations and workings used for critique
Executive Summary
This critique approaches the report with some caution, given that it is the subject of political lobbying and media spin. The main part of the critique concentrates on the actual research design, method and analysis.
The design of the research has ignored important variables that have been highlighted in previous research in this area as influencing outcomes. Frontier Economics has admitted that these were not used in the analysis of their data set, even though they were part of the survey.
While the report claims that comparisons have been done on a like for like basis, the critique shows that this has not been done, even though claim and claimant characteristics were asked for in the survey. Instead the comparisons are made purely on actual money received within each claim group rather than whether the claims are actually comparable, as they do not take into account problems such as type and severity of injury or damage. In the language of basic research or statistical design, this is a classic case of comparing apples with pears.
In this way the results are unreliable and thus make for meaningless conclusions.
Even if the results were reliable, the statistical analysis has some important flaws. After some criticism of the results, Frontier Economics firstly provided an annex of further results and then more recently ‘corrections’ to their data which solved some of the wilder anomalies but still left the analysis with crude averages and irregular groupings of observations meaning that whole sections of the results are unreliable. The ‘corrections’ also contained further errors.
There are other sources of research detailed below that have very different conclusions and also look at qualitative issues. These should be considered in conjunction with the criticism based on the lack of ability to actually perform ‘like for like’ comparisons in the research in reviewing its validity.
This report has also been published by Which?. Download the PDF file (72kb).
Notes on the author of this critique
Brian Critchley lectures in both the Business School and the faculty of Law, Governance & International Relations at London Metropolitan University. His research interests focus mainly on issues of social justice, employment and representation. Brian also undertakes work independently and has worked with the Trades’ Union Congress, Communication Workers’ Union, National Union of Teachers and Communications International (now UNI).
Background research
The report gives very little attention to work that has been done in this area. It does not identify other relevant research of both quantitative and qualitative nature that differs significantly in conclusions from the reports own conclusions. This is an extremely important area, as it will identify issues that have already arisen and proved important factors in influencing previous research results and thus must be taken into account when constructing the design of any further research. Independent research would also want to challenge findings that contradict with its own results and this is ignored.
Examples of such studies would include:
- An APIL survey1 that identifies
- where settlement offers are made to unrepresented claimants, they are generally 50 per cent less;
- average time spent on a case was 16.82 hours;
- 73 per cent of respondents indicated that there were issues of complexity within the case;
- 69 per cent of respondents’ case load were general damages of less then £5000;
- 63 per cent of cases did not go on to issue proceedings.
1. This research was carried out among Association of Personal Injury Lawyers (APIL) members in January and February 2005. 782 completed, or partially completed, surveys were completed. These surveys contained quantitative data in respect of 2,274 settled cases of personal injury, 2,242 of them with a final general damages award below £5,000.It showed that for cases valued at up to £5,000, the difference between the first offer made and the final settlement for the cases analysed was, on average, almost £1,000. This represents an average increase of around 50% from the first offer to the final settlement. This suggests that if claimants were to accept the insurer's first offer, without the aid of legal help, they could be dramatically under compensated.
- Work by Professor Richard Moorhead2 that highlights issues around access to justice when claimants are not represented and highlights important research design issues such as claim and claimant characteristics3.
- Thompsons’ survey evidence4 that suggests that two thirds of all respondents believe that without union legal service representation, their case would not have been handled fairly.
- Mori poll evidence for APIL5 that suggests that:
- 73% of respondents would be unable to work out the value of their claim in relation to personal injury if left to their own devices;
- and 80% of those respondents also felt that they would not be confident that what they would be offered by insurers would necessarily be the correct and proper level of compensation justified by their claim.
The Frontier Economics report does recognise some literature that identifies claim and claimant characteristics as important factors6, and there are questions that attempt to identify these in their survey; however, either the data was too onerous to collect or it has been ignored. This is a very serious drawback and undermines the ability of the work to claim that it “allows an initial view to be developed about the empirical relationship between representation and compensation”. In fact, the use of ‘initial view’ may be the author’s way of recognising the problem of not accounting for these variables and hinting that further work needs to done to account for them.
2. Moorhead R (2005) Litigants in person: ghosts in the machine Legal Action (2005) 8-9; Moorhead R (2003) Access or aggravation? Litigants in person, McKenzie friends and lay representation Civil Justice Quarterly, 22 (Mar) (2003) 133-155; Moorhead R (2005) Litigants in Person: Unrepresented litigants in first instance proceedings; http://www.dca.gov.uk/research/2005/2_2005.htm for the Department of Constitutional Affairs
3. Claim and claimant characteristics are raised on p.8 of the report, only to say that further research could be done using these.
4. “Small Claims Big Deal” was a survey of UNISON members carried out in March 2005.
5. From Allan Gore QC (APIL) giving evidence to the House of Commons Select Committee on Constitutional Affairs 11 October 2005
6. Browne & Schmit (1998) Insurance Claims and Legal Filings: Trends from 1977-1997; Connelly et al (2005) Insurance Claim Propensity and Claim Size for Injured Road Users; Fenn at al (2006) The Funding of Personal Injury Litigation: comparisons over time and across jurisdictions.
In designing the research these factors should have played a significant part in identifying that the data should have been broken down further to control for different claim and claimant characteristics. This means that any comparison of represented and unrepresented claims is done purely on the value of compensation rather than of similar types of claim. The only measure that is being made is the average of how much net compensation is awarded to represented and unrepresented claimants and the lifecycle of the case.
This effectively means that you cannot measure the impact of representation, as it is not a properly controlled variable. The results are therefore meaningless as there cannot be a ‘like for like’ comparison of claims.
Example:
An unrepresented claimant of a particular age group and particular type of background received £6,000 net compensation and the lifecycle of the case is 650 days for a specific severity of a specific type of injury.
For this to be compared in order to measure the effect of representation, it would have to be compared to a level compensation and case lifecycle in a claim where a represented claimant shared all these claimant and claim characteristics.
The report should also have ensured that extra data was examined for evaluating the outcomes of represented and unrepresented unsuccessful claims and the omission of this again means that the research is very unfocussed and open to skewing the real impact of representation even if claim and claimant characteristics were controlled for.
The report also does not locate the research within the wider context and thus leaves it open to accusations of bias:
- The research coincides with moves, backed by the insurance industry, to raise the threshold for recovery of legal costs in personal injury claims from £1000 to £2,500, which covers the majority of claims. The ABI are campaigning to raise it to an even higher figure of £5000.
Data & Analysis
With the caveat that the comparison of sums of money without knowing whether they are from similar types of claims is going to lead to results that are not reliable, even if we look at the data there are some serious causes for concern.
Frontier Economic will not release the data set, citing confidentiality. This makes it extremely difficult to perform any serious analysis; however, this critique of the data and analysis will be based on what data can be gleaned from the report and the subsequent corrections, ignoring some obvious mistakes in the new data that was supplied.
The actual data used represents 92,934 claims not all the 112,634 claims in the data set, as the report does not look at claims under £1,000. As this lower figure is the only data that has been used to produce results, this critique will generally take it as 100% of observations, as this is more kind to the results; however, at points it is interesting to compare sample sizes with the wider data set. These are then crudely averaged before being broken down into subsections of different value bands. On the basis of the crude averages some conclusions are drawn which are extremely difficult to justify; although, interestingly, the figures in the post publication annex show a small benefit for claimants that are represented in the overall average for claims between £1,000 and £5,000 – there is no comment on this and, of course, it shows the opposite of what the ABI would want to find to back up their campaign on the small claims limit.
When the results for the different bands are looked at, there are also some results that are not made clear in the executive summary and press release and are also problematic for the research design. For instance, there is cause for concern when the annex that was only published after requests for more detail, states that: “Across all motor and public liability claims sub-samples, there is an empirical relationship between legal representation and higher compensation payments for all three compensation value bands.”(p.28 in annex) but the executive summary of the published report states that: “there appears to be no empirical link, on average, between compensation received by represented and unrepresented individuals.”(p.3) While the second statement is technically correct as it refers to the average, it should be qualified by the reports own findings. This method of finding this average figure also highlights some serious shortcomings.
The report shows that for motor claims, there is an empirical relationship between representation and higher compensation for all the value bands except when claims for over £15,000 are included. This raises an important issue: why does this £15,000 to £25,000 category cover a range of £10,000 whereas the lower categories are measured at £5,000 intervals and the number of observations at this higher level is comparatively low (just over 3% of all motor claims and also just under 3% of all claims above £1,000 – see table below), making the results less reliable.
| Claim grouping observations as a percentage of sample size and overall data set | ||
|---|---|---|
| Of claims over £1000 | Of overall data set | |
| All claims between £1000 & £250000 | 100.00% | 82.51% |
| Claims between £1000 & £5000 | 55.21% | 45.55% |
| Claims between £5000 & £10000 | 32.26% | 26.62% |
| Claims between £10000 & £15000 | 9.55% | 7.88% |
| Claims between £15000 & £25000 | 2.98% | 2.46% |
If we look at these sample sizes as percentage of the overall data set, it looks even worse
| Breakdown of sample sizes as a percentage of the total data set including observation under £1,000 | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
|
All claims between £1000 & £250000 |
66.986% | 15.524% | 57.952% | 15.128% | 7.718% | 0.222% | 1.315% | 0.174% |
| Claims between £1000 & £5000 | 37.163% | 8.390% | 30.313% | 8.106% | 5.928% | 0.152% | 0.923% | 0.132% |
| Claims between £5000 & £10000 | 21.664% | 4.953% | 20.054% | 4.881% | 1.331% | 0.042% | 0.279% | 0.031% |
| Claims between £10000 & £15000 | 6.328% | 1.550% | 5.906% | 1.523% | 0.336% | 0.017% | 0.085% | 0.010% |
| Claims between £15000 & £25000 | 1.830% | 0.631% | 1.680% | 0.617% | 0.123% | 0.012% | 0.028% | 0.002% |
R - Represented
U - Unrepresented
The Frontier Economics report uses bands that overlap (1,000-5,000, 1,000-10,000 & 1,000-15,000) justifying this as a means to trap data that may be comparable between bands. While the critique has already shown that comparability has not been controlled for in any meaningful sense, it is useful to break down the observations by their different value intervals sequentially to calculate the number of actual observations and check reliability.
For employer liability this problem is also evident in two of the value bands with both represented and unrepresented claims within the £15,000 to £25,000 band representing only 1.69% of observations in this category and both represented and unrepresented claims within the £10,000 to £15,000 representing only 4.45% of observations.
| Sample size as percentage of its own claim grouping | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 81.19% | 18.81% | 79.30% | 20.70% | 97.20% | 2.80% | 88.30% | 11.70% |
| Claims between £1000 & £5000 | 45.04% | 10.17% | 41.48% | 11.09% | 74.66% | 1.91% | 61.95% | 8.85% |
| Claims between £5000 & £10000 | 26.26% | 6.00% | 27.44% | 6.68% | 16.76% | 0.53% | 18.75% | 2.05% |
| Claims between £10000 & £15000 | 7.67% | 1.88% | 8.08% | 2.08% | 4.24% | 0.21% | 5.72% | 0.66% |
| Claims between £15000 & £25000 | 2.22% | 0.76% | 2.30% | 0.84% | 1.54% | 0.15% | 1.89% | 0.14% |
R - Represented
U - Unrepresented
It is here that another anomaly appeared in the higher band range; in the published report, the number of samples for unrepresented cases was extremely high and out of proportion with the general balance between represented and unrepresented cases. The 57:94 split in the report represented a ration that is seriously at odds with the stated general split in this category, which is around 97:2. When this was pointed out, Frontier provided new figures (see table below) that produced different results and gave us a ratio of 91:9; although the new Frontier report cites this as around 94:6 which cannot be correct if the overall numbers given for the other claim value bands is correct.
| Percentages of represented and unrepresented claims based on different claim groups and value groupings | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 81% | 19% | 79% | 21% | 97% | 3% | 88% | 12% |
| Claims between £1000 & £5000 | 82% | 18% | 79% | 21% | 98% | 3% | 88% | 13% |
| Claims between £5000 & £10000 | 81% | 19% | 80% | 20% | 97% | 3% | 90% | 10% |
| Claims between £10000 & £15000 | 80% | 20% | 80% | 21% | 95% | 5% | 90% | 10% |
| Claims between £15000 & £25000 | 74% | 26% | 73% | 27% | 91% | 9% | 93% | 7% |
| Total as percentage of data used | 100% | 89% | 10% | 2% | ||||
R - Represented
U-Unrepresented
A good test of whether results are reliable is to see if the pattern remains when a few observations are removed, if this was done here then the conclusions would show that there is, on average, an empirical relationship between representation and higher compensation across both public liability and motor claims.
In fact, in all three claims groups, this higher band provides significant statistical problems due to the small number of observations. On the published results the analysis also produces a minus figure for the number of public liability represented claims. This has now been updated with the new figures from Frontier Economics but, as with the previous problems, it does not inspire confidence. In fact, at the meeting held with Thompsons Solicotors, ABI and Frontier Economics, it was Frontier who continually stressed the importance of having a high number of observations in gaining reliable results and from the table below we can see that for the higher band we have respectively only 13 and 2 unrepresented claims as observations for the employer and public liability categories.
The following tables show the actual numbers of observations (claims) based on the figures in the report.
| Actual numbers of observations for value grouping (broken down further) | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 75448 | 17486 | 65274 | 17039 | 8693 | 250 | 1482 | 196 |
| Claims between £1000 & £5000 | 41859 | 9450 | 34142 | 9131 | 6677 | 171 | 1040 | 149 |
| Claims between £5000 & £10000 | 24401 | 5579 | 22587 | 5498 | 1499 | 47 | 315 | 34 |
| Claims between £10000 & £15000 | 7127 | 1746 | 6653 | 1715 | 379 | 19 | 96 | 11 |
| Claims between £15000 & £25000 | 2062 | 710 | 1892 | 695 | 138 | 13 | 32 | 2 |
| Total | 92934 | 82313 | 8943 | 1678 | ||||
Lastly, the inclusion of a section looking at legal costs as a proportion of compensation is not logical, as it seems irrelevant to the terms of reference of the report and is not part of the main body of research, as this looks at net compensation after legal costs and other disbursements have been taken from the overall figure. So it begs the question, why is it there? The obvious conclusion would be that it is there because they like the figures and think that the ABI would like them as well, given the ABI campaign to raise the small claims limit. This raises questions about the independence of the research.
Annex: calculations and workings used for critique
| Number of Observations | |||||
|---|---|---|---|---|---|
| Motor | Employer Liability | Public Liability | Total Sample | Percentage | |
| All | 99498 | 10660 | 2476 | 112634 | 100.00% |
| All claims above £1000 | 82313 | 8943 | 1678 | 92934 | 82.51% |
| All claims between £1000 & £5000 | 43273 | 6848 | 1188 | 51309 | 45.55% |
| All claims between £1000 & £10000 | 71358 | 8394 | 1537 | 81289 | 72.17% |
| All claims between £1000 & £15000 | 79726 | 8792 | 1644 | 90162 | 80.05% |
| Note that the statistics are only performed on claims above £1000, therefore the actual data being used in 83% of the overall figure, or 92,934 observations rather than the full data set of 112,634. | |||||
| Number of Observations within each value grouping | |||||
|---|---|---|---|---|---|
| Motor | Employer Liability | Public Liability | Total Sample | Percentage | |
| All claims between £1000 & £250000 | 82313 | 8943 | 1678 | 92934 | 100.00% |
| Claims between £1000 & £5000 | 43273 | 6848 | 1188 | 51309 | 55.21% |
| Claims between £5000 & £10000 | 28085 | 1546 | 349 | 29980 | 32.26% |
| Claims between £10000 & £15000 | 8368 | 398 | 107 | 8873 | 9.55% |
| Claims between £15000 & £25000 | 2587 | 151 | 34 | 2772 | 2.98% |
| Percentages slit between represented and unrepresented claims | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All | 76 | 24 | ||||||
| All claims above £1000 | 81 | 19 | 79.3 | 20.7 | 97.2 | 2.8 | 88.3 | 11.7 |
| All claims between £1000 & £5000 | 82 | 18 | 78.9 | 21.1 | 97.5 | 2.5 | 87.5 | 12.5 |
| All claims between £1000 & £10000 | 81 | 19 | 79.5 | 20.5 | 97.4 | 2.6 | 88.1 | 11.9 |
| All claims between £1000 & £15000 | 81 | 19 | 79.5 | 20.5 | 97.3 | 2.7 | 88.2 | 11.8 |
R - Represented
U - Unrepresented
| Actual number of observations (claims) by value bands | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All | 85602 | 27032 | ||||||
| All claims above £1000 | 75277 | 17657 | 65274 | 17039 | 8693 | 250 | 1482 | 196 |
| All claims between £1000 & £5000 | 42073 | 9236 | 34142 | 9131 | 6677 | 171 | 1040 | 149 |
| All claims between £1000 & £10000 | 65844 | 15445 | 56730 | 14628 | 8176 | 218 | 1354 | 183 |
| All claims between £1000 & £15000 | 73031 | 17131 | 63382 | 16344 | 8555 | 237 | 1450 | 194 |
R - Represented
U - Unrepresented
|
Actual numbers of observations for value grouping (broken down further) |
||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 75448 | 17486 | 65274 | 17039 | 8693 | 250 | 1482 | 196 |
| Claims between £1000 & £5000 | 41859 | 9450 | 34142 | 9131 | 6677 | 171 | 1040 | 149 |
| Claims between £5000 & £10000 | 24401 | 5579 | 22587 | 5498 | 1499 | 47 | 315 | 34 |
| Claims between £10000 & £15000 | 7127 | 1746 | 6653 | 1715 | 379 | 19 | 96 | 11 |
| Claims between £15000 & £25000 | 2062 | 710 | 1892 | 695 | 138 | 13 | 32 | 2 |
| Total | 92934 | 82313 | 8943 | 1678 | ||||
R - Represented
U - Unrepresented
| Percentages of claims based on value groupings | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 81% | 19% | 79% | 21% | 97% | 3% | 88% | 12% |
| Claims between £1000 & £5000 | 82% | 18% | 79% | 21% | 98% | 3% | 88% | 13% |
| Claims between £5000 & £10000 | 81% | 19% | 80% | 20% | 97% | 3% | 90% | 10% |
| Claims between £10000 & £15000 | 80% | 20% | 80% | 21% | 95% | 5% | 90% | 10% |
| Claims between £15000 & £25000 | 74% | 26% | 73% | 27% | 91% | 9% | 93% | 7% |
| Total as percentage of data used | 100% | 89% | 10% | 2% | ||||
R - Represented
U - Unrepresented
| Sample size as percentage of its own value claim grouping | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 81.19% | 18.81% | 79.30% | 20.70% | 97.20% | 2.80% | 88.30% | 11.70% |
| Claims between £1000 & £5000 | 45.04% | 10.17% | 41.48% | 11.09% | 74.66% | 1.91% | 61.95% | 8.85% |
| Claims between £5000 & £10000 | 26.26% | 6.00% | 27.44% | 6.68% | 16.76% | 0.53% | 18.75% | 2.05% |
| Claims between £10000 & £15000 | 7.67% | 1.88% | 8.08% | 2.08% | 4.24% | 0.21% | 5.72% | 0.66% |
| Claims between £15000 & £25000 | 2.22% | 0.76% | 2.30% | 0.84% | 1.54% | 0.15% | 1.89% | 0.14% |
R - Represented
U - Unrepresented
| Breakdown of sample sizes as a percentage of the total data set including observation under £1,000 | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Motor | Employer Liability | Public Liability | |||||
| R | U | R | U | R | U | R | U | |
| All claims between £1000 & £250000 | 66.986% | 15.524% | 57.952% | 15.128% | 7.718% | 0.222% | 1.315% | 0.174% |
| Claims between £1000 & £5000 | 37.163% | 8.390% | 30.313% | 8.106% | 5.928% | 0.152% | 0.923% | 0.132% |
| Claims between £5000 & £10000 | 21.664% | 4.953% | 20.054% | 4.881% | 1.331% | 0.042% | 0.279% | 0.031% |
| Claims between £10000 & £15000 | 6.328% | 1.550% | 5.906% | 1.523% | 0.336% | 0.017% | 0.085% | 0.010% |
| Claims between £15000 & £25000 | 1.830% | 0.631% | 1.680% | 0.617% | 0.123% | 0.012% | 0.028% | 0.002% |
R - Represented
U - Unrepresented
| Claim grouping observations as a percentage of sample size and overall data set | ||
|---|---|---|
| Of claims over £1000 | Of overall data set | |
| All claims between £1000 & £250000 | 100.00% | 82.51% |
| Claims between £1000 & £5000 | 55.21% | 45.55% |
| Claims between £5000 & £10000 | 32.26% | 26.62% |
| Claims between £10000 & £15000 | 9.55% | 7.88% |
| Claims between £15000 & £25000 | 2.98% | 2.46% |
| Range of observations where, on average, there can be shown to be a significant link between representation and higher compensation |
|---|
| 81404 observations which represents: 88% of all the data actually examined in the study |