NHS Breast Pathology EQA FAQs

Team Leader: Dr Sue Moss

Location: Sir Richard Doll Building, Sutton

Section: Section of Epidemiology

NHS Breast Screening Pathology External Quality Assurance (EQA) Frequently Asked Questions (FAQs)

• How do I enrol in the scheme?

• What if I forget my unique identification number?

• How do I find out who my regional co-ordinator is?

• When do I have to get my returns in for the current circulation?

• Can trainees participate in the scheme?

• Does the EQA scheme identify substandard performance?

• How can I work out my measure of agreement?

• What is a Kappa statistic?

How do I enrol in the scheme?

Please apply directly to the CSEU based at Sutton, Surrey. We will register the participant and issue the unique identification number together with the details of the scheme by post.

What if I forget my unique identification number?

The CSEU will send you a reminder either by post or email on request.

How do I find out who my regional co-ordinator is?

Who's Who in the NHSBSP EQA scheme:

When do I have to get my returns in for the current circulation?

Timetable for current circulation (2006/1)

The readings for those pathologists reaching the CSEU before the first closing date are analysed (1st analysis) prior to the meeting of the National Co-ordinating Committee for Breast Screening Pathology. At the meeting, the consensus for each case is agreed but inevitably discussion focuses on cases where agreement is poor. The CSEU reanalyses (2nd analysis) the readings when sufficient time has elapsed to allow the slides to circulate to all participants in the regions. The timing is a compromise between letting the slides circulate to all pathologists who wish to participate (currently over 500) and providing feedback on the cases as soon as possible. Forms from pathologists who miss the second closing date will be entered onto the computer but once the analysis has been distributed to the regions no more readings will be entered.

Can trainee’s participate in the scheme?

At the discretion of Regional Coordinators, trainees may view the slides in order to perform self-assessment. They should preferably not submit their forms, but, if they do, they should indicate that they are trainees so their diagnoses may be excluded from the assessment process.

Does the EQA scheme identify substandard performance?

A measure of agreement on the major diagnostic category has been devised and was introduced in 1995. The scheme is seeking CPA (UK) accreditation so that Royal College of Pathologists and National External Quality Assurance Advisory Panel procedures can be used to identify substandard performance and subsequent action. Members of the scheme can contact their NHSBSP Pathology Regional Co-ordinators for further information and advice.

How can I work out my measure of agreement?

The measure of agreement is determined from the four major diagnostic categories:

• Benign (including radial scar)
• Atypical hyperplasia
• In-situ carcinoma (including micro-invasive)
• Invasive

If the participant’s diagnosis accords with the majority opinion of the coordinators, a score of 3 is given. A score of 2 is awarded if the diagnosis deviates by one group, 1 if it deviates by two groups and 0 if it deviates by three groups. Thus for a majority diagnosis of invasive carcinoma, scores of 3, 2, 1 and 0 would be awarded for diagnoses of invasive carcinoma, in-situ carcinoma, atypical hyperplasia and benign respectively. Each participant’s scores are then added together and expressed as a percentage of their total possible score from the number of slides they read. Only cases where there is a majority diagnosis amongst coordinators of at least 80% and deemed appropriate at the coordinators meeting are included in the measure of agreement.

The example uses a total of 10 slides (the consensus on slide 9 was below 80% so this slide was omitted and the participant did not read slide 12). The total possible score is 30 (3 points for each of the 10 slides). The participant gained a total of 21 so the measure of agreement is 21/30*100=70%.

The participant did not read slide 12, this may have been because the slide had been broken. However, given that all those taking part in the Scheme will regularly be reporting breast specimens, and cases are included for assessment only where the majority opinion is made by 80% of the coordinators, it is not acceptable for participants to omit any cases unless the slide cannot be read.

Interest in the scheme has developed to measure performance; however, the educational element remains the most important feature of the scheme.

What is a Kappa statistic?

The Kappa statistic is a measure of agreement that can be applied to different diagnostic categories amongst pathologists. If we assume that the majority diagnosis is correct, which is not necessarily so, then the simplest measure of agreement is to see how many pathologists agree with the majority diagnosis. There is a weakness if we were to use this measure. If pathologists were just to make guesses then we would expect some agreement by chance. Kappa provides a more reasonable measure by considering the agreement in excess of the amount of agreement that we would expect by chance.

The Kappa statistic allocates a score of zero if the agreement between the readers is no better than would be expected by chance. Perfect agreement gives a score of 1. Scores can also be negative if there is consistent disagreement. One problem with Kappa statistics is that values lack a clear intuitive meaning. It is conventional to accept the following guidelines (Landis and Koch, 1977 Biometrics 33, 159-174).

The Kappa statistic always compares one diagnosis versus the rest (eg invasive versus the other possible diagnoses). Consequently, a weakness of Kappa is that it takes no account of the degree of disagreement (if the majority opinion is invasive, a diagnosis of benign is treated the same as a diagnosis of in-situ). We present Kappa's for individual diagnosis and a summary Kappa, which is a weighted average of the individual Kappa's. If there are only two diagnostic categories then the individual Kappa's and the summary Kappa will all be the same.

Another problem with Kappa is that the value depends on the case mix for each circulation so it can be misleading to compare Kappa's between circulations.