Response to the HEFCE Consultation

Future approaches to quality assessment in England, Wales and Northern Ireland

Question 1: Do you agree with our proposed principles to underpin the future approach to quality assessment in established providers?

Agree. We should like to be clear whether QA (and the TEF) apply to all degree-awarding providers or only to the ‘publicly funded’ sector as defined in http://www.hefce.ac.uk/glossary/ “any provider of higher education which also receives direct, recurrent public funding for any purpose.”

Notes on individual principles:

Institutional autonomy is very important, and it is also very important to recognize that different institutions have different missions and differing student cohorts. Care must nevertheless be exercised in allowing too much freedom to determine what is the ‘most appropriate academic experience’. The External Examiner/Peer Review system should prevent any abuse of this freedom.
Surely peer review is part of external scrutiny.
Student involvement, when appropriately managed, is very valuable; it must be recognized however that students have limited experience. (The European standards and guidance document (May 2015) states (Sec 1.3) “Institutions should ensure that the programmes are delivered in a way that encourages students to take an active role in creating the learning process, and that the assessment of students reflects this approach.” We are unsure whether the last phrase is feasible in practice.)
We agree that students must be well-informed about what a HEI’s degree programme will deliver and what knowledge, skills and experience they have the opportunity to develop as a result of completing it.
Again we agree that diversity is an important part of the HE landscape. There is no ‘one size fits all’ for both the content of the various degree programme codes and their mode of assessment in MSOR.
In particular any QA activity must dovetail with the proposed Teaching Excellence Framework. Duplication of effort and burden on HEIs is absolutely to be avoided.
As above, ‘consistency’ must not lead to ‘uniformity’ or ‘one size fits all’.

Question 2: Do you agree that our current proposals for the use of meaningful external scrutiny as set out on pages 9-10 are sufficient? If you do not agree, please indicate what additional or different external scrutiny you propose and provide the reasons for this.

Neither Agree nor Disagree

Para. 33a includes ‘improvement of student outcomes’. This should not mean ‘steadily better (class of) degree results’. But this may well be what it is taken to mean so the phrase is better omitted.

Para. 34, degree classification algorithms. With the wide range of structure for degree programmes (in itself a strength of the system) too rigid a standardisation would not be feasible and in any case may conflict with institutional autonomy. Perhaps across parts of the sector which have ‘similar missions and student cohorts’ some consistency might be achieved. The same applies to ‘calibration of degree standards’.

The possible role of PSRBs would be extremely dependent on the resources which they can provide. In the case of MSOR there is no statutory or regulatory body (except in the case of certain financial areas such as accountancy and actuarial mathematics) though there is a Professional Body, namely the IMA. The IMA accredits degree programmes but its resources are inadequate for frequent and widespread involvement with the peer review process. The IMA’s reviewers are when possible practising academics, rather than ‘professional reviewers’ whose time is limited.

Governing bodies: is this genuine oversight or a rubber-stamping exercise?

Question 3: Do you agree that future approaches to quality assessment should be based on an assumption that ‘one size’ can no longer sensibly fit all?

Strongly Agree.

Question 4: Do you agree that there should be a baseline requirement for the quality of the academic experience for students, and that this should be published and maintained?

‘Quality of academic experience’ is very vague, and a ‘baseline requirement’ correspondingly so, but we assume this refers essentially to the Quality Code http://www.qaa.ac.uk/assuring-standards-and-quality/the- quality-code which is broadly accepted by the sector. This is already published and maintained.

Question 5: For England, do you agree with the proposal that an individual provider, once it has passed the gateway for entry into the publicly funded system in England, should not be repeatedly externally retested against the baseline requirements for an acceptable student academic experience, unless material evidence suggests otherwise?

We understand that the QAA’s current procedure for granting TDAP (teaching degree awarding powers) is already very rigorous. Assuming that something similar would be the case under the new arrangements then repeated testing is unnecessary. However if the procedure for TDAP becomes less rigorous then this situation may change.

Question 6: For Northern Ireland, do you agree that providers should provide annual evidence and assurance that they are meeting the baseline requirements for an acceptable student academic experience?

Genuinely light-touch annual evidence may be the only realistic way forward.

Question 7: Do you agree that the funding bodies’ verification of an institution’s review methodology provides a reasonable mechanism through which to operate risk-based scrutiny of a provider’s arrangements to secure a good and improving student academic experience and student outcomes?

Agree. The IMA strongly supports Para. 41 a.-d.

Para. 41e. refers to ‘continuous improvement’ of the ‘associated outcomes that matter to students’. If this means the classes of degree awarded (surely the outcome of most interest to students) then we find this idea misguided.

Regarding Para 4, if we note that annual review at subject level would not be sustainable by institutions, and would also not be possible for the smaller PSRBs (such as the IMA) to contribute to.

Question 8: Do you agree that student outcomes data should provide the basis for continuous improvement activities within an individual provider?

Metrics need to be shown to be robust and reliable before any trust can be placed in them by the HE community. There is strong evidence that metrics alone are inadequate for assessing research and we believe the same to be true regarding learning and teaching. Undoubtedly numerical data has its place but there needs to be a place for qualitative data as well.

Progression, awards and employment are relatively unambiguous numerical data, but other aspects such as the NSS, staff details and contact hours, without expensive extraction of detail, can easily be misleading.

Question 9: Do you agree that we should take forward into detailed design and pilot phases further work on the use of student outcomes data to identify patterns and trends and on the development of approaches for monitoring and supporting institutions as they address areas of concern?

Agree. We agree so long as the discussions about data are, as they say in Sec. 53, ‘robust, sophisticated and nuanced’. It is clear to us that thorough design and piloting of the use of ‘student outcomes data’ is essential before it is used to judge institutions.

Question 10: In Northern Ireland, do you agree with the approach outlined above to introduce more effective and consistent arrangements for collecting and analysing feedback from higher education learners?

The IMA covers all UK mathematical sciences but we do not feel we have sufficient knowledge to answer this question, other than to say that the proposals look quite expensive.

Question 11: Do you agree with the proposal that more emphasis should be placed on the role of a provider’s governing body to provide assurances about the quality of the student academic experience and student outcomes in line with the Higher Education Code of Governance? If you agree, please indicate what, if any, additional support they should receive to provide such assurances.

Agree but we are very concerned that (a) governing bodies do not in general have the expertise to seriously engage with assurance about the quality of the student experience and (b) there might be conflicts between the governing body and the academic board or Senate.

A governing body would need to set up a subcommittee with co-opted members. The latter should not be internal auditors who do not know about academic standards; where would they come from? In any case a great deal of training would be needed before there was really meaningful engagement with the quality assessment and assurance process. Workload on governing bodies would certainly increase and the difficulty of recruiting suitable governors also increase. (If remuneration is introduced this would consume valuable resources which would be better used elsewhere.)
We are concerned that the governing body should not have a management or executive role in the process.

Question 12: For England, do you agree that, for English institutions, HEFCE should develop and use the existing external accountability mechanisms, particularly the HAR, in the ways described on page 20?

Agree, but to say that there is ‘a requirement [from HEFCE] that the governing body should receive assurances that the institution has an effective framework’ via internal audit is quite different from saying that the governing body should itself provide such assurances. This would require a change to the HAR.

Question 13: For Northern Ireland, do you agree that DEL should develop and use the existing accountability mechanisms in the ways described on pages 20-21?

No comment.

Question 14: Do you agree that there should be a ‘probationary period’ for new entrants to the publicly funded sector in England?

Agree. A full year of activity needs to be assessed for a new entrant as the new institution’s assessment regulations will not have been even partially checked after 1 calendar year. The end of the year 1 + review process would be better? The assessment should be risk-based in the same way as for established HEIs. A very strong showing after the first assessment should end the probationary period.

Question 15: Do you agree that international activities should be included in the remit of future quality assessment arrangements as described on pages 22-23?

Agree. It might be difficult for external examiners to visit international campuses so alternatives such as video link should be available.

Question 16: Do you agree that a future quality assessment system must provide reliable assurances to students and other stakeholders about the maintenance of academic output standards and their reasonable comparability across the UK higher education system?

Agree but the key word is ‘reasonable’. In MSOR there are many ways to challenge students and different institutions can perfectly well adopt different ways. We resist the suggestion that degrees from all institutions ‘mean the same thing’ since they patently don’t. The most that can be claimed is that institutions of a similar nature might have similar standards and deliver similar outcomes for the students.

Question 17: Do you agree that the external examining system should be strengthened in the ways proposed, i.e. through additional training and the establishment of a register?

Agree. Who will provide the training, and who will be the trainers? Will training need to be repeated after a certain time? Will the Register consist precisely of those who have ‘taken’ the training or will there be an assessment to check that they have ‘passed’ it?

The IMA does not favour a group of professional external examiners. Also training must be done in a time- efficient way otherwise it will be extremely difficult to recruit EEs.

Academics have many priorities and being an EE is time-consuming enough as it is and not usually recognized by universities as contributing to service, something which should change.

Sensible fees, perhaps in line with consultancy, might help.

There is a danger that a new bureaucracy would change the EEs role into that of solely ticking boxes, and remove the important element of providing academic advice in a supportive and understanding way.

Question 18: Do you agree that our proposals in relation to the external examining system are sufficient, i.e. do they go far enough to provide the necessary assurances about academic output standards to students and other stakeholders?

Agree. External examiners should concentrate on assessments, student work and programme structure rather than on quality assurance.

Question 19: Do you agree that it would be helpful to explore approaches to the calibration of academic output standards in different disciplinary and multi-disciplinary contexts?

We believe that the MSOR community is too diverse to reach meaningful conclusions. We do not pretend that all institutions are the same or all degrees the same, at any rate beyond the first year of a degree programme. Going beyond the discipline of MSOR would be even more difficult.

Perhaps the degree classification system should once again be reconsidered.

Question 20: Do you agree that providers should use the accreditation activities of at least some PSRBs more centrally in future approaches to quality assessment?

In principle the IMA agrees that PSRBs are well placed to advise on quality assessment. But the IMA does not have the resources to engage in any kind of annual audit of large numbers of universities.

Question 21: Do you agree with the proposal that we should place more emphasis on the role of the governing body of a provider with degree awarding powers to provide assurances about security and reasonable comparability of the academic output standards of students?

Disagree. This goes further than Qu 11 in the ‘reasonable comparability’ part. The wording given is the standards of awards for which we are responsible have been appropriately set and maintained, and are reasonably comparable with those awarded to students by other UK degree awarding bodies.

We do not see how could such a statement be made with any authority.

Question 22: Do you agree with the proposal to develop guidance to providers on a sensible range of degree classification algorithms at the pass/fail and 2i/2ii borderlines?

Agree, but the ‘range’ of algorithms would have to reflect the range of structures of degree programmes and there are many of these. The IMA is not in favour of forcing rules like ‘all modules must be passed’ on the entire community. (The MSOR Benchmark Statement is clear about this.)

Question 23: Do you agree with our proposals to develop and implement a strengthened mechanism to investigate rapidly when there is an indication of serious problems within an individual provider which has not been addressed in a satisfactory and timely manner?

Agree. But we do not know what would trigger an indication of serious problems.

Question 24: Should the mechanism to investigate problems in an individual provider require, in addition to the investigation of the specific issue of concern, the re-testing of the arrangements in the provider under review against the baseline requirements set out for the gateway for entry to the higher education system?

No. Not unless the problem is really serious.

Question 25: Do you agree with the proposal that providers seeking entry to the publicly funded sector in England and Northern Ireland should be tested, through an external peer review scrutiny process, against a set of baseline requirements for quality?

Agree. Particularly important is that statement of Para. 114:

the ‘threshold bar ’ for entry should be set at a level sufficient to ensure that students would receive an appropriately high quality academic experience, that academic output standards would be set appropriately and remain secure, and that the reputation of the system as a whole would be protected.

Question 26: Are there any particular areas of our proposals that you feel we should concentrate on as we undertake a more detailed design phase?

Use of metrics especially if it forms a large part of quality assessment needs to be very carefully thought out. See the response to Qu. 8.

Question 27: Are there proposals not referred to above that you feel we should have in consideration? If so, what are they and what is the rationale for their inclusion?

The definitions of ‘standards’ and ‘quality’ have not been helpful in recent years and these should be looked at with a view to replacing them with phrases that the general public would recognise and expect to see.

Question 28: Are there any particular areas pertinent to the devolved nature of higher education in Wales and Northern Ireland that you feel we should have considered further? If so, what are they and what is the rationale for their inclusion?