Public Sector Equality Duty
Question 1
a) What are your views on the potential equality impacts of the proposals and other plans in this consultation?
There is a severe danger that increasing fees for institutions whose teaching is highly rated will have a negative impact on equality. The Institute of Mathematics and its Applications believes that fee increases and measures of teaching quality should be de-coupled.
Universities which tend to recruit greater numbers of students from (i) minority ethnic groups, and (ii) social groups who traditionally have progressed relatively poorly or who have to work part-time while studying may perform poorly on metrics. In addition, any measures of ‘teaching quality’ must be carefully constructed, piloted and calibrated to support providers that are really making the contribution to equality impact.
Indeed, it is proposed (p.83, para 16 of Annex A) that the metrics that feed into the TEF assessment should be benchmarked to take account of the different outcomes that might be expected from, among other things, the demographic profile of its student intake. This is clearly a very difficult matter, and we oppose any attempt to rush through a ‘one size fits all’ methodology. HE institutions differ very greatly not only in their demographic profile but in their mission, and this diversity is a source of strength in the system.
b) Are there any equality impacts that we have not considered?
Not sure.
Teaching Excellence Framework (TEF) (Part A: Chapters 1-3)
Question 2
How can information from the TEF be used to better inform student and employer decision making? Please quantify these benefits as far as you can.
If the TEF is institution-based, then it is not clear how it conveys information about the ‘quality of teaching’ in any particular discipline. To achieve parity with the REF, hence to make teaching of equal importance to research, the TEF would have to be at discipline level.
Any TEF has to be very carefully constructed. There are serious doubts about what existing ‘metrics’, or proxy measures of teaching quality, actually measure. For example, it has been argued in some public discussions that better, more demanding teaching at higher academic levels may result in poorer results for the students if they do not respond to the greater challenge—a challenge which might well prepare them better for the workplace. Such exposure to higher level and high quality teaching may well be interpreted by many students as a “poorer student experience”. The responses of students in the NSS are highly self-focussed and there is no weighting for the student’s involvement and commitment to their programme.
Para 19 (p.12) of the consultation: “There are many examples of excellent teaching within the higher education system but, as NSS data suggests, teaching quality is variable.” This is an example of over-reliance on untested and uncalibrated data.
Para 35 (Part A, Chapter 1, p.25): “We envisage incorporating new common metrics, for example on engagement with study (including teaching intensity) and learning gain, once they are sufficiently robust and can be used to compare providers.” We welcome the idea that metrics should be robust but we emphasize again the difficulty of achieving robustness before a proper trial is carried out, bearing in mind the very wide diversity of missions, teaching methods, intended outcomes and degree programmes even within a single discipline.
We note that “A technical consultation will be run in 2016 which will cover the operational detail of metrics and of the assessment criteria, process and outcomes, as well as looking at the evidence to be submitted alongside applications and how it will be used for provider level assessment.” (p.25, para. 32 of Part A, Chapter 1). A consultation is not enough here; the process will need to be piloted and reassessed before implementation.
Question 3
Do you agree that the ambition for TEF should be that it is open to all HE providers, all disciplines, all modes of delivery and all levels?
Yes
Given that the set of established, publicly funded universities already displays “all disciplines, all modes of delivery and all levels”, it is not clear why TEF would only be available to a selected subset of providers.
Question 4
Where relevant, should an approved Access Agreement be a pre-requisite for a TEF award? What other mechanism might be used for different types of providers?
The Office for Fair Access is best placed to comment on Access Agreements. These would presumably become the responsibility of the proposed Office for Students.
Again we emphasize that widening participation and fair access are threatened by a coupling of fee level with ‘teaching quality’.
Question 5
Do you agree with the proposals on:
a) what would constitute a ‘successful’ QA review
No
b) the incentives that should be open to alternative providers for the first year of the TEF
No
c) the proposal to move to differentiated levels of TEF from year two?
No
The whole emphasis of the relevant paragraphs appears to be to put through some kind of untested and uncalibrated TEF which may end up causing unintended but real harm and not be fit for purpose. The timetable is, as stated elsewhere, much too short.
Question 6
Do you agree with the proposed approach to TEF assessments on Timing?
Not sure
Assessment panels?
Not sure
and process?
Not sure.
Assessment panels must contain independent experts, of course. We note that “In time, it is envisaged that panels will be convened for each discipline (subject) and include experts in that discipline to make relevant and robust judgements.” Until this happens, there is unlikely to be any information on the ‘quality of teaching’ in any individual discipline, except such as is provided by untested proxy measures of doubtful robustness. On the other hand discipline-level assessment might prove to be a large administrative burden (comparable to the old TQA).
We do not understand the details of the “process” well enough to comment.
Question 7
How can we minimise any administrative burdens on institutions? Please provide any evidence relating to the potential administrative costs and benefits to institutions of the proposals set out in this document.
The BIS paper emphasizes a “light touch” and “low administrative burden”. However, if the eventual outcome really is for differentiated fee levels being allowed, institutions will be unlikely to treat the assessment as light touch. (As above, we are strongly against the coupling of fees with assessment of teaching quality.) There is likely to be investment on the level of the REF.
We are in favour of the TEF and QA being a common process to reduce the burden.
If TEF becomes subject specific (the only circumstance in which it would be of use to intending applicants) then more care needs to be taken to keep the administrative burden on departments/discipline areas down, while, crucially, maintaining the credibility of the exercise.
Question 8
Do you agree with the proposed approach to differentiation and award as TEF develops over time?
Not sure
It is unclear to us how a crude system of ‘levels’ can encapsulate complex data of the kind envisaged in a TEF. (There may also be problems with legal challenges to the award of a level.)
Question 9
Do you agree with the proposed approach to incentives for the different types of provider?
No
As above, we do not agree that fees should be linked to ‘teaching quality’. See particularly Qu. 1 and 2.
Question 10
Do you agree with the focus on teaching quality, learning environment, student outcomes and learning gain?
Yes
The question is, how do you measure these things? Rushing into an ill-thought-out TEF with unknown and unintended consequences is not the way forward.
Question 11
Do you agree with the proposed approach to the evidence used to make TEF assessments – common metrics derived from the national databases supported by evidence from the provider?
Not sure
As stated many times above, devising a set of metrics which really reflect teaching quality over a highly diverse provision, with diverse teaching methods, missions, student bodies and intended degree outcomes, even within a single discipline such as mathematical sciences, is a very difficult task, and the ‘national databases’ are unlikely to be adequate. To use them, uncalibrated and without proper analysis, for a hastily arranged TEF which might have serious unintended consequences would be folly. As examples of the one aspect of different teaching methods compare (i) an intense tutorial system supported by lectures at Oxbridge for highly selective students, (ii) a lecture led environment supported by loose exercise support systems (large civic universities) (iii) distance learning (the Open University).
Social mobility and widening participation (Part A: Chapter 4)
Question 12
a) Do you agree with the proposals to further improve access and success for students from disadvantaged backgrounds and black and minority ethnic (BME) backgrounds?
Not sure
The point was made at a public meeting (Royal Society, 8 December 2015) that minorities and women actually do relatively well in higher education; more worrying is the poor performance of white male students from the lowest socio-economic strata which unless cleverly weighted in institution outcomes may discourage wider engagement with such groups.
b) Do you agree that the Office for Students should have the power to set targets where providers are failing to make progress?
Not sure
c) What other groups or measures should the Government consider?
Whatever incentives (sticks or carrots) are envisaged for the TEF, the Government will have to encourage engagement by HEIs in widening participation at the discipline level.
Question 13
a) What potential benefits for decision and policy making in relation to improving access might arise from additional data being available?
No comment.
b) What additional administrative burdens might this place on organisations? If additional costs are expected to be associated with this, please quantify them.
No comment.
Opening up the sector to new providers (Part B: Chapter 1)
Question 14
Do you agree with the proposed single route into the higher education sector?
Please give reasons for your answer, including information quantifying how the potential cost of entry would change as a result of these proposals.
No comment.
Question 15
a) Do you agree with the proposed risk-based approach to eligibility for degree awarding powers (DAPs) and university title?
No comment.
b) What are your views on the options identified for validation of courses delivered by providers who do not hold DAPs?
No comment.
Question 16
Do you agree with the proposed immediate actions intended to speed up entry?
No comment.
Provider exit and student protection (Part B: Chapter 2)
Question 17
Do you agree with the proposal to introduce a requirement for all providers to have contingency arrangements to support students in the event that their course cannot be completed?
Please give reasons for your answer, including evidence on the costs and benefits associated with having a contingency plan in place? Please quantify these costs where possible.
No comment.
Simplifying the higher education architecture (Part C)
Question 18:
a) Do you agree with the proposed changes to the higher education architecture?
b) To what extent should the Office for Students (OfS) have the power to contract out its functions to separate bodies?
c) If you agree, which functions should the OfS be able to contract out?
d) What are your views on the proposed options for allocating Teaching Grant?
Option 1: BIS Ministers set strategic priorities and BIS officials determine formula.
Option 2: BIS Minister sets strategic priorities and allocation responsibilities divested to OfS
Please give reasons for your answer,
We have no views on this question.
Question 19
Do you agree with the proposal for a single, transparent and light touch regulatory framework for every higher education provider?
Yes
Please give reasons for your answer, including how the proposed framework would change the burden on providers. Please quantify the benefits and/or costs where possible.
We have severe doubts about the ability for the TEF to be “light touch”, if it is to be fit for purpose and to work at discipline level. If the TEF does not go down to discipline level then it is of no use to students in making a choice as to where to study a particular discipline. But if it does go down to discipline level then are we back with the TQA which was a huge burden on departments?
Of course more transparent information for students, so long as it is “reliable and robust”, is a good thing, and that would be at discipline level.
Question 20
What steps could be taken to increase the transparency of student unions and strengthen unions’ accountability to their student members?
We have no views on this proposal.
Question 21:
a) Do you agree with the proposed duties and powers of the Office for Students?
Not sure
b) Do you agree with the proposed subscription funding model?
Not sure
We have no views on this proposal.
Question 22
a) Do you agree with the proposed powers for OfS and the Secretary of State to manage risk?
Not sure
This will be very complex process for the OfS given that Teaching excellence and effectiveness will be, in part, reflected in the “social status” of the HEI, examination quality of the student intake, social mix of the student body etc. . We still don’t know how the teaching excellence will be determined. Therefore, we would really doubt that at this stage there is much understanding of the potential difficulties that might ensue from a broad TEF implementation after the initial year. The minister wants to improve teaching, and the DfE, OfS need to find some metrics that they think will mirror the improvement -this is the current situation. That is not much information so far for anyone to assess and manage risks.
b) What safeguards for providers should be considered to limit the use of such powers?
Question 23
Do you agree with the proposed deregulatory measures?
Not sure
Please give reasons for your answer, including how the proposals would change the burden on providers. Please quantify the benefits and/or costs where possible.
We have no views on this.
Reducing complexity and bureaucracy in research funding (Part D)
Question 24
In light of the proposed changes to the institutional framework for higher education, and the forthcoming Nurse Review, what are your views on the future design of the institutional research landscape?
The separation of teaching and research is a serious concern. Regulating each of these as if they were the sole focus of universities is likely to result in a combined burden that is excessively onerous to the detriment of delivery.
The TEF should be designed to acknowledge the value of university research and industrial links to the education of students. Research is what prevents education becoming moribund and out of date. It should be noted that universities employ, in the main, the same staff to deliver both teaching and research in part because the cross over effects are so highly beneficial.
Question 25
a) What safeguards would you want to see in place in the event that dual funding was operated within a single organisation?
If RUK is in charge of both QR and grant distribution then there must be a clear separation in governance terms within RUK between these two operations.
b) Would you favour a degree of hypothecation to ensure that dual funding streams, along with their distinctive characteristics, could not be changed by that organisation?
Yes
The REF should be designed to rate the quality of research. Research grant income is won via a competitive process involving peer review, and so as part of a broad set of indicators of quality we think it has a place. It should never be the major factor within mathematics, however, because much high quality mathematics research does not require large grants, and further, the grant money available to mathematics is so small that not obtaining a grant cannot be seen as an indication of poor quality.
Question 26
What are the benefits of the REF to a) your institution and b) to the wider sector? How can we ensure they are preserved?
a) Not applicable. This is the response of the IMA and not that of an individual institution.
b) The benefits of the REF to UK mathematics departments is high. The REF has shown that there is excellence in mathematics research in the length and breadth of the country. This is brilliant news for the
government, and this breadth of excellence needs to be protected, not undermined, for example by metrics which penalise emerging and small departments, such as the environment score seem to do.
The importance of a diverse research base cannot be overestimated, see for example the report, “The value of structural diversity – Assessing diversity for a sustainable research base” by Digital Science.
A diverse research base means flexibility, the ability to respond quickly to emerging national needs, and a route for talent to emerge that can respond to new needs. It should be noted that funding only a research ‘elite’ (however determined) would drive UK research into being inward looking, old fashioned and rigid, within an academic generation (10 years). In fact, the result of funding only an ‘elite’ would be to have all the problems of the research bases of small countries. This is because university departments hire their staff to fit their own teaching and research interests and the contracts are open ended. There is absolutely no need to have small country problems in a country the size of the UK, and having the range and quantity of talent residing in the UK.
Another strong plus of a diverse research base is the availability to the vast majority of UK students, of teachers whose practice is enlivened and deepened by research and industrial consultation. Again, restricting research to an ‘elite’ would greater lower the experience of the vast majority of UK students.
Protecting the benefits of diversity in research involves using indicators and metrics which clearly value research diversity, and on the other hand, do not bias against small and emerging departments, such as the environment score which so far has correlated with the assumption that big is better.
Question 27
How would you suggest the burden of REF exercises is reduced?
The object of the exercise is to come up with a score for a body of work (as submitted by a department, say) which is commensurate with its quality. If metrics can be found which broadly correlate with the outputs of a peer review process at the UoA level then we should use them. Both the use of metrics and peer review have problems. However, metrics are seen as low burden and low credibility, while peer review is seen as high burden and higher credibility.
Question 28
How could the data infrastructure underpinning research information management be improved?
Researchfish needs to be debugged and easier to use. It seems every bit of data they can think of is required, and that they hope to find a use for it all later. Instead, it needs to be decided in advance what they are going to do with the data and ask for the minimum set that achieves their aim.
Higher Education Service Area and Research Committee
15 January 2016
