Breaking the impasse – an effective process to assess Engagement and Impact: IRU Submission

The strong support for measuring the value of university research for end-users has been hindered by the lack of a viable way to do so.  Discussion is caught between the difficulty of ascribing meaning to an array of data sets, many of which are highly subject to manipulation, and the lack of breadth in an assessment of particular case examples.

Previously, in its policy statement ‘Measuring Research Value for End Users’, the IRU argued  that judgment by a panel of relevant people is crucial to creating a plausible rating of performance with the panel working through the data and individual examples put forward.  That proposal still lacked clarity about the focus of the assessment.

At page 15, the Australian Research Council (ARC)’s consultation paper gets to the essence of how to resolve the impasse through its proposal that a “focus on the processes or approaches to impact used by universities may be more appropriate”.  Under this approach, to make an assessment, the panel would consider a set of information about how the university is engaged with end-users as well as evidence of the value of those interactions for the end-users. The inclusion of short vignettes highlighting individual and verifiable impact stories would work as exemplars demonstrating the pathways to impact.

The approach would move away from the researcher-centric attempt to track individual research outputs over a period from idea to application.  If Excellence in Research for Australia (ERA) assesses the value of the research as judged by fellow researchers, an engagement and impact measure should focus on the value for end-users. 

To be meaningful, this engagement and impact measure must rise above a set of data indicators to encompass a knowledge-based judgement of the value of research for end-users.

A number of iterations over different rounds will probably be necessary until an optimal measure is developed.

Overview

The IRU submission to the ARC Engagement and Impact Assessment Consultation Paper covers:

  • the purpose of creating an assessment of engagement and impact with end-users;
  • the importance of having a focus on end-users in designing the assessment;
  • key aspects for the assessment process;
  • amendments to the proposed principles; and
  • a response to the various questions the Consultation Paper poses.

Full submission attached below (7 pages).

Towards a comprehensive impact and engagement measure for Australian research

ATSE’s final report on its ‘Research Engagement for Australia’ project is a useful contribution towards a comprehensive impact and engagement measure for Australian research as envisaged by the National Innovation and Science Agenda.

“The ATSE report completes their work and remains consistent with the original proposal for a simple measure of engagement.  Over 2015, IRU engaged with ATSE to test and strengthen its model including through the participation of three IRU members in the pilot phase of the project,” said Mr Conor King, Executive Director of the IRU.

“IRU stressed the need for judgement of the available evidence, including financial data, to form an assessment of the value of university research for its end users.  Hence we welcome the inclusion of panel-based judgements to the REA approach and the provision to include additional information to the three indicators ATSE has developed.”

Read more…

Not Another Metric

The release of Research Engagement for Australia by the Australian Academy of Technological Sciences and Engineering (ATSE) does not provide the balance to the Excellence in Research for Australia (ERA) assessment needed to recognise the excellence of university research in support of industry and public needs.

The IRU recognises the work ATSE has done to extract from current data metrics about engagement with research users in industry and the public sector. However, ATSE is wrong in its guiding assumption that a thorough qualitative assessment of the value of research for users outside the academic communities is not feasible. Read more…

Rankings – over 50% of Australia’s universities in the world’s top 500

Today’s release of the QS (Quacquaelli Symonds) World University rankings confirms Australia’s continued strong standing in the world rankings with 25 of Australia’s 39 universities in the world’s top 500.

‘There are at least 11,000 universities worldwide1. International rankings identify the top 500 or top 5%, inclusion in which indicates a university of significantly high standing, said Professor Ian O’Connor, Chair IRU.

‘Six of the IRU group are listed in the QS top 500, with The University of Newcastle in the top 300.