The government has announced a review of the Australian Research Council in response to the vocal calls by the academic and scientific community for the systemic problems related to the grants process to be fixed.
The Education minister Jason Clare announced the review last Tuesday and provided the Australian Research Council (ARC) with a Statement of Expectations for its operation prior to the review reporting to government.
Professor Sue Bennett stated that “the much criticised ‘national interest test’, introduced by former Coalition Education Minister Dan Tehan in 2018 will stay.”
The ‘national interest test’ is a statement that researchers provide with their submission that explains their research in non-scientific language that a small panel uses to judge whether the research should be funded. This review is carried out separately to the assessment of submissions by academic experts.
The nit-picking, as this process is known, is a major cause of concern among the research community.
At the heart of the problem is the reduction in research funding that has occurred over the past decade.
Academics commit a significant amount of time and effort to prepare ARC grant applications and the low success rates are a cause for frustration amongst researchers.
There is also concern among the research community that the lack of research funding and the method used to assess grant submissions is driving academics to adopt controversial and frankly unfair practices.
There are several issues that the government’s review should tackle yet does not appear likely to do so.
The first is the over reliance on citations as a measure of a researcher’s achievements. This means that an academic who has achieved significant research impact or industrial outcomes will be rated lower than an academic that has a higher number of citations but little research impact.
This drives some researchers to form ‘citation circles’ where researchers form groups that heavily cite the papers of other members of the group and don’t cite likely competitors for ARC grants.
Researchers know that it is difficult to prove that a ‘citation circle’ is in effect and the effort involved to identify citation circles could be considerable.
Funding to develop an automated approach that utilised ORCID (researcher identities), Google Scholar and journal publication bibliographies would be money well spent.
The second is the practice of reviewers to give low scores to grant applications that don’t include their colleagues and high scores to grant applications that include their colleagues.
Again, proving that this practice occurs is difficult, but a thorough investigation of ARC grant submission reviews for the past five years could be enlightening.
Finally, the biggest concern and frustration with the ARC review process is that reviewers can make statements supporting their scores that are false, misleading and in some cases unintelligible.
Researchers can ask that a review be ignored, however, the list of reasons that the ARC might entertain disregarding a review does not include false, misleading and unintelligible statements.
The term “death through faint praise” aptly describes the method used by some reviewers to justify low review scores.
The researchers that are invited to carry out ARC grant reviews are not required to provide evidence nor references for their statements.
Applicants can submit a rejoinder in response to the expert reviews. However, an application that has a low score will not be saved by the rejoinder.
Many researchers, and academics, don’t apply for ARC grants due to the low success rates and the frustration of having to deal with the soul-destroying statements made by some of the reviewers.
What this means is that the pool of potential researchers that will participate as an ARC reviewer is small, and there is a focus on getting onto the ARC college of experts – the expert panels that ultimately rank grant applications.
An increase in government research funding will not fix the systemic problems with the ARC grant review process.
The grant submission format should not be changed every year. This mechanism is used to prevent applicants from responding to reviewer comments in a subsequent submission.
The grant review process should be broken into two steps: a project review and a review of the applicants.
By focusing on the project, the ultimate purpose of research funding is placed central to the review process.
The applicant review step should occur only for the highly ranked projects and should not provide opportunities for reviewers to denigrate applicants for one reason or another.
A simplified list of criteria for the applicant review should be developed.
The vast majority of applicants are capable of completing an ARC grant.
Comments related to age or the number of years that a person has been a researcher is discrimination. The idea that junior academics, either separately or in a team, cannot successfully complete an ARC grant is fanciful, discriminatory and biased towards older researchers.
Ranking researchers based on the number of citations achieved is nonsense. This measure does not include research impact, nor does it include confidential industry related research.
An appeal process over the applicant review should be provided because the number of reviews that make unacceptable statements about one or more of the applicants is high.
Some of the most important research papers in my field have been published as conference papers, yet it appears to be a misstep to list a conference paper in the list of top 10 research outcomes.
Similarly, research may be published in a second quartile journal simply because there are no first quartile journals that publish in the field, for example, public policy related to an Australian industry.
The ignorant statements by some reviewers about the value of research published in a conference paper, book or other publication should be grounds for appeal.
Publications that are eligible to be included in an organisations Excellence in Research for Australia (ERA) submission should be treated equally.
The pool of reviewers should be increased, and this may mean that researchers are paid to carry out the reviews. Junior researchers should be provided with training and encouraged to become ARC grant reviewers.
All statements made by reviewers should be intelligible, and backed by evidence and references. Personal opinions have no place when carrying out grant reviews.
The fact is that the national research funding is low, and this means that there is a need to ensure that research funds are distributed.
A researcher should be limited to participation in one ARC grant at a time. This is likely to be a contentious issue that could undo any discussion about reforming the ARC because there are research teams that have achieved remarkable funding success and rely on the ARC grants to continue their research.
But what about researchers that for one reason or another fall foul of the application process?
By restricting the number of grants that a researcher can participate in at a time there may be a benefit in limiting the negative practices adopted by some researchers to ensure grant success.
A pool of international experts should be created to ensure that grant submissions are reviewed by at least one international expert.
As an international expert in my field, I carry out grant submission reviews for the Hong Kong government and receive a small honorarium. The process is very professionally carried out and provides an excellent model that should be adopted by the ARC.
The government’s review into the ARC should consider the many complaints that researchers have about the organisation and processes that it oversees.
Concerns about the government’s contribution to research funding are high, but this is another matter that should be argued separately and not be used to cloud the ARC review.
Mark Gregory is an Associate Professor in the School of Engineering at RMIT University
Do you know more? Contact James Riley via Email.