Australian Research Council's Administration of the National Competitive Grants Program
Entities audited: Australian Research Council
Introduction
3.1
The Australian Research Council (ARC) is a Commonwealth entity which ‘advises the Australian Government on research matters, administers the National Competitive Grants Program, a significant component of Australia’s investment in research and development, and has responsibility for Excellence in Research for Australia’.
3.2
The agency’s purpose is to ‘grow knowledge and innovation for the benefit of the Australian community through funding the highest quality research, assessing the quality, engagement and impact of research and providing advice on research matters’.
3.3
The ARC is established by the Australian Research Council Act 2001 (the ARC Act) as an independent body, and reports to the Minister for Education. The ARC is subject to the Commonwealth Grants Rules and Guidelines (CGRGs) as a non-corporate Commonwealth entity.
3.4
The National Competitive Grants Program (NCGP) is a competitive grants program designed to support high-quality fundamental and applied research and research training. The NCGP comprises 13 schemes under two funding programs: Discovery and Linkage. The Discovery program provides support to fundamental research to encourage innovation, while the Linkage program is designed to promote collaboration and research partnerships to support research and development.
3.5
Grant rounds are run annually for all schemes and programs, attracting a large number of applications. Between 2016 and 2018, the ARC received 17 556 applications in total across the NCGP schemes and programs, and commenced funding for 3493 grants totalling $1.99 Billion.
Audit rationale and details
3.6
The Australian National Audit Office (ANAO) observed that government-funded research is a critical component of Australia’s broader social and cultural life, in addition to contributing to the economy and international competitiveness. The ANAO stated:
The audit provides assurance to Parliament about whether this significant source of research and development funding is being administered by the ARC in an effective manner and is meeting the objectives set for the NCGP by the Government.
3.7
The ANAO provided an outline of its methodology in undertaking the audit, including: reviewing ARC records such as guidelines and standard operating procedures, analysis of a random sample of grants awarded over a number of years, and interviews with relevant ARC officers and a number of universities and peak bodies.
3.8
The audit was conducted at the cost of approximately $400 000 and utilised six audit staff.
Overview of performance audit findings
3.9
The ANAO reported that, on the whole, the ARC’s administration of the NCGP is ‘effective, except that its performance indicators and monitoring and assurance arrangements should be strengthened’.
3.10
The ANAO found that the NCGP guidelines were appropriately aligned with the CGRGs and the Australian Government’s research and innovation objectives, and these were effectively communicated to stakeholders.
3.11
The ANAO approved of a number of aspects of the ARC’s grants management. The ANAO described the ARC’s grant assessment processes, conflict of interest management and funding recommendations as ‘mature and effective’.
3.12
Performance management of the NCGP was identified as an area which required improvement. The ANAO found that the ARC’s key performance indicators (KPIs) were found to be ‘relevant, but not all are reliable’. The audit further identified that the ARC’s assurance arrangements could more strongly focus on risk management in order to ensure administering organisations comply with requirements under grant agreements.
3.13
The ANAO made three recommendations:
Recommendation 1: the ARC review the practice of issuing NCGP guidelines annually;
Recommendation 2: the ARC ensure that its KPIs for the NCGP are reliable and include efficiency; and
Recommendation 3: the ARC ensures that its monitoring and assurance activities, in particular institutional reviews, are risk-based and contribute to the ARC’s assurance that NCGP objectives are being achieved.
3.14
The ARC agreed to all three recommendations.
Application assessment process
3.15
The ARC administers the NCGP grants program in its entirety, including the creation of guidelines, calling for and assessing grant applications, processes for the recommendation and approval of proposals to the Minister, and providing funding for successful grants.
3.16
Eligible organisations are able to make grant proposals in either the Discover or Linkage programs. The NCGP guidelines were found by the ANAO to provide clear direction regarding general and scheme-specific eligibility criteria, which is made available on the ARC’s website. The guidelines also provide ‘clear and comprehensive assessment criteria and criteria weightings’ for both schemes.
3.17
The assessment process for applications is detailed within the NCGP guidelines. The process includes the following key features:
Grant proposals are assessed according to the eligibility criteria in addition to their relative rank against other proposals;
Proposals are assessed by assigned assessors who are expert in the specific field of work being addressed. Assessors are chosen by the College of Experts and subsequently form a Selection Advisory Committee (SAC) responsible for overseeing the peer review process;
Feedback and rejoinder processes; and
Recommendations are provided to the Minister in accordance with the findings of the SACs, and further appeals processes after ministerial approval is granted.
3.18
In examining the ARC’s grant application assessment process, the ANAO found that:
The ARC has appropriate assessment, eligibility, scrutiny and appeal processes to support well informed and transparent grant assessment and funding decisions.
3.19
The ANAO stated that the ARC’s ‘mature and effective’ assessment processes were effective, in part due to its grant management information technology (IT) systems which supported the receipt and assessment of grant applications. The ANAO stated that the use of expert advice, general and detailed assessors, and appropriate advice was indicative of a ‘robust process’ to examine grants applications.
3.20
The ANAO issued no recommendations in relation to the ARC’s assessment processes.
3.21
The Auditor-General stated that the ARC managed its grant programs ‘probably as well as what we’ve seen in terms of the processes around it, and certainly their approach to dealing with probity-type issues is very good’. Given this, the Committee recognises that the ARC could operate as a best practice model for other agencies in terms of the implementation of grant guidelines, the management of conflicts of interest and general processes.
Consistency of NCPG Guidelines with CGRCs
3.22
The audit examined NCGP guidelines issued between 2015 and 2018 to determine whether the assessment of applications were consistent with the NCGP guidelines. The report stated that the ARC utilised:
[A] multi-stage process for the development of the NCGP guidelines and associated supporting materials that is governed by a standard operating procedure … and related work instructions.
3.23
The CGRGs require that officials ‘develop guidelines for all new grant opportunities and issue revised guidelines where significant changes have been made to a grant opportunity’. The ARC was found to be compliant with the mandatory requirements of the CGRGs in relation to creating guidelines, obtaining Ministerial approval for them, and releasing them publicly. Risk assessments were conducted for all guidelines developed for grants commencing between 2016 and 2020, all of which were found to be ‘low risk’.
3.24
The ANAO found that:
The NCGP guidelines align with the CGRGs and the government’s research and innovation objectives, and have recently increased the focus on government priorities. The guidelines clearly outline the elements of the NCGP and are effectively communicated to stakeholders.
3.25
The audit report recommended that, given the relatively minor changes in the guidelines between each year, the ARC publish new guidelines for the NCGP ‘only when necessary; for example, when significant changes are made to the grant eligibility or assessment criteria’.
3.26
The ARC agreed to the ANAO’s recommendation, advising that it would ‘review the practice of issuing NCGP guidelines annually with a view to having multi-year guidelines’. The ARC confirmed that this task had been completed and multi-year issuing of guidelines had been implemented in February 2020.
Decision-making processes regarding funding
3.27
The ARC utilises the NCGP guidelines in making decisions regarding the funding of grants applications. Once applications are assessed, SAC meetings are held to ‘review and consider applications for funding and to make recommendations to the ARC’s Chief Executive Officer (CEO)’. The CEO then submits advice relating to all applications to the Minister for Education, with recommendations on which applications are recommended, or not recommended, for funding. The Minister is required by both the NCGP guidelines and the ARC Act not to approve funding for any grant proposal that fails to meet the eligibility test contained in the guidelines.
3.28
The ARC also has an appeals process outlined in the standard operating procedures (SOPs) and the NCGP guidelines that enable unsuccessful NCGP applicants to have the decision regarding their grant proposal re-examined. Appeals are managed by the NCGP Appeals Committee, which considers issues relating to administrative process. The ANAO found that only a small number of cases are brought to the Appeals Committee, representing less than one percent of all proposals received by the ARC.
3.29
The ANAO found that the ARC had been compliant with the processes contained in the guidelines, the SOP, and the ARC Act in relation to its decision-making processes. All funded applications were found to be eligible in accordance with the requirements set out by the guidelines.
3.30
The audit found that the SAC appropriately ranked and recommended all grants for funding that were examined by the ANAO, and that the ARC CEO provided funding recommendations for all applications to the Minister.
3.31
In relation to advice provided to the Minister, the ARC was found by the ANAO to have:
…provided the Minister for Education clear advice and funding recommendations consistent with requirements of the ARC Act, the CGRGs and the NCGP guidelines.
Performance reporting
3.32
Performance reporting for Commonwealth agencies is directed by the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and its associated Rule and guidance issued by the Department of Finance (Finance). Taken together, this framework provides information in relation to aspects of public performance, particularly in relation to the connection between an entity’s purpose to its outcomes, and the clear reporting of how that entity uses its outcomes to achieve its purpose.
3.33
The ARC’s performance management and governance framework for the NCGP is contained in the agency’s Planning and Reporting Framework, which provides details regarding the ARC’s key activities and processes that meet its obligations under the Commonwealth framework. The ARC also utilises an annual Performance Measurement Framework, which provides information on performance measures as outlined in the Corporate Plan for the relevant year.
3.34
At the time of the ANAO’s audit, the ARC utilised 20 key performance indicators (KPIs) to assess and report its activities for the administration of the NCGP. These KPIs were incorporated into the ARC’s Corporate Plan 2018-19.
3.35
The audit report stated that:
The performance management framework established by the ARC for the NCGP is largely appropriate. Most of the KPIs are relevant but they require further refinement to be reliable and do not include any efficiency measures.
3.36
The ANAO assessed the KPIs for the 2018-19 reporting period in the context of their relevance and reliability for the NCGP program. The ANAO found that 17 of the KPIs were relevant, while the remaining three did not have enough information to allow the KPI to be clearly understood. Further, the ANAO’s assessment found that:
Seven performance indicators were assessed as being reliable, eight as partly reliable and five indicators were assessed as being not reliable due to insufficiently described measurement methods or insufficiently defined targets.
3.37
The audit further found that internal reporting on the NCGP included information such as point-in-time data and current status, but would be improved by historical or contextual information to strengthen the ARC’s oversight and reporting of the program. The ANAO recommended that the ARC improve the specificity of its performance reporting targets, ensuring that ‘its KPIs for the NCGP are reliable and include efficiency’.
3.38
Universities Australia (UA) agreed with the ANAO’s findings that performance reporting could be improved, but warned against overly strict reporting frameworks:
UA supports the use of baseline and trend data to show changes but advocates for caution in the use of absolute values as targets as these may distort the system in unanticipated ways. For example, a number of indicators under the ‘ARC funding supports research training and development’ element in the 2018-19 ARC corporate plan measure the absolute number of researchers. It would be more helpful to have a measure that shows these as a share of an appropriate total (e.g. the percentage of researchers on ARC-funded grants).
3.39
UA concurred with the ANAO’s assessment that the strategy ‘[c]ultivate a system-wide culture of research integrity’ required clarification.
3.40
The ARC agreed to the ANAO’s recommendation, and confirmed that it would review its KPIs for the NCGP program to include a measure of efficiency as outlined in the Resource Management Guidelines issued by Finance. The ARC stated:
The ARC has commenced an assessment of KPIs from other similar granting agencies both within Australia and internationally. Preliminary review has indicated that similar granting agencies, which fund long-term research, have found it difficult to develop efficiency targets. Internationally there appears to be no single indicator or evaluation method that adequately captures the results of research and development.
3.41
The ARC further explained that it was challenging for the entity to create and implement appropriate KPIs due to the nature of the industry. The ARC stated:
The KPIs contained in the Corporate Plan seek to reflect the nature of the research environment and the role of the ARC in that environment. Devising relevant and reliable targets for the research environment is a complex undertaking. Noting each individual research project funded by the ARC is unique and has different needs. To predetermine the success rates or projects funded, may lead to a result that is at odds with the ARC’s purpose, which is to fund the best quality projects that contribute to the growth of knowledge and innovation for the benefit of the Australian community.
3.42
The entity stated that it was also examining alternative methods ‘to demonstrate the qualitative value of funding excellent research’.
Managing conflicts of interest
3.43
The ARC utilises a range of processes in order to manage conflicts of interest. The agency has a conflict of interest and confidentiality policy, which requires that:
… all material personal interests are disclosed and that conflicts of interest are identified and managed in a rigorous and transparent way to ensure the integrity, legitimacy, impartiality and fairness of ARC processes.
3.44
The policy applies to all ARC staff and relevant personnel, including committee members, assessors and others engaged in work on behalf of the ARC, which includes organisations and individuals administering or receiving grants. The guidelines provide further details for grant applicants or organisations on managing conflicts of interest, outlining processes to appropriately identify and address potential conflicts of interest.
3.45
The ARC also uses information technology tools to manage potential conflicts of interests for ARC staff and contracted. The agency uses the Research Management System (RMS) to monitor relationships between assessors and program participants. The ARC prevents assessors from working on a particular proposal if there is evidence of a known relationship between the assessor and the participant or organisation involved.
3.46
The ANAO reported that the ARC has ‘established appropriate arrangements to manage actual and perceived conflicts of interest in the NCGP process’. The audit found that the ARC’s NCGP conflict of interest process was:
… robust, with well documented processes and business rules. The process for [all] Assessors is automated and embedded within the RMS. As all grant proposals are submitted through the RMS, all applicant and assessor conflicts of interest are dealt with through this system, and are compliant with the relevant NCGP conflict of interest SOPs and work instructions developed by the ARC.
Communication with stakeholders
3.47
Communication with stakeholders was a key theme during the ANAO’s audit and in the inquiry. The ARC has a number of communication strategies relating to grants, including: an external communications strategy, an industry communications strategy, a consultation framework, and an outreach and engagement strategy. Reports on communications activities are written at the end of each calendar year, and include a broad picture of the ARC’s various communications activities and outreach programs.
3.48
The ANAO reported that:
The ARC has implemented strategies to effectively communicate with key stakeholders. The ARC uses a variety of communications activities, targeting different audiences and key stakeholders, to provide information on NCGP guidelines and its elements, as well as NCGP-funded research outcomes.
3.49
The ANAO stated that stakeholders had expressed concerns regarding the communication of grant opportunities:
Stakeholders also advised the ANAO that improved engagement with the universities around the timing of grant opportunities would help avoid grant proposal submission dates falling during peak academic peaks, February and March.
3.50
Particular objections were raised regarding the timing of announcements in relation to the results of grant rounds. The International Australian Studies Association (IASA) stated that the ARC’s past practice has involved clearly articulated guidance regarding application deadlines and estimated periods of grant announcements. The IASA also stated that the ARC’s prior practice of alerting administering universities to successful grants applications had changed, which delayed the publication of outcomes and did not concurrently advise all applicants of outcomes.
3.51
Recent practice had changed, which the IASA explained means that:
Instead of posting all successful grants on the [RMS] page at the same time, the Minister for Education and/or parliamentary members made staggered announcements about individual grants over several weeks.
3.52
This concern was shared by UA, which stated that:
The sector has expressed concern about potential impacts to the management of research programs and staff due to the recently introduced practice of progressive announcements of ARC grant outcomes by the Minister and Government members and senators.
3.53
The IASA explained that the inconsistency in the timing of announcements could result in researchers being advised about the outcome of their proposal at different times, and delaying the commencement of successful projects. UA further explained that research organisations often partially fund projects from other sources prior to receiving the outcome of a grant proposal from the ARC. UA stated that the lack of communication from assessors regarding the outcome of applications ‘encourages an inefficient process, wherein applicants spend considerable time preparing budgets that may then be subject to shortfalls’.
3.54
The IASA recommended that the ARC appoint specific dates for the announcement of successful grants, and should be identified when grant applications open.
3.55
In response, the ARC explained that the progressive announcements process referred to by the inquiry participants was introduced in late 2019, and, following feedback, had since been paired with an early release of results to researchers advising on the outcome of grant applications. The rationale for this change was stated to enable researchers to begin planning and work on projects. The ARC explained that this process was similar to the grant announcements model used by the National Health and Medical Research Council.
3.56
Nonetheless, officials explained that announcements were ultimately made at the discretion of the Minister for Education.
3.57
Recommendation 5 of this report addresses improvements to the timely announcement and communication to stakeholders of grant opportunities and outcomes of grant programs.
Committee comment
3.58
The Committee recognises the results achieved by the ARC in administering the NCGP, highlighted by the findings of the ANAO. The Committee particularly notes the comments made by the ANAO that suggested that the ARC performed in accordance with best practice principles.
3.59
The Committee agrees with the ANAO’s findings that the ARC’s grants administration is robust and mature, particularly in areas such as conflict of interest management, application assessment processes, and decision-making frameworks.
3.60
There is scope for improvement in relation to certain aspects of the NCGP administration, such as identifying more specific key performance indicator targets. The Committee recognises the unique challenges faced by the ARC in identifying particular KPIs regarding the efficiency of its programs. The Committee agrees, however, with the ANAO’s observation that specific performance targets are essential in public governance and administration to achieve strong results in grants administration in addition to providing certainty to Parliament that the agency is meeting its reporting obligations.
3.61
The Committee observes that the ANAO did not provide substantial comment on the communication practices of the ARC in making grants announcements. The ARC’s explanation of action provided since the audit’s conclusion, however, addresses some of the concerns raised by submitters in relation to the timing of grants announcements and providing sufficient notice for researchers and organisations to commence successful grants.
3.62
The Committee also recognises the practical difficulties experienced by universities and other administering organisations in managing grants, such as managing partially-funded projects which are awaiting grant funding decisions, and providing sufficient feedback to applicants.
3.63
The Committee encourages the ARC to work with stakeholders to identify methods of communicating grants outcomes and ensuring researchers and organisations are appropriately advised of outcomes in a timely way.