Chapter 2 Audit Report No. 07 2009-10 Administration of Grants by the National Health and Medical Research Council

Chapter 2 Audit Report No. 07 2009-10 Administration of Grants by the National Health and Medical Research Council

Introduction[1]

2.1                   The National Health and Medical Research Council (NHMRC) is a statutory agency within the Health and Ageing portfolio, with a total annual budget of around $1 billion. The agency, which has existed in various forms since 1936, is widely regarded as one of Australia’s peak bodies in the area of evidence-based health advice, and is a significant provider of grants to support health and medical research in Australia.

2.2                   Over the years, NHMRC grants have contributed to progress in many areas of health and medical science, from advancing knowledge and treatment of cancer to preventing cardiovascular disease and improving the health of Aboriginal and Torres Strait Islander Australians.[2] NHMRC investment in health and medical research, on behalf of the Australian Government, is estimated at 16 per cent of the total national investment by both public and private sectors.[3] In 2008, the NHMRC administered 3843 new and continuing grants, accounting for $595 million in expenditure.

2.3                   The grants are a vital source of income for many health and medical researchers. Individual researchers can apply to the NHMRC via their universities or research organisations for grants to cover research projects or multi-component research programs, salaries and infrastructure support. The grant process is highly competitive, with less than 30 per cent of applications receiving funding each year.

2.4                   Grant applications are assessed on the basis of scientific merit through a process of peer review and expert panels – the objective being to select the highest calibre research for funding. This selection process relies heavily on the participation of NHMRC grants. The integrity of the selection process is therefore fundamentally important, as it underpins the advice that the NHMRC provides to the Minister for Health and Ageing for approval of the grants with the highest potential to deliver beneficial outcomes for Australia.

Changes to the NHMRC since 2006

2.5                   In July 2006, the NHMRC became a statutory agency with responsibilities specified under the 2006 amended National Health and Medical Research Act 1992 (NHMRC Act). The NHMRC Act defines the NHMRC as the Chief Executive Officer (CEO), the Council and its committees and the staff of the NHMRC. The NHMRC is also a prescribed agency under the Financial Management and Accountability Act 1997 and the Public Service Act 1999.

2.6                   Since 2006, the NHMRC has experienced a period of transition, facing several challenges as it separates its administrative functions from the Department of Health and Ageing (DoHA) and adjusts it governance and administrative arrangements to support its legislative responsibilities and core business – particularly grant administration. The agency has also had a substantial change agenda, particularly in developing new IT systems to improve its data capacity and grant management functions.

Increased funding for NHMRC grant programs 2000-08

2.7                   Funding for NHMRC grants is administered through a special account, the Medical Research Endowment Account (MREA), established under section 49 of the NHMRC Act. From 2000 to 2008, a series of government initiatives to bolster Australia’s research capacity resulted in more than a three-fold rise in the NHMRC’s grant budget and a corresponding two-fold increase in active (new and continuing) grants. Over this period, the NHMRC awarded more than eight thousand grants, an investment in research exceeding $3.2 billion.

The NHMRC grant process

2.8                   Each year, the NHMRC invites researchers in eligible Australian universities and research organisations to apply for funding through its range of scholarships and research programs. Grant programs generally fall into three groups based on the intended use (or type) of the grant: Research Support; Infrastructure Support; and People Support. In 2007-08 funding for Research Support was $440 million, with the largest scheme, Project Grants, accounting for $283 million of this amount.

2.9                   NHMRC grant programs are based on a competitive selection process. Grant applications are reviewed and ranked by a process of peer review, using external assessors and expert Grant Review Panels (GRPs), with a view to selecting research of the highest calibre for funding.

2.10               The NHMRC also calls each year for academics to participate as assessors and members of the GRPs. To comply with the NHMRC’s policies and guidelines, and prior to accepting grants for review, these individuals are required to declare any conflicts of interest that could affect their impartiality in assessing and selecting grants.

2.11               In 2008, the NHMRC received over 2586 applications for Project Grants – the largest NHMRC grant scheme. For this scheme, over 449 assessors, 42 GRPs and 499 GRP members were involved in the grant selection process. Based on the selection process, advice is provided to the Minister of Health and Ageing, who has responsibility for the final approval of grants for funding. Success rates vary between the different schemes.

Deed of Agreement with Administering Institutions

2.12               Administering Institutions (mainly universities) play an important role in the NHMRC’s grant process, by acting as a conduit for grant enquiries, submission of applications and post-award management of grants.

2.13               Under NHMRC policy, only approved Administering Institutions may receive NHMRC grants. Each Administering Institution must sign a Deed of Agreement (the Deed) with the NHMRC, which establishes the parameters and expectations for the management of grant funds, accountability and reporting requirements. Under this arrangement, each Administering Institution has responsibility for the effective management of the NHMRC research projects and associated grant funds provided by the Commonwealth. An important role for the NHMRC is in managing the relationship with the Administering Institutions to achieve effective and accountable administration of grants.

Previous audit coverage

2.14               A previous ANAO audit, Audit Report No. 29 2003-04, Governance of the National Health and Medical Research Council, examined the governance of the NHMRC and made six recommendations.[4] Subsequent to that audit report, the accountability and governance arrangements of the NHMRC were amended (post Uhrig Review) to reflect a whole of government shift to improved governance and accountability.[5] The NHMRC’s revised governance arrangements are examined in the current audit, in the context of grant administration.

The Audit

Audit objective[6]

2.15               The audit objective was to form an opinion on the effectiveness of the NHMRC’s grant administration. To meet this objective the NHMRC was assessed against four criteria:

Overall audit conclusion

2.16               The ANAO made the following overall audit conclusion:

The National Health and Medical Research Council (NHMRC) has a key role in providing grants to support health and medical research in Australia. NHMRC grants are an important source of income for many health and medical researchers, and constitute a substantial Government investment in research and innovation in Australia. Over the period 2000 to 2008, Government initiatives to strengthen Australia’s research capability resulted in more than a three-fold increase in NHMRC grant funding, with a corresponding two-fold rise in the number of grants. The NHMRC’s investment in research during this time exceeded $3 billion.

Against this background, since 2006 the NHMRC has been adjusting to its new responsibilities and expectations as a statutory agency. Consistent with the revised National Health and Medical Research Council Act 1992 (NHMRC Act), high level governance arrangements are in place: a Chief Executive Officer (CEO); established governance structures which include the Council and its committees; and defined responsibilities for each of these governing entities. These arrangements constitute a sound basis for the agency’s governance and a platform from which to address challenges and expectations arising from broader Government initiatives to enhance investment in Australia’s health research sector.

However, the NHMRC is an agency in transition, with a substantial change agenda. Particularly evident is the gradual transfer of key administrative functions from the Department of Health and Ageing (DoHA), culminating in the NHMRC’s growing administrative independence. In recognising weaknesses in its own management of grants, the NHMRC has also reviewed its grant processes and compliance framework, and commenced a $3 million project to develop a new grant management system.[7]

2.17               The ANAO found a number of shortcomings in the NHMRC’s administration of the grant program:

2.18               To improve the overall grant administration process, the ANAO suggests that the NHMRC focus on the following:

2.19               The ANAO made the following overall comment on improving these aspects of the grant administration process:

Enhancing management of key aspects of the grant selection process

Selection of grants for funding involves a process of peer review, with appraisal of applications by external assessors and a Grant Review Panel (GRP) comprised of relevant experts. This process carries inherent risks for the NHMRC, as it relies on the commitment of experts from within the research community, who, at times, are members of the NHMRC Council and its committees, assessors and members of GRPs, or are themselves recipients of NHMRC grants. As NHMRC grants are highly competitive, the selection of the highest calibre grants is largely reliant on the NHMRC’s ability to maintain a fair and defensible peer review process.

The NHMRC provides guidelines and procedures to assist reviewers in conducting peer review and grant selection, and expects them to adhere to conflict of interest provisions. However, the NHMRC was not consistent in its application of key elements of the grant selection process, including grant eligibility requirements, recording of grant scores and key actions of the GRPs, and implementation of conflict of interest provisions.

Closer monitoring and scrutiny of the selection process is required to provide the NHMRC with the confidence that its policies and guidelines are being consistently and appropriately implemented. Clear recording of the GRP’s key actions and recommendations, and the reasons underpinning these, will promote a more defensible grant selection process and better position the NHMRC in responding to unsuccessful applicants or contested grant decisions. Overall, these improvements will allow the NHMRC to achieve greater transparency and probity in its grant selection process.

Improving assurance of the appropriate management and use of grant funds

To provide confidence that Commonwealth funds will be used appropriately and for the purpose they are intended, grants are awarded only to approved Administering Institutions, and administered under a Deed of Agreement (the Deed) that sets out the terms and conditions for the management of grants.

Owing to several shortcomings in the certification of Administering Institutions, and the monitoring and management of grants, the NHMRC is not well placed to provide adequate assurance about the use of grant funds. There is a general lack of compliance monitoring around reconciliation and reporting of grants, with NHMRC’s main grant management systems having no monitoring capability. This has diminished the NHMRC’s ability to account for grant funds, reducing its efficiency in its own policy for approval of Administering Institutions or a compliance framework for post-award management of grants.

It will also be necessary for the NHMRC to implement a workable risk-based certification process for Administering Institutions and a systematic and sustainable approach to monitoring compliance with the Deeds, reconciliation of grants and recovery of debts.

Implementing an appropriate grant management system

A suitable automated grant management system can assist in monitoring the progress and outcomes of grants. This is particularly the case for the NHMRC given its considerable investment in research and the large volume of applications processed each year.

The NHMRC’s information systems do not adequately support the NHMRC’s core business – grant management. Its primary grant management system contains substantial data anomalies. Furthermore, the system does not accommodate the monitoring of grants’ financial and progress reporting requirements, or capture qualitative information from submitted grant reports. This diminishes the NHMRC’s capacity to gather and evaluate valuable information for reporting against program outcomes.

The NHMRC was advancing development of a new grant management system, and a data repository designed to improve the NHMRC’s data capacity. To obtain the most benefit from its new systems will require the NHMRC to focus on system interfaces, adopting a more rigorous but sustainable program of data maintenance and improving staff training in grant management. It is important that the grant system incorporates adequate controls to allow better management of eligibility issues and non-compliance against the Deed.[10]

ANAO recommendations

Table  2.1 ANAO recommendations, Audit Report No. 07 2009-10            

1.

To provide adequate assurance that the NHMRC grant funds are being managed appropriately by Administering Institutions, the ANAO recommends that the NHMRC:

·         complete the development and implementation of a risk-based assessment for approval of Administering Institutions, and systematically maintain complete records of those approvals; and

·         implement arrangements to improve monitoring of the Administering Institutions’ compliance with the requirements of the Deed of Agreement, including conducting audit activity where a high risk is indicated or persistent non-compliance evident.

 

NHMRC response: Agreed

2.

To improve the transparency and probity of its peer review process, the ANAO recommends that the NHMRC:

·         monitor the incidence and reasons underpinning the allocation of Grant Review Panel (GRP) members’ application to their own GRP for assessment; and

·         enhance the documentation of key actions and recommendations of the GRPs, in order to provide a defensible record of the selection proceedings and strengthen feedback to applicants.

 

NHMRC response: Agreed

3.

In order to improve the identification and management of conflict of interest, the ANAO recommends that the NHMRC:

·         amend its conflict of interest guidelines to strengthen guidance on acceptable and unacceptable conflicts of interest; and

·         develop a risk-based strategy for more systematic monitoring and review of conflict of interest compliance, including a register of private interests.

 

NHMRC response: Agreed

4.

To improve accountability of grant funds, the ANAO recommends that the NHMRC implement risk-based arrangements including enhanced systems to:

·         manage overdue annual financial reports and final acquittal statements;

·         recover debt due to overpayments and unspent funds; and

·         achieve timely receipt, review and analysis of grants’ progress and final reports.

 

NHMRC response: Agreed

5.

To strengthen the NHMRC’s management of grants, the ANAO recommends that the NHMRC include as part of the new Research Grant Management System (RGMS):

·         appropriate compliance controls to identify breaches of legislative, key policy and eligibility requirements for all grant applications;

·         a suitable interface between RGMS and the NHMRC’s financial system to allow accurate information exchange and regular reconciliation of the systems;

·         a regular program of data verification and cleansing to prevent corruption of future NHMRC data; and

·         a structured training program and complete documentation for all key processes.

 

NHMRC response: Agreed

The Committee’s review

2.20               The Committee held a public hearing on Wednesday 12 May 2010, with the following witnesses:

2.21               The Committee took evidence on the following issues:

Research areas

2.22               The Committee understands that the National Health and Medical Research Council Act 1992 requires the NHMRC to determine its targets for research grants in accordance with major national health issues identified in consultation with the Minister for Health and Ageing. According to the  NHMRC Strategic Plan the research areas remain flexible to accommodate the changing needs of the Australian community over the period of the Plan:

NHMRC will help Australia deal successfully with health issues as they arise. These include emerging issues for the health system or individuals, or new health and medical research developments. ... NHMRC, therefore, needs to be flexible to meet unforeseen challenges that may arise during the period covered by this Strategic Plan.[11]

2.23               The Committee sought clarification regarding how the NHMRC narrows its research priority areas for grant funding and whether or not the agency places any restrictions on research areas.  The NHMRC explained to the Committee that there are no restrictions provided an applicant satisfies the eligibility requirements.[12] The CEO added that NHMRC has developed a series of funding vehicles to ensure a balance of funding between a variety of types of research including laboratory research, clinical research and public health research.[13]

2.24               The Committee asked specifically whether or not complementary health and alternative health were included in the targeted research areas. The NHMRC confirmed that both areas are included in the current Strategic Plan.[14] The ANAO report noted that in 2009, NHMRC received 35 applications for research in these areas and that 12 (34.3 per cent) were funded at a cost of $4.5 million.[15]

2.25               The Committee expressed concern that research aimed at discrediting complementary medicine may be funded and asked for examples of the types of successful projects. The NHMRC maintained that a grant application that showed a ‘clearly prejudiced expectation’ would not be considered scientifically sound and would not therefore be funded.[16] Professor Anderson described the focus of a number of successful projects in this research area:

There have been quite a lot looking at Chinese traditional medicine, about the effectiveness of that. There have been some looking at Indigenous Aboriginal traditional medicines. There have been grants looking at the chemistry of extracting of herbs to increase purity and that sort of stuff.[17]

Commercialisation

2.26               The Committee was particularly interested in the commercialisation of research funded through the NHMRC grant program and asked the agency if it had data on the issue. The NHMRC informed the Committee that no rigorous study had been undertaken into the overall monetary benefits of the program but that limited research has been done by Access Economics and the NHMRC itself. The CEO told the Committee that these studies indicated considerable benefits are flowing to the Australian economy from the investment in research:

(The Access Economics study found) that the benefits to the Australian economy of the cochlear ear implant and CSL, including Gardasil, are about equivalent to the entire government’s investment in health and medical research over that period of time. We also did a study a couple of years ago – and we are repeating it – where we looked at 1,208 grants and asked the grantees what the benefits were. Quite apart from the rapid growth in patents and intellectual property protection they have done, they also reported on their leverage of funds into Australia. I think for every government dollar these people levered about 30c one way or another on top of that.[18]

Conflict of interest

2.27               The ANAO acknowledged that conflicts of interest were inevitable in the peer review process used by NHMRC to assess grant applications, given the ‘small pool of researchers available to assess applications in specialist areas’ in Australia.[19]  However, the ANAO found that the NHMRC guidelines needed to be strengthened to provide ‘greater clarity on the types of conflict of interest and situations in which these are relevant’ and that conflict of interest declarations needed to be monitored for compliance.[20]

2.28               The Committee asked the ANAO how many potential reviewers declared a conflict of interest. The ANAO replied that, over a two year period, 1,200 researchers had declared a conflict of interest.[21]

2.29               The Committee asked the NHMRC to clarify how the conflict of interest process works. The NHMRC explained to the Committee that every reviewer is asked to declare on every grant application if they have a conflict of interest.[22] If the reviewer declares a conflict of interest they will not see that particular grant application and will be excluded from all consideration of that application:

The first step is that you do not even get to see the grant in the first place on the panel. Then, when the panel meets, you are outside the room. When the panel is ranking the grants, you are outside the room and the ranking is blind to you.[23]

2.30               Nevertheless, the NHMRC admitted that a grant can go to someone who is on a selection panel. Asked by the Committee to explain how this occurs, the NHMRC reiterated that the reviewer would not be in the room when the application was being assessed but would be present when the cluster of applications was being considered:

... they are not there when they are being ranked and not there when they are being reported.[24]

2.31               The NHMRC added that the research community expects an open and transparent system and that the panels are independently monitored to ensure this:

We have our own staff as well as observers. So we also recruit lay people to look at these panels and report directly to us on how the panel has been operating. This is a very important perception that the research community keep us very much up to the mark on. They want a level playing ground, so they are very diligent with us ...[25]

Assessment and selection of grants

Fairness

2.32               The Committee raised the issue of the fairness of the assessment of grant applications and asked the NHMRC if a reviewer’s bias could affect the success of a grant application particularly with regard to fashionable or popular research topics. The NHMRC assured the Committee that the assessment process mitigated the risk of personal bias or prejudice influencing a decision:

These panels have 10 to 12 people, and the chair’s role is to make sure that they are all put to the test in what they are saying. Remember that not only do the panel have another 10 people looking at what they are doing but there are one or two written external reviews on every grant, so there is an independent review by experts. It is not impossible but hard to bring personal prejudice to the table because in this scheme we are funding one in five applications and they are scoring as ‘outstanding’ and ‘excellent’ – the very top of the very good. For every one we fund, our panel has ranked three more as worth funding but unable to be funded. The pressure is so high that something that is just a passionate view is very unlikely to get up.[26]

Documentation and procedures

2.33               The ANAO found that the documentation for grant approval often lacked a clear trail and that it was difficult to establish the reason why scores had been altered or budgets reduced.[27] The ANAO told the Committee that records were incomplete and that online databases and hard copy records could not be reconciled.[28] While the ANAO emphasised that it was not questioning the legitimacy of grant approval decisions, it identified the issue as a serious impediment to transparency and accountability.[29]

2.34               The Committee asked the NHMRC what steps had been taken to improve documentation and procedures. The NHMRC told the Committee that it is developing and implementing a new Research Grant Management System (RGMS) that will enable online application, appraisal and management of grants.[30] The system is expected to streamline and standardise the process, eliminating many of the discrepancies identified by the ANAO audit.[31]

2.35               Other improvements identified by the NHMRC include the capacity to undertake the peer review process in-house which will allow greater control of documentation and the appointment of an independent chair for every grant review panel.[32]

Peer review process

2.36               Given the inherent nature of the peer review process, the Committee questioned whether or not the ANAO’s recommendations for greater accountability and transparency were too demanding. The CEO admitted he had initially had doubts about the practicality of the recommendations but has come to the conclusion that the process will be improved by meeting the requirements:

I think in the discussions during the review I probably did have some concerns that there was perhaps a misunderstanding about what would work and what does not. Peer review is eventually an opinion by somebody who is worthy to give that opinion, but at the end of the day I think that the recommendations are compatible with nevertheless having high-quality peer review without influencing that. Indeed ... it will improve that. For example, this year for our panels we are going to not only document it more but we are going to be making them nail their colours to the wall on each of our selection criteria, not just the overall one.[33]  

Administering Institutions

2.37               The ANAO report was critical of the both the NHMRC’s certification process for Administering Institutions and the ongoing monitoring of the Institutions for compliance.[34] The ANAO noted that NHMRC’s inability to ‘provide assurance that Administering Institutions are sound administrative entities with suitable capability to administer grants’ presents a significant risk to Commonwealth funds.[35] The NHMRC had advised the ANAO that it was revising its certification policy and developing and implementing a new compliance framework for Administering Institutions.[36]

2.38               The Committee asked the NHMRC what steps had been taken to implement the new framework. The NHMRC informed the Committee that the agency has set up a separate, independent quality and regulation branch to consolidate the staff and processes involved in the certification procedure.[37] This branch also handles complaints.[38]

2.39               Additionally, the NHMRC has revised the Deed of Agreement, the legal contract between the agency and Administering Institutions setting out terms and conditions for the administration of grants.[39] As at May 2010 the new Agreement is out for consultation with the community and, after relevant feedback is incorporated, will be forwarded to the NHMRC research advisory committee and Council.[40]

2.40               Further, the NHMRC told the Committee that the overall policy framework for Administering Institutions has been updated and as at May 2010 is out for comment.[41] The CEO explained that there have been some delays outside the agency’s control in finalising the new policy documentation but it expects completion by the middle of 2010:

The first is that the Department of Innovation, Industry, Science and Research had just changed their rules about the support of the indirect costs of research, and that has impacted on the relationship between universities and medical research institutes. That has affected the way we need to approach our policy. As you are probably aware too, the government’s considerations around health reform, the role of research in that, and the comments about supporting the current costs of research and training also mean that we need to make sure that our policy does not stand in the way of the cooperation that we hope for. So there have been a couple of background things that have led us to be a little slower than we want.[42]

Hospitals and smaller institutions

2.41               The Committee raised concerns regarding the impediments faced by hospitals and smaller institutions attempting to access NHMRC grant funding. Members of the Committee had received anecdotal evidence that smaller institutions faced significant on-costs in applying for and administering grants.

2.42               The NHMRC admitted that it is easier for larger institutions to absorb on-costs and suggested that ‘smaller institutions need to think about whether their overhead costs compared to their research activities are not disproportionate’.[43]  However, the NHMRC is aware that their size allows smaller institutions to be more flexible and innovative in their research proposals.[44]

2.43               The NHMRC advised the Committee that it encourages collaborative research approaches to ensure that the capacity of smaller institutions and hospitals can be tapped.[45] The agency explained that Australia has an excellent reputation for collaborative research and that the majority of its grants are awarded to research teams rather than individuals:

That is the way of medical research these days. You often need, say, a cutting-edge geneticist, a behavioural scientist, an epidemiologist and then a clinical oncologist on a grant. I do think there is something in the Australian characteristic, if you like, that makes that collaboration fairly easy. We hear this from Australians – who think this is a good thing – who come back to Australia after years in, say, the United States, where the culture around this is very different.[46]

2.44               The Committee asked for clarification regarding the difficulties hospitals face in accessing research grants. While emphasising the importance of clinical research, the NHMRC explained that currently hospital research does not attract a distinct stream of funding and that research projects may not be a high priority for hospital administrators:

It does not mean that some hospitals have not been fantastic about that, but, if you are a hospital administrator – if you have ambulance bypass problems or patients stacked up in emergency – you can understand the priority. So I think it is important for the future that the support for clinical research is there beside the patients in the hospital and that the health services research is there in the health system and is supported in that sort of way.[47]

Post-award grant management

2.45               The ANAO found a number of deficiencies in the NHMRC’s administration of post-award grant management.[48] The ANAO noted that this has historically been a problematic area for the NHMRC but acknowledged that the agency is taking steps to rectify the issues.[49]

2.46               The Committee asked the NHMRC what changes it had made to achieve the improvements and what success it had had to date. The NHMRC told the Committee it had set up a separate section to handle post-award management and had instituted rigorous processes and procedures to address the issues raised by the ANAO:

We set up a specific section, developed some very comprehensive standard operating procedures and really focused on the acquittal process.[50] 

2.47               The Committee was pleased to hear that outstanding acquittals had been reduced from 1275 in 2008 to approximately 100 as at May 2010.[51]

Conclusion

2.48               The Committee acknowledges the benefits to Australian health flowing from research supported by the NHMRC grants system and notes the upheaval caused by the NHMRC’s recent separation from the Department of Health and its move to a statutory agency. The Committee recognises that NHMRC is still consolidating its new status but is pleased to see evidence of a tightening in administrative practices, processes and procedure. 

2.49               While acknowledging the difficulties inherent in the peer review process, the Committee urges the NHMRC to implement the ANAO recommendations aimed at strengthening accountability and transparency throughout the peer review process. In particular, the Committee is concerned that conflict of interest issues may encourage a perception that the assessment and selection of grant applications is not open and fair.

2.50               The Committee commends the NHMRC for the ongoing improvement in post-award grant management but urges the agency to implement the ANAO recommendation to implement risk-based arrangements to ensure better management of Commonwealth monies.

Navigation: Previous Page | Contents | Next Page