OIG audits of Global Fund grants to Rwanda and Benin raise issues related to data availability and quality

3. NEWS
19 Mar 2019
For grants in Rwanda, which follows a results-based financing model, availability of good and reliable data is especially critical

The recent audit of Global Fund grants in Rwanda by the Office of the Inspector General (OIG) was primarily about data –– specifically, the availability and quality of data. Similar issues were raised in the OIG’s recent audit of grants in Benin. This article provides a summary of the OIG’s findings for both audits.

The OIG published a report on the Rwanda audit on 25 February, and one on the Benin audit on 8 March.

Rwanda Audit

“Rwanda continues to achieve impressive programmatic results with the support of the Global Fund and partners,” the OIG observed. Rwanda has experienced a decline in HIV- and TB- related deaths, with an increased number of people on antiretroviral treatment and relatively high TB treatment success rates. HIV-related deaths fell from 9,600 in 2007 to 3,100 in 2017, partly due to the implementation of a “treat all” policy.

The audit found that although Rwanda’s systems and controls to safeguard data quality are adequately designed, there are weaknesses in implementing the systems and controls, especially for malaria. (Financial assurance is generally effective, the OIG observed.)

The audit covered all three active grants in Rwanda, one for each disease, for the period January 2016 to 30 June 2018. The principal recipient (PR) for all three grants –– RWA-H-MOH, RWA-T-MOH and RWA-M-MOH ––was the Ministry of Health (MOH). However, the OIG said that its findings concerning data were limited to the malaria grant, which accounts for 20% of the Global Fund’s active grants in Rwanda. The OIG found fewer issues related to data in the HIV and TB grants.

Despite prevention efforts, the OIG said, malaria cases rose threefold, from 1.7 million in 2014 to 4.9 million in 2017, the highest increase in Africa. The OIG observed that Rwanda’s data quality measures are not robust enough to respond to the increase in cases. The OIG recommended that the effectiveness of Rwanda’s prevention activities be re-examined.

The Global Fund has been using a results-based financing model, also known as national strategy financing, in Rwanda since 2014. According to the OIG, Rwanda is the only major portfolio using this model. Under the model, annual disbursements are directly linked to reported programmatic results, so the availability of good and reliable data is critical.

The OIG rated the systems and processes to safeguard the reliability of the data as “partially effective.” It rated overall assurance and oversight mechanisms as “needing significant improvement.” (The OIG has a four-tiered rating scheme: effective; partially effective; needs significant improvement; ineffective.)

The OIG said that the weaknesses it identified “do not necessarily call into question the relevance of the results-based financing model in Rwanda, a country that has demonstrated strong accountability mechanisms and a track record of effective program implementation.”

The OIG noted that a separate review, commissioned by the Secretariat in 2018, confirmed the continued relevance of the model (see GFO article from 7 November 2018). “However,” the OIG said, “failure to strengthen controls over data and the scope of assurance would undermine, over time, the reliability of the results based on which the Global Fund is supporting Rwanda’s health programs.”

Problem areas

The audit identified four main problem areas, as follows:

  • There are weaknesses in data systems and controls.
  • There are differences between consumption and patient data.
  • Limited survey results are used to validate programmatic achievements.
  • There is insufficient oversight and assurance over data.

 

Below, we briefly describe each problem area.

Data systems and controls

HMIS

Rwanda uses an electronic health management information system (HMIS) to record and report program results. The OIG said that there a number of controls in place –– e.g., qualified data management staff; monthly and quarterly data validation –– but that these controls are not being effectively implemented and that weaknesses exist that may affect the quality of data. This was an issue primarily for the malaria grant.

The audit revealed a number of control deficiencies. (There was a similar finding in the previous OIG audit conducted in 2014.) For example, the OIG said, although standard operating procedures outline the required approval levels before reported results can be altered, two-thirds of the health facilities visited during the audit had changed the malaria results without obtaining approval after they had already been reported. The changes resulted in an increase of 19% in the reported number of malaria cases at the facility level.

Incomplete malaria registers

National guidelines require that all results reported in HMIS be supported by primary records, referred to as “registers.” However, the OIG noted, almost two-thirds of the 100 community health workers (CHWs) sampled during the audit did not have registers for prior periods.

The audit identified problems with the data entered on registers. For example, the results of malaria tests were generally not recorded in the registers, making it difficult to determine the outcome of the diagnosis and subsequent treatment recorded in the HMIS. (A spokesperson for the OIG explained to the GFO that the register is the main source document. If it is not properly maintained, it is not possible to ensure that the HMIS data is correct.)

Inadequate supervision of community health workers

The 45,000 CHWs in Rwanda have limited capacity to record data in a timely manner, the OIG stated. As a result of a resurgence in malaria cases, in December 2016 the MOH revised its guidelines to enable CHWs to manage malaria cases in adults as well as children under 5. This   further strained the CHWs’ capacity.

Staff in the health centers are supposed to visit each CHW monthly. However, for more than three-quarters of the CHWs sampled, the OIG found no evidence of any visits for the 12 months preceding the  audit. Since the audit, the OIG said, the MOH has started redesigning the supervision framework.

Routine data checks not consistently performed

Staff at health facilities are supposed to perform monthly and quarterly data validation exercises before reporting their results. The OIG found that only 59% of the health facilities it visited were performing the validations.

Part of the problem, the OIG explained, is that a new structure for health facilities was implemented in June 2017, resulting in some changes in personnel. The MOH is rolling out an electronic learning platform for newly recruited data managers and supervisors.

Differences between consumption and patient data

The MOH has installed an electronic Logistics Management Information System (eLMIS) in all district pharmacies, district hospitals and health centers to record medicines received and issued to patients and CHWs. The Central Medical Store utilizes an inventory management system known as “SAGE.” The SAGE and eLMIS systems had yet to be integrated at the time of the audit.

According to the OIG, the malaria data showed that the number of patients diagnosed with test kits and treated with anti-malaria drugs was about 41% higher than the quantity of test kits issued. The number of patients treated for uncomplicated malaria was 34% higher than the quantity of anti-malaria medicines consumed.

“These results are contrary to audit findings in most other countries which typically show that more test kits and anti-malaria medicines have been used compared to the number of cases recorded,” the OIG declared.

Use of limited survey results

The MOH uses two main mechanisms to collect the data reported to the Global Fund: routine data and survey data. For 2015-2017, nine of the performance indicators for the malaria grant were assessed through surveys (which are conducted only once every two years) while 14 indicators were monitored using routine data.

The OIG said that some indicators assessed through surveys could more easily be monitored using routine data. The latest community level survey for malaria case management covered only 57 of the 45,000 CHWs, the OIG said, and may not be representative of the national average. The OIG also said that the survey methodology should be reviewed.

Insufficient oversight and assurance over data

In line with the principle of country ownership, the OIG stated, the Secretariat relies on Rwanda’s existing national systems and controls rather than independently verifying reported results. Following the OIG’s 2014 audit, the Secretariat undertook to routinely review the systems and controls; however, this practice has not been effectively implemented, the OIG said.

The MOH is responsible for conducting bi-annual integrated supportive supervision and data quality audits (ISS-DQAs). These audits verify the systems to validate reported results for one indicator (out of 14) for each grant. The OIG said that the audits did not cover the effectiveness of the IT systems and controls that produce the results. In addition, the audits were inconsistently performed.

Through the local fund agent, the Secretariat conducted a review in January 2016 which found that the systems issues identified in the 2014 audit had not been fully addressed.

Agreed management actions (AMAs)

There were four AMAs (see table). All are owned by the Head, Grant Management Division and all have a target date of 31 December 2019.

Agreed management actions
1.

Following the actions taken by the country since the audit, the PR, with support from the Secretariat and in collaboration with partners will:

a.  Further strengthen the electronic HMIS and associated IT controls including: ensuring automatic locking of the HMIS after each reporting period; updating data validation rules to prevent entry of negative numbers; securing access for users to physical servers; and restricting super-user access to a minimum; and
b.  Improve community level data and the supervision arrangements by recruiting and training additional Community and Environmental Health Officers in data management and supervision of CHWs; and standardizing use and management of registers across health facilities and communities.

2.

In order to improve the reliability of supply chain data, the Secretariat will support the PR and partners to:

a. Perform further analysis of the differences identified in the audit report in order to identify the underlying reasons for discrepancies between consumption and patient data in line with terms of reference that are to be agreed with the OIG;
b. Develop an action plan including timelines and responsible parties to address the identified underlying causes; and
c.  Conduct at least annually data triangulation or external consistency checks in line with WHO guidelines on data quality review.

3.

a. In light of the progress made in the country’s routine data systems, the Secretariat in collaboration with partners will support efforts to ensure the PR uses available routine data from the HMIS to report on malaria treatment; and
b. The Secretariat will review and approve all survey protocols developed for the collection of survey-based data used by the PR for reporting on the agreed indicators in the grants. The protocols will include sample sizes and data collection methodologies.

4.

a. The Secretariat will update the portfolio’s assurance plan to include independent verification of data systems and related IT systems that produce the results. This plan will include: assurance activities relative to the identified controls on data systems; the frequency of these activities; and the assigned assurance providers; and
b. The Secretariat will ensure that the PR develops an action plan to address outstanding audit recommendations from Rwanda’s Office of the Auditor General.

Note: The Community and Environmental Health Officers supervise the CHWs. At the time of the audit, there was only one officer at each health facility.

 

++++++++++++++++++++++++++++++++

Benin Audit

The audit of grants to Benin found that the country has made significant progress in the fight against the three diseases despite challenges related to program implementation. The audit covered the period June 2016 to June 2018 and included all five active grants –– two HIV, one TB, one malaria and one RSSH.

The PRs for the HIV grants were the national AIDS program (BEN-H-PSLS) and Plan International (BEN-H-PlanBen). The PR for TB grant was the national TB program (BEN-T-PNT); for the malaria grant, the national malaria program (BEN-T-PNLP); and for the health systems strengthening grant, the Health System Performance Program (BEN-S-PRPSS). 

Examples of the progress Benin has achieved are as follows:

  • Benin has one of the highest rates of HIV treatment coverage in West and Central Africa.
  • At the time of the audit, Benin’s TB treatment success rate exceeded 89%, compared to an average of 79% in the region.
  • Malaria mortality rates dropped significantly between 2011 and 2016. Malaria cases confirmed through testing rose from 354,223 to 1,219,975 due to increased use of rapid diagnostic tests.

 

The audit identified three overarching issues and risks:

  • Non-compliance with procurement plans and guidelines;
  • Uneven quality of services; and
  • Deficiencies in the quality of data.

 

Below, we briefly describe each of these issues and risks.

Procurement plans and guidelines

Procurement of health commodities in Benin is in line with national guidelines, the OIG said.  Annual quantification exercises are carried out, leading to the development of a consolidated procurement plan.

The OIG said that there are “stock tensions” at the central level and stockouts at the facility level due to a failure to comply with the consolidated procurement plan. (A spokesperson for the OIG told the GFO that “stock tension” refers to a situation where there are persistent low stocks, with buffer stocks becoming depleted.)

The audit found delays in delivering health commodities: Malaria drugs that were supposed to be supplied by the government were not procured in 2017 and 2018 due to budget insufficiencies and complex public procurement processes. Delays in some of the partners’ procurement processes aggravated the situation, the OIG observed.

At the central level, the audit found, there was less than the required three-month buffer stock of malaria drugs. In addition, there was a stock-out of rapid diagnostic tests.

In addition, the OIG noted, managers at district and community levels rarely complied with procurement guidelines. The OIG said that a key root cause of these problems is the low number of trained pharmacists: There were no pharmacists at eight district warehouses and hospitals sampled by the audit.

Quality of services

The audit found that access to quality malaria services is low, particularly at the community level.

Malaria treatment is not consistent with what the treatment guidelines call for, the OIG stated. Registers show more people treated than people testing positive; people testing negative receiving treatment; and people testing positive not receiving treatment. The OIG said that these problems were due to the limited capacity of CHWs, who lack malaria health care–related training as well as support and supervision from health facility nurses.

The audit found that for HIV, there are no appropriate national guidelines for “therapeutic education” for prevention of mother-to-child transmission (PMTCT). (“Therapeutic education” refers to the advice and support midwives provide to pregnant women.) Services vary depending on each midwife’s understanding, the OIG said, and at times some services are not even performed. In addition, there are delays in performing tests to diagnose HIV infection in infants.

Data quality

The OIG observed that the unavailability of data and inconsistencies in the data impede effective monitoring and decision-making.

Community data for malaria are not being reported at the national level, the audit found. The data reported from each community is not aggregated at the health facility level due to a lack of adequate tools. Instead, the OIG said, the health facility sends a detailed report for each community to the health district. Limited human resources and a heavy workload at the district level –– a single statistician at the district covers on average 270 CHWs –– makes it challenging to capture the disaggregated data. Since health facilities do not send community reports to upper levels directly, the OIG noted, the reporting of community data in the national information health system is lacking.

The audit showed that the community data being reported to the Global Fund comes from a parallel system maintained by a sub-recipient, Catholic Relief Services (CRS). This is not sustainable, the OIG said, because it is not based on the national system; it only covers regions supported by the Global Fund and does not guarantee strong assurance over the information reported.

The uneven quality of data and services has led to gaps in the services provided to pregnant women, the OIG stated. In 2017, 28% of HIV-positive pregnant women did not give birth in PMTCT sites, meaning that deliveries were not necessarily performed using the recommended protocol. Although 83% of children born to HIV-positive mothers are tested for HIV using polymerase chain reaction (PCR), one-third of these tests were not performed within the required timeline of 6-8 weeks post-partum (to detect very early infections before antibodies have developed), and there is no system in place to monitor whether HIV-positive babies are put on treatment.

Ratings

The OIG concluded that the programs are currently only partially effective in providing adequate quality of services to patients and reliable data for decision-making. The OIG said that activities related to procurement and supply management have also been only partially effective.

Agreed management actions (AMAs)

There were two AMAs (see table). Both are owned by the Head, Grant Management Division.

Agreed management action Target date
1. The Secretariat will support the Ministry of Public Health (MPH), the (U.S.) President’s Malaria Initiative and Chemonics to develop a logistic surveillance report on monthly basis ­to monitor the stock status and the buffer stock available of malaria health commodities at central, district, zonal warehouse and health facility level. 28 Feb 2020
2.

The Secretariat will support the MPH and partners to:

·   Update the health facilities reporting template to integrate the malaria community data;

·   Conduct a reconciliation exercise for the malaria community data reported through the national system with data reported by CRS; and

·   Update the therapeutic education standards and tools and re-train the midwives and the health staff working in PMTCT sites on the updated therapeutic education standards and tools.

30 Jun 2020
Note: Chemonics is a private sector technical partner involved in procuring health products and managing stockouts for USAID.

 


Share |

Leave a comment

Leave a comment