While eight of the Global Fund’s 13 key performance indicators for which results are available are performing well, the organization is at risk of not meeting the indicators for the other five. This information was contained in a report on KPIs provided to the Board at its meeting on 16-17 November.
There are 16 indicators in all, but the Board paper provided results for only 13 of them.
The current set of KPIs were put in place to monitor and assess how well the goals, targets and objectives of the 2012-2016 Global Fund Strategy were being met.
The Board paper recommended 2016 targets for 10 of the KPIs. (Some KPIs already had 2016 targets.)
The Fund has not managed to meet its access to funding goal (KPI 7), which called for disbursements to be made within 10 months of receipt of the concept note for 75% of approved grants. In addition, it looks unlikely that the Fund will meet its targets for service delivery (KPI 3), health systems strengthening (KPI 5), value for money (KPI 10), and protecting human rights (KP1 12).
“We must acknowledge shortfalls, and point out that we are behind on some KPIs,” said Global Fund Executive Director Mark Dybul in a separate report to the Board.
Indicators that are on track
Of the eight KPIs that are on track, three exceeded their goals. KPI 6 measures the percentage of investments in countries where Global Fund support is reported on national budgets and tracks whether the Fund’s support is being integrated in national processes. The Fund’s target was 92%, but the result has come in at 94%. The main driver is improved reporting. The proposed target for 2016 is 94%.
KPI 14 tracks domestic financing for the three diseases by measuring the percentage of funded programs that receive the minimum counterpart financing from their government. The target for 2015 was 90% of programs, but the result came in at 93% (77 of 83 programs). The proposed target for 2016 is 90%.
The efficiency of grant management operations (KPI 15) is also looking strong. This KPI benchmarks the operating expenditure of the Fund against similar organizations and tracks those expenses as a percentage of grants under management. The target for 2015 was to be below 2.75%, and the result is forecasted to come in at 2.4%. The proposed target for 2016 is unchanged from 2015.
Also worth noting is that the goal of saving 10 million lives and averting 140 million infections for the period 2012-2016 (KPI 1: performance against strategic goals) is on target. Results for the period 2012-2014 are 6.2 million lives saved and 97 million infections averted. The Fund believes that the 2016 target will be exceeded.
The Board paper also reviewed the performance of four other indicators (KPIs 4, 9, 11, and 13) that are on track. KPI 4 (efficiency of Global Fund investment decisions) is particularly opaque. It tracks how well investment decisions are aligned with county needs. It is said to be on track, but the Fund acknowledged in the past that this indicator does not effectively reflect the efficiency of investment decisions.
KPI 9 tracks the ability of Fund recipients to mitigate potential risks. It works off a risk assessment rating, which has remained unchanged since 2013, and is within the target range.
Also coming in within its target range was KPI 11 (grant expenses forecast), which monitors whether grants are being effectively planned, managed and implemented. Finally, KPI 13 (resource mobilization), which monitors pledges as a percentage of the replenishment target and the percentage of pledges that become actual contributions, is reported to be on target to hit 100% in 2016.
Indicators that have not met or are unlikely to meet their targets
In many ways, the KPIs are still a work in progress. Dr Dybul and others at the Fund have noted that the reasons the Fund has not met some of its targets are in some cases quite complex. Here is a what the Board paper said about the KPIs that are unlikely to be met.
KPI 7 – Access to Funding
Like most of the KPIs, the results of this one will only be clear in 2016, but the Fund is already acknowledging that it is unlikely it can be met. This indicator measures how long it takes from submission of the concept note to receipt of the first disbursement. The goal was that 75% of grants emanating from concept notes submitted in 2014 would take 10 months or less, and that 75% of those submitted in 2015-2106 would take eight months or less.
At the end of the second quarter of 2015, only 54% had met the 10 months or less target. The Board paper said that “achieving the 10-month target for 2014 submissions looks increasingly challenging, and the 8-month target for 2015 submissions unlikely.” The explanation? “The grant making phase of the new funding model is taking longer than anticipated.” The Fund is reviewing its grant-making processes to identify what is holding them up and how the system could be improved.
KPI 3 - Performance against strategic service delivery targets
The Fund believes that the 2016 targets for three of seven services that make up this indicator may not be met: (1) number of TB cases treated; (2) number of long-lasting insecticide nets distributed; and (3) number of HIV positive pregnant women who receive antiretroviral therapy to reduce mother to child transmission.
Between 2012 and 2016, the Fund was hoping to treat 21 million cases of TB according to the DOTS approach. The data indicate that only 8.5 million cases have so far been treated, but the Fund believes this may be a case of under-reporting – i.e. that some treated cases are not being accurately attributed to support from the Global Fund, and so the Fund may have come closer to hitting this target than the data seem to indicate.
The Fund had targeted distributing 390 million LLINs by 2016, but only 219 million have so far been handed out. The Fund says that 86% of the shortfall comes from 10 countries. To address this issue, the Fund intends to improve performance monitoring and use its Implementation Through Partnerships project to accelerate service delivery where implementation is behind schedule. (The project covers 20 countries. GFO has learned that eight of the 10 countries that account for the shortfall in LLIN distribution are among the 20 countries in the ITP project.)
The Fund targeted providing ARVs to 2.7 million HIV-positive pregnant women, but the data gathered suggests it has only managed to reach 1.6 million to date, with almost the entire shortfall attributable to 10 countries. Again, the Fund thinks this may be a question of under-reporting, not under-achievement. One way in which the data can misrepresent the uptake of ARVs is if HIV-positive pregnant women are prescribed the drugs for life. The Fund intends to address this by improving its performance monitoring, and the quality of data provided by partners.
KPI 5 – Health System Strengthening
This KPI is intended to measure the proportion of countries in which the HIV, TB and malaria service availability and readiness score improve by five percentage points by 2016. These scores rate how well countries are able to deliver prevention, treatment, and care to those affected by the three diseases. The target is 60% of countries showing this improvement, but data is currently only available for one country. That country showed a four percentage point improvement in service availability and readiness. Measuring this KPI requires two data points. Survey results for nine more countries are due at the end of 2015, but only one of these will provide the second data point required to calculate the results.
KPI 10 – Value for Money
A significant portion of the support the Global Fund provides is invested in commodities. The Fund believes it can achieve savings by leveraging its purchasing power. It set a target to reduce the amount it spends on those commodities by 8% in 2015. The second quarter result for 2015 shows a 4% savings. The Secretariat believes that because many agreements are in place for two years, most of the savings for 2015 and 2016 have already been realized. Therefore, it is unlikely that the Fund will reach its 8% target for the full 2015 year. The proposed target for 2016 is 4%.
KPI 12 - Human Rights Protection
The Global Fund seeks to have human rights complaints against its supported programs identified through risk assessment tools, and resolved under the Fund’s human rights complaints procedure. For 2013-2104, 30% of complaints were resolved in this fashion. This KPI targeted a year-on-year improvement, with the goal to eventually reach 100%. However, the second quarter result for 2015 was still only 30%: Nine complaints have been received so far this year, but only three are under investigation under the human rights complaints procedure. The remaining six did not meet the criteria and so are being dealt with under the standard review processes of the Office of the Inspector General. The Fund is currently looking at developing a new human rights KPI for the next strategy.
Other comments on KPIs
Opinions differ regarding the usefulness of the KPIs, as is evidenced by the fact that several other Board papers mentioned KPIs. The papers expressed concern about the value of KPIs generally, as well as reservations about some KPIs in particular.
The 2015 Strategic Review (see article in this issue) commended the Fund as an organization with a level of scrutiny that matches or exceeds most other international organisations, but questioned the value of the KPIs for monitoring Secretariat performance. The review, commissioned by the Technical Evaluation and Reference Group, pointed out that the value of KPIs lies in understanding changes in an indicator, which requires considerable understanding of the context. According to the review, “when corporate KPIs are intended for use by the Board as much as by management, as in the Global Fund, problems arise when Board members lack the same level of contextual understanding.”
In its report, the Board’s Coordinating Group raised concerns about how data is collected for several of the KPIs, particularly when it comes to in-country systems that are not up to standard. One indicator they took particular note of is KPI 5, which tracks health system strengthening. According to the report, this measure “is not representative and should be entirely revisited.”
In the Report of the Executive Director (see article in this issue), Dr Dybul said that KPI 7, which measures access to funding and will not be met this year, has exposed that there are inadequacies in the Fund’s data systems. “We need a fresh and innovative approach to framing and organizing our basic grant data, so that our managers can accurately identify themes and trends and take proactive moves to come up with solutions,” he said.
With respect to KPI 10 (Value for money), Dr Dybul said that the results of this KPI show that “the impressive gains seen in commodity cost savings over the past three years are beginning to be constrained by the limited volume of procurement spending that can access the lower prices available through the Global Fund’s strategic tendering process.” The Coordinating Group expressed concerns about KPI 10, however, saying that it “relies more on cost-savings than added value.”
The Board had been scheduled to vote on the proposed 2016 targets at its meeting on 16-17 November. However, several Board members asked for more information on the performance of certain KPIs. The Board may vote electronically on the 2016 targets in the coming weeks.
The Mid-Year 2015 Corporate KPI Results & 2016 Targets, Board Document GF-B34-08, should be available shortly at www.theglobalfund.org/en/board/meetings/34. The Report of the Executive Director, Board Document GF-B34-02; the Strategic Review 2015, Board Document GF-B34-10; and the Report of the Coordinating Group, Board Document GF-B34-03 – should also be available shortly at the same site.