Implementing the Strategic Action Programme for the Yellow Sea Large Marine Ecosystem: Restoring Ecosystem Goods and Services and Consolidation of a Long-term Regional Environmental Governance Framework

TERMS OF REFERENCE

 

UNDP-GEF Mid-Term Review Consultant (International)

 

Implementing the Strategic Action Programme for the Yellow Sea Large Marine Ecosystem: Restoring Ecosystem Goods and Services and Consolidation of a Long-term Regional Environmental Governance Framework

1. INTRODUCTION

This is the Terms of Reference (ToR) for the UNDP-GEF Midterm Review (MTR) of the full-sized project titled Implementing the Strategic Action Programme for the Yellow Sea Large Marine Ecosystem: Restoring Ecosystem Goods and Services and Consolidation of a Long-term Regional Environmental Governance Framework, which is to be undertaken in March 2018. This four-year project was signed in July 2014, launched in July 2017, and is expected to terminate in July 2018. An project extension is anticipated by the participating countries based on the findings and recommenddations of the mid-term evaluator. In line with the UNDP-GEF Guidance on MTRs, this MTR process will be initiated before the submission of the second Project Implementation Report (PIR). This ToR sets out the expectations for this MTR.  The MTR process must follow the guidance outlined in the document Guidance For Conducting Midterm Reviews of UNDP-Supported, GEF-Financed Projects.

2. PROJECT BACKGROUND INFORMATION

Yellow Sea Large Marine Ecosystem is a water body bordered by China, RO Korea and DPR Korea, covering an area of 400,000 km2. Rivers discharge about 1.6 billion tons of sediment and 1,500 billion tones of freshwater into the Yellow Sea. The low flushing rate between Yellow Sea and East China Sea of one every seven years, combined with weak water circulation, makes this sea vulnerable to pollution and its coastal areas highly susceptible to localized pollution discharges. Qingdao, Dalian, Shanghai, Seoul/Incheon (RO Korea) and Pyongyang/Nampo (DRP Korea) are the five cities with over tens of millions of inhabitants bordering the sea. This population replies on the Yellow Sea LME’s ecosystem carrying capacity to provide capture fisheries resources in excess of two million tonnes per year, mariculture over 14 million tonnes per year, support for wildlife, provision of bathing beaches and tourism, and its capacity to absorb nutrients and other pollutants. Yet fishing efforts increased threefold between the 1960s and early 1980s, during which time the proportion of demersal species, such as small and large yellow croakers, hairtail, flatfish and cod, declined by more than 40 percent in terms of biomass. Other major transboundary problems include increasing discharge of pollutants; changes to ecosystem structure leading to an increase in jellyfish and harmful algal blooms; 40 percent loss of coastal wetlands from reclamation and conversions projects. Severe environmental degradation has cost the country approximately nine percent of its gross national income in 2009. This situation has been further exacerbated by incomplete legislation and insufficient enforcement. The environmental foundation needed to sustain economic growth may be irreversibly altered, and the important human health implications of a deteriorating environment such as increased agriculture and food contamination and air and water pollution, have resulted in a series of efforts to improve the environment.

The objective of the regional project is to achieve adaptive ecosystem-based management of the Yellow Sea Large Marine Ecosystem bordered by China, RO Korea and DPR Korea by fostering long-term sustainable institutional, policy and financial arrangements for effective ecosystem-based management of the Yellow Sea in accordance with the YSLME Strategic Action Programme (YSLME SAP) adopted by China and RO Korea in 2009.

To achieve this objective, the project will support the formation of the YSLME Commission oversee the implementation of the YSLME SAP, innovate institutional arrangements, improve management capacity and quality of function. This includes, developing robust governmental coordination mechanisms, strengthening regulatory mechanisms while strengthening the incentive structure to promote environmental protection, developing mechanisms to link land and sea and resource use to carrying capacity, and systems for the participation of a range of stakeholders.

There are four components in the project:

  1. Sustainable national and regional cooperation for ecosystem based management.
  2. Improved Ecosystem Carrying Capacity with respect to provisioning services.
  3. Improved Ecosystem Carrying Capacity with respect to regulating and cultural services.
  4. Improved Ecosystem Carrying Capacity with respect to supporting services.

The key outcomes sought are:

  1. Establishment of a self-sustaining cooperative mechanism for ecosystem-based management.
  2. Recovery of depleted fish stocks and improved mariculture production and quality.
  3. Improved ecosystem health;
  4. improved inter-sectoral coordination and mainstreaming of ecosystem based management principles at the national level, maintenance of habitat areas, strengthened stakeholder participation in management and improved policy making.
  5. Skills and capacity significantly developed for region-wide ecosystem-based management.

3. OBJECTIVES OF THE MTR

The MTR will assess progress towards the achievement of the project objectives and outcomes as specified in the Project Document, and assess early signs of project success or failure with the goal of identifying the necessary changes to be made in order to set the project on-track to achieve its intended results. The MTR will also review the project’s strategy, its risks to sustainability.

4. MTR APPROACH & METHODOLOGY

The MTR must provide evidence based information that is credible, reliable and useful. The MTR Consultant will review all relevant sources of information including documents prepared during the preparation phase (i.e. PIF, UNDP Initiation Plan, UNDP Environmental & Social Safeguard Policy, the Project Document, project reports including Annual Project Review/PIRs, project budget revisions, lesson learned reports, national strategic and legal documents, and any other materials that the consultant considers useful for this evidence-based review).

The MTR Consultant is expected to follow a collaborative and participatory approach[1] ensuring close engagement with the focal agencies of the two participating countries, UNDP Country Office, the UNDP Regional Techincal Advisor for International Waters and UNOPS.

Engagement of stakeholders is vital to a successful MTR.[2] Stakeholder involvement should include interviews with stakeholders who have project responsibilities, including but not limited to; executing agencies, senior officials’ component leaders, key experts and consultants in the subject area, Project Board, project stakeholders, academia, local government and CSOs, etc. Additionally, the MTR Consultant is expected to conduct a field mission to all three countries and selected project sites. Interviews will be held with the government focal agencies per country and as well as other stakeholders.

The final MTR report should describe the full MTR approach taken and the rationale for the approach making explicit the underlying assumptions, challenges, strengths and weaknesses about the methods and approach of the review.

5. DETAILED SCOPE OF THE MTR

The MTR Consultant will assess the following four categories of project progress. See the Guidance For Conducting Midterm Reviews of UNDP-Supported, GEF-Financed Projects for extended descriptions.

i.    Project Strategy

Project design:

  • Review the problem addressed by the project and the underlying assumptions.  Review the effect of any incorrect assumptions or changes to the context to achieving the project results as outlined in the Project Document.
  • Review the relevance of the project strategy and assess whether it provides the most effective route towards expected/intended results.  Were lessons from other relevant projects properly incorporated into the project design?
  • Review how the project addresses country priorities. Review country ownership. Was the project concept in line with the national sector development priorities and plans of the country (or of participating countries in the case of multi-country projects)?
  • Review decision-making processes: were perspectives of those who would be affected by project decisions, those who could affect the outcomes, and those who could contribute information or other resources to the process, taken into account during project design processes?
  • Review the extent to which relevant gender issues were raised in the project design. See Annex 9 of Guidance For Conducting Midterm Reviews of UNDP-Supported, GEF-Financed Projects for further guidelines.
  • If there are major areas of concern, recommend areas for improvement.

Results Framework/Logframe:

  • Undertake a critical analysis of the project’s logframe indicators and targets, assess how “SMART” the midterm and end-of-project targets are (Specific, Measurable, Attainable, Relevant, Time-bound), and suggest specific amendments/revisions to the targets and indicators as necessary.
  • Are the project’s objectives and outcomes or components clear, practical, and feasible within its time frame?
  • Examine if progress so far has led to, or could in the future catalyse beneficial development effects (i.e. income generation, gender equality and women’s empowerment, improved governance etc...) that should be included in the project results framework and monitored on an annual basis.
  • Ensure broader development and gender aspects of the project are being monitored effectively.  Develop and recommend SMART ‘development’ indicators, including sex-disaggregated indicators and indicators that capture development benefits.

 

ii.    Progress Towards Results

 

Progress Towards Outcomes Analysis:

  • Review the logframe indicators against progress made towards the end-of-project targets using the Progress Towards Results Matrix and following the Guidance For Conducting Midterm Reviews of UNDP-Supported, GEF-Financed Projects; colour code progress in a “traffic light system” based on the level of progress achieved; assign a rating on progress for each outcome; make recommendations from the areas marked as “Not on target to be achieved” (red).

 

Table. Progress Towards Results Matrix (Achievement of outcomes against End-of-project Targets)

Project Strategy

Indicator[3]

Baseline Level[4]

Level in 1st  PIR (self- reported)

Midterm Target[5]

End-of-project Target

Midterm Level & Assessment[6]

Achievement Rating[7]

Justification for Rating

Objective:

 

Indicator (if applicable):

 

 

 

 

 

 

 

Outcome 1:

Indicator 1:

 

 

 

 

 

 

 

Indicator 2:

 

 

 

 

 

Outcome 2:

Indicator 3:

 

 

 

 

 

 

 

Indicator 4:

 

 

 

 

 

Etc.

 

 

 

 

 

Etc.

 

 

 

 

 

 

 

 

 

Indicator Assessment Key

Green= Achieved

Yellow= On target to be achieved

Red= Not on target to be achieved

 

In addition to the progress towards outcomes analysis:

  • Compare and analyse the GEF Tracking Tool at the Baseline with the one completed right before the Midterm Review.
  • Identify remaining barriers to achieving the project objective in the remainder of the project.
  • By reviewing the aspects of the project that have already been successful, identify ways in which the project can further expand these benefits.

 

iii.   Project Implementation and Adaptive Management

 

Management Arrangements:

  • Review overall effectiveness of project management as outlined in the Project Document.  Have changes been made and are they effective?  Are responsibilities and reporting lines clear?  Is decision-making transparent and undertaken in a timely manner?  Recommend areas for improvement.
  • Review the quality of execution of the Executing Agency/Implementing Partner(s) and recommend areas for improvement.
  • Review the quality of support provided by the GEF Partner Agency (UNDP) and recommend areas for improvement.

Work Planning:

  • Review any delays in project start-up and implementation, identify the causes and examine if they have been resolved.
  • Are work-planning processes results-based?  If not, suggest ways to re-orientate work planning to focus on results?
  • Examine the use of the project’s results framework/ logframe as a management tool and review any changes made to it since project start. 

Finance and co-finance:

  • Consider the financial management of the project, with specific reference to the cost-effectiveness of interventions. 
  • Review the changes to fund allocations as a result of budget revisions and assess the appropriateness and relevance of such revisions.
  • Does the project have the appropriate financial controls, including reporting and planning, that allow management to make informed decisions regarding the budget and allow for timely flow of funds?
  • Informed by the co-financing monitoring table to be filled out, provide commentary on co-financing: is co-financing being used strategically to help the objectives of the project? Is the Project Implementing Partner meeting with all co-financing partners regularly in order to align financing priorities and annual work plans?

Project-level Monitoring and Evaluation Systems:

  • Review the monitoring tools currently being used:  Do they provide the necessary information? Do they involve key partners? Are they aligned or mainstreamed with national systems?  Do they use existing information? Are they efficient? Are they cost-effective? Are additional tools required? How could they be made more participatory and inclusive?
  • Examine the financial management of the project monitoring and evaluation budget.  Are sufficient resources being allocated to monitoring and evaluation? Are these resources being allocated effectively?

 

Stakeholder Engagement:

  • Project management: Has the project developed and leveraged the necessary and appropriate partnerships with direct and tangential stakeholders?
  • Participation and country-driven processes: Do local and national government stakeholders support the objectives of the project?  Do they continue to have an active role in project decision-making that supports efficient and effective project implementation?
  • Participation and public awareness: To what extent has stakeholder involvement and public awareness contributed to the progress towards achievement of project objectives?

 

Reporting:

  • Assess how adaptive management changes have been reported by the project management and shared with the Project Board.
  • Assess how well the Project Implementing Partner and country-partners undertake and fulfil GEF reporting requirements (i.e. how have they addressed poorly-rated PIRs, if applicable?)
  • Assess how lessons derived from the adaptive management process have been documented, shared with key partners and internalized by partners.

Communications:

  • Review internal project communication with stakeholders: Is communication regular and effective? Are there key stakeholders left out of communication? Are there feedback mechanisms when communication is received? Does this communication with stakeholders contribute to their awareness of project outcomes and activities and investment in the sustainability of project results?
  • Review external project communication: Are proper means of communication established or being established to express the project progress and intended impact to the public (is there a web presence, for example? Or did the project implement appropriate outreach and public awareness campaigns?)
  • For reporting purposes, write one half-page paragraph that summarizes the project’s progress towards results in terms of contribution to sustainable development benefits, as well as global environmental benefits.

iv.   Sustainability

  • Validate whether the risks identified in the Project Document, Annual Project Review/PIRs and the ATLAS Risk Management Module are the most important and whether the risk ratings applied are appropriate and up to date. If not, explain why.
  • In addition, assess the following risks to sustainability:

Financial risks to sustainability:

  • What is the likelihood of financial and economic resources not being available once the GEF assistance ends (consider potential resources can be from multiple sources, such as the public and private sectors, income generating activities, and other funding that will be adequate financial resources for sustaining project’s outcomes)?

Socio-economic risks to sustainability:

  • Are there any social or political risks that may jeopardize sustainability of project outcomes? What is the risk that the level of stakeholder ownership (including ownership by governments and other key stakeholders) will be insufficient to allow for the project outcomes/benefits to be sustained? Do the various key stakeholders see that it is in their interest that the project benefits continue to flow? Is there sufficient public / stakeholder awareness in support of the long-term objectives of the project? Are lessons learned being documented by the Project Implementing Partner on a continual basis and shared/ transferred to appropriate parties who could learn from the project and potentially replicate and/or scale it in the future?

Institutional Framework and Governance risks to sustainability:

  • Do the legal frameworks, policies, governance structures and processes pose risks that may jeopardize sustenance of project benefits? While assessing this parameter, also consider if the required systems/ mechanisms for accountability, transparency, and technical knowledge transfer are in place.

Environmental risks to sustainability:

  • Are there any environmental risks that may jeopardize sustenance of project outcomes?

Conclusions & Recommendations

The MTR Consultant will include a section of the report setting out the MTR’s evidence-based conclusions, in light of the findings.[8]

Recommendations should be succinct suggestions for critical intervention that are specific, measurable, achievable, and relevant. A recommendation table should be put in the report’s executive summary. See the Guidance For Conducting Midterm Reviews of UNDP-Supported, GEF-Financed Projects for guidance on a recommendation table.

The MTR Consultant should make no more than 10 recommendations total.

Ratings

The MTR Consultant will include its ratings of the project’s results and brief descriptions of the associated achievements in a MTR Ratings & Achievement Summary Table in the Executive Summary of the MTR report. See Annex E for ratings scales. No rating on Project Strategy and no overall project rating is required.

Table. MTR Ratings & Achievement Summary Table

Measure

MTR Rating

Achievement Description

Project Strategy

N/A

 

Progress Towards Results

Objective Achievement Rating: (rate 6 pt. scale)

 

Outcome 1 Achievement Rating: (rate 6 pt. scale)

 

Outcome 2 Achievement Rating: (rate 6 pt. scale)

 

Outcome 3 Achievement Rating: (rate 6 pt. scale)

 

Etc.

 

Project Implementation & Adaptive Management

(rate 6 pt. scale)

 

Sustainability

(rate 4 pt. scale)

 

 

6. TIMEFRAME

The total duration of the MTR will be approximately 6 weeks starting March 2017, and shall not exceed four (2) months from when the consultant is hired. The tentative MTR timeframe is as follows:

TIMEFRAME

ACTIVITY

December 20, 2017

Application closes

5 January 2018

Select MTR Consultant

Within 1 week after contract signing

Prep the MTR Consultant (handover of Project Documents)

2 weeks after contract signing (March 1)

Document review and preparing MTR Inception Report

 

Finalization and Validation of MTR Inception Report- latest start of MTR mission

14 days (2 weeks)

MTR mission: stakeholder meetings, interviews, field visits (Incheon – Seoul – Rudong – Weihai – Qingdao – Dalian)

1 day

Mission wrap-up & presentation of initial findings to the Project Steering Committee

10 days

Preparing draft report

2 days

Incorporating audit trail from feedback on draft report/Finalization of MTR report

 

Preparation & Issue of Management Response

23 April 2018

Expected date of full MTR completion

Options for site visits should be provided in the Inception Report.

7. MIDTERM REVIEW DELIVERABLES

#

Deliverable

Description

Timing

Responsibilities

1

MTR Inception Report

MTR Consultant clarifies objectives and methods of Midterm Review

No later than 2 weeks before the MTR mission

MTR Consultant submits to the Commissioning Unit and project management

2

Presentation

Initial Findings

End of MTR mission

MTR Consultant presents to project management and the Commissioning Unit

3

Draft Final Report

Full report (using guidelines on content outlined in Annex B) with annexes

Within 3 weeks of the MTR mission

Sent to the Commissioning Unit, reviewed by RTA, Project Coordinating Unit, GEF OFP

4

Final Report*

Revised report with audit trail detailing how all received comments have (and have not) been addressed in the final MTR report

Within 1 week of receiving UNDP comments on draft

Sent to the Commissioning Unit

*The final MTR report must be in English. If applicable, the Commissioning Unit may choose to arrange for a translation of the report into a language more widely shared by national stakeholders.

8. MTR ARRANGEMENTS

The principal responsibility for managing this MTR resides with the Commissioning Unit. The Commissioning Unit for this project’s MTR is UNDP China. The commissioning unit will contract the consultant – after review of the selected candidate by UNDP CO together with the Project Management Office - and ensure the timely provision of per diems and travel arrangements to all countries to be visited for the MTR Consultant.  UNDP CO will be responsible for liaising with the MTR Consultant to provide all relevant documents, set up stakeholder interviews, and arrange field visits. The MTR Consultant will meet virtually with the UNDP CO and UNDP RTA to discuss the evaluation’s scope and objectives, as well as to debrief the UNDP on the evaluation’s findings.

9. QUALIFICATIONS

The consultant cannot have participated in the project preparation, formulation, and/or implementation (including the writing of the Project Document) and should not have a conflict of interest with project’s related activities. 

The selection of consultant will be aimed at maximizing the overall qualities in the following areas:

  •  Experience applying SMART indicators and reconstructing or validating baseline scenarios (5%);
  •  Competence in adaptive management, as applied to sustainable fisheries (5%)
  •  Previous Experience working with the GEF or GEF-evaluations (20%);
  •  Experience working in the East Asian Region, particularly China and RO Korea (15%)
  •  Work experience in the field of ocean governance, or fihseries management, or ecosystem-based management, preferably at the LME level for at least 10 years (15%);
  •  Demonstrated understanding of issues related to gender; experience in gender sensitive evaluation and analysis (10%).
  •  Excellent communication analytical skills (10%);
  •  Project evaluation/review experiences within United Nations system will be considered an asset (10%);
  •  A Master’s degree in environmental management, fisheries management, community development, or other closely related field (10%).

The International Consultant, will primarily cover the tasks, but not limited to the following:

  1.  Prepare the MTR Inception Report including a detailed plan of the mission with an interview schedule, evaluation questions and provide it to the UNDP and CPMU no later than 2 weeks before the MTR mission
  2.  Ensure the conduct of evaluation activities as agreed on with PMO and UNDP; (including visits to China and RO Korea)
  3.  Consolidate and analyze data and information gathered during the evaluation;
  4.  Finalize the MTE Report;

In consultation with the Consultant and as requested, the UNDP CO will make available all relevant documentation and provide contact information to key project partners and stakeholders, and facilitate contact where needed. The Consultant will request UNDP CO to assist in organizing any briefing de-briefing meetings including coordination of stakeholders’ input in the evaluation draft report.

10. PAYMENT MODALITIES AND SPECIFICATIONS

Consultants will be contracted by UNDP and remunerated according to the reviewed and accepted financial proposal. The contract will be output-based and payment issued only upon delivery of satisfactory outputs/milestones.

Table 6. Payment Schedule

%

Milestone

20%

Following submission and acceptance of the MTR mission Inception Report

40%

Following submission and approval of the 1ST draft MTR report

40%

Following submission and approval (UNDP CO and IW RTA) of the final MTR report

 

11. APPLICATION PROCESS

Applicants are requested to apply online (https://jobs.undp.org/cj_view_job.cfm?cur_job_id=76306). Individual consultants are invited to submit applications together with their CV for these positions.

The application should contain a current and complete C.V. in English with indication of the e‐mail and phone contact. Shortlisted candidates will be requested to submit a price offer indicating the total cost of the assignment (lumpsum / daily fees only).

UNDP applies a fair and transparent selection process that will take into account the competencies/skills of the applicants as well as their financial proposals. Qualified women and members of social minorities are encouraged to apply.

TOR ANNEX A

LIST OF DOCUMENTS TO BE REVIEWED BY THE MTR Consultant[9]

1.       UNDP Project Document

2.       Project Inception Report

3.       All Project Implementation Reports (PIR’s)

4.       Terms of reference of Interim Commission Council and subsidiary bodies

5.       Quarterly progress reports and work plans of the various implementation task teams

6.       Finalized GEF focal area Tracking Tools at CEO endorsement and midterm (fill in specific TTs for this project’s focal area)

7.       Annual Reports (2016 and 2017)

8.       Minutes of Mangement, Science and Technical Panel (MSTP) and Interim Commission Council meetings

9.       Minitues of six Regional Working Group Meetings

10.    Workplan (2017-2019)

11.    Draft review report of National SAP

12.    Technical reports

 

ToR ANNEX B: Guidelines on Contents for the Midterm Review Report[10]

i.

Basic Report Information (for opening page or title page)

·         Title of UNDP supported GEF financed project

·         UNDP PIMS# and GEF project ID# 

·         MTR time frame and date of MTR report

·         Region and countries included in the project

·         GEF Operational Focal Area/Strategic Program

·         Executing Agency/Implementing Partner and other project partners

·         MTR CO members

·         Acknowledgements

 

ii.

Table of Contents

 

iii.

Acronyms and Abbreviations

 

1.

Executive Summary (3-5 pages)

·         Project Information Table

·         Project Description (brief)

·         Project Progress Summary (between 200-500 words)

·         MTR Ratings & Achievement Summary Table

·         Concise summary of conclusions

·         Recommendation Summary Table

 

2.

Introduction (2-3 pages)

·         Purpose of the MTR and objectives

·         Scope & Methodology: principles of design and execution of the MTR, MTR approach and data collection methods, limitations to the MTR

·         Structure of the MTR report

 

3.

Project Description and Background Context (3-5 pages)

  • Development context: environmental, socio-economic, institutional, and policy factors relevant to the project objective and scope
  • Problems that the project sought to address: threats and barriers targeted
  • Project Description and Strategy: objective, outcomes and expected results, description of field sites (if any)
  • Project Implementation Arrangements: short description of the Project Board, key implementing partner arrangements, etc.
  • Project timing and milestones
  • Main stakeholders: summary list

 

4.

Findings (12-14 pages)

 

 

4.1

 

 

Project Strategy

·         Project Design

·         Results Framework/Logframe

 

4.2

Progress Towards Results

·         Progress towards outcomes analysis

·         Remaining barriers to achieving the project objective

 

4.3

Project Implementation and Adaptive Management

·         Management Arrangements

·         Work planning

·         Finance and co-finance

·         Project-level monitoring and evaluation systems

·         Stakeholder engagement

·         Reporting

·         Communications

 

4.4

Sustainability

·         Financial risks to sustainability

·         Socio-economic to sustainability

·         Institutional framework and governance risks to sustainability

·         Environmental risks to sustainability

5.

Conclusions and Recommendations (4-6 pages)

 

 

  5.1 

 

 

Conclusions

·         Comprehensive and balanced statements (that are evidence-based and connected to the MTR’s findings) which highlight the strengths, weaknesses and results of the project

 

  5.2

Recommendations

  • Corrective actions for the design, implementation, monitoring and evaluation of the project
  • Actions to follow up or reinforce initial benefits from the project
  • Proposals for future directions underlining main objectives

 

6.

Annexes

·         MTR ToR (excluding ToR annexes)

·         MTR evaluative matrix (evaluation criteria with key questions, indicators, sources of data, and methodology)

·         Example Questionnaire or Interview Guide used for data collection

·         Ratings Scales

·         MTR mission itinerary

·         List of persons interviewed

·         List of documents reviewed

·         Co-financing table (if not previously included in the body of the report)

·         Signed UNEG Code of Conduct form

·         Signed MTR final report clearance form

·         Annexed in a separate file: Audit trail from received comments on draft MTR report

·         Annexed in a separate file: Relevant midterm tracking tools (METT, FSC, Capacity scorecard, etc.)

 

 

 

 

         

ToR ANNEX B: Midterm Review Evaluative Matrix Template

Evaluative Questions

Indicators

Sources

Methodology

Project Strategy: To what extent is the project strategy relevant to country priorities, country ownership, and the best route towards expected results?

(include evaluative question(s))

(i.e. relationships established, level of coherence between project design and implementation approach, specific activities conducted, quality of risk mitigation strategies, etc.)

(i.e. project documents, national policies or strategies, websites, project staff, project partners, data collected throughout the MTR mission, etc.)

(i.e. document analysis, data analysis, interviews with project staff, interviews with stakeholders, etc.)

 

 

 

 

 

 

 

 

Progress Towards Results: To what extent have the expected outcomes and objectives of the project been achieved thus far?

 

 

 

 

 

 

 

 

Project Implementation and Adaptive Management: Has the project been implemented efficiently, cost-effectively, and been able to adapt to any changing conditions thus far? To what extent are project-level monitoring and evaluation systems, reporting, and project communications supporting the project’s implementation?

 

 

 

 

 

 

 

 

Sustainability: To what extent are there financial, institutional, socio-economic, and/or environmental risks to sustaining long-term project results?

 

 

 

 

 

 

 

 

 

TOR ANNEX C: MTR RATINGS

Ratings for Progress Towards Results: (one rating for each outcome and for the objective)

6

Highly Satisfactory (HS)

The objective/outcome is expected to achieve or exceed all its end-of-project targets, without major shortcomings. The progress towards the objective/outcome can be presented as “good practice”.

5

Satisfactory (S)

The objective/outcome is expected to achieve most of its end-of-project targets, with only minor shortcomings.

4

Moderately Satisfactory (MS)

The objective/outcome is expected to achieve most of its end-of-project targets but with significant shortcomings.

3

Moderately Unsatisfactory (HU)

The objective/outcome is expected to achieve its end-of-project targets with major shortcomings.

2

Unsatisfactory (U)

The objective/outcome is expected not to achieve most of its end-of-project targets.

1

Highly Unsatisfactory (HU)

The objective/outcome has failed to achieve its midterm targets, and is not expected to achieve any of its end-of-project targets.

 

Ratings for Project Implementation & Adaptive Management: (one overall rating)

6

Highly Satisfactory (HS)

Implementation of all seven components – management arrangements, work planning, finance and co-finance, project-level monitoring and evaluation systems, stakeholder engagement, reporting, and communications – is leading to efficient and effective project implementation and adaptive management. The project can be presented as “good practice”.

5

Satisfactory (S)

Implementation of most of the seven components is leading to efficient and effective project implementation and adaptive management except for only few that are subject to remedial action.

4

Moderately Satisfactory (MS)

Implementation of some of the seven components is leading to efficient and effective project implementation and adaptive management, with some components requiring remedial action.

 

Moderately Unsatisfactory (MU)

Implementation of some of the seven components is not leading to efficient and effective project implementation and adaptive, with most components requiring remedial action.

2

Unsatisfactory (U)

Implementation of most of the seven components is not leading to efficient and effective project implementation and adaptive management.

1

Highly Unsatisfactory (HU)

Implementation of none of the seven components is leading to efficient and effective project implementation and adaptive management.

 

Ratings for Sustainability: (one overall rating)

4

Likely (L)

Negligible risks to sustainability, with key outcomes on track to be achieved by the project’s closure and expected to continue into the foreseeable future

3

Moderately Likely (ML)

Moderate risks, but expectations that at least some outcomes will be sustained due to the progress towards results on outcomes at the Midterm Review

2

Moderately Unlikely (MU)

Significant risk that key outcomes will not carry on after project closure, although some outputs and activities should carry on

1

Unlikely (U)

Severe risks that project outcomes as well as key outputs will not be sustained

 

ToR ANNEX D: MTR Report Clearance Form

(to be completed by the Commissioning

 

Midterm Review Report Reviewed and Cleared By:

 

Commissioning Unit

 

Name: _____________________________________________

 

Signature: __________________________________________     Date: _______________________________

 

UNDP-GEF Regional Technical Advisor

 

Name: _____________________________________________

 

Signature: __________________________________________     Date: _______________________________

 

 

ANNEX F: EVALUATION CONSULTANT CODE OF CONDUCT AGREEMENT FORM

Evaluators:

 

1.        Must present information that is complete and fair in its assessment of strengths and weaknesses so that decisions or actions taken are well founded.

 

2.        Must disclose the full set of evaluation findings along with information on their limitations and have this accessible to all affected by the evaluation with expressed legal rights to receive results.

 

3.        Should protect the anonymity and confidentiality of individual informants. They should provide maximum notice, minimize demands on time, and respect people’s right not to engage. Evaluators must respect people’s right to provide information in confidence, and must ensure that sensitive information cannot be traced to its source. Evaluators are not expected to evaluate individuals, and must balance an evaluation of management functions with this general principle.

 

4.        Sometimes uncover evidence of wrongdoing while conducting evaluations. Such cases must be reported discreetly to the appropriate investigative body. Evaluators should consult with other relevant oversight entities when there is any doubt about if and how issues should be reported.

 

5.        Should be sensitive to beliefs, manners and customs and act with integrity and honesty in their relations with all stakeholders. In line with the UN Universal Declaration of Human Rights, evaluators must be sensitive to and address issues of discrimination and gender equality. They should avoid offending the dignity and self-respect of those persons with whom they come in contact in the course of the evaluation. Knowing that evaluation might negatively affect the interests of some stakeholders, evaluators should conduct the evaluation and communicate its purpose and results in a way that clearly respects the stakeholders’dignity and self-worth.

 

6.        Are responsible for their performance and their product(s). They are responsible for the clear, accurate and fair written and/or oral presentation of study imitations, findings and recommendations.

 

7.        Should reflect sound accounting procedures and be prudent in using the resources of the evaluation.

Evaluation Consultant Agreement Form[11]

 

Agreement to abide by the Code of Conduct for Evaluation in the UN System

 

Name of Consultant: __________________________________________________

 

Name of Consultancy Organization (where relevant): ________________________

 

I confirm that I have received and understood and will abide by the United Nations Code of Conduct for Evaluation.

 

Signed at (place) on date

 

Signature: ________________________________________

 

TOR ANNEX G

EVALUATION REPORT OUTLINE[12]

Opening Page

  • Title of UNDP supported GEF financed project
  • UNDP and GEF project ID#s.
  • Evaluation time frame and date of evaluation report
  • Region and countries included in the project
  • GEF Operational Program/Strategic Program
  • Implementing Partner and other project partners
  • MTR Consultant
  • Acknowledgements

Executive Summary

  • Project Summary Table
  • Project Description (brief)
  • Evaluation Rating Table
  • Summary of conclusions, recommendations and lessons

Acronyms and Abbreviations

(See: UNDP Editorial Manual[13] )

1. Introduction

  • Purpose of the evaluation
  • Scope & Methodology
  • Structure of the evaluation report

2. Project description and development context

  • Project start and duration
  • Problems that the project sought to address
  • Immediate and development objectives of the project
  • Baseline Indicators established
  • Main stakeholders
  • Expected Results

3. Findings

(In addition to a descriptive assessment, all criteria marked with (*) must be rated[14] )

3.1 Project Design / Formulation

  • Analysis of LFA/Results Framework (Project logic /strategy; Indicators)
  • Assumptions and Risks
  • Lessons from other relevant projects (e.g., same focal area) incorporated into project design
  • Planned stakeholder participation
  • Replication approach
  • UNDP comparative advantage
  • Linkages between project and other interventions within the sector
  • Management arrangements

3.2 Project Implementation

  • Adaptive management (changes to the project design and project outputs during implementation)
  • Partnership arrangements (with relevant stakeholders involved in the country/region)
  • Feedback from M&E activities used for adaptive management
  • Project Finance
  • Monitoring and evaluation: design at entry and implementation (*)
  • UNDP and Implementing Partner implementation / execution (*) coordination, and operational issues

3.3 Project Results

  • Overall results (attainment of objectives) (*)
  • Relevance(*)
  • Effectiveness & Efficiency (*)
  • Country ownership
  • Mainstreaming
  • Sustainability (*)
  • Impact

4. Conclusions, Recommendations & Lessons

  • Corrective actions for the design, implementation, monitoring and evaluation of the project
  • Actions to follow up or reinforce initial benefits from the project
  • Proposals for future directions underlining main objectives
  • Best and worst practices in addressing issues relating to relevance, performance and success

5. Annexes

  • ToR
  • Itinerary
  • List of persons interviewed
  • Summary of field visits
  • List of documents reviewed
  • Evaluation Question Matrix
  • Questionnaire used and summary of results
  • Evaluation Consultant Agreement Form

 

 

ANNEX H: EVALUATION REPORT CLEARANCE FORM

(to be completed by CO and UNDP GEF Technical Adviser based in the region and included in the final

document)

Evaluation Report Reviewed and Cleared by

 

UNDP County Office

 

Name:_________________________________

 

Signature:______________________________ Date:______________________________

 

UNDP GEF RTA

 

Name:

 

Signature:___________________________ Date:______________________________

 

Annex I

CO-FINANCING TABLE FOR UNDP-SUPPORTED GEF FINANCED PROJECTS

 

Co Financing Types/Sources

IA Own Financing

(Million US $)

Government

(Million US $)

Other Sources[15]

(Million US $)

Total Financing

(Million US $)

Total Disbursement

(Million US $)

Proposed

Actual

Proposed

Actual

Proposed

Actual

Proposed

Actual

Proposed

Actual

Grant

 

 

 

 

 

 

 

 

 

 

Credits

 

 

 

 

 

 

 

 

 

 

Equity

 

 

 

 

 

 

 

 

 

 

In Kind

 

 

 

 

 

 

 

 

 

 

Non grant instruments[16]

 

 

 

 

 

 

 

 

 

 

Other Types

 

 

 

 

 

 

 

 

 

 

TOTAL

 

 

 

 

 

 

 

 

 

 

 

 

[1] For ideas on innovative and participatory Monitoring and Evaluation strategies and techniques, see UNDP Discussion Paper: Innovations in Monitoring & Evaluating Results, 05 Nov 2013.

 

 

[2] For more stakeholder engagement in the M&E process, see the UNDP Handbook on Planning, Monitoring and Evaluating for Development Results, Chapter 3, pg. 93.

 

 

[3] Populate with data from the Logframe and scorecards

 

 

[4] Populate with data from the Project Document

 

 

[5] If available

 

 

[6] Colour code this column only

 

 

[7] Use the 6 point Progress Towards Results Rating Scale: HS, S, MS, MU, U, HU

 

 

[8] Alternatively, MTR conclusions may be integrated into the body of the report.

 

 

[9] This list will be updated before MTE as more documents become available.

 

 

[10] The Report length should not exceed 40 pages in total (not including annexes).

 

 

[11] www.undp.org/unegcodeofconduct

 

 

[12] The Report length should not exceed 40 pages in total (not including annexes).

 

 

[13] UNDP Style Manual, Office of Communications, Partnerships Bureau, updated November 2008

 

 

[14] Using a six-point rating scale: 6: Highly Satisfactory, 5: Satisfactory, 4: Marginally Satisfactory, 3: Marginally Unsatisfactory, 2: Unsatisfactory and 1: Highly Unsatisfactory, see section 3.5, page 37 for ratings explanations.

 

 

[15] Other Sources refer to contributions mobilized for the project from other multilateral agencies, bilateral development cooperation agencies, NGOs, the private sector, etc. Specify each and explain “Other sources” of co-financing when possible.

 

 

[16] Describe “Non-grant instruments” (such as guarantees, contingent grants, etc.)

 

 

UNDP Around the world

You are at UNDP China 
Go to UNDP Global