Evaluation Methodology

???????????????????????????????Using empirical evidence to evaluate outcomes and impact is vital to adaptive management and fostering best practices. A number of systems for evaluation of conservation impacts, notably the Open Standards for Conservation (Conservation Measures Partnership), are available to guide development of indicators on conservation outcomes and data needed to assess them. Indicators have also been developed for captive animal welfare (ENDCAP), companion animal and farm animal welfare (for example European Union (EU) Welfare Quality Project and International Companion Animal Management Coalition’s (ICAM) Dog Population Management monitoring and evaluation guidance). Wildlife Impact develops custom indicators for each project’s specific wildlife conservation and welfare outcomes that build on these tools. Using these measures, along with other emerging systems of impact analysis (for example Black and Groombridge, 2010), the Wildlife Impact program is addressing the need for outcome evaluation by providing independent verification of project outcomes and valuable strategic planning tools, specifically: project-specific outcome indicators and data required for assessment; synthesis of project outcomes; an overall benchmark for project and for each outcome category; and specific steps to improve outcomes.

Wildlife Impact’s evaluation methodology, below, is always open for review and comments.

Impact Evaluation Methodology

Evaluation Process

  1. Applicant project/organization (applicant) provides organizational documents and or other written confirmation to demonstrate adherence to Wildlife Impact (WI) criteria for evaluation. This process typically takes 15 -30 days following receipt of documentation.
  2. Applicant pays agreed on fee to begin initial review. Upon receipt of funds, WI provides a list of documents and data needed based on goals defined in organizational documents from step 1, strategic plans and organization’s public messaging.

Minimum written documents required are:

  • Strategic plan if available; any available documentation on goals, mission and vision, and activities if no strategic plan
  • Management plan if available
  • Priorities/outcomes of key project funders
  • MoUs or agreements with key stakeholders, especially government
  • Policies and protocols for staff, management and operations
  • Program overview – descriptions, costs, staffing
  • Last fiscal year budget
  • Most recent annual report or annual work summary
  • List of actions and associated measures or indicators used to track progress, if available
  • Documentation of work accomplished, and data on progress towards goals or specifically against indicators, if available

The initial review process typically takes 45 – 60 days following receipt of documents.

  1. WI consults with applicant on results of initial review and whether applicant wishes to pursue an Outcome Analysis Framework or full Impact Evaluation. If not, the fee is non-refundable but can be applied towards full analysis fees within a two year period. Where a lack of strategic planning is cited as a key gap, WI can provide planning services that the fee can be applied to in lieu of a full Impact Evaluation.
  2. If an Outcome Analysis Framework or full Impact Evaluation is agreed to, WI provides invoice and applicant pays remainder of evaluation fee and moves forward to on-site visit and evaluation. All evaluations involve an on-site visit to gather data, talk to project staff and other stakeholders, and visually evaluate work and outcomes. Site visits are typically from 1 – 5 days depending on the project type and evaluation needs. Please be aware that coordination and scheduling of the site visit can sometimes add unexpected delays to the evaluation process.
  3. Upon receipt of funds, WI will provide applicant:
  • CV(s) of evaluator(s) to confirm there are no conflicts of interest. Alternate evaluator(s) will be provided if a conflict exists. Once evaluator is confirmed, WI will ask applicant to arrange for site visit by evaluator. Applicant arranged and pays for the reasonable travel, accommodation and board of the evaluator (on site accommodation and board are recommended to decrease cost and ensure convenient access).
  • a list of documents and data needed to proceed and a list of the indicators to be evaluated and what data will be used to do so.
  1. Following the on-site visit, WI will evaluated the data gathered and produce a comprehensive report synthesizing outcomes of the project, assigning an overall grade for the project as a whole and within each outcome category, as well as detailing successes and challenges and specific steps that can improve outcomes. WI may communicate with the applicant, stakeholders and the evaluator(s) if there are follow up questions or information needed. The evaluation process typically takes 90 – 150 days depending on the complexity of the applicant’s project(s) or organization.
  2. WI will communicate the names of applicants on its website and social media, newsletters and e-communications, and will promote evaluations with passing grades to donors and the public. Actual results of the evaluation are confidential to the applicant unless otherwise permitted by applicant. We encourage applicants to transparently communicate their results for the purposes of building trust with donors and the public; encourage greater support for the sector; identify common areas where support is required; and to help WI target and tailor its future programs to develop capacity where it is most needed by projects in the field. Scoring results and general outcomes will be used anonymously (without identification of the applicants) in compilations of all WI evaluation results, e.g. annual percentage of evaluations scoring a B grade, and percentage of projects that need additional funding support for strategic planning.

Evaluation methods and indicators

WI evaluation is based on principles of strategic planning, evaluation and adaptive management. Ideally, projects operate following a strategic process:

  1. Define what you want to accomplish
  2. Analyze the current situation and threats, including identifying the stakeholders and underlying social, political, economic and ecological systems
  3. Envision the desired situation and outcomes
  4. Identify strategies and actions to create these outcomes
  5. Develop measures to monitor progress and impacts
  6. Implement strategies and monitor progress and impacts
  7. Evaluate and adjust plans as needed to achieve success
  8. Share lessons learned. (Adapted from The Open Standards for the Practice of Conservation; Foundations of Success (FOS), The Nature Conservancy (TNC), and World Wildlife Fund (WWF)

In practice, strategic planning and evaluation are often difficult when projects struggle with few resources and staff to address time-sensitive issues in complex settings. WI helps organizations with strategic planning tools to identify appropriate indicators and monitoring actions, and provides independent evaluation of indicators and outcomes.

WI has developed a database of published and open source indicators, and develops specific indicators for each applicant depending on their goals and activities and the potential ramifications of their work in its geographic, biodiversity and sociopolitical contexts. This database is being finalized and will be available via the Wildlife Impact website for download and review.

We rely on indicators that meet the Biodiversity Indicator Partnership (BIP) guidance (2010):

  1. Scientific validity, meaning “a) there is an accepted theory of the relationship between the indicator and its purpose, with agreement that change in the indicator does indicate change in the issue of concern; b) the data used is reliable and verifiable.”
  2. Measured by data already available or data that could reasonably be collected by the project
  3. Responsive to change in the issue of concern
  4. Can be understood and communicated “a) conceptually, how the measure relates to the purpose, b) in its presentation, and c) the interpretation of the data”
  5. Relevance to the applicant’s needs
  6. Can reasonably be used, monitored and evaluated by the applicant (BIP, 2010).

WI evaluation processes, indicators and monitoring recommendations are based on the work of the Conservation Measures Partnership’s Open Standards for Conservation, Conservation Action Plans (The Nature Conservancy) Ferraro & Pattanayak (2006), Baylis et al. (2015), Conservation Excellence Model (Black et al., 2011), ENDCAP, European Union Animal Welfare Quality, International Companion Animal Management Coalition, Biodiversity Indicators Project, IUCN Action Plans for species or taxa (IUCN Species Survival Commission Specialist Groups), and Wildlife Impact.

Benefits

Benefits including outreach/promotion to funders and the public, and scholarships for capacity development are available as funding permits for two years following each analysis. Projects’ specific needs that would enable improved evaluation scores will be highlighted to funders and the public on Wildlife Impact’s website.

Wildlife Impact Principles

Wildlife Impact holds itself accountable to the following principles:

  • We are committed to improving conservation of wildlife and natural habitats and wildlife welfare in developing countries
  • We abide by and encourage application of national and international wildlife laws and conventions
  • We provide fair and safe work conditions and equal opportunity employment
  • We recognize human rights, social justice and cultural sensitivity in interactions with human communities.

Impact Evaluation Criteria
To seek an evaluation from Wildlife Impact, projects/organizations need to demonstrate they:

  • Have an institutional commitment to wildlife conservation and/or wildlife welfare
  • Projects act within and uphold local, national and international laws
  • Meet the minimum criteria below.

Minimum Criteria

Wildlife Impact cannot conduct evaluations of projects that participate in the following activities:

  • Breeding of wildlife (excepting animals that are part of an active reintroduction project with confirmed release site and reintroduction plan)
  • Research that compromises the well-being of individual animals or species
  • Use of wildlife as pets, for the purposes of financial gain, or for entertainment of any kind
  • Projects allowing the general public (individuals who are neither staff nor volunteers) to have physical contact with wildlife. Policies for staff/volunteer physical contact with wildlife must be based on precautionary principles to avoid disease transmission and deleterious behavioral modifications
  • Projects not abiding by CITES conventions or other wildlife/conservation laws.

Our minimum criteria conform to the standards of the Global Federation of Animal Sanctuaries (GFAS) and the Conservation Initiative on Human Rights (CIHR). WI encourages applicants with questions or concerns to contact us directly at info@wildlifeimpact.org.

Evaluation grading process

Wildlife Impact (WI) evaluates projects’ conservation and welfare impacts using accepted or accredited standards and approaches. WI creates specific indicators for each applicant depending on their goals and activities and the potential ramifications of their work in its geographic, biodiversity and sociopolitical contexts. WI uses indicators that meet the Biodiversity Indicator Partnership (BIP) guidance (2010). Our evaluation processes, indicators and monitoring recommendations are based on the Conservation Measures Partnership’s Open Standards for Conservation, Conservation Action Plans (The Nature Conservancy) Ferraro & Pattanayak (2006), Baylis et al. (2015), Conservation Excellence Model (Black et al., 2011), ENDCAP, European Union Animal Welfare Quality, International Companion Animal Management Coalition, Biodiversity Indicators Project, IUCN Action Plans for species or taxa (IUCN Species Survival Commission Specialist Groups), and Wildlife Impact. WI evaluations are conducted and reviewed by professionals with expertise in monitoring and evaluation, strategic planning, conservation, wildlife ecology and welfare, and organizational management.

Prior to evaluation, projects or organizations must first demonstrate, via sharing organizational documentation or other evidence, that WI evaluation candidate criteria have been met. Follow up on these criteria may also be part of the on-site evaluation. For a passing grade (>50%), project evaluation provides evidence that all WI criteria have been met, and evidence of some positive outcomes for wildlife conservation and/or welfare.

Wildlife Impact Evaluation Benchmarks
Benchmark type Description
Descriptive Grade Score
Exceptional A+ 100% Significant positive impacts, no noted negative impacts; sustainable; regular evaluationand strategic planning; goals are being met
Excellent A 90-99% Significant positive impacts; minor improvements possible but not neededNo negative impacts or negative unintended consequences needing to be addressed
High standards of work; Goals are being met
Project or organization is able to sustain activities to continue meeting goals
Regular strategic planning; Regular evaluation and adaptive management is undertaken orbeing implemented
Very Good B 80-89% Significant positive impacts; few minor improvements needed
No significant negative impacts
Good C 70-79% Many positive impacts but room for improvement; Few to no negative impacts
Needs Improvement D 60-69% Some positive impacts but could be improved; Few negative impacts
Needs Significant Improvement E- F 0 -59% Range from:
Few positive impacts but significant work needed to improve outcomes + Some negative impacts from activities or unintended consequencesTo:No impact or outcomes on stated goals AND/OR
Significant negative impacts or unintended consequences AND/OR
Publicized goals are not relevant to actual activities AND/OR
Activities have no discernible impact conservation or welfare AND/OR
Evaluation criteria found not to have been met (e.g. illegal trades, human rights issues)
Data Needed for Assessment Unable to make an assessment based on available data. Suggestions will be provided on indicators necessary to assess progress, data needed and collection methodologies, as appropriate.