CLM 5 Recipe 

1. Mystery Clients - are representatives from key populations who visit facilities as anonymous evaluators. One client per facility.

2. Facility Observation Tool (Version 3.0). This tool is used by mystery clients to observe and evaluate various indicators related to the services availed at the facility. 

3. CLM Monitors Daily Log. The mystery clients maintain a daily log where they record their experiences, observations, and interactions during the facility visits. This log serves as documentation of their visit and provides additional context to support the assessment findings.

4. Data Dictionary.  The data dictionary provides a standardized set of definitions and explanations for the indicators and variables used in the mystery client assessment. It ensures consistency and accuracy in data collection and analysis)

5. Facility Profiling Tool. The profiling tool is used to gather information about the facility, including its services, staff, and resources. This information helps in contextualizing the mystery client visit and understanding the overall capacity and performance of the facility.

Source: CLM Documentation Report, June 2023

Preparation stage

Step 1

Selection of Mystery Clients. Representatives from key populations, known as mystery clients, are chosen to conduct assessments of facility services. These mystery clients act as anonymous evaluators and provide valuable insights based on their firsthand experiences.

Step 3

Prepare the tools and brief/train the Mystery clients on how to use the tools.

Step 3

Facility Visit Scheduling. Mystery client visits are scheduled to occur every four months. This regularity allows for consistent monitoring and evaluation of the facility’s performance over time and allows for the identification of trends, gaps, and improvements in the quality of services provided.

Actual data collection

Step 1

Mystery clients go to the assigned facilities and utilize various tools to collect data, including the Facility Observation Tool (Version 3.0), CLM Monitors Daily Log, Data Dictionary, and Profiling tool. During their facility visits, mystery clients interact with staff, observe services, and ask specific questions related to the assessment indicators.

Mystery Clients carefully document their observations, interactions, and experiences using these tools. The scorecard provided in the Facility Observation Tool is completed based on their interactions and observations, capturing different aspects of the facility's performance. Additionally, mystery clients maintain a daily log to provide further context and support for their assessments. This documentation serves as valuable data for analysis and evaluation.

Step 2

Submission and Analysis. After completing the assessment, the mystery clients submit their scorecards and documentation to the KPAC CLM supervisor. The supervisor analyzes the data, validates the findings, and identifies areas for improvement.

Step 3

Facility Validation and Improvement. The assessments conducted by mystery clients, along with facility exit interviews, contribute to the validation of facility performance. The findings help identify areas that require attention or improvement, enabling targeted interventions to enhance service quality.

Source: CLM Documentation Report, June 2023

Enhanced Service Understanding

• CLM 5 provides an alternative perspective for service improvement.

• Balances client views on facility attributes and service quality. Example: Delays due to accessibility challenges Clients' understanding fosters a balanced perspective and better relationships.

Data-Driven Improvements

• CLM 5 employs mystery client assessments for data collection.

• Shared assessment data with the government to enhance facility services.

Source: CLM Documentation Report, June 2023

Data Quality and Documentation: 

• • Gaps identified in data completeness, accuracy, and cleanliness.

  • CLM 5 noted as the least developed recipe for data collection.

  • Challenge: Ensuring reliable and consistent data capture.

Training and Capacity Building:

• • Challenge in training KP data collectors (mystery clients).

  • Ensuring bias-free data collection process is a concern.

Source: CLM Documentation Report, June 2023

Tools List (as of June 2023)

1. 1. 1) CLM 5 Tool – Facility Observations through Mystery Client Scorecard. Version 3.0. 2023.04.06. [word]

   2. 2) CLM 5 Tool – Facility Observer – DAILY LOG Summary by Supervisors. Version 1. 2023.05.11. [word]

   3. 3) CLM 5 Tool – Facility Observation through fMystery Client DATA DICTIONARY. Version 1. 2023.05.05. [word] 

Descriptions (Annotated Bibliography)

Data Collection Tools

---. (2023). CLM 5 Tool – Facility Observations through Mystery Client Scorecard. Version 3.0. 2023.04.06. [word].

https://www.kpacpng.org/elibrary.

The Facility Mystery Client – Scorecard also measures quality of services for KPs in PNG. In this tool, members of key populations act as “mystery clients” to assess services and providers at facilities every four months using a scorecard to coincide and to triangulate the findings with the other CLM platforms like the Facility Exit Interviews (CLM 3) and the Community Forums (CLM 4). At least three KP or KP leaders who represent FF, IH and KC (selected among the monitors) act as “mystery clients”. The responses to be collected are focused on indicators for services availed and health services scorecard. These are collected without showing the questionnaire. There is a portion in the questionnaire for qualitative data concerning explanations on scores (1) Strongly Disagree or (2) Disagree. Additional observations on element of quality as observed by Mystery Client (third party of simulated client) are also included in the tool that are scored from 0-4.  Once the facility observer is done completing the details. He/she Data collection management details at the end of the form are to be filled-out by the CLM monitor, supervisor and encoder.

---.(2023).  CLM 5 Tool – Facility Observer – DAILY LOG Summary by Supervisors. Version 1.2023.05.11. [word].

This Daily Log Summary by Supervisors for CLM 5 is for tabulation of the data collected by facility observers using the Scorecard. The form contains entries about the master control numbers, instrument code numbers, CLM Monitor’s name and facility name. Key data include the total number of respondents interviewed and finished, total number approached but refused, and the total number approached (finished + refused). It also indicate the status check by supervisor, as well as notes regarding reasons for refusal or any other relevant observations. The log details are found at the lower end of the form.

---. (2023).  CLM 5 Tool – Facility Observation through Mystery Client – DATA DICTIONARY. Version 1. 2023.05.05. [word].

https://www.kpacpng.org/elibrary.

Data Dictionary for CLM 5 is also a reference for the variable names (for example, bi01, bi02 etc.) in the Facility Mystery Client Scorecard. This also guides the monitors in relation to the indicators, definitions and attributes/representation about the data being collected in the questionnaire. For example, Indicator #6 on specific services for STI could be consultation, testing, clinical examination, diagnosis and treatment. In coding, Code as 1 means that all encoded data came from respondents who gave consent. Entries for the data collection management has to be filled-up by all parties involved in the data collection and encoding.

Training Tools

---. (2023). CLM 5 – Facility Observation Through Mystery Client – Refresher & Planning Session #1. Annotated Program. 2023.03.05. [word].

The annotated program described the schedule, activities, process, lead facilitator, target outputs and needed materials/equipment for the refresher training on CLM 5. The expected results of this refresher training were: i) Drill CLM 5 Version 3 and filled out the form correctly, ii) Practice the written part of the observation, iii) Enable the facilitiy observers to ask questions for profiling facilities accurately while on field, iv) Supervisor will check the CLM 5 form with Monitors – MC, and v) Assigned MC to respective facilities. In general, the participants were expected to know how to conduct CLM 5 or facility observation by mystery clients and be familiarized with logistics assigned to 16 facilities.

Source:
CLM Documentation Report, June 2023
CLM Documentation Annotated Libray, June 2023