CLM 3 Recipe 

1. At least 9 trained Community Monitors - dispersed to cover up to 3 facilities per day for 10-15 days (currently designed to be conducted every 4 months).

2. The main CLM 3 tool is a 4-page facility scorecard with each interview done within 10 – 15 minutes.

3. Teams with Team Leaders.  Monitors are divided into 3 teams, each consisting of  up to 3 monitors. One of the monitors is designated as team leader who oversees logistics and data collection activities of the team and leads data quality assurance

4. Supervisor. Responsible for overseeing the entire process and ensuring second-level data quality assurance.

5. Trainings, workshops, and exercises on Exit Interviews. Community monitors receive series of workshops and trainings on how to conduct the exit interviews, including a data collection simulation to practice their skills

6. Updated Directory of Facilities and profile of available services for referrals

7. Available tools. Facility Exit Interview Scorecard, Data Dictionary, CLM Monitor's Daily Log, Supervisor's Daily Log Summary, and Facility Data Profiling.

8. Daily debrief between monitors and supervisor.  For discussion about the quality of data collected, including completeness and accuracy, as well as any challenges encountered.

Source: CLM Documentation Report, June 2023

Preparation stage

Step 1

Coordinate with government officials and comply with Protocols.  During the initial coordination meetings, the CLM team introduce the CLM concept and start generating buy-in. Obtain permission to visit the facilities and conduct exit interviews.

Step 2

Organize CLM Teams. KPAC forms three teams, each consisting of three monitors, one of which is designated as team leader, who will be responsible for conducting the interviews and collecting responses from the KP community members who just accessed an HIV or health service in a facility .

Step 3

Prepare the logistics for the facility exit interview.

Step 4

Meeting of the CLM teams to discuss the plan for the exit interview.

Conduct of Facility Exit Interview

Step 1

CLM teams visit designated health facilities according to schedule and pay courtesy call to the facility head and scout interview venues.

Step 2

Conduct exit interviews with the respondents. The respondents are individuals from key populations who have accessed health services. CLM monitors approach a client after receiving services, greet, make introductions and obtain consent to interview; if consent is given, the monitor proceeds in administering the Exit Interview questionnaire (scorecard)

Step 3

Use the CLM 3 Tool-Exit Interview Scorecard. Monitors will use the CLM 3 Tool-Exit Interview Key Populations Scorecard (latest v.8.0 2023.05.25), a template designed to gather information about the respondent's background, their visit to a health facility, and their perception or experience of the quality of services received.

Continue with Health Services Scorecard: The scorecard section of the template measures the quality of services based on various indicators. Ask the respondents to rate their experience using a scale of 1 to 4 for each indicator, covering the 7 As of quality of services: awareness, accessibility, availability, affordability, appropriateness, acceptability, and accountability. Ask for  Additional Information on specific responses explaining low scores on specific indicators of the 7 As, inquiries about any other services needed, and dates.

Step 4

Record Data on Hard Copy Forms (scorecard): Monitors record the responses obtained during the exit interviews on hard copy forms. The data collected includes information such as the date of last access, province, district, name of the facility, main service availed, additional services availed, sex, gender identity, gender expression, gender preference, key population group, age, birth month, and birth year.

Step 5

Conclude the interview and thank the respondent.

Step 6

First-Level Quality Assurance Check. After completing all the forms, the monitors submit the filled-out forms to the team leader for a first-level quality assurance check. The team leader reviews the forms to ensure they are properly-filled and the data is accurate.

Step 7

Second-Level Checking and Coding. The team leader then submits all the quality-checked forms to the supervisor for second-level checking and coding. The supervisor further examines the forms for quality assurance and assigns appropriate codes to the collected data.

Step 8

Signatures and Submission. The monitors, team leader, and supervisor sign the completed forms to confirm their involvement and validate the data collected. Once all the necessary forms are completed, they are submitted to the appropriate officer for further analysis and decision-making.

The above description outlines the general steps involved in the data collection process based on the information provided and observed during the consultants’ visits and observation of Round 3 exit interviews in 2 clinics.

Source: CLM Documentation Report, June 2023

KPAC CLM 3 (2022-2023)

Client-Centered Approach

• Clinic adopts a client-centered approach to better understand and address client needs and expectations.

• Transition from name-calling to using codes for patients, enhancing friendliness and confidentiality.

• Change based on feedback from exit interviews, where KP clients expressed discomfort with being called by their names in a general population clinic.

Increased Staff Sensitization

• CLM process sensitizes clinic staff, especially those new to sensitive healthcare settings or working with KP groups.

• Improved confidentiality practices observed among service providers.

• Positive impact seen in NCD clinics where CLM 3 is implemented.

Source: CLM Documentation Report, June 2023

Probing Skills of Monitors

• • Inadequate probing skills of monitors were identified as a gap.

  • Monitors need to improve their ability to engage in meaningful conversations and probe for deeper insights.

Lack of Structured Format for Qualitative Data Gathering

• • Lack of a structured format poses challenges for monitors collecting qualitative data during exit interviews. Monitors require skills to capture qualitative information effectively.

Operational, Logistics, and Management Skills

• • CLM 3 Exit Interview initiative involves complex operational, logistics, and management skills.

  • Similar to managing a full survey research project in terms of complexity.

Retention and Attrition of Trained Monitors

• • Retaining trained monitors is a challenge that can disrupt data collection quality.

  • Addressing factors contributing to attrition is important, such as providing ongoing support and incentives.

Availability of Training Manual

• • Absence of a readily available training manual for monitors is a gap.

  • A comprehensive manual can guide monitors, covering qualitative data collection, real-time analysis, and clinic-specific action points.

  • A well-documented training resource ensures consistency and equips monitors with the necessary skills.

Source: CLM Documentation Report, June 2023

Tools List (as of June 2023)

1. 1. CLM 3 Tool – Exit Interview_Key Populations_Scorecard. Version 8.0. 23.05.25 [word]

   2. CLM 3 Tool -  Exit Interview - DAILY LOG by Monitors.Version 7.  2022.10.18. [word]

   3. CLM 3 Tool – Exit Interview – DAILY LOG Summary by Supervisors. Version 2.0. 2022.10.18. [word

   4. CLM 3 Tool – DATA DICTIONARY - Exit Interview_Key Populations_Scorecard. Version 6.0. 2022.09.09. [word]

   5. CLM 3 Monitors Schedule. [pdf] 

Descriptions (Annotated Bibliography)

Data Collections Tools

---. (2023). CLM 3 Tool – Exit Interview Key Populations_Scorecard. Version 8.0. 23.05.25. [word].

https://www.kpacpng.org/elibrary.

The Exit Interview – Scorecard for CLM 3 is a tool for measuring quality of services for key populations in Papua New Guinea. The seven categories of service quality are awareness, accessibility, availability, affordability, appropriateness, acceptability of services, and accountability of service providers. The four (4) page-scorecard is used during  facility exit interviews among members of key populations and people living with HIV that are conducted every four months by Community Champions acting as community monitors. For easy retrieval and verification of respondent’s answers, a master control number and an instrument code are indicated in the scorecard. Getting the consent of the respondent to participate by indicating yes or no is a requirement before asking questions. The questionnaire is divided into three sections: 1) Background information (15 indicators); 2) Health services card (14 questions); and 3) Additional notes if there are responses in the scorecard that indicate (1) Strongly Disagree and (2) Disagree (this is the qualitative data part of the questionnaire). It is important for the monitor to get the consent of the respondent to participate after properly introducing herself/himself and the objectives of the CLM. Information on the length of time spent on the interview gathered at the start and end of the interview is part of the data entries. For scoring, this is the guide: 1 – Strongly Disagree, 2 – Disagree, 3 – Agree, and 4 – Strongly Agree. And 0 for items that are Not Applicable. Data collection management for the scorecard is done by filling-up the required details at the end portion of the tool.

---. (2022). CLM 3 Tool -  Exit Interview - DAILY LOG by Monitors.Version 7.  2022.10.18. [word].

https://www.kpacpng.org/elibrary.

The Monitor’s Daily Log for Exit Interview – Scorecard is another tool for CLM 3. This form is where successful interviews for respondents who were approached and finished answering the questions are logged. The CLM Monitors are instructed to update this log after each interview is done and instrument has been reviewed for completeness and accuracy. The form includes the master control number, instrument code, start and end of interview, sex, age and KP group. It also requires indication of whether the data is complete and accurate, and checked by both team leader and supervisor. The column for Notes  are for observations and remarks, if any. At the lower portion, the CLM Monitor needs to fill out log details at the beginning of the day, and calculate the totals at the end of the day.

---. (2022). CLM 3 Tool – Exit Interview – DAILY LOG Summary by Supervisors. Version 2.0. 2022.10.18. [word].

https://www.kpacpng/elibrary.

In the Supervisor’s Daily Log Summary for Facility Exit Interview – Scorecard, three key information are tabulated and checked by the supervisor: i) Total number interviewed and finished; ii) Total number approached and refused; and iii) Total number approached (finished + refused). The form also indicates the master control numbers, instrument code numbers, name of the monitor and facility name. The column for Notes is intended for stating/describing the reason for refusal and other observations. Log details referring to the date and name of supervisor are filled out at the lower portion of the logsheet.

---. (2022). CLM 3 Tool – DATA DICTIONARY - Exit Interview_Key Populations_Scorecard. Version 6.0. 2022.09.09. [word].

https://www.kpacpng.org/elibrary.

An additional tool for CLM 3 is the Data Dictionary as reference for the variable names in the Exit Interview – Scorecard. The purpose of this tool is to provide the monitors with a guide in relation to the indicators, definitions and attributes/representation about the data being collected in the questionnaire. For example, Indicator #6 on specific services  for HIV could be consultation, HIV conseling, HIV testing, and HIV (ART) treatment. In coding, Code as 1 means that all encoded data came from respondents who gave consent. Entries for the data collection management has to be filled-up by all parties involved in the data collection and encoding.

---. (2022). CLM 3 Monitors’ Data Collection Schedule. [pdf].

https://www.kpacpng.org/elibrary.

The CLM 3 Monitor’s Data Collection Schedule for Facility Exit Interview indicates the a mini calendar plan with dates of visit, day, number of facility, group that will conduct the interview and name of health facility. It also specify the status of the exit interview, whether completed or not. This tool is for use of the three groups in each round of the exit interview. This monitors schedule is for the month of June 2022, Round 1.

Data Analysis and Visualization Tools

KPAC. (2023). CLM 3 Online Dashboard.

https://www.kpacpng.org/elibracy.

This Online Dashboard for CLM 3 uses the Microsoft Power BI software. The data in the Exit Interview – Scorecard that were analyzed and in an Excel spreadsheet are visualized into coherent and interactive insights through the online dashboard. In the health facility exit interviews for September 2022, Round 2, the overall score for 16 facilities and 406 respondents is reflected, including the specific scores for: i) awareness of services, ii) accessibility of services, iii) availability of services, iv) affordability of services, v) appropriateness of services, vi) acceptability of services, and vii) accountability of services. With this online dashboard, the number of respondents for each facility, scores ny monitor and date of interviews are indicated. A search on pertinent data by age group can also be done.

---. (2022). CLM 3 Round 1 Overall Results. v.2022.08.22 [pdf].

https://www.kpacpng.org/elibrary.

This presentation of CLM 3 Round 1 Overall Results consist of 18 powerpoint slides. The scorecard results are categorized as: 3.5-4.0 Excellent, 2.5-3.49 Good, 1.5-2.49 Fair, and below 150 Poor. Round 1 was conducted for 8 days on 14-16 and 20-24 June 2022. Nine  monitors with 2 supervisors collected data feom 361 clients in 15 out of the 23 facilities in the National Capital District (NCD). Results showed that: i) KPAC monitors were put in place, ii0 361 paper forms filled-out, iii) valuable collaboration/linkage was established with health facilities in NCD, iv) Baseline database was built using KOBO webform in June 2022 with World Vision International support, and v) based on data analysis, average score for 15 facilities was 3.7 or excellent.

---. (2022). Facility Scorecard – KPAC CLM 3 Scores. Template Sample v.1 [word]

https://www.kpacpng.org/elibrary.

This sample template visualizes the health services scorecard for Six (6) Mile Facility in NCD. Out of 4.0, the facility scored 3.52 or with an excellent rating. Of the seven (7) categories for quality, its highest score was on awareness (3.91), the lowest was on acceptability (3.47) and the rest of the scores were 3.62 and above. A certificate of excellent rating is given to the facilities which they use in turn as a material for promoting their facilities and services. This certificate is jointly signed by the NCD Provincial Health Authority Manager and KPAC Executive Director. 

TRAINING TOOLS

---. (2023). CLM 3 – Refresher Training #1 Annotated Program. 2023.05.23. [word].

The annotated agenda for the refresher training on CLM 1 on May 24-25, 2023 consisted of schedule, activities, process, lead facilitator, target outputs and needed materials/equipment. The expected outputs of this training were to: i) To reorient  and clarify the CLM 3Tool (v.7.0), Data Dictionary, Monitors Daily Log and Supervisors Daily Logbook to the 9x Monitors & 1x Supervisor), ii) Monitors and Supervisors practiced completing   the CLM 3 Tool, iii) Monitors and Supervisors highlighted and corrected common repeated mistakes on CLM 3 – Round #1 & Round #2, iv) Signed TOR for 4x CLM Monitors (new) and 1x Supervisor, and v)  Developed Planning Schedule for CLM 3 – Round #3 on 29th May -16th June 2023.

---. (2022). CLM 3 Round 1 – Training of Monitors – Annotated Program. 2022.05.31.[word].

https://www.kpacpng.org/elibrary.

The revised version of the Annotated Training Program on CLM 3 Facility Exit Interview was developed to present in detail the activity outline, estimated time for each activity, methodology, facilitator and other resources needed. This training was conducted on June 8-9, 2022 to ensure that Monitors are able to to fill-out the CLM 3 – Exit Interview Form accurately. The focus of the training was on the following activities: 1) Overview of CLM 3, CLM 4 -Tool A & Tool B, 2) Filling of CLM 3 -Facility Exit Scorecard, 3) Practical - Filling in Facility Exit Interview Scorecard, 4) Participant’s Feedback on-Practical Filling of the Facility Exit Interview Scorecard, 5) Filling in Community Forum Scorecard – Tool A & Tool B, 6) Practical -Filling in Community Forum -Tool A & Tool B, 7) Participant’s Feedback on-Practical Filling of the Facility Exit Interview Scorecard, 8) Explain Scoring, 9) Practical – Coding and Encoding of data using the filled forms being scored, 10) Participants Feedback on –Practical Coding and Encoding of Data. The target number of participants for this training was 25 (4 partners (KPAC recruited CLM 3 Exit Interview Monitors, PHA, NDoH)

# 11 participants (CLM team CMM rep) including facilitators.

Source:
CLM Documentation Report, June 2023
CLM Documentation Annotated Libray, June 2023