The Framework Method (framework analysis): steps, example and use in evaluation

Scope: interview and focus group data in applied research and evaluation. All example data are invented.

Short answer: the Framework Method (framework analysis) is a structured way to analyse qualitative data: you sort the data under a thematic framework and summarise it in matrices with cases as rows and themes as columns, so you can compare across cases and within each case and trace every summary back to the transcript (Ritchie & Spencer, 1994; Gale et al., 2013). The steps are:

1. Familiarisation with the data.
2. Coding a first set of transcripts.
3. Building a working analytical framework.
4. Indexing all the data with it.
5. Charting summaries into case-by-theme matrices.
6. Mapping and interpretation.

In an evaluation, the method organises the evidence but does not judge. To answer "how well did the programme work?" you also need explicit criteria and standards, set before the analysis.

Where the method comes from and what it is for

Jane Ritchie and Liz Spencer developed the Framework Method at the UK's National Centre for Social Research in the late 1980s for large-scale applied policy research (Gale et al., 2013). It suits applied studies with specific questions and a limited time frame, where commissioners and reviewers need to see how conclusions were reached. It is not tied to one epistemological position and can be used inductively, deductively or both (Gale et al., 2013).

Ritchie and Spencer (1994) distinguish four kinds of question in applied research. They are a useful way to sort evaluation questions before you start:

The Framework Method is a method of analysis, not an evaluation design. It arranges and describes data. Judging the merit, worth or significance of a programme requires criteria (what matters) and standards (what level counts as good enough) (Scriven, 1991; Davidson, 2005).

The stages, with an invented example

Invented example. All names and data are invented for illustration. Rovina Regional Hospital pilots a new rota on three wards: twelve-hour shifts and a shared team of nurses who move between wards. After three months, two focus groups follow the same topic guide: FG-A, eight ward nurses (N1–N8), and FG-B, six ward managers (W1–W6). Management wants a formative and process evaluation to decide whether and how to extend the pilot.

[A-13] N3: On the noticeboard. One day it was just up there. Nobody asked us,
           and then they were surprised that we swap shifts among ourselves.
[A-14] N5: It suits me, actually. I have a small child, and twelve-hour shifts
           mean more days off. If only we knew more than a week ahead.
[B-31] W2: The nurses complain, but overtime has gone down. That's a fact.
[B-32] W4: It's gone down on the spreadsheet. When they swap a shift between
           themselves, nobody records it as overtime.
[B-33] W2: But that's not the rota's fault.

Ritchie and Spencer describe five stages. Gale et al. (2013) break the same logic into seven: they start with transcription and split the framework stage into coding and building a working framework. In an evaluation, two add-ons come before and after the method itself.

In an evaluation: questions, criteria and standards first

Write each evaluation question (EQ) with an ID and label its kind. For each evaluative question, set a criterion and a standard; label the others as descriptive. An evaluation matrix makes this explicit and records what interviews cannot carry:

Criteria are value choices, often negotiated with the commissioner, drawn from programme objectives, professional standards or general sets such as the OECD DAC criteria: relevance, coherence, effectiveness, efficiency, impact and sustainability (OECD, 2019). Rubrics that describe what "excellent", "adequate" and "poor" look like make those values visible (King et al., 2013). If you refine a criterion after reading the data, record the change and the reason, so that criteria do not drift towards the results.

1. Prepare the data and define the case

Decide what a case is, because every comparison depends on it: participant, group or site. In the example, rows are participants grouped by focus group, and the group is the level of comparison. Note limits such as who was absent or barely spoke. Anonymity matters more than usual, because the commissioner is the participants' employer.

2. Familiarisation

Read all the transcripts and field notes, and note anything the evaluation questions did not foresee: unintended effects are an evaluation finding. The typical mistake is to read only for answers to the EQs.

3–4. Code, then build a working framework

Code the first few transcripts openly, using a priori codes from the EQs alongside codes from the data. Ideally two analysts code the first transcripts independently and compare (Gale et al., 2013); the point is to discuss differences, not to compute an agreement coefficient. Then group codes into numbered categories with short definitions, and add an "other" code under each category so that data that do not fit are not lost (Gale et al., 2013). Map categories to EQs and keep data-derived categories that fit no EQ.

1 Implementation of the pilot (EQ1)
  1.1 Informing staff about the change
  1.2 Staff involvement in preparation
  1.3 Deviations from the plan
  1.9 Other
2 Barriers and facilitators (EQ2)
  2.1 Shared team: induction and knowing the patients
  2.2 Differences between wards
  2.9 Other
3 Observed changes (EQ3)
  3.1 Workload and overtime
  3.2 Predictability of the rota
  3.3 Life outside work
  3.9 Other
4 Conditions for rollout (EQ4)
5 Outside the EQs: informal shift swaps and how they are recorded

Try the framework on another transcript and revise it. A framework derived only from the EQs, without coding any data, shows you only what you already expected.

5. Indexing

Apply the framework to all the data: label passages with category numbers, allowing more than one per passage. A-13 carries 1.1, 1.2 and 5; B-32 carries 3.1 and 5. If "other" keeps growing, revise the framework and re-index what you have already done.

6. Charting

For each main category, build a matrix: rows are cases, columns are subcategories, cells hold a short summary in the participant's terms with a reference to the transcript. Keep each summary close to the case, preserve relevant context and the source reference, and keep this descriptive charting apart from the cross-case interpretation that comes later. Charting already selects and condenses, so it is not interpretation-free.

Chart for category 1, Implementation (cells reflect only the excerpt above):

Chart for 3.1 and category 5 (cells reflect only the excerpt above):

Empty cells need an explanation. The matrix draws attention to contradictions, deviant cases and empty cells (Gale et al., 2013). Use explicit labels for the reason a cell is empty: not asked (the topic did not come up in the guide or the discussion), not mentioned (the participant did not speak to it), said not applicable, or ambiguous. An empty cell is never evidence that "there was no problem".

Focus groups. You may add a row for the group as a whole, but the cell must keep the internal differences and their sequence: "W2 reports lower overtime; W4 objects that swaps are not recorded; W2 replies that this is not the rota's fault." A dominant speaker's view is not the group's view. Interaction, agreement and disagreement belong in the summary (Kitzinger, 1994; Kidd & Parshall, 2000). See also How to analyse focus group data.

7. Mapping and interpretation

Read the matrices in both directions. Down a column: what is the range of views on informing staff, and where do the groups differ? Along a row: how do categories connect within one case? N5 values the extra days off but wants the rota earlier. Then build typologies, look for associations, and describe deviant cases separately. Keep participants' explanations (W2: "But that's not the rota's fault") apart from your own, and do not infer causation from a pattern in a small sample.

In an evaluation: from matrix to judgement

This is where an evaluation differs from a research report. The working logic of evaluation is to set criteria, set standards, gather evidence of performance against them, and synthesise that into a judgement (Fournier, 1995; Davidson, 2005). For each EQ, write:

1. The answer, worded to match the evidence. EQ1: "Nurses report that the rota was available only about a week ahead (N5) and that they were not consulted (N3); managers did not address either point. The publication dates can be checked against the published rotas."
2. The judgement against the standard. "Fidelity to plan: partial, provisional. The element 'rota published four weeks ahead' appears not to have been met; check the rota records."
3. The confidence you place in the finding, by rules set in advance. How many speakers and groups support a finding is one descriptive consideration, not a confidence score. Also assess how rich and relevant the data are, how coherent the finding is with the data, the counter-evidence, and the limits of the sources (who was in the room, what the guide asked). Use "not assessed" when the data are not enough. In the example, EQ1 rests on one speaker per point in one group. GRADE-CERQual assesses confidence in findings of qualitative evidence syntheses through methodological limitations, coherence, adequacy of data and relevance (Lewin et al., 2018). It can inspire such rules, but it was not designed for a single evaluation, and adapting it is a judgement you should state.
4. Perceived versus demonstrated effects. Interviews document how participants perceive change and the mechanisms they describe. Whether overtime actually fell needs the records, and B-31 to B-33 show that even the records are disputed. Report the disagreement; do not resolve it quietly.

Keep five layers apart: what a participant said, the summary in the cell, your interpretation, the evaluative judgement and the recommendation. A participant's wish ("If only we knew more than a week ahead") is a finding; a recommendation is the evaluator's separate judgement, linked to a finding and addressed to someone.

Causal questions ("did the pilot reduce overtime?") are beyond what interviews can answer on their own. Use a design built for them, such as contribution analysis (Mayne, 2012) or realist evaluation (Pawson & Tilley, 1997), together with other data. For process evaluation of complex interventions, the UK Medical Research Council guidance is a good frame for what interviews can contribute (Moore et al., 2015).

Framework analysis vs thematic analysis

The Framework Method belongs to the codebook family of thematic analysis, alongside template analysis (Braun et al., 2019). Unlike reflexive thematic analysis, which has no codebook and aims at interpretive depth, it fixes a framework early, indexes all data under it and keeps each case visible in the matrix. For how to choose between approaches, see How to choose a qualitative analysis method.

When the Framework Method is not the right approach

• Your question is causal. See above; interviews give perceived effects.
• Your data are highly heterogeneous. The method cannot accommodate data that do not cover similar topics (Gale et al., 2013), for example interviews, documents and observation without a shared structure.
• You want deep interpretation of meaning, not comparison of cases. Consider reflexive thematic analysis, or IPA, narrative or discourse analysis, depending on what you are analysing.
• The evaluation is developmental and the questions are still being found. Developmental evaluation supports innovation in complex, changing settings (Patton, 2011), and a framework fixed early can get in the way.
• You need a result in days, not weeks. Consider rapid qualitative analysis, and state what it leaves out.

Quality checklist

• Every evaluative question has a criterion and a standard; the others are labelled descriptive.
• The case is defined; claims about how widely a view is held count speakers and groups, not turns.
• All data are indexed, and "other" and "outside the EQs" are reviewed.
• Every cell can be traced to the transcript; empty cells carry a reason.
• Confidence rules were set before the judgement; "not assessed" is used where needed.
• The trail runs from each judgement through the matrix and index to the text.

How the app supports this

The Framework Method evaluation analysis in SurveysAI Qualitative Analysis Workflow follows this logic. You provide transcripts and an analysis brief with an evaluation aim, choose the evaluation purpose, and can add "Evaluation questions or criteria" (for example, the OECD DAC criteria) and a "Theory of change". The analysis produces an evaluation matrix with criteria and standards, indexing of every segment, charting by group, evidence strength rated by rules fixed in advance (strong, moderate, weak, contested or not assessed), and a judgement against the criteria for each evaluation question, reported by question with an executive summary.

If you do not supply criteria, the analysis proposes them and the report labels them as proposed, not agreed. The method section states that interviews show perceived effects and the mechanisms participants describe, not demonstrated effects, and that AI assisted the analysis. You choose whether you decide at key points yourself or leave them to the workflow; the report says which. An AI critic reads key outputs of the analysis; it is the same AI model that wrote them, not a human and not a second model, so it is not independent validation. Quotations are checked word for word against the transcripts by program code. The app produces draft analytic outputs; you remain responsible for reviewing, revising and owning the judgement before anyone acts on it.

Run it on your evaluation: create an account with Framework Method evaluation, upload your transcripts, and add your evaluation questions or criteria in the "Evaluation questions or criteria" field. If the analysis is not preselected, choose Framework Method evaluation in the start dialog.

References

• Braun, V., Clarke, V., Hayfield, N., & Terry, G. (2019). Thematic analysis. In P. Liamputtong (Ed.), Handbook of research methods in health social sciences (pp. 843–860). Springer. https://doi.org/10.1007/978-981-10-5251-4_103
• Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. SAGE.
• Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 1995(68), 15–32. https://doi.org/10.1002/ev.1017
• Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013). Using the Framework Method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology, 13, Article 117. https://doi.org/10.1186/1471-2288-13-117
• Kidd, P. S., & Parshall, M. B. (2000). Getting the focus and the group: Enhancing analytical rigor in focus group research. Qualitative Health Research, 10(3), 293–308. https://doi.org/10.1177/104973200129118453
• King, J., McKegg, K., Oakden, J., & Wehipeihana, N. (2013). Rubrics: A method for surfacing values and improving the credibility of evaluation. Journal of MultiDisciplinary Evaluation, 9(21), 11–20. https://doi.org/10.56645/jmde.v9i21.374
• Kitzinger, J. (1994). The methodology of focus groups: The importance of interaction between research participants. Sociology of Health & Illness, 16(1), 103–121. https://doi.org/10.1111/1467-9566.ep11347023
• Lewin, S., Booth, A., Glenton, C., Munthe-Kaas, H., Rashidian, A., Wainwright, M., Bohren, M. A., Tunçalp, Ö., Colvin, C. J., Garside, R., Carlsen, B., Langlois, E. V., & Noyes, J. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings: Introduction to the series. Implementation Science, 13(Suppl. 1), Article 2. https://doi.org/10.1186/s13012-017-0688-3
• Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–280. https://doi.org/10.1177/1356389012451663
• Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350, Article h1258. https://doi.org/10.1136/bmj.h1258
• OECD. (2019). Better criteria for better evaluation: Revised evaluation criteria definitions and principles for use. OECD DAC Network on Development Evaluation. https://www.oecd.org/en/publications/better-criteria-for-better-evaluation_15a9c26b-en.html
• Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.
• Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE.
• Ritchie, J., & Spencer, L. (1994). Qualitative data analysis for applied policy research. In A. Bryman & R. G. Burgess (Eds.), Analyzing qualitative data (pp. 173–194). Routledge. https://doi.org/10.4324/9780203413081_chapter_9
• Scriven, M. (1991). Evaluation thesaurus (4th ed.). SAGE.