/
Text
Praise for this Book
This is one of the clearest and most accessible research methods books I have read as a
scholar. I highly recommend for both students and teachers alike. It is an excellent survey
of both qualitative and quantitative research methods.
—Natalie Danielle Baker, Sam Houston State University
Students writing a thesis or dissertation will benefit from this text’s approach to the development of and design of research. It provides a strong theoretical understanding of the
process of moving from an idea to a research proposal to the final research project. A good
addition to the library of someone who is doing research.
—Hugh Clark, Florida Gulf Coast University
This book provides a detailed outline of research methods. The authors break the concepts
and content into a manner in which it is easily understood by the reader. They also provide
activities for the reader to use to apply the content.
—Jaimee L. Hartenstein, University of Central Missouri
The authors describe the complexities of research and clearly describe the many details that
go into a well-designed and executed study. They have done an amazing job describing the
process of research design. Any student learning about research will benefit from this book.
—Lauren Hays, University of Central Missouri
This is an excellent text for those instructors who recognize the importance of giving
research design more comprehensive treatment than is typical of most textbooks.
—Scott Liebertz, University of South Alabama
I will add this book as required reading for my course and personally add this to my collection of research material to aid in my writing and evaluation projects.
—Sherill Morris-Francis, Mississippi Valley State University
This text is a very approachable conversation-based introduction to research design for
social science students. It provides an inviting approach to the research process that draws
the reader in as a participant instead of a spectator. It’s focus on research as an iterative and
reflexive process rather than a linear set of steps is refreshing. It engages the reader in the
process and introduces questions to assist in understanding the complexity and interconnectivity of each part of the process.
—Isla A. Schuchs Carr, Texas A&M University-Corpus Christi
Research Design
Sara Miller McCune founded SAGE Publishing in 1965 to
support the dissemination of usable knowledge and educate a
global community. SAGE publishes more than 1,000 journals
and over 600 new books each year, spanning a wide range of
subject areas. Our growing selection of library products includes
archives, data, case studies, and video. SAGE remains majority
owned by our founder and after her lifetime will become owned
by a charitable trust that secures the company’s continued
independence.
Los Angeles | London | New Delhi | Singapore | Washington DC
| Melbourne
Research Design
Why Thinking About Design Matters
Julianne Cheek
Elise Øby
FOR INFORMATION:
Copyright © 2023 by SAGE Publications, Inc.
SAGE Publications, Inc.
2455 Teller Road
Thousand Oaks, California 91320
E-mail: order@sagepub.com
All rights reserved. Except as permitted by U.S. copyright
law, no part of this work may be reproduced or distributed
in any form or by any means, or stored in a database or
retrieval system, without permission in writing from the
publisher.
SAGE Publications Ltd.
1 Oliver’s Yard
55 City Road
London, EC1Y 1SP
United Kingdom
SAGE Publications India Pvt. Ltd.
B 1/I 1 Mohan Cooperative Industrial Area
Mathura Road, New Delhi 110 044
India
All third party trademarks referenced or depicted herein
are included solely for the purpose of illustration and are
the property of their respective owners. Reference to these
trademarks in no way indicates any relationship with, or
endorsement by, the trademark owner.
Printed in [the United States of America / Canada]
Library of Congress Control Number: 2022914343
SAGE Publications Asia-Pacific Pte. Ltd.
18 Cross Street #10-10/11/12
China Square Central
Singapore 048423
Acquisitions Editor: Helen Salmon
Product Associate: Audra Bacon
Production Editor: Astha Jaiswal
Copy Editor: Diane DiMura
Typesetter: diacriTech
Proofreader: Barbara Coster
Cover Designer: Gail Buschman
Marketing Manager: Victoria Velasquez
This book is printed on acid-free paper.
22 23 24 25 26 10 9 8 7 6 5 4 3 2 1
BRIEF CONTENTS
Preface
xix
Acknowledgments
xxvii
About the Authors
xxix
Chapter 1
Research Design
1
Chapter 2
Ethical Issues in Research Design
27
Chapter 3
Developing Your Research Questions
47
Chapter 4
Why Methodology Matters When Designing Research
69
Chapter 5
Qualitative and Quantitative Approaches to Designing
Research
91
Chapter 6
Obtaining Data Using Qualitative Approaches
119
Chapter 7
Analyzing and Interpreting Qualitative Data
151
Chapter 8
Foundational Design Issues When Using Quantitative
Methods
181
Collecting Data Using Quantitative Methods
211
Chapter 9
Chapter 10 Designing Research Using Mixed Methods
241
Chapter 11 Why Knowing and Declaring Your Research Design Hand
Matters
271
Glossary
289
References
299
Index
313
CONTENTS
Preface
xix
Acknowledgments
xxvii
About the Authors
xxix
Chapter 1 Research Design
1
Purposes and goals of the chapter
1
Introduction: What Is Research Design?
2
Designing Research Is an Iterative Process
2
Research Design as a Messy, Complex, and Demanding Thought-Driven Process 4
Research Design: Working With the Literature
5
Using Relevant Literature When Designing Research
Working With the Literature Is Not the Same as Simply Reviewing It
5
7
How Do You Make Decisions About Which Literature to Trust or Rely On?
8
Journal Articles
Books and Book Chapters
Other Types of Literature That Might Be Useful if Used With Care
Research Design: Considering Methodology and Methods
Methods
9
10
11
12
13
Research Design: Considering Theory
14
The Importance of Reflexive Thinking When Designing Research
18
What Does Reflexivity Mean?
Putting Reflexive Thinking Into Practice When Designing Research
Ethics: Much More Reflexive Thinking Still to Do
18
19
20
Conclusions
21
Summary of Key Points
22
Key Research-Related Terms Introduced in This Chapter
23
Supplemental Activities
23
Further Readings
24
Notes
24
Chapter 2 Ethical Issues in Research Design
27
Purposes and Goals of the Chapter
27
What Is Research Ethics?
28
Putting Informed Consent Into Practice
29
Informed Consent—Who, What, and When
Informed Consent in Relation to “Vulnerable” Populations
30
31
x Research Design
Putting Confidentiality and Anonymity Into Practice
The Use of a Pseudonym Does Not Necessarily Ensure Anonymity
32
33
What You Need to Think About When Reusing, Repurposing, and Sharing Data 35
How to Address These Types of Questions?
36
What You Need to Think About When Using Information on the Internet as Data 37
Blurring the Boundary Between Public and Private
Working With Ethics Committees
Focusing on the Principles, Not the Requirements
37
39
40
Conclusions
41
Summary of Key Points
43
Key Research-Related Terms Introduced in This Chapter
43
Supplemental Activities
43
Further Readings
45
Notes
45
Chapter 3 Developing Your Research Questions
47
Purposes and Goals of the Chapter
47
Bringing Research Questions Into Focus
48
Feasibility Considerations
51
Putting the Idea of “Think Big, Plan Big, but Do a Small, Well-Contained
Study”3 Into Practice.
51
Using the Literature When Developing Research Questions
53
What Is Missing in the Existing Body of Knowledge in the Literature Related to
Your Problem?
Caution: Mind the Gap
Beyond the Gap
Different Forms of Reasoning and How They Shape the Form That Research
Questions Take
Deductive Reasoning
Inductive Reasoning
53
54
54
55
56
57
Putting Iterative and Reflexive Research Question Development Into
Practice—Learning From Others
58
Scratching the Underbelly of Research Design: Developing Clear Research
Question(s). Reflections by Maxi Miciak and Christine Daum.7
59
A Bit About Us, Our Projects, and What Makes Us “Qualified” to Write This.
In the Beginning There Was . . .
Generating the Question(s)
Embracing Rather Than Running From Critique
Landing on a Question(s)
Key Messages
59
60
60
63
64
64
Conclusions
65
Summary of Key Points
66
Key Research-Related Terms Introduced in This Chapter
66
Supplemental Activities
67
Further Readings
67
Notes
67
Contents xi
Chapter 4 Why Methodology Matters When Designing Research
69
Purposes and Goals of the Chapter
69
Thinking Methodologically
70
Data: A Concept Shaped by Methodological Assumptions
71
The Importance of Bringing Methodological Considerations Related to
Data Into Focus
72
Paradigms: Sets of Basic Beliefs That Guide Methodological Thinking
74
Onto-Epistemological Derived Assumptions Underpin
Methodological Thinking
76
Inquiry Paradigms and How They Connect to Methodological Thinking
78
Positivism
Critiques of Positivism
Post-Positivism
Constructivism
Inquiry Paradigms Affect Thinking About Whether Research Is Credible
Why Is Thinking About Your Paradigmatic Stance Important?
78
80
80
81
82
84
The Importance of Asking Methodological Questions of Your Research Design 85
Avoiding the Misuse of Methods
86
Conclusions
86
Summary of Key Points
87
Key Research-Related Terms Introduced in This Chapter
88
Supplemental Activities
88
Further Readings
90
Notes
90
Chapter 5 Qualitative and Quantitative Approaches to Designing
Research
91
Purposes and Goals of Chapter
91
Qualitative and Quantitative Research Strategies
92
Qualitative and Quantitative Approaches Reflect Different Research Purposes
A Word of Caution
Qualitative and Quantitative Approaches Reflect Different Logic of Inquiry
Quantitative Approaches Employ Deductive Reasoning
Qualitative Approaches Predominantly Draw on Inductive Reasoning
Another Word of Caution
Rounding Off Our Introductory Discussion of Qualitative and Quantitative
Research Strategies
Common Features Associated With Quantitative and Qualitative Approaches
Common Features Associated With Quantitative Ways of Thinking When
Designing Research
Common Features Associated With Qualitative Ways of Thinking When
Designing Research
Variation Within Quantitative and Qualitative Research Approaches
Quantitative Inquiry as a Diverse Approach
Quantitative Approaches Vary in the Methods That They Use
Capturing the Variety in Quantitative Approaches
92
93
93
93
94
95
96
96
96
98
101
101
103
103
xii Research Design
Qualitative Research Approaches as Diverse Strategies of Inquiry
More Specialized Forms of Qualitative Research
Ethnography as an Example of a Specialist Qualitative Approach
Discourse Analysis—Another Form of Specialized Qualitative Inquiry
How Many Specialist Types of Qualitative Inquiry Are There and
What Are They?
Capturing the Variety of Qualitative Approaches
Summing Up: Design Considerations in Light of the Variety Within Qualitative
Research
105
106
106
107
108
109
111
Which Are Better: Qualitative or Quantitative Research Approaches?
112
Conclusions
114
Summary of Key Points
115
Key Research-Related Terms Introduced in This Chapter
116
Supplemental Activities
116
Further Readings
117
Notes
117
Chapter 6 Obtaining Data Using Qualitative Approaches
119
Purpose and Goals of the Chapter
119
Qualitative Methods Are Not Stand-Alone Data Collection Techniques
121
Different Qualitative Methods Use Different Strategies of Inquiry
122
Key Questions to Ask Yourself When Choosing Types of Qualitative Methods
or Strategies of Inquiry
Navigating the Diversity Between and Within Qualitative Strategies of Inquiry
When Designing Your Research
How Structured Will Your Qualitative Interviews Be and Why?
Choices About Structure Are Choices About the Degree of Control You Have
Over the Interview
Using the Same Reflexive Thinking When Collecting Data Using Other
Qualitative Methods
Will You Interview Your Participants Individually or in Some Form of Group
and Why?
Focus Groups—A Specific Type of Interview
Which to Choose?
What Will You Ask Your Participants in the Interview and Why?
Developing Lines of Inquiry
How Many Lines of Inquiry and Associated Questions Are Ideal for an Interview Guide?
Designing Good Interview Questions
Ask One Question at a Time
Avoid Asking Dichotomous and Therefore Redundant or Limiting Questions
Don’t Ask Leading Questions or Make Leading Comments, When Interviewing
Try Out Your Draft Lines of Inquiry and Questions Before You Do
Your Interviews
Applying the Same Type of Thinking to Other Types of Qualitative Methods
Who Will You Interview and Why?
Choosing Between Different Types of Purposeful Sampling Plans in Your
Study Design
Putting Purposeful Sampling Into Practice When Designing Your Research
123
123
125
126
127
129
130
130
132
132
134
135
135
136
136
136
137
140
140
142
Contents xiii
Conclusions
146
Summary of Key Points
146
Key Research-Related Terms Introduced in This Chapter
147
Supplemental Activities
147
Further Readings
148
Notes
148
Chapter 7 Analyzing and Interpreting Qualitative Data
151
Purposes and Goals of the Chapter
151
Analysis of Qualitative Data: An Iterative and Dynamic Strategy
152
When Does Analysis “Begin” When Designing and Conducting Qualitative
Research?
153
Using Memos to Capture Your Analytic Thinking and Hunches
Why You Should Not Wait to Begin Analyzing Your Qualitative Data Until
All Your Data Is Collected
Developing an Iterative Qualitatively Driven Analytic Strategy
Strategies for Organizing the Data You Collect and Keeping Track of Your
Analytical Thinking About That Data
Strategies for Deciding What Parts of the Data You Have Collected Are
Relevant for Addressing Your Research Problem
The Process of Data Condensation
154
156
157
157
159
159
Coding—A Strategy to Condense Your Data
161
More Choices and Decisions to Make When Putting Coding Into Practice
163
Methodological Choices About Whether to Employ an Inductive or Deductive
Approach to Your Coding
Choosing a Coding Strategy Congruent With the Theoretical Pillars of
Your Design
Coding in Grounded Theory
Rounding Off Our Discussion of Coding
The Art of Interpretation
Ways of Establishing the Credibility of the Interpretations You Make and
Therefore the Rigor and Trustworthiness of Your Research
Collecting and Analyzing Data—When Do You Know That You Are “Done”?
Connecting Analytical Considerations to Decisions About Sample Size
Principles to Guide Sample Size Considerations in Your Qualitative Research
Design
163
165
166
167
168
170
171
172
173
Conclusions
175
Summary of Key Points
176
Key Research-Related Terms Introduced in This Chapter
177
Supplemental Activities
178
Further Readings
178
Notes
179
xiv Research Design
Chapter 8 Foundational Design Issues When Using
Quantitative Methods
181
Purposes and Goals of the Chapter
181
What You Need to Think About in Order to Design Credible
Quantitative Research
182
Key Questions to Ask Yourself When Designing Quantitative Research
Why You Need to Ask Yourself All These Key Questions Simultaneously
Where to Begin? Deciding Who You Will Collect Numerical Data From
and Why
What You Will Need to Think About When Using a Sample in Your
Research Design
How Do I Design a Sampling Strategy That Enables a Representative Sample?
Are There Other Sampling Strategies I Can Consider if Probability Sampling
Is Not Feasible?
Rounding Off: Important Things to Keep in Mind if You Decide to Use a Sample
in Your Research Design
Choosing an Analysis Procedure Suitable for Answering Your
Research Question
Research Questions About What Is Going On in a Study Population
Descriptive Procedures
Correlational Procedures
Research Questions About Why Something Happens in a Study Population
When the Research Question Takes the Form of a Hypothesis
Aspects That Affect Whether or Not the Conclusions From Testing a Hypothesis Are
Statistically Reasonable
Still More Thinking to Do
What Types of Data Are There?
Nominal Data
Ordinal Data
Interval/Ratio Data
How Different Types of Data Enable Different Types of Knowledge
Why Making Sure You Collect Data of the “Right” Type Is Not Enough to
Ensure That Your Research Design Is Statistically Reasonable
Conclusions
184
184
185
187
188
189
190
192
193
193
194
195
197
197
198
199
199
200
201
203
205
205
Summary of Key Points
207
Key Research-Related Terms Introduced in This Chapter
208
Supplemental Activities
208
Further Readings
209
Notes
209
Chapter 9 Collecting Data Using Quantitative Methods
211
Purposes and Goals of the Chapter
211
Measuring Variables to Enable Valid Research Findings
212
Are You Measuring What You Think You Are?
Making Sure the Measurements You Make Measure What They Claim to
Example 1
Example 2
What Can We Conclude From These Examples?
213
214
215
215
216
Contents xv
Taking a Closer Look at What Enables Variables to Be Measured
216
How to Make Abstract Variables Measurable
218
Step 1: Clarify Your Understandings (i.e., Develop the Constructs) of the Variables
Step 2: Identify Quantifiable Factors That Will Enable Measuring the Constructs
Developing Measurement Items to Actually Measure the Variables
Developing a Measurement Instrument
Developing Measurement Items That Measure What You Intend
Them to Measure
Do Your Measurement Items Demand Too Much From the Respondents?
Will the Measurement Items Enable Consistent Measurements of the
Quantifiable Aspects They Are Intended to Measure?
Is the Form a Measurement Item Takes in Keeping With What That Item Is
Supposed to Measure?
Takeaways From This Section
Putting a Measurement Instrument Into Practice
Response Rates and How They Relate to Sampling and Validity
Ways of Collecting Data When Doing Survey Research, and Why They Matter
Scenario 1
Scenario 2
Scenario 3
Take-Home Messages From These Scenarios
218
219
220
222
222
224
226
227
229
229
230
231
232
233
233
234
Conclusions
235
Summary of Key Points
237
Key Research-Related Terms Introduced in This Chapter
239
Supplemental Activities
239
Further Readings
240
Notes
240
Chapter 10 Designing Research Using Mixed Methods
241
Purposes and Goals of the Chapter
241
What Is a Mixed Methods Research Approach?
242
Mixed Methods Research as Combining Qualitative and Quantitative
Research Approaches
Snapshot 1: Mixed Methods as a Method That Combines Qualitative and Quantitative
Approaches
Snapshot 2: Mixed Methods as a Way of Thinking That Combines Aspects of Qualitative
and Quantitative Thought
A Different Definition and View of What Mixed Methods Research Is
Snapshot 3: Mixed Methods As a Single Study, but Where One of the Methods Is
Incomplete and Cannot Stand Alone
What Can We Learn From These Snapshots About What Mixed Methods Is?
Why Use a Mixed Methods Research Design?
The Importance of Thinking About Why You Might Use a Mixed
Methods Approach
Priority and Timing of the Components in a Mixed Methods Study
Thinking About Matters Related to Priority or Weighting of Components
Thinking Through, and Deciding About, Matters Related to the Timing of
the Components
243
243
244
244
244
246
247
248
249
249
250
xvi Research Design
An Example of How to Connect Purpose, Priority, and Timing and Why
This Matters
Option One: A QUAL → Quan Design
Option Two: A QUAN → Qual Design
What Can We Learn From This Example?
Mixing—A Central Consideration in Mixed Methods Research
“Mixing” as a Concept Not Just a Procedure
Incorporating Different Levels of Focus Into Your Thinking About Mixing
What About Paradigmatic-Related Considerations When Mixing Methods?
Summing Up Our Discussion of Mixing
Strategies for Navigating the Complex and Contested Field of Mixed
Methods Research
Strategy One: Use Diagramming (Not Just Diagrams) as a Way of Keeping
Track of the Decisions You Make About Your Emerging Mixed
Methods Design.
Strategy Two: Keep Folding Back Reflexively on Your Own Thinking, and the
Decisions You Are Making or Have Made, When Designing and Reporting
Your Mixed Methods Research.
Strategy Three: Don’t Go It Alone. Get Help and Find Support Along the Way.
253
254
255
256
257
258
258
259
261
262
262
263
265
Conclusions
265
Summary of Key Points
266
Key Research-Related Terms Introduced in This Chapter
267
Supplemental Activities
268
Further Readings
269
Notes
269
Chapter 11 Why Knowing and Declaring Your Research Design Hand
Matters
271
Purpose and Goals of the Chapter
271
Knowing and Declaring What Your Research Design Related Hand Is
273
Pulling Up a Chair at the Research Design Table
Putting All of Your Cards on the Table and Declaring Your Hand:
An Important Part of Research Design
274
276
Declaring Your Hand: Missing in Action in Much of the Reporting of Research 276
What Happened Along the Way?
277
How to Declare Your Research Hand: Circling Back to Tell the Story of the
Designing of Your Research
279
How to Reflexively Circle Back to Tell the Story of Your Project When
Reporting on Your Research
281
Capturing All This in a Diagram of Some Sort
Developing, and Then Diagramming, an Overview of the Process of
Research Design
The Importance of If . . . Then Thinking
282
283
284
Conclusions
285
Summary of Key Points
286
Key Research-Related Terms Introduced in This Chapter
288
Contents xvii
Supplemental Activities
288
Further Readings
288
Notes
288
Glossary
289
References
299
Index
313
PREFACE
WHY WRITE A BOOK ABOUT RESEARCH DESIGN? AND
EVEN MORE IMPORTANT, WHY SHOULD YOU READ IT?
Why did we write this book?
D
esigning research is about making decisions to transform a research idea into a
research plan able to provide answers about some sort of research problem or question. Thinking about, and then making these decisions results in what we call our research
design—the plan that we will follow to put our research into practice and address our
research problem or questions.
Making these decisions begins the moment that we begin to think about a topic that
we want to know more about. For example, What specifically do we want to know about
this topic and why? What contribution is the research that we are designing intended to
make to the development of knowledge related to this topic? How can we obtain that type
of knowledge in a way that is credible? When we think through and address these types of
questions, we are designing research.
A few years ago, we began developing a reading list for a course we were going to teach
about research design. We had assumed, given the explosion in research related textbooks
and articles published in the past decade, that our problem would be which texts to choose
given that there would be so many to choose from. Well, we were wrong. What we found
on closer examination was, that in fact, there were very few books about research design
itself.
If the thinking sitting in and behind designing research was overtly discussed at all,
it was usually done so in relation to an aspect of research design that the textbook was
focused on—not research design as a whole. For example, some books discussed research
design in relation to a specific type of method, or how to write a specific part of that design
such as a literature review. Others discussed research design in relation to qualitative or
quantitative approaches to research more generally. The result of this was that research
design was usually discussed as part of a book that was focused on something else such as a
specific method or an approach to research.
It seemed to us that the focus on designing research was getting lost in all this, and
becoming secondary to a focus on matters of procedures and techniques associated with
the research methods that formed part of that design. While methods and procedures for
putting those methods into place are part of what research design is, in themselves, they
are not what designing research is all about. Research design is about the layers of interconnected thinking and decisions that gave rise to that choice of method or procedure to
be part of the overall research design.
This prompted us to take a closer look at some of the syllabi for courses about research
design—what was being taught about research design and how? We discovered the same
thing as we had when looking for research design related textbooks. There was much more
focus on methods and how to design them in these syllabi and course outlines, and much
less focus on the thinking needing to be done before deciding on a particular method or
xx Research Design
approach to be a part of an overall research design. In effect, research design itself had once
again been relegated to a secondary focus and discussed in relation to something else—
the procedures that make up those methods.
We wondered if we had missed something in all this, or maybe even just got it wrong.
However, looking into this further, we found that what we were noticing, others were
commenting on too. For example, Denzin and Giardina (2016) observed that students
learning about research and research design are
more often than not taught particular “methods of data collection” (such as interviews, case studies, focus groups, ethnography, other basic research design techniques, etc.) within the context of research methods or research design courses; it
is few and far between that philosophy of science and philosophy of inquiry seminars are required of graduate students—and even fewer still, we would contend,
that call into question or contest the very notion of data or evidence itself. (p. 6)
This trend troubled us. It was this that provided the initial trigger for us to start
thinking about writing this book. Writing a book about research design, rather than
methods of data collection was the goal. When doing so we wanted to develop and explore
the idea of developing a research design where the emphasis was put on the process (the
designing), not just the product (the design). To achieve this, we wanted to overtly return
thinking to where it rightfully belongs in the process of designing research. This is at the
center of that process to which, and from which, all paths and decisions lead and intersect
as our research design emerges.
This is because the form that a research design comes to take affects every part of the
process of designing research—from the questions that are, or even can be asked, through
to the way that data and the ways of obtaining that data are thought about. We agree with
Kuntz (2015) that to be methodologically responsible, and for our purposes here, being
responsible when designing research, cannot simply be “reduced to maintaining the integrity of research procedure . . . which method to use, where to use it, and how to interpret
the data it produces” (p. 11). Rather it is thinking through, and justifying, the way that a
research design became the way that it is, that is important for maintaining the integrity of
the research design, the data produced as a result of putting that design into practice, and
the interpretations made of that data.
This is a thinking that requires us to ask a lot of questions—both about all areas of
the research design, and the assumptions that we bring with us when we are designing our
research. As Swaminathan and Mulvihill (2017) note “[a]sking good questions is fundamental to the heart of research, critical thinking, creative thinking, and problem solving”
(p. 1). It is also fundamental when designing research.
Therefore, there is much intellectual work to do, and many decisions to make, when
thinking about your research design. This book is about this intellectual work and those
decisions. For as Becker (1998) reminds us,
[E]very subject we study [for our purposes here that includes research design] has
already been studied by lots of people with lots of ideas of their own . . . about what
it’s about and what the objects and events in it mean. These experts by profession
or group membership usually have an uninspected and unchallenged monopoly of
ideas on “their” subject. Newcomers to the study of the subject, whatever it is [for
our purposes read research design] can be easily seduced into adopting those conventional ideas as the uninspected premises of their research. . . . [W]e need ways
Preface xxi
of expanding the reach of our thinking, of seeing what else we could be thinking
and asking, of increasing the ability of our ideas to deal with the diversity of what
goes on in the world [or for our purposes, in and when designing research]. (p. 7)
Making a contribution to expanding the reach of our thinking and seeing what else we
could be thinking and asking about research design, and of ourselves when designing that
research, is why we wrote this book.
Why do we think that you should read this book?
So, keeping in mind the above discussion about why we wrote the book, why do we think
you should read this book?
First, the book is written in such a way as to open up, rather than shut down, discussions of research design. It introduces and develops the idea of a research design process in
its own right, but at the same time acts as a resource for indicating what the next steps are
in obtaining more information about each part of that process. Put another way, it gives
you enough knowledge to know what you do not know about the process of designing
research, and therefore what you need to find out more about, and how you might do that.
Second, throughout the book we engage in a dialogue with you the reader to provide
a serious but accessible introduction to research design able to be used by you as a guide
when designing your own research and/or reading and making judgments about reports
of other’s research. The style of this dialogue is in the form of a conversation about what
we, and others, have learned as researchers and teachers of research that we wish someone
had told us about before we began reading about, and attempting to design our research.
Third, in this dialogue we actively encourage you to think with, and through, what is
written in the book, and then bounce off that thinking to do even more thinking. These
recurring cycles of thinking continue throughout the reading of this book, just as they do
throughout the “lifecycle of the research [design] process” (Swaminathan & Mulvihill,
2017, p. 2). They will require you to focus on, think through, and make informed decisions about a range of interrelated and interconnected theoretical, methodological, and
ethical considerations that shape what we term, or know as, a research design. These
recurring cycles of thinking, and the understandings that this enables, is an important
part of ethical and responsible research and research design. This is so no matter what
research approach we are using. It is not an optional extra.
Fourth, we do not pretend that such thinking will be a straightforward or easy process.
It will be challenging and messy at times. And, as Becker (1998) notes, “[I]t's more work
than if you did things in a routine way that didn’t make you think at all” ( p. 7). However,
the payoff is a well thought through research design able to be defended, and consequently
research that is able to be trusted. Therefore, maybe you should read the book because of
what it is not—an “easily digestible” (Koro-Ljungberg, 2016, p. 6) oversimplification of
the complex process of designing research.
Fifth, it is not often that we get access to accounts and reflections of how this complex
process was navigated as it is often hidden behind what Morse (2008, p. 1311) calls “the
elegance of the end product”—in this case the research design that is produced. Therefore,
we have included in the book accounts of examples of how this complexity was navigated
by researchers and students when they were thinking about, and through, their research
design. What did they do and why? These reflexive accounts provide you with unique
insights into the “real story” of designing research and are a powerful way of learning from
the hard-earned wisdom of others.
xxii Research Design
WHO IS THIS BOOK FOR?
This book is for anyone with an interest in better understanding the process of designing
research. The discussion in the book will assist you to think through questions such as,
What needs to be thought about, and through, when designing research? When and why?
What are the effects of that thinking on the way that research is designed and a research
design produced? What do you need to think about when reading about the way research
was designed in order to make decisions about if you trust that design?
The discussion is grounded in our experience of teaching research to undergraduate
and postgraduate students, and also of working with people who are expert in a variety of
practice fields who want to know more about research. The material is presented in such
a way that it can be overtly linked to readers’ development of their own research designs
for their research projects. Readers not actually engaged in conducting research can use
the material to test the thinking, and expose the assumptions, in research reports that they
may be presented with, or are using, in their practice areas. Therefore, the book will appeal
to, and be useful for, a wide range of readers.
Students attending courses in research (e.g., research design, as well as specific methods) and also students who are actually doing research (especially for the first time) will
find the book a useful companion, and at times sparring partner, as they are thinking
about and designing their research. The book will also interest those teachers of courses
about research design or research methods who are looking for a text that challenges,
opens up, and extends students’ thinking about research design rather than limiting it to a
focus on research design in relation to specific research methods.
Another group of readers often overlooked when thinking about who might find a
book like this useful are practitioners and managers who want to use research to provide
evidence for their practice or the practices that they manage. This might be either by
doing that research themselves, or by being able to make informed decisions about the
trustworthiness of research that has been reported. For those managers commissioning
research, the book provides an overview of the research design matters that are important
to consider when writing the brief for the research, evaluating the progress of the research,
and using the results of the research.
HOW THE BOOK WORKS
The book focuses on the thinking that underpins and shapes the development of a
research design. This thinking affects the choices and decisions we make when designing
our research, as well as the way we write about, and represent, the research design produced as a result of that thinking and those choices. Thus the “red thread” that holds the
chapters together is the idea of designing research as a process in which there is constant
thinking through, and revisiting of, decisions about that design as it is developed. There
are many layers of thinking that sit behind a description, diagram, or representation of
what we call a research design.
Capturing the interconnected and layered thinking that sits around, and in, any
research design led to quite a few challenges when we were writing this book. Chief among
them was how to capture a complex, nonlinear, and dynamic process of thinking in a static
book that by necessity presents material from beginning to end. We do this by overtly
linking all the chapters in the book. Hence, points introduced in one chapter pick up on
Preface xxiii
points from earlier chapters, or are picked up on and developed in the chapters that follow
and then returned to again.
We use tip boxes and sometimes endnotes to indicate where in the book those discussions occur to guide those readers who want to explore how points made in the chapter
they are reading are connected to discussions in other chapters. They then can return to
rejoin the discussion in the chapter they were reading originally. In this way, we actively
encourage you, our readers, to use the chapters in this book, and the thinking in them,
“when it looks like they might move your work along—at the beginning, in the middle, or
toward the end of your research” (Becker, 1998, p. 9).
Consequently, just as thinking about research design cannot be reduced to a linear
process, neither can reading this book. When thinking about and writing this book and
the individual chapters in it, we never intended that the book or the chapters must, or even
could, be read in a linear form in a single direction. Reading and interacting with this
book involves moving backward and forward in the thinking that is at the heart of the
research design process.
The Structure of the Book
Each chapter begins with an outline of the purposes and goals of the chapter. Here we
present the issues to be discussed and thought through in the chapter. The chapter takes
the form of an extended discussion of those goals. At the end of each chapter, the key
points from that chapter are summarized, thereby providing a way for readers to do a
self-check of how they went when reading the chapter. Throughout the chapters, we use
the device of different types of boxes (Tip, Activity, and Putting It Into Practice boxes) to
emphasize specific points made in the text, to encourage you to think about the practical
implications of what we have discussed, and to provide tips about where to find more in
depth or detailed discussions of such points.
To consolidate the discussion in the chapter, after the summary of the key points in
each chapter, we have included several activities for you to try. These activities are designed
to reinforce central points and ideas raised in the chapter. They focus on putting these
ideas into practice when you are actually designing your research. This is because it is one
thing to have a discussion about central points related to research design, but quite another
to actually put that discussion into practice! These activities can be done individually or
with others such as in a class or tutorial group. Each chapter then concludes with a list of
some suggested further readings related to the content in the chapter.
To build your research design related vocabulary, for each chapter we provide a list of
any new key research design related terms introduced for the first time in the book in that
chapter. While we will have discussed and defined these terms throughout the discussion in that chapter (and placed them in bold where first introduced), a concise working
definition of each of these terms can be found in the Glossary at the end of the book. This
Glossary will assist readers who may need to refresh their memory about terms first introduced in earlier chapters.
Recommended Ways of Reading the Book
Without appearing to suggest the obvious, we recommend that you enter our conversation
about research design by reading Chapter 1, in which we take a closer look at the idea of
designing research and what that actually means and involves. Chapter 1 is a foundational
xxiv Research Design
chapter as it introduces you to many of the terms you will come across, and concepts
that you will need to think about, when designing your research. Focusing on designing
research as a nonlinear process involving recurring cycles of thinking about each aspect of
your research design, the chapter provides an overview of, and introduction to, the areas
that together make up your research design. Areas such as methodological, theoretical,
and ethical considerations.
We also look at the role that relevant literature plays in the development of your
research design. The chapter calls for reflexivity (we take a close look at this idea—what it
is and why it matters) on the part of the researcher to acknowledge and make transparent
the theoretical and methodological assumptions shaping their research design. We argue
that reflexivity and ethical considerations are central to designing responsible research.
The rest of the book unpacks and develops the points raised in Chapter 1. The following points are designed to give you a taste of what is to come:
• In Chapter 2, we pick up on the point of ethical thinking permeating the entire
research design process. The chapter focuses on reflexively thinking about, with,
and through ethical considerations and what they mean for the way that you will
design your research. Allowing ethical thinking to frame your research project in
this way means acknowledging that ethical thinking in research involves much
more than getting ethics approval from an ethics committee.
• Chapter 3 is about the process of focusing, and refocusing, your thinking to
work out what actually the problem area or issue that your research is being
designed to address is, and what the questions are that you want to ask about that
problem or issue and why. We look at how this thinking, and the decisions that
you make as a result of it, affects what form your research questions take, and the
possible answers that they can give. The chapter includes an invited contribution
from Maxi Miciak and Christine Daum which provides a unique reflexive
account of how these researchers used cycles of thinking to work out what it was
that they really wanted to focus their doctoral research on and why.
• In Chapter 4, the focus is on the connections between method, methodology,
epistemology, and ontology, and how these connections affect the way we
design our research. We focus on how any specific research method is shaped
by methodological assumptions and understandings, which in turn are shaped
by onto-epistemological assumptions. Thus, methods are not an isolated set of
rules or neutral procedures for data collection. The role they play in, and the
effect they have on shaping, a research design is the result of the thinking and
understandings that gave rise to those methods in the first place.
• Chapter 5 picks up on the connections between method, methodology,
epistemology, and ontology discussed in Chapter 4 to explore the effects of these
connections on the way that we think about data, evidence, and what are known
as qualitative and quantitative approaches and data when designing our research.
• Chapters 6 through 10 are a series of chapters in which we discuss methods
as part of the interconnected web of decisions that make up a research design.
In these chapters, we demonstrate that methods are far more than procedures.
They are part of a research design that provides both the rationale and context
for their use. Therefore, detailed discussions of specific individual methods are
Preface xxv
not our focus in this suite of chapters. Rather, we concentrate on the thinking
you need to do with and about methods when designing research. This is the
thinking that underpins the choices you make about which methods to use, and
why you decided to use them. Consequently, individual methods are discussed as
exemplars of this thinking rather than as methods per se.
Chapters 6 and 7 focus on qualitative approaches to research and what
you will need to think about when using them as part of your research design.
Chapters 8 and 9 look at what you will need to think about and why when using
quantitative approaches to research. In Chapter 10, we build on our discussions
in Chapters 6 through 9 and look at what you will need to think about when
designing research using mixed methods—an approach that combines more than
one method in some way in your research design.
We have used all of the above approaches in our own empirical work. In
Chapters 6 through 10 we draw on this experience to share insights that we have
learned, sometimes the hard way, about thinking about, and then putting these
approaches into practice.
• Chapter 11 rounds off the book by returning in many ways to where we began—
thinking about and exposing the assumptions that all of us are making when
we design our research. Focusing on the idea of declaring your hand about the
thinking that underpins your research design, the chapter pulls together much
of what we have discussed in earlier chapters. It uses many examples when doing
so. Therefore, rather than a sign-off statement or reiteration of what has been
covered, the discussion is more of an inspirational one designed to encourage
you to pursue your interest in research design, keep thinking reflexively about
designing research, and continue asking questions about, and of, any research
design—including the one that you might be developing.
Coda: Coming clean—Why we were a bit annoyed when we began writing
In the interests of being completely honest about why we wrote this book, and in the way
that we have, we think that we should “come clean” and declare that this was also because
we were a bit annoyed. We were annoyed that research design is often viewed as a subject
that is dry, boring, tedious, and to be endured. Further, being asked to teach a course in
research design is often considered as drawing the short straw when teaching allocations
are made. We don’t agree!
Therefore, a large part of the impetus for writing this book is our belief as educators about
research, and active practitioners of what we teach, that research design and research can be
taught and discussed in ways that are relevant, interesting, and exciting. It need not, and
should not be, impenetrable, boring, tedious, or dry. Neither should textbooks about it.
We hope that our book length conversation about research design will excite, encourage, maybe at times even provoke you, but never bore you, as you think through, and with,
the material that we present.
Instructor Resources
An instructor resource site at http://edge.sagepub.com/cheek1e is available to support
this book. It includes editable PowerPoint slides, and suggested essay questions to accompany the book.
ACKNOWLEDGMENTS
T
his book would not have been possible without the support of our respective families
who encouraged us, put up with us working long days and over weekends on the
book, and who never once doubted that we would complete it—even if we sometimes
did!
Nor would it have been possible without the support of friends who listened to our
struggles, encouraged us over cups of coffee in person and over Skype while reminding us
that we could do this.
We also would like to acknowledge the support for, and interest in, this project from
colleagues and managers at Østfold University College where both of us were working
when most of the book was written. This support was an important part of making the
book possible.
The expert and honest feedback of Helen Salmon, our acquisitions editor, has been
a major factor in assisting us to write the book we wanted to and in a way that speaks to
the range of readers we wanted it to. We also appreciated the assistance given to us along
the way by various people at SAGE —in particular that given by Audra Bacon and Olivia
Weber-Stenis as the book neared completion.
Finally, the book would not be what it is without the lively discussions and feedback
from all the students, fellow researchers, mentors, and educators that we have had the
privilege to work with, and learn from, over many decades. They have taught us much
about what really matters when thinking about research design and it is from their questions about, and at times struggles with, designing research that the idea for this book
emerged. Thank you all.
SAGE and the authors are also grateful for feedback from the following reviewers during the development of this text:
Natalie D. Baker, Sam Houston State University
Hugh G. Clark, Florida Gulf Coast University
David R. Dunaetz, Azusa Pacific University
Sarah Fineran, Des Moines Area Community College
Jaimee L. Hartenstein, University of Central Missouri
Lauren Hays, University of Central Missouri
Scott Liebertz, University of South Alabama
Sherill Morris-Francis, Mississippi Valley State University
Howard J. Moskowitz, Capella University
Sarah Raskin, Virginia Commonwealth University
Isla A. Schuchs Carr, Texas A&M University–Corpus Christi
Kyle J.A. Small, Anderson University
Stephen E. Sussman, Barry University
Anne Whitesell, Ohio Northern University
ABOUT THE AUTHORS
Julianne Cheek is a professor at Østfold University College, Norway. Her publications
reflect her ongoing interest in qualitative inquiry and the politics of that inquiry. From
2010 to 2012, she had the honor of serving as the vice president of the International
Association of Qualitative Inquiry and currently serves on the External Advisory Board
of the International Congress of Qualitative Inquiry, held annually at the University of
Illinois. She has had the privilege of serving in the capacity of editor in chief of the wellrespected journal Qualitative Health Research, as well as editorial board member and international advisory member of a number of core journals and significant Handbooks related
to qualitative inquiry. She has a long-term interest in the teaching and development of
courses in research design and methods at master’s, doctoral, and postdoctoral levels.
Julianne has formal qualifications in teaching and education and began her working life
teaching science in secondary schools. Her PhD study, which was in social sciences, used
qualitative research approaches. Her first academic post was teaching research methods
and psycho-social science to nursing students. Since then she has taught research design
and methods to students in a number of health-related areas including physiotherapy,
nutrition, medicine and sport sciences, as well as to students in leadership and organizational studies. Throughout her career she has held senior administrative and development
research related university posts, including Dean of Graduate Studies, Director of Early
Career Development, and Dean Research.
Elise Øby is an associate professor at Kristiania University College, Norway. She holds
a PhD in mathematics and a master’s degree in organization and leadership. Her master’s
study employed qualitative research. She has held academic positions in multiple universities and colleges in Norway. She has also worked in public administration and held a
management position at a university college, part of which was to manage the allocation of
resources for teaching and research. She started her academic career teaching mathematics and statistics across a variety of study programs including teacher education, finance,
business administration, and engineering. Teaching mathematics and statistics in applied
areas such as finance and business administration sparked her interest in the decisions
involved when using mathematics and statistics as tools to learn something about complex
problems. It also highlighted the importance of ensuring that researchers and students
recognize that designing research that includes the use of such tools involves much more
than simply choosing a statistical procedure for analyzing empirical data. The thinking
that sits behind the choice of such tools in research projects was missing in many discussions about research design. Her current teaching includes courses in research design,
quantitative methods, and qualitative methods. She has an interest in enabling students
and researchers to design research in such a way that the findings from that research are
credible, reliable, trustworthy, and defensible.
xxx Research Design
NOTE
1. From Stake (2010, p. 78).
1
RESEARCH DESIGN
What You Need to Think About and Why
PURPOSES AND GOALS OF THE CHAPTER
The purpose of this chapter is to introduce the idea of research design: what it is, and what
you will need to think about when developing your research design. The focus is on the
thinking that affects the choices and decisions you make when designing your research.
The chapter provides the overall conceptual framework for the book and introduces
core areas that you will need to think about when you are designing your research. This
includes considerations about literature, methods, methodology, theory, and ethics: what
they are and what effect they have on the shape that your research design takes.
We highlight how designing your research requires you to join existing conversations
in relevant research literature related to the various areas of that design. Areas such as
methodological, theoretical, and ethical considerations. We explore how the way that we
navigate those conversations, what parts of them we join, and what parts of them we ignore
affects the way we think when we make decisions about our developing research design.
Throughout the chapter we emphasize the part that reflexivity plays in the thinking
about, and development of, any type of research design. When doing so we highlight how
thinking reflexively forces us to constantly think through all decisions about that design
as it develops.
The goals of the chapter are to
• Establish what research design is.
• Introduce the idea of research design as an iterative, nonlinear process.
• Identify foundational decisions and considerations that make up a research
design.
• Consider how theoretical, methodological, and ethical decisions shape any
research design.
• Illustrate that research design is much more than simply selecting methods or
techniques that will be used to collect data.
• Highlight the importance of linking the purposes of the planned research to how
that research will be designed and conducted.
• Demonstrate the use of relevant research literature to assist in the development of
the research design.
1
2 Research Design
• Provide information about how to make decisions about the relative merit of
using different types of literature when designing research.
• Emphasize that designing research requires reflexivity on the part of the
researcher.
• Present, and explain, the conceptual framework for the book.
INTRODUCTION: WHAT IS RESEARCH DESIGN?
Put simply, research design refers to the process by which a research idea is developed into
a research project or plan that can then be carried out by a researcher or research team.
It results in “a logical plan for getting from here to there, where here may be defined as the
initial set of questions to be answered, and there is some set of conclusions (answers) about
these questions” (Yin, 2009, p. 26).
Research design is not simply about research methods or procedures. While methods
and procedures are one of the areas you will need to think about when designing research,
there are many other areas that make up that design. These areas will need to be thought
through as well. This includes theoretical, methodological, and ethical considerations,
each of which is discussed in later parts of the chapter. When discussing these areas we
highlight that any thinking that we do about any of them (e.g., ethics) will affect other
areas of your research design (e.g., the methods you choose to use and how you use them).
It is the thinking that we do, and the decisions that we make, about these areas that
shapes and makes up what we term a research design—our plan for getting from here to
there.
TIP
A DIAGRAM OF A SPECIFIC RESEARCH DESIGN ≠ RESEARCH DESIGN
It is important not to confuse the diagram of a research design with what research
design is. There is a difference between research design as a thinking-based process
and “a” specific research design usually represented in the form of a diagram. The
diagram of a specific research design is a summary or representation of the result of
that thinking. Any diagram of a research design cannot be understood apart from the
thinking that gave rise to it in the first place.
DESIGNING RESEARCH IS AN ITERATIVE PROCESS
Designing research is an iterative process. Put simply, an iterative process is “doing something again and again, usually to improve it.”1 It involves cycles of thinking where you
begin with an idea, think it through, and then revisit the initial idea that you had, refine or
change it in line with that thinking, and then think that change through and so on. This
continues until you have landed on a research design that you believe will be able to get
you from here (about to start your research) to there (completing that research in a credible, systematic, and well-thought-through way).
Chapter 1
•
Research Design
3
Designing research iteratively involves cycles of visiting and revisiting, examining and
reexamining, modifying and then modifying again, each area of your emerging research
design. It is about thinking carefully about what we are proposing to do, and why. It will
require us to think backward and forward through the various areas of the research design
process as our thinking refines or modifies decisions and ideas we first had.
For example, if we rethink and change in some way the methods we are proposing
to use, then we will need to revisit the ethics related thinking we have done to see what
changes we might have to make to that thinking in light of the methods related changes
we have made.
Iterative is not an easy concept to define concisely or precisely. Nor is it an easy concept to put into practice when designing your research. You might find it helpful to think
of iterative research design as an active and “constant, continuous process of making and
unmaking” what will eventually emerge as your research design (Jackson & Mazzei, 2012,
p. 1). When making and unmaking your research design, you will continually ask yourself questions about the decisions you have made about the emerging design in order to
modify or confirm those decisions.
The goal of asking these questions is to improve and refine the emerging research
design. “Asking good questions is fundamental to the heart of research, critical thinking, creative thinking, and problem solving” (Swaminathan & Mulvihill, 2017, p. 1) and
occurs throughout the entire “lifecycle of the research process” (Swaminathan & Mulvihill,
2017, p. 2). In the box below we provide an example of putting this type of questioning and
iterative thinking into practice.
PUTTING IT INTO PRACTICE
EXAMPLES OF PUTTING ITERATIVE THINKING INTO
PRACTICE WHEN DESIGNING RESEARCH
You decide that you will study the process of older people moving into nursing homes.
You begin to think about this idea some more and realize that you will need to think
about, and then make, quite a few more decisions, in order to be able to design your
research study.
For example, what exactly do you want to know about that process of moving into
nursing homes? Costs (e.g., of providing care for these older people or costs they
incur when moving)? Or the characteristics of the older people making that move (e.g.,
age, gender, ethnicity)? Or the effect of the move on the older person or their families?
These are just a few of the foci your study might take depending on what you decide it
is that you want to find out something about—your research questions.
After thinking this through, you decide that you want to know more about how older
people themselves experience the process of moving into a nursing home to live. This
decision means that you will return to modify your initial decision that your study was
about the process of older people moving into nursing homes, and adjust it to reflect
what it actually is about that process that you have decided you are interested in,
namely, how older people themselves experience that process.
You then think more about this focus of how older people themselves experience
the process. This leads you to decide that you need to think some more about what
4 Research Design
older people you are interested in knowing more about and why. You decide that
the group of older people you are interested in are those who move from hospital
to nursing homes. This is because you are interested in an unplanned move as the
result of some form of acute health crisis. This leads you to revise your research
focus to how older people who move from hospital to nursing homes experience
that focus.
In this iterative process, there are cycles of visiting and revisiting, examining and
reexamining, modifying and then modifying again your thinking and decisions about
the focus of your study.
Diagrammatically we can represent this process as in Figure 1.1 below.
FIGURE 1.1 CYCLIC THINKING THROUGHOUT THE PROCESS OF DESIGNING
RESEARCH
Study the process
of older people
moving into
nursing homes
Study how older people
themselves experience
the process of
moving into nursing
homes to live
Study how older people
moving from hospital to
nursing homes experience
the process of moving
into nursing homes
to live
What exactly is it about
What older people are
this process that you
Making a
Making a
you interested in?
want to know about?
decision
decision
You will continue this
cyclic thinking until you
have landed on what the
focus is that you want to
take in your research
You will then continue this process of cyclic thinking throughout the entire process
of designing your research as you think through each of the decisions you make, and
their effects on the way that your research will need to be designed.
Research Design as a Messy, Complex, and Demanding
Thought-Driven Process
Our discussion of designing research as an iterative process has highlighted that research
design is not about linear, discrete, step-by-step thinking. It is a much messier, complex,
and demanding process than that. There are a series of interrelated decisions needing
to be made. These decisions enable us to turn our research idea(s) into a well-thoughtthrough, and therefore designed, research study. In order to make thought-through decisions rather than thought-less ones, we need to think carefully about what we are doing
and why at all points of undertaking our research. It is this thinking, and the iterative cascades of decisions resulting from that thinking, that research design, and designing research,
is all about.
TIP
We take a more extended look at an example of the messy process of iteratively
“making and unmaking” (Jackson & Mazzei, 2012, p. 1) what will eventually emerge
as our research design in Chapter 3 where Maxi Miciak and Christine Daum discuss
the way their research questions developed iteratively when they were designing their
research, and why they developed in the way that they did.
Chapter 1
•
Research Design
5
RESEARCH DESIGN: WORKING WITH THE LITERATURE
A research design is not developed in isolation. When we begin thinking and writing
about any aspect of a research design, we become part of a series of long conversations
(Hesse-Biber & Leavy, 2006; Locke et al., 2014) others have had before us, and will continue to have after us, about designing research. For example, throughout the process of
developing your research design you will need to be aware of, and take into account, what
is already known substantively about the problem that your research is being designed to
address. Similarly, when thinking about how you might do your research you will need to
be aware of, and take into account, the methodological conversations about how research
might be done as well as how the way that you are proposing to do your research relates to
those conversations.
In other words, you will need to join conversations in the body of knowledge that
has been built up by the work of others, and which is relevant to the various areas that
make up your research design. These conversations have been going on for many years
among researchers. In these conversations, some voices may be louder than others, and
some voices might be silenced and/or lost. There is not always agreement about the various
areas of research design being discussed. This means that you will need to know enough
about these conversations to make a decision, and justify that decision, about which parts of
them you will use in your research design and which you will not.
Where will you find these conversations? Most of them you will find in what is referred
to as “the literature” related to the various areas of your research design.
Using Relevant Literature When Designing Research
To assist you in developing your research design you will draw on, and interact with, relevant literature throughout the entire process of designing your research. Relevant literature
refers to theoretical writing and reports of empirical work “that have important implications for the design, conduct, or interpretation of the study, not simply those that deal with
the topic, or in the defined field or substantive area, of the research” (Maxwell, 2006,
p. 28). Reading, thinking about, and interacting with relevant literature enables us to
join ongoing conversations between scholars and researchers about the different aspects
of designing research that we will need to think about when designing our own research
studies.
When designing research, it is important to become aware of these conversations in
the relevant literature, think about the ways the conversations have been had, and then
decide and declare the position you will take in relation to those conversations. Taking a
position involves considering what parts of the conversations you agree with, what parts
you do not, what parts you will use, what parts you will not, as well as what parts of that
conversation your work and thinking might add to. It also involves justifying the choices
that you make. This will require you to add an “and why?” to each of these considerations.
For example, we might be reading literature about empirical2 work in an area related
to our initial thinking about, and framing of, what we think our research problem is.
However, after reading that empirical work, we may realize that there are aspects of the
problem that we need to think about differently, or read more about. This might involve
reading other empirical studies that take a slightly different focus. Or, it might involve
reading about different, or additional, theoretical concepts that can help reframe the
6 Research Design
problem. Reading about new and different ways of thinking about the problem our
research is being designed to address forces us to revisit our thinking about that research
problem. This may lead us to modify our initial thinking about what the problem is in
some way.
Such an iterative process of interacting and thinking with literature occurs throughout the entire research design process, not just when we are thinking about our research
problem. All parts of our research design require us to join a range of analytic conversations—about substantive issues, about other empirical work, about theoretical matters,
about methodological and method related matters, and about the various interconnections between these different conversations. Our research design is the result of which
conversations we have decided to join and those we have not, what decisions we have
made as a result of joining them or not joining them, and what conversations we want our
research to be part of when it is completed.
PUTTING IT INTO PRACTICE
USING RELEVANT LITERATURE THROUGHOUT THE
RESEARCH DESIGN PROCESS
Different types of literature (in terms of its focus) will be used for different purposes
at different parts of the process of developing your research design. For example,
when you are thinking about your research problem or area, you will work with literature relevant to your substantive problem area to find out what others have and
have not done and how this might affect what you choose to focus on (or not focus
on) in that area.
You will also work with literature that theoretically is relevant to your research
problem or area. For example, if you are interested in the theoretical idea of moral
distress, you will read literature related to that theoretical concept and how it is, and
might be, defined. You will then be able to use the result of your thinking about what
that literature is about to inform other parts of your research design. For example,
if you are going to measure moral distress in a group of workers using some sort of
quantitative survey, then what you will actually measure will be influenced by what
others have identified in the literature to be key aspects of moral distress.
There is also a lot of methodological and methods related literature that you will
need to think through, and which can help you when designing your research. No
matter what research method(s) you use, you will need to read, and think through,
literature related to those methods—ways they have been, and might be, used. You
will use this literature to inform what you need to think about when collecting and
analyzing data using your selected methods. You will also need to find out about the
strengths and weaknesses of those methods so that you are able to make it clear
what the way you have designed your research enables you to say (i.e., the type of
conclusions you can make using those methods) and equally importantly what it
does not.
And of course, from minute one of thinking about your research, you will need to
join conversations in the literature about research ethics both generally, and specifically related to the substantive focus of your research, and the way you put your methods of choice into practice.
Chapter 1
•
Research Design
7
Figure 1.2 captures this process of using different literature at different parts of
the process of developing your research design.
FIGURE 1.2 ■ The Use of Different Literature at Different Parts of
the Research Design Process
Some sort of problem/hunch
Literature
Substantive literature
Research area
Literature on relevant
theory/knowledge and more
refined substantive literature
Research problem/questions
Methodology and methodsfocused literature
Research methodology
Ethical
considerations
Associated research methods to
collect data to answer question(s)
Analysis according to accepted
chosen methodology’s principles
Development of
findings/discussion/conclusion in
line with chosen methodology’s
principles
Specific methods literature
Literature/knowledge about
accepted conventions
of analysis
Literature to enable
theoretical generalizability
Working With the Literature Is Not the Same as Simply Reviewing It
Working with literature when designing research is part of the entire research design process. It is not limited to the production of some sort of one-off, static, review of selected
(the reasons for which are not always declared) literature. Instead, working with the literature is central to the iterative process that underpins the development of a research
design. It
enables the researcher to (a) understand the conversations already happening
within and across relevant fields; (b) figure out how to add to these conversations;
and (c) identify the best means of doing so theoretically and methodologically.
(Ravitch & Riggan, 2017, p. 32)
The importance of thinking with literature as a process, rather than as a one-off static
product often called “the” literature review (Ravitch & Riggan, 2017), is picked up on,
and explored in detail, throughout this book. For example, in Chapter 2, we join conversations in the literature about ethical considerations when designing research and what
8 Research Design
implications those conversations might have both for how we design our research and also
what we consider ethical matters to be. In Chapter 3, we focus on what we need to think
about when developing our research problem or questions and the part empirical, methodological, and theoretical literature plays in that development. In Chapters 4 and 5, we join
methodological related conversations in the literature that impact the way we think about
data and what type(s) of knowledge our research is being designed to contribute to those
conversations. In Chapters 6 through 10, we participate in, and make decisions about,
conversations related to the way that data can be collected at the level of specific methods.
PUTTING IT INTO PRACTICE
WORK WITH THE LITERATURE, DON’T JUST REVIEW IT
There are many textbooks written about “how to” review the literature. Often, they are
more about the techniques of finding and summarizing literature, and less about the
importance of thinking about that literature as part of an ongoing iterative research
design process. While it may be useful to read about ways of searching for, finding, and
summarizing journal articles or other relevant literature, it is important to remember that thinking about, and reviewing, the literature when designing research is not
simply producing some sort of descriptive overview of what seems to be relevant literature. Rather, working with relevant literature to help inform your thinking about
various aspects of your research design is an iterative process which will require constant reference to more, new, and different literature as the design, and the thinking
that underpins and shapes that design, develops and unfolds. You will have a series
of ongoing and different conversations with literature during an iterative research
design process.
As you engage in conversations with literature, we strongly suggest you keep track
of what you read as you read it. For example, when you read an article, make a note
about the key points from that article when you read it. At the same time, be careful
to note the name of the article, the author, the journal, and the date of publication.
Backtracking to figure out where you found something that you later want to check is
almost impossible as the amount of literature with which you have your conversations
increases rapidly as your design develops.
You will need to know these details so that you can cite the author and article from
which you obtained your ideas, or from which you used some text. A citation is a reference to somebody else’s work to acknowledge that the idea was not originally yours
or to show that the idea that you are coming up with builds on that person’s work in
the first place. This is an important part of responsible and ethical research design.
Plagiarism is when we use other people’s work without fully acknowledging that the
idea or the words came from that person(s) in the first place. In effect, we are taking
those ideas and representing them as our own.
HOW DO YOU MAKE DECISIONS ABOUT WHICH
LITERATURE TO TRUST OR RELY ON?
Research related literature is usually categorized in terms of where it has been published
and what review process it has gone through. It is important to be aware of these categorizations as not all types of literature are afforded equal weight in terms of their scientific
standing and trustworthiness. This difference in scientific standing and trustworthiness
Chapter 1
•
Research Design
9
can affect perceptions about the trustworthiness of your research design if doubts are
raised about the credibility of the source of the literature that you are using to base aspects
of that design on.
Journal Articles
Reports of empirical or theoretical research are usually found in peer reviewed journals
and are afforded high status and trustworthiness by most researchers and scholars. This
is largely because of the expertise of the editorial board of those journals and the process of peer review that the journal undertakes. In this process of peer review, authors
submit their manuscripts, reporting their research, to the journal editor to be reviewed
and considered for publication. The manuscript (mostly with the author’s names and
affiliations removed, which is called “blind” peer review) is then sent by the editor to
at least two peer reviewers who are experts in the area of the research being reported.
These expert reviewers then read, and make scientific judgments about, the quality of
the manuscript.
Such “blind” peer review is designed to make sure, as far as possible, that the focus of
the review is influenced by what is said in the manuscript about the research design, the
findings, and their significance. Publications in journals that employ this type of review
are considered a reliable and credible form of scientific- and research-related literature
because of this form of rigorous review by peers in the field. This type of literature is often
called “scientific” literature.
There are two major publication models for scientific journals. One is the traditional
subscription model where the author does not pay fees to the publisher of a journal to cover
the costs of the peer review process and, if accepted, publishing the article. Instead, these
costs are recovered by the publisher of the journal by charging readers a fee for accessing
the articles in the journal. This fee can be in the form of annual subscriptions to the journal, or it can be in the form of paying a fee to access individual full text articles over a set
period of time such as 24 hours, after which time access is lost.
The other model is what is known as Open Access publishing. The article still undergoes rigorous peer review and rejection rates for articles submitted to many Open Access
journals are in line with those of traditional subscription model journals. However, in
Open Access journals the costs of reviewing and publishing the article are paid by the
author if, and when, the article is accepted. There are no costs for individuals wanting
to read that article—hence the description of this model as Open Access. Access is open
to everyone as there is no payment involved. Hence the reach and access of an article in a
reputable Open Access journal may be greater than in a journal where not all readers have
access to that article because, for example, institutions do not subscribe to that journal so
staff and students will need to pay to read those articles.
There are different types of Open Access possible. These differences are related to
the degree of, and how, access is given to the article. The one we have described above is
known as Gold open access. However, there is also what is called Green open access where
although there is not completely open access to the article such as on the journal’s website,
authors are able to post on their personal or institutional website a version of the article
able to be accessed by readers. There is also an in-between model where journals that use
a subscription-based publishing model will make an individual article openly available to
everyone if the authors pay a fee to enable this when the article is published (see Richtig et
al., 2018 for a good discussion of this).
10 Research Design
Increasingly funding bodies are requiring those gaining funding for their research to
enable some form of open access to any articles reporting on research from that funded
project so that anyone can read that article, anywhere and without any time limit. This
is because the goal of the funding is to enable the development and dissemination of the
knowledge gained from the funded research as widely as possible and not depend on a
reader having the resources to be able to pay for that access.
Books and Book Chapters
Book chapters and books that are published by what are often described as “good quality”
or “reputable” national and international publishing houses are also given credibility and
standing in terms of the hierarchy of research and scientific literature.
Defining just exactly what a quality or reputable national or international publisher of
scientific books and book chapters is, is not clear cut. There is no standard way to identify
such publishers. However, to help you make decisions about this you will find that most
universities and research institutions have developed their own lists (formal or informal)
of whom they consider to be reputable publishers. In some countries, government bodies,
drawing on input from researchers, have developed lists of publishers that publish books
and book chapters that are recognized as credible and of good quality.3
Publishers deemed reputable have in place similar processes to the peer reviewed journal process. Draft chapters or draft books will be sent out for review by peers and published only if favorable reviews are received or revisions to the chapter or book have been
made in line with reviewers’ recommendations. These publishers will cover basic costs
associated with the production of the book or chapter, and not charge the author fees
unless it is in relation to some sort of recognized Open Access publishing model. Other
factors that can indicate that a publisher is reputable include who the authors are whose
work is published by them and how the publisher distributes their books. Guidelines for
publication that the publisher provides to authors can also provide a guide to the credibility of the submission and review processes of that publisher—what are they, how detailed
are they, and are they used in practice?
TIP
BE AWARE OF WHAT ARE KNOWN AS PREDATORY JOURNALS AND
PREDATORY PUBLISHERS
These are journals and publishers that mimic the Open Access model. The goal of
these journals is to take the author’s money for their own profit, rather than to ensure
that any fees charged are used to make the knowledge gained from the author’s work
available as widely as possible and not dependent on a reader having the resources to
be able to pay for that access.
Consequently, predatory journals will “sell” submitting a paper to the journal by
promising authors a very short review time (often a few days) and advertising very high
acceptance rates. They will also charge considerable fees at the time of submission.
Such fees are usually nonrefundable either fully or in part, even if the paper is not
accepted.
Characterized by the use of widespread spamming, predatory publishers obtain
lists of research groups or publications by researcher and then contact potential
authors to ask them to submit their work to them—even if the author might not be
working or publishing in the area of the journal’s focus.
Chapter 1
•
Research Design
11
The result is that many or most papers published by many of these journals are
often of poor quality and do not meet the standards set by reputable Open Access journals with expert editors, editorial boards, and reviewers enabling credible, thorough,
and transparent peer review processes.
You need to be aware of the existence of predatory journals so that you can make
decisions about the credibility of an article you are reading in terms of where it is published and the editorial and review processes that article has been through. Can you
trust what it being reported in the article?
You also need to be alert when you are thinking about where you might publish your
own research. How can you make sure that the journal you are thinking of submitting
your work to is not a predatory one? One excellent resource to help you do this is the
guide for what to look for when deciding if a journal is a legitimate one or a predatory one provided by Victoria Glasson (2017) in her post “6 Ways to Spot a Predatory
Journal.” She gives the following advice for spotting predatory journals:
1. Always check the journal website thoroughly.
2. Check what professional publishing and/or editors’ organizations or bodies the
journal is a member of.
3. Check the journal’s contact information.
4. Research the editorial board.
5. Check if the journal has a peer review process and publication timelines.
6. Read through past issues of the journal. (See Glasson, 2017.)
If you would like to read more about any of the six pieces of advice above, we
encourage you to go on the site and take a look at the additional advice Glasson offers
in drop-down text attached to these points. 4
You will also find useful resources on most major mainstream and reputable publishers’ web pages about what to look out for, and think through, in relation to predatory journals. For example, on SAGE’s website, you will find Natalie Gerson’s (2019)
very useful post “How to Protect Yourself From Predatory Publishers and Other Open
Access FAQs.” Advice is also provided to authors on the Taylor and Francis website
about making decisions about whether Open Access journals are of good quality.5
Another useful resource is the Journal of Human Lactation editorial statement
and policy on the use of references from predatory publishers in articles submitted
to that journal (JHL Editorial Team, 2020). In addition, another article in that journal,
“Understanding Quality in Research: Avoiding Predatory Journals” by Strong (2019), is
very helpful as well. Finally, the website https://thinkchecksubmit.org/ provides useful tools and resources to make sure you are submitting your research to a journal or
publisher that can be trusted.
Other Types of Literature That Might Be Useful if Used With Care
There are also some forms of literature that have not undergone such a formal process or
blinded peer review but which still can prove very useful in terms of providing ideas and context for aspects of a research design being developed. This does not necessarily mean that these
non-peer-reviewed articles or chapters are not trustworthy or not able to be used by researchers.
It does mean, though, that they have not undergone quite as rigorous review process as peer
reviewed books, book chapters, and journal articles. Examples include articles in non-peerreviewed journals that may be in more profession- or practice-based journals, and books and
collections of chapters self-published in-house by a researcher or group of researchers.
Another type of literature that may be useful when designing research are reports of
some sort. These can be, for example, government or technical reports, policy or reform
12 Research Design
documents, or government regulations.6 This type of literature is useful in terms of providing contextual material for the study. At times parts of this literature even form part of
the actual texts that the research is being designed to analyze, for example, if the research
draws on some form of document or textual analysis as its theoretical and methodological
inspiration.
Questions arise about the trustworthiness of using online information, such as
Wikipedia and blogs, general encyclopedias, reports in newspapers and popular scientific
books, as part of the literature in a study. While there is no absolute or straightforward
answer to how to make decisions about the trustworthiness of this information, it is often
the case that the further away whatever is being reported or discussed is from data or findings of actual research studies, the less scientific the source can be considered. That said,
we agree with Stake (2010) who noted over a decade ago that “Wikipedia is a valuable
resource, in spite of the potential mischief of open editing. Wikipedia information begs to
be checked, doubted, presented with caution” (p. 116).
With respect to using newspaper articles or other forms of journalistic reporting, it is
important to see if the article or report tells us what the information and conclusions are
based on. Some articles in some newspapers do this. However, often we get sensational
headlines such as “Being rich and successful really IS in your DNA: Being dealt the right
genes determines whether you get on in life.”7 Yet, when we read that article there is very
little there to trust, or convince us to trust that article. This is because there is very little
reporting of any details of the research on which such claims are based.
RESEARCH DESIGN: CONSIDERING
METHODOLOGY AND METHODS
Methodological related thinking shapes the form that the research design takes. Put simply, methodology refers to “the strategy, plan of action, process or design lying behind the
choice and use of particular methods and linking the choice and use of methods to the
desired outcomes” (Crotty, 1998, p. 3). Methodological considerations force us to think
about if, and if so how, a particular method gives us the type of data to generate the type
of knowledge that we need to address our research question(s). Therefore, “when we are
examining methods, comparing them or thinking about the kinds of knowledge that they
produce, then we are doing methodology” (Greener, 2011, p. 5).
Methodological related questions we might ask ourselves when thinking about the
design of our research include the following: What type(s) of knowledge or data will I
need to address the research question(s) that the research is being designed to answer? Will
the use of a particular method contribute this type of data—why or why not? In this way,
we use methodological thinking to provide a rationale “for the choice of methods and the
particular forms in which the methods are employed” (Crotty, 1998, p. 7) in our research
design.
When designing your research, you will need to read widely in order to join existing
conversations about both methodology itself and the assumptions different methodologies make about research and how to do that research and the effect these assumptions
have on the way that research is thought about and designed.
For example, in what are termed “qualitative” approaches to research,8 the research
will be designed in such a way as to enable the emergence of rich and qualitative
Chapter 1
•
Research Design
13
interpretations of the perceptions or experiences of people about a specific aspect(s) of
the everyday context(s) in which they exist. This type of approach is often referred to as
naturalistic or interpretive inquiry. It aims for in-depth understandings of peoples’ perceptions and experiences of whatever is the focus of the study.9 Although procedures for
the research may be identified beforehand, qualitative research designs are characterized
by “built-in flexibility, to account for new and unexpected empirical materials and growing sophistication” (Denzin & Lincoln, 2005b, p. 376).
On the other hand, in what are often termed “quantitative” approaches to research,10
one can assume that the researcher has a commitment to, and has identified the need for,
some form of quantification of specific characteristics of a group of people or other objects
of interest. This is because “[q]uantitative research works with statistics or numbers that
allow researchers to quantify the world” (Stockemer, 2019, p. 8). To employ this sort of
approach to inquiry you will need to follow set mathematical and statistically based procedures. This is so that appropriate forms of numerical data are produced that are able to be
used to make valid probabilistic and statistically based interpretations about those specific
characteristics.11 This type of research design12 requires and “places a premium on the
. . . specification of the research strategies and methods of analysis that will be employed”
(Denzin & Lincoln, 2005b, p. 376).
Which approach you choose as part of your research design will be the result of your
thinking about what type of knowledge you want your research to produce and how you
can obtain that knowledge. Once you have decided on your methodological approach,
you will then need to make decisions about what methods you will use as part of your
research design.
Methods
Research methods refer to “the techniques or procedures used to gather and analyze data
related to some research question or hypothesis” (Crotty, 1998, p. 3, italics added). Each
research related method has specific procedures and techniques associated with it that
are designed to obtain a particular type of knowledge or information—usually referred to
as data.13 The methods chosen must be consistent with the type of knowledge or data we
want to obtain from our research (our methodological considerations), which in turn must
be consistent with the nature of the research problem that is shaping the entire research
design.
For example, if we want to know about individual people’s experience of losing their
jobs, we will need to choose methods that enable us to get in-depth information about that
experience from the point of view of each individual participant—possibly using some
form of individual interview. If, however, we want to know how many people from different segments of a defined population group experience anger, sadness, or any of the other
possible characteristics that we have identified as being of interest related to the experience
of losing one’s job, then we will need to use different methods in our research. We will
need methods that allow us to generate the type of numerical data we will need to answer
questions related to how many people experience those characteristics of interest.
This is why research methods are not, and should not be thought as, stand-alone
techniques simply able to be selected and inserted into a research design. Decisions about
methods are part of the overall research design process. Their choice must be closely
related to the purpose of the research and consistent with all other parts of the design.
14 Research Design
While designing research does involve thinking about how to use or employ particular
methods, this occurs after having thought about why choose those methods in the first
place—a methodological consideration. Methodological related thinking and decisions “guide a researcher in choosing methods and shape the use of the methods chosen”
(Crotty, 1998, p. 3).
Methodological thinking related to designing research is thus at a higher level of focus
than is thinking about how to do specific methods. For example, in a research design
where interviews of some sort are the method of choice, methodological related decisions
will be about why interviews themselves are an appropriate method or way to collect the
data in our study in terms of the type of knowledge that we will need to address our
research questions. Is the choice of interviews, and a particular type of interview, as the
method we will use for data collection consistent with the aims and desired outcomes of
the research?
You may see the terms methodology and method being used interchangeably in some of
the writing about research design, and when research is reported. Or you may see methods
as a required or standard heading in research reports and methodology missing altogether.
This can lead to thinking of methods as stand-alone techniques, rather than methods as
arising from, and unable to be understood apart from, methodological ways of thinking. Neither methods nor the data produced by them can be understood apart from the
methodological considerations and choices in the overall research design in which they are
embedded. If removed from these understandings, methods are reduced to being merely
techniques—sets of procedural rules to follow.
We develop this introductory discussion of methodology further in Chapters 4
and 5 of this book. In addition, all chapters in the book at some point put the spotlight on the effect of methodological related thinking on the aspect of research design
being scrutinized in that chapter, such as ethical implications arising from a particular
methodology and the methods associated with it (Chapter 2), the way research questions are formulated (Chapter 3), and the way methods are put into practice (Chapters
6, 7, 8, 9, 10).
RESEARCH DESIGN: CONSIDERING THEORY
Theoretical considerations and choices, like methodologically focused considerations and
choices, provide orienting ideas that shape the form a research design takes. Assumptions
about what theory is and is for, as well as ideas or concepts from specific theories, provide
orienting ideas that influence all aspects of the research design. The theoretical framework
of your research is “[t]he system of concepts, assumptions, expectations, beliefs and theories
that supports and informs your research” (Maxwell, 2013, p. 39). This includes the questions that are asked, the way data is collected, the analysis of that data, the interpretation of
that analysis, and consequently the conclusions that can be drawn from the research.
Despite its wide use both in everyday and more academic contexts, it is difficult
to come up with a precise definition of theory. Theory tends to be one of those words
that everyone uses, but struggles to give a precise meaning to. This is because the long
conversations about theory that researchers, philosophers, and others have been having
for many hundreds of years have not resulted in agreement among them about “the” or
Chapter 1
•
Research Design
15
“right” definition of theory. Thus, theory remains a term used, and understood, differently
by researchers depending on how they position their thinking, and subsequently their
research design, in relation to that conversation.
In the physical and natural sciences, theory traditionally has been viewed as “a set
of abstract (ideally mathematical) propositions, some of which take the form of ‘laws,’
that predict a range of specific events or results” (Maxwell & Mittapalli, 2008, p. 877).
Adopting this view, or understanding, of theory affects all aspects of the way designing
research is thought about, including methodological ones.
If this view of theory is adopted, then
• theory is understood as providing a set of propositions, or a priori concepts, that
have been standardized or stabilized in some way, in order for them to be able to
be tested to either support or disprove that theory.
• the goal of both the research, and the theory that the research is designed to
test in some way, is to enable generalization of what you learn from the data to
specific population groups of interest. Such generalization will involve the use of
mathematically derived fixed principles and procedures.
These orienting ideas emerging guide the researcher’s thinking when developing their
research design.
However, this is not a view or understanding of theory, or research, held by all
researchers. In much of the thinking in the social sciences, thinking about theory is not
restricted to theory as providing a set of propositions, or a priori concepts, able to be tested
to either support or disprove that theory. Instead, theory is thought of more as an enabler
for the research. The role of theory in this view is to provide what one of the foundational
thinkers in the area of social theory, Herbert Blumer (1954), called “sensitizing concepts”
(p. 7) for the research. A sensitizing concept is not prescriptive but, but merely “suggest
directions along which to look” (p. 7).
If this view of theory is adopted, then it follows that the role of theory in research
design is to
• provide initial orienting concepts for the framing of the study rather than
standardized or stabilized a priori concepts to be tested.
• contribute to, and build, our understandings of the way things work or are
understood and/or are experienced, in social contexts and settings.
• enable exploration and building of aspects of theoretical concepts.
Rather than focusing on only one theory, or one or several theoretical concepts and
testing that theory or those concepts, the research is designed to add layers and richness to
our understandings of that theory or those concepts themselves.
The following box uses an example of how different views of theory affect research
design using the example of studying motivation. It also demonstrates that what theoretical view the researchers adopted about motivation affected the methodology they used in
their study.
16 Research Design
PUTTING IT INTO PRACTICE
HOW DIFFERENT VIEWS OF THEORY AFFECT RESEARCH
DESIGN – THE EXAMPLE OF STUDYING MOTIVATION
To demonstrate how theoretical understandings, and the orienting concepts derived
from those theoretical understandings, affect the way that research is designed and
put into practice, we will compare two different ways that research was designed to
study the same broad substantive focus, namely motivation and school students. While
both studies are about this same broad substantive area, they differ with respect to
the orienting ideas that underpin the thinking in their respective research designs.
The different methodologies used in each study reflect the different ways that theory is thought about in each study. Study One (Martin, 2001, 2003) focuses on specific
theoretical concepts that have been identified in previous research as being related
to motivation in schools. The study is designed to test how those concepts relate to
motivation in schools and how they relate to each other (Martin, 2001, 2003). Study
Two (Friels, 2016) is designed to build theory by adding layers and richness to existing
theoretical understanding of both the idea of motivation itself and how motivation is
perceived and experienced and “works or does not work” in a specific school context.
Study One
The Student Motivation Scale developed by Martin (2001, 2002, 2003) is underpinned
by a wide range of theoretical contributions from theories about motivation. The scale
is an instrument for measuring motivation, designed by separating motivation into
factors reflecting enhanced motivation (“boosters”) and factors reflecting reduced
motivation (“guzzlers”). The Student Motivation Scale therefore is able to not only
focus on “the energy and drive of students,” but also on “their ability to deal with pressure and setback” (both quotes from Martin, 2002, p. 34).
Understanding what are the boosters and what are the guzzlers of each individual student, both in the student’s life and in the classroom, requires a measurement
instrument able to measure several aspects of motivation. The Student Motivation
scale is able to do just that. Moreover, because “motivation is students’ energy and
drive to learn, work hard, and achieve at school” (Martin, 2001, p. 1), such a multidimensional understanding of motivation has the potential to assist and aid educators
“operating in contexts in which students require assistance to sustain motivational
strengths and address areas of motivation that may be of some concern” (p. 20).
In other words, by measuring a student’s motivation using the Student Motivation
Scale, educators can acquire the knowledge they need to “keep high boosters high;
keep low guzzlers low; increase low boosters; and reduce high guzzlers” (Martin,
2002, p. 42) for that specific student.
In the 2003 study, Martin examined the Student Motivation Scale by collecting data
from 2,561 high school students. The data were analyzed to test the proposed categorization of factors into guzzlers and boosters. The replies were analyzed according to
statistical procedures with the intent to generalize beyond the sample of people surveyed in the study. The analysis showed that “the Student Motivation Scale is psychometrically sound and can be usefully implemented to determine groups of students at
risk of disengagement, disinterest, and underachievement” (Martin, 2003, p. 88)
Study Two
On the other hand, Friels’s study (2016) involved interviewing four African American
high school female students from low-income families using semistructured14
face-to-face interviews. The interviews were designed to “capture the stories of the
Chapter 1
•
Research Design
17
students as they share their experiences” (p. 8) related to motivational factors, and
their perceptions of the role various factors (such as the community they grew up
in, peers, and family) play in their academic success or failure at school. Qualitative
analysis of these interviews provided new insights into the students’ perceptions
of what it was that motivated them and why. In this way, the research built more
nuanced and contextual understandings of the idea of motivation itself that can
be used by educators and policy makers when developing targeted programs to
provide effective support to African American high school female students from
low-income families.
With the previous discussion in mind, useful questions to ask yourself when designing
your research include the following:
• What theoretical assumptions are you bringing with you to the table when you
are designing your research?
• How do these assumptions affect the way that you design your research?
• Does this matter?
Having this type of discussion with yourself when thinking and writing about your
research is a central, but often overlooked, part of designing research. Such thinking will
make you aware of why you made the decisions about your research design that you did—
including exposing any assumptions you may be making about theory, research, and science when doing so.
Exposing these assumptions will enable you to use theory well, and not be used, and
therefore constrained, by it. It will help you avoid becoming bogged down in questions or
assumptions about “the” “best” theory to use, or “the” “right way” to use theory.
Answers to these types of questions are study specific and related to what it is that you
want to know about and why you want to know about it. In other words, what type of
knowledge do you want your research to provide in order to contribute better understandings of the problem that your research is designed to address.
TIP
REMEMBER: THEORETICAL CONCEPTS ARE COMPLEX
Theoretical concepts and theories are complex. Therefore, you will need to know
enough about these theories to be able to think through this complexity in order to
make an informed decision about what understanding of a particular theory or concept
you will put into practice in your research design.
For example, if you are using a theoretical concept such as power in your research
design, you will need to move beyond common sense or assumed understandings of
what power is, or making simplistic statements such as “power is a guiding theoretical
concept for the study.” This statement disguises the complexity of the idea of power
which is a concept made up of a repertoire of diverse perspectives drawn from diverse
theoretical positions (see Hindess, 1996).15
18 Research Design
THE IMPORTANCE OF REFLEXIVE THINKING
WHEN DESIGNING RESEARCH
Thinking about, and developing, a research design iteratively is challenging, and at times
confronting. We are forced to ask questions of our developing research design and expose
and examine the assumptions we are making about that design. It is much easier to think
of research design as a predetermined type of recipe, or a step-by-step diagram, to follow.
This recipe or diagram can be selected from some sort of procedure manual or textbook
about “how to do” research and followed step-by-step without really having to do too
much thinking except how to do what the steps require.
However, simply following a series of predetermined steps does not allow us the thinking space to consider the assumptions about research, existing knowledge, theory, and
our role as researcher that we bring to both the design of our research and the way that we
subsequently go about putting that research design into practice. The steps, like research
designs, are not neutral, value and theory free. An instrumental focus and reducing thinking about research design to “how to do or follow the steps” ignores the assumptions about
research, knowledge, and our role as researcher that are embedded in those steps. There is
a lot of theory-in-use in the assumptions that any seemingly neutral or objective steps and
procedures in that research design make—even if this is not declared or even acknowledged. Thinking about these assumptions and the effect that they have on the cascade of
decisions that must be made when designing research requires reflexivity on the part of the
researcher.
What Does Reflexivity Mean?
Defining reflexivity is not an easy task because, as Lumsden (2019) points out, “[T]here
are numerous definitions and operationalizations of reflexivity” (p. 2).16 Put simply,
reflexivity is a type of folding or bending back (Finlay & Gough, 2003) on our own thinking to work out why we have come to think about something in the way that we do. What
assumptions do we make when we think in this way? Based on what?
When designing your research, such folding or bending back takes the form of “critical self-reflection of the ways in which researchers’ social background, assumptions, positioning and behavior impact on the research process” (Finlay & Gough, 2003, p. ix). It
helps you to understand “the significance of the knowledge, feelings, and values” (Attia &
Edge, 2017, p. 35) you bring with you when designing your research, and how this affects
all aspects of how you develop that design from the research questions you ask, to the
methods you use and analytical lenses you employ (Attia & Edge, 2017). For example, are
there methods that you believe to be “better” than others? What do you base this on? Can
you justify this assumption? How did you come to have such an assumption in the first
place? In this way, reflexivity challenges us to work out why we think what we do when
designing our research, and whether there are other possible ways of thinking about that
design.
Hence reflexivity is a form of thinking about research design that is dynamic—not
static or linear. It requires adding an “and why” to all the thinking we do about our design.
This type of and why thinking forces us to expose, examine, and challenge our thinking
and the choices that that thinking resulted in throughout the research design process.
This includes choices made before we begin designing the research; while we are designing the research; when we are putting that design into practice; and even after we have
Chapter 1
•
Research Design
19
completed the research design. For example, what we will report, or not report, about our
research.
Putting Reflexive Thinking Into Practice When Designing Research
Between us we have many years of experience both doing research ourselves and acting as
advisors for students’ research. We have noticed that the researchers or students who navigate the challenges of the process of designing research well are those who take the time to
reflexively think through, ask questions of, and then declare the decisions that they have
made related to the design of their studies. They are constantly asking themselves a series
of interrelated questions about that emergent design. Questions such as these: What type
of knowledge will I need to address the problem or questions I want to ask? What is an
appropriate way to obtain that knowledge? Appropriate in what sense? Methodologically?
Ethically? Feasibility wise ?
They are also the researchers or students who ask themselves questions about the effect
that the way they answer the above types of questions has on the overall research design.
Questions such as these for example:
• What happens to this part of the design if I make this decision and not another?
• How might this affect decisions that I have already made about other parts of my
research design?
• Why am I thinking about these questions and adopting a stance in relation to
them, in the way I am?
Asking these types of questions of yourself requires you to fold back on your own
thinking. In so doing, it enables you to reveal and understand your own research standpoints. It also enables you to recognize and think through the effects of those standpoints
on the choices you have made in that design process. This is because reflexive thinking
challenge[s] some of your ways of knowing. . . . You may need to unlearn . . . what
you bring to the learning and to see your knowledge and experiences as foundations on which you will continue building. (Skukauskaite et al., 2018, p. 340)
Put another way, reflexive thinking enables the design of our research to be understood not only in terms of what it is but how it became to be the way that it is, thereby
providing justification for the way that it is. Thus, “[r]eflexivity can be a way to examine the
complete research process and a vital procedure for enhancing validity” (Lahman, 2018,
p. 35) of all types of research—a point we return to many times in this book.
Activity
Understanding Our Own Research Standpoints: An Example of
Thinking Reflexively
In a reflexive piece of writing, Sharlene Hesse-Biber thinks through and then declares
how her background and assumptions, that is, her research stance, affected why and
how she wrote her book Mixed Methods Research: Merging Theory With Practice (2010b).
20 Research Design
We include some excerpts of this reflexive writing below, all of which are taken from
page 25 of her book.
I am a feminist qualitative researcher who has a particular perspective on social
reality. As a feminist, I am interested in asking a set of research questions that
often trouble the waters of traditional knowledge building by including issues of
difference in the research process.
I am interested in issues of power, authority, and control while conducting
research as well as asking such questions as: What is studied? From whose perspective? Who is being studied? Who is left out and needs to be included in this
study? I guess you might say that I am a methods interloper—an outsider and
an insider to mixed methods research.
As a sociologist who has had traditional training in quantitative methods and
the positivist paradigm, I am an insider in that I practice and teach both methods and have in fact conducted several mixed methods projects. As a feminist,
I am often the outsider who asks new questions, yet I will utilize a range of
tools—quantitative and qualitative—as needed to answer my questions. I am
not wedded to one specific method or set of methods. I use whatever methods will
facilitate getting answers to my research problem(s).
As a researcher, my agenda is one of promoting a comprehensive approach and
understanding of the use of methods techniques by placing the practice of methods more firmly within a research context.
I am cognizant of the importance of living within the contradictions and tensions of the research process. I enter into dialogue with this process. To dialogue
means confronting our assumptions, suspending judgment, and embracing difference. To dialogue also means to hone our listening skills, with a stance toward
understanding.
SHARLENE HESSE-BIBER
These excerpts provide an excellent example of reflexive thinking in action. In this
writing, Hesse-Biber is aware of how her background and interests might affect the
way she thinks about, writes about, and conducts her research.
Now think about your own research stance. How might your background and
assumptions affect how you think about designing research?
Ethics: Much More Reflexive Thinking Still to Do
Reflexive thinking will force us to consider another set of related considerations when
designing our research. These are considerations related to the ethical dimensions of our
research. Research ethics are concerned with moral behavior in research contexts (Wiles,
2013). Thinking through ethics at all points of the research design process is part of
responsible research (Kuntz, 2015). It is part of becoming a responsible methodologist
(Kuntz, 2015) and a researcher who thinks “about how to become a more responsible
author, scholar, individual, citizen“ (Koro-Ljungberg, 2016, p. 126).
Chapter 1
•
Research Design
21
Thinking about research ethics and how to put that ethical related thinking into practice impacts all aspects of our research design. This includes thinking about if the research
area or the problem related to that area that we are thinking of researching is something
that should be researched at all, through to what we report about how we did our research
and what our findings are. Lahman (2018) calls this “poking and prying with a purpose
into what is good, bad, right, or wrong in research” and in fact uses this as her working
definition of what research ethics is (p. 4).
In the next chapter, we devote the entire chapter to taking a closer look at how we
might put ethical principles into practice when designing our research. For now, the point
to hold on to is that we will need to constantly think through the ethical dimensions and
implications of decisions we make throughout the entire process of designing research.
Thinking with ethics must be a visible and central part of the iterative thinking–based
process from which a research design emerges. Ethical considerations sit in, and around,
all aspects of the process undertaken to develop a research design.
CONCLUSIONS
Designing research is about making decisions to transform a research idea into a research
plan. These decisions begin the moment that we begin to think about a topic that we
want to know more about. This topic is the substantive focus of our research. What specifically do we want to know about this topic and why? What contribution is the research
that we are designing intended to make to the development of knowledge in this substantive area?
All researchers come to their research (or for our purposes, their research design) with
“orienting ideas” (Miles et al., 2014, p. 19). Orienting ideas give a direction for the thinking that is done when designing research, as well as when putting that design into action.
What we decide about what it is that we want to know more about and why provides the
basis for the formation of the questions that our research is being designed to address.
Once we have developed those questions, we can then make decisions about how we will
obtain the type of knowledge needed to address them. This will involve making decisions about what research methodology, and which research methods associated with that
methodology, we will use in our study design.
The methodological approach we employ provides the logic and rationale for the methods
we choose to obtain the information or data that we need to answer our research questions. However, even when we have decided on those methods and how we will put them
into practice, we are not finished making decisions about our research design. We will
need to think about what we will do with the information we obtain from putting those
methods into practice. This will include making decisions about how we will analyze the
data or information produced by those methods, as well as how we will link our findings to the existing body of knowledge about the substantive area our research is being
designed to contribute to.
Thus, research design involves much more than simply selecting research methods or
techniques that can be used to collect data. While research methods are part of a research
design, they are not all of it. Rather, research design is a process. The decisions that we
have made about our research design at every point when getting from here to there must
be transparent—as must the reasons for why those decisions were made.
22 Research Design
These decisions and choices include (1) what will be studied, more specifically the research
problem and the questions that are asked about, and of, that problem; (2) the type of
knowledge that the research is designed to produce, in other words, methodological considerations; (3) the way that that type of knowledge is produced, or more specifically, the
methods used to collect and analyze information or data in the research; and (4) what the
research is being designed to be able to say something about or be used for. For example, will it
support or disprove a theory or proposition, or will it add nuanced or new information to
build theoretical understandings? Or will it be used to do both if we are using combinations of methods in our research design?
When designing research, our thinking about research design cannot be limited to
focusing on putting together some sort of linear plan comprised mainly of data collection procedures and techniques stripped of the assumptions and thinking that gave rise
to them in the first place. The thinking underlying the entire process of designing that
research remains invisible and undeclared. Research design then becomes reduced to a
diagrammatic representation of a linear series of steps or procedures without any accompanying text to explain that diagram and the way that it was developed. Producing, or
in many cases simply copying and pasting, a diagrammatic representation and summary
of a research design from some sort of textbook (usually about methods) becomes what
research design, and designing research, is all about.
What is overlooked, or even ignored, in all this is that when you cut and paste a diagram
of a research design somebody else has developed, you are also copying and pasting a
whole heap of (usually undeclared) assumptions and choices that the person developing
the diagram made about, for example, what research is and how it should be done.
All research designs are full of assumptions and choices made by the person designing the
research. These are assumptions and choices about what the purpose of the research is, what
type of knowledge the research design will need to enable to be produced in order to address
the research problem, and how that knowledge can be produced using methods and techniques to do so. These assumptions and choices provide the context for understanding the
research design and how it was designed. Therefore, such assumptions need to be thought
about, surfaced, acknowledged, and declared when we design our research.17
We have covered a lot of ground in this opening chapter. We will return to these ideas at
various points in the chapters to follow. Like research design itself, this book is not meant
to be read or thought about linearly. Nor are the chapters meant to be read in isolation
from each other. Points made in one chapter are returned to and developed in later parts
of the book. In the next chapter, we explore in more detail how thinking about ethical
considerations is a central part of iterative research design.
SUMMARY OF KEY POINTS
Research design
• is a strategy that guides a specific research project.
• is about making decisions about what form various parts of that project will take.
• is about linking the purposes of the planned research to how that research will be
conducted.
Chapter 1
•
Research Design
23
• addresses a specific research problem and related research questions.
• is more than the identification of methods or techniques that will be used to collect
data.
• is made up of theoretical, methodological, and ethical considerations that shape the
design.
• uses relevant and credible research literature at all points of the research design
process to assist in the development of that design.
• is an iterative, nonlinear, process.
• requires reflexivity on the part of the researcher throughout the entire research design
process.
• Rather than simply being a set of individual procedures or steps, research design is a
thoughtful, reflective, and ultimately reflexive process that constantly requires us to
pause in order to consider what we are doing and why.
• What emerges from this process is what is called a “research design,” the shape and
substance of which is made up of decisions and choices made about a number of
areas.
• All of these decisions and choices are interconnected and cannot be viewed or made
in isolation.
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
reflexivity/reflexive thinking
relevant literature
research design
research ethics
substantive area of research
theory
empirical
iterative process
methodology
methods
qualitative approaches
quantitative approaches
SUPPLEMENTAL ACTIVITIES
Try one or both of the following exercises designed to assist you develop the type of reflexive thinking that is central to research design and that you can use to ask yourself questions about research reports you are reading or the research you are designing yourself.
1.
Obtain a report outlining the findings of a research study. Look for the level of detail
about the way the research was designed and what was discussed and what was not.
Here are some examples of what to look for:
• What was the research about and why was it about this?
• How did the researchers choose to do their research and why did they do it in
this way?
• Do they say what actually happened when putting aspects of the design into
practice, and why this happened?
24 Research Design
• Do they talk about any changes in their thinking about the research design
during its development and also when putting it into practice?
• Do they discuss methodological, theoretical, and ethical considerations that
impacted on their research design as it took shape?
2.
Journal the decisions you make and why you make them when designing your
research. If you are in the process of designing research, as you think about and
work through the various chapters of this book, keep a diary or journal of what
implications the discussion in each chapter has for the way that you will design
that research. For example, after reading this chapter, write about your thinking
concerning the role of theory in that design, and what assumptions you are basing
that thinking on. Why are you thinking about theory in that way, and how does this
impact decisions you might make about what type of data or information you will
need as a result of that thinking? The diary or journal then becomes a record of the
types of decisions you made, and why, related to the various areas that make up your
research design. This provides a record of the reflexive thinking that underpins the
design of your project.
FURTHER READINGS
Becker, H. S. (1998). Tricks of the trade. How to think about your research while you’re doing it. The
University of Chicago Press.
Lumsden, K. (2019). Reflexivity: Theory, method, and practice. Routledge.
NOTES
1. Cambridge online dictionary, https://dictionary.cambridge.org/dictionary/english/
iterative accessed 28/3/2020
2. Merriam-Webster online dictionary defines empirical to mean “originating in or based
on observation or experience.” Merriam-Webster.com Dictionary, Merriam-Webster,
https://www.merriam-webster.com/dictionary/empirical. Accessed 25 Feb. 2021.
3. For example, in Norway underpinning the annual collection of data about each
researcher’s publications is a list of recognized journals, articles from which will
be included in that collection. Decisions are also made about which books and book
chapters will be recognized based on where they have been published and by which
publisher.
4. See https://rxcomms.com/blog/6-ways-spot-predatory-journal/vglasson/
https://authorservices.taylorandfrancis.com/are-open-access-journals-good5. See
quality/
6. Research that is published informally or noncommercially or remains unpublished
is sometimes referred to as gray literature. Gray literature can include non-peerreviewed but still useful sources such as government reports, statistics, patents, conference papers, etc.
7. The article with this headline was first published on the digital platform www.daily
mail.co.uk of The Daily Mail on 9 July 2018 (https://www.dailymail.co.uk/sciencetech/
article-5934673/Being-rich-successful-really-genes-study-suggests.html). Initially,
the article claims that “[S]cientists have found social mobility is partially written into
our genes, which can make us high-flyers or high-earners” (paragraph 1). Reading
a bit more, the article says that “[t]he authors say our genes explain only roughly four
Chapter 1
•
Research Design
25
per cent of differences in social mobility” (paragraph 13) and that “the effect of the
‘genes for education’ on any one child’s life is small” (paragraph 27).
8. See Chapter 5 for what we mean by qualitative approaches to research.
9. This point is picked up and developed in Chapters 6 and 7.
10. See Chapter 5 for what we mean by quantitative approaches to research.
11. This point is picked up and developed in Chapters 8 and 9.
12. These designs are known as positivist or post-positivist; see Chapters 4 and 5 where
this is discussed in detail.
13. See for example the discussion of the qualitative research interview in Chapter 6 and the
quantitative survey in Chapter 9.
14. See Chapters 6 and 7 for a detailed discussion of this type of interview.
15. Hindess (1996) argues that there have been two conceptions of power that “have dominated Western political thought in the modern period” (p. 1). One “is the idea of power
as a simple quantitative phenomenon,” and the “second, more complex understanding
is that of power as involving not only a capacity but a right to act, with both capacity and
right being seen to rest on the consent of those over whom the power is exercised” (p. 1).
Theoretical writing and understandings of power in use may draw on one, two, or several theoretical traditions or variants thereof.
16. If you would like to read further about this, Lumsden (2019) in her Introduction to her
book Reflexivity: Theory, Method, and Practice provides a good (and accessible) introduction to, and discussion of, this complex idea.
17. Chapter 11 provides good examples of researchers declaring their hand in terms of the
assumptions and thinking that underpin their research design, and discusses why such
declaring of your hand is part of being a responsible researcher.
2
ETHICAL ISSUES IN
RESEARCH DESIGN
PURPOSES AND GOALS OF THE CHAPTER
In this chapter, we introduce and develop the idea of research ethics as an orienting idea
framing and permeating the entire research design process.
We explore the thinking that you will need to do about ethics when developing your
research design. We stress the point that thinking about research ethics, just like research
design itself, is a constant process. When doing so, we distinguish between research ethics
and gaining ethics approval from a research committee.
We take a close look at foundational concepts associated with research ethics such
as the principles of participant confidentiality, anonymity, and consent. We focus on
what we will need to think about when putting these ethical principles into practice when
designing our research.
Ethical challenges and dilemmas arising from contemporary trends such as the
repurposing and sharing of existing data, and the increasing rapid digitalization of many
aspects of social life, are also explored. In addition, we discuss the role of ethics committees, emphasizing that they are more than simply regulatory authorities.
Throughout the discussion we highlight that thinking about ethics is an important
part of the reflexivity1 that underpins the entire research design process (see Chapter 1).
Such thinking impacts on all aspects of the development of our research design, from the
moment we have an idea about a possible research problem, to well after we complete our
research including what we say, and how we write, about that research.
Having this discussion early in the book (the second chapter) is a deliberate choice
made to emphasize the centrality of reflexive thinking with ethics when designing our
research. We develop the ideas and issues introduced in this chapter throughout the rest
of the book. In this way, we model and capture an iterative way of thinking with ethics when designing our research. This is important because ethical matters related to
research design are multifaceted and dynamic—they do not keep still and often emerge as
a research design develops and then is put into practice.
The goals of this chapter are to
• Introduce the idea of research ethics.
• Focus on the thinking that you will need to do about ethics when developing
your research design.
• Explore participant confidentiality, anonymity, and consent as foundational, but
at times problematic, concepts in research ethics.
27
28 Research Design
• Consider the impact that sharing or repurposing data, using digital platforms
when collecting data, or using nonresearch generated sets of digital data as part of
our research design has on putting ethics into practice.
• Explore the role of ethics committees.
• Emphasize the centrality of reflexive thinking with ethics when designing research.
• Provide examples that illustrate the impact that thinking about ethics has on the
way we design our research and put it into practice.
WHAT IS RESEARCH ETHICS?
From the minute we begin thinking about what we might study and why, and then how
we will study it and why, we are constantly interfacing with ethical considerations. Ethics
is a term that all readers will have come across at some stage. But what exactly are we talking about when we talk about ethics or research ethics? Answers to this question range
from whole books, to a few sentences, and all places in between! Wiles (2013) provides us
with a useful entry point and orienting framework for exploring this question. She writes,
“Ethics is the branch of philosophy which addresses questions about morality” (p. 4). This
then raises the question of what morality is, and how morality is linked to ethics. Morality
focuses on the “personal set of values and beliefs which guide self-discipline (including
respect for others). . . . Ethics is an attempt to codify and regulate morality by stipulating
norms and principles for behaviour” (Duncan & Watson, 2010, p. 50). It is not always easy
to distinguish between morality and ethics as types of reasoning. This is because they are
so closely linked, and because they are often used interchangeably.
Research ethics are concerned with moral behavior in research contexts. Principles
and issues often identified in relation to thinking with research ethics in the design and
conduct of research include “respecting human dignity, respecting persons, and being
concerned for welfare and justice” (van den Hoonaard & van den Hoonaard, 2013, p. 15).
Therefore, you will often see research ethics discussed in terms of doing no harm to participants (nonmaleficence); having some positive benefit to participants or society (beneficence); and respecting participants’ decisions about what happens to them in the research,
including if they choose to participate in it (autonomy or self-determination). In addition,
no participant, or group of participants, should be either advantaged or disadvantaged
over others (Anderson & Corneli, 2018).
Putting ethical principles into practice when designing research will require us “to continually reflect on the ethical implications of researching people’s lives” (Duncan & Watson,
2010, p. 52) throughout the entire design process. This includes thinking about and asking
ourselves questions such as these: What will we tell our participants about our study and
when? What will we write about what they tell us—how and why? What will we do if our
research uncovers issues of abuse or illegality? How will we look after our data and who has
access to it? What will we say about how and why we did our research in a particular way?
What will we include in, and what we leave out of, the reporting of our research and why?
Such reflexivity is central to developing what van den Hoonaard and van den Hoonaard
(2013) refer to as a researcher’s “inner ethical poise” (p. 13). In the discussion to follow, readers
are encouraged to develop awareness of, and explore, both their inner ethical poise and why
that awareness matters in terms of the way we think and act when designing our research.
Chapter 2
•
Ethical Issues in Research Design
29
Regardless of which research methods we intend to employ in our research design, we
will be forced to consider and think reflexively about what van den Hoonaard and van den
Hoonaard (2013) describe as “keystones in any ethical research” (p. 39), namely participant confidentiality, anonymity, and consent. They note that these keystones form a complex triangle of connected ethical related issues. This triangle is bent into various forms by
the circumstances and context of each individual research project and design. For “[w]hile
there are a number of ‘common’ ethical issues [such as informed consent, anonymity and
confidentiality, and risk and safety] . . . research is always situated and contextual and the
specific issues that arise are often unique to the context in which each individual research
project is conducted” (Wiles, 2013, p. 9).
In the next sections of this chapter, we will take a closer look at what it means to put
these keystones of ethical research into practice when designing our research.
PUTTING INFORMED CONSENT INTO PRACTICE
Before we begin collecting information from people in our study, it will be necessary to
gain their consent to both collect and use that information. How to obtain that consent
and how to ensure that it is informed consent must be a central part of our thinking from
the outset when we are designing our research—not an afterthought. But what is informed
consent? Informed consent is when participants agree or consent to participate in the
research. It means that the participant understands both that they are giving consent and
what they are consenting to. Such consent relies on full disclosure by the researcher of
what participating in the research involves.
To make an informed decision about whether to consent to participate in a research
study, potential participants must be given both transparent, and sufficient, information
in an appropriate form about the study. This is so that participants know
• what the study is about;
• who designed the research and decided the way it will be done;
• what their role as a participant in the study will be;
• who will actually do the research or collect information from them;
• that they can decide whether or not they want to participate;
• that they have the right to withdraw, or withdraw any information collected
about or from them, at any time from the study without having to give reasons;
• what benefit they may or may not receive from being in the study;
• who else might benefit from the research and how;
• any risks that they may encounter by participating in the study;
• any costs to them (including time) by participating;
• what will be done with the information that they provide to researchers;
• who the researchers are, their affiliations, and the source of any funding for the
research received.
30 Research Design
Each of these points should be thought about and addressed as you are designing your
research. This means that you will be thinking with, and about, ethics at all stages of
the development of your design. To be able to give clear information about each of them
means making our research design, and our thinking about that design, as clear, honest,
and transparent as possible to potential participants. This includes the actual research
questions, the aims and the rationale for the research, the ways in which the research will
be conducted, and what we will tell about the research and to whom, after it is completed.
For example, we cannot deceive people about aspects of our research by telling them
that our research is for one thing when in fact it is for another. Nor can we imply some
advantage to participate when that might not be the case, therefore raising false hopes
and expectations on the part of our participants.2 This is because an important part of
informed consent is that the consent is given freely and not because the participant is
offered inducements (such as financial gain) to participate in the study, nor because there
are negative consequences for them if they do not participate.
Informed Consent—Who, What, and When
We will also need to think about how much information, in what form, to give, when and
to whom (Wiles, 2013). For example, we will need to think about how to keep the language that we use when describing our study in information sheets and consent forms, for
example, as concise and simple as possible. Thought will also need to be given to participant group appropriateness of that information in terms of the way that the information
is presented, including the assumed level of literacy of the reader. A way of doing this is to
ask “persons from the sample population of interest to review the consent form(s) prior to
using it” (Lahman, 2018, p. 74).
PUTTING IT INTO PRACTICE
SEEKING CONSENT IN AN APPROPRIATE WAY
Lahman’s suggestion to ask “persons from the sample population of interest to review
the consent form(s) prior to using it” (Lahman, 2018, p. 74) could usefully be extended
to include asking these persons from the sample population of interest to also review
the way in which that consent will be sought. For example, is a written consent form
the best means of gaining consent in the population or community of interest? There
may be culture-specific considerations that we need to take into account that make
the use of a written consent form inappropriate. 3 Persons from the sample population
of interest could also be asked to comment on from whom that consent needs to be
gained. For example, Duncan and Watson (2010) found, when researching in different
cultural contexts, that in some communities it was necessary for them to obtain verbal
community consent before attempting to get written consent from individuals in those
communities.
Sometimes when conducting our research, the design of the research may change. For
example, in many qualitative research approaches, aspects of the research design emerge
as the study progresses. This can make giving information about exactly what participating in a study may mean problematic.4 Therefore, in such cases we will need to consider
Chapter 2
•
Ethical Issues in Research Design
31
when and who we will need to get informed consent from and overtly incorporate these
considerations into every part of our research design including constantly reviewing them
as the design of our research evolves. The same consideration applies if our study design is
some form of longitudinal study:
In longitudinal studies or research with repeated stages of data collection it may
be appropriate to provide information, and gain consent, for each stage of data
collection. . . . This approach highlights the importance of viewing consent as a
process that is ongoing throughout a project rather than as a one-off event (Wiles,
2013, p. 28).
Other issues related to informed consent arise if we intend to reuse data that has been
collected in other studies. We return to this point in a later section of the chapter.
Informed Consent in Relation to “Vulnerable” Populations
There are some individuals and groups of people formally deemed “vulnerable” by ethics committees, government regulatory authorities, and professional associations. For
example, the Declaration of Helsinki5 explicitly states that “[S]ome groups and individuals
are particularly vulnerable and may have an increased likelihood of being wronged or of
incurring additional harm. All vulnerable groups and individuals should receive specifically considered protection” (World Medical Association, 2018, item 19).
In general, a group is considered vulnerable if there is good reason to believe that individuals in that group may, for some reason, have difficulty providing free and informed
consent to participate in research. In the Declaration of Helsinki (World Medical
Association, 2018), individuals incapable of giving informed consent, individuals likely
to consent under duress, individuals with increased likelihood of incurring additional
harm, and individuals that do not benefit from the results of the research are all considered vulnerable.
TIP
BE AWARE OF LOCAL UNDERSTANDINGS AND REQUIREMENTS
RELATED TO WHO IS DEEMED VULNERABLE
Individual countries may also designate who vulnerable groups are. For example, in
the United States, groups designated as vulnerable populations include “children, prisoners, individuals with impaired decision-making capacity, or economically or educationally disadvantaged persons” (Department of Health and Human Services, 2018).
Therefore, it will be important to make sure when designing research that you are
aware of the local understandings and requirements related to who is considered a
vulnerable participant and what that means for gaining consent.
The use of the designation “vulnerable,” while well intentioned, has been critiqued
on a number of grounds. One ground is that this designation may actually work against
the best interests of groups deemed vulnerable. This is because increased regulation and
requirements for accessing participants may make research with these groups increasingly
difficult and thereby reduce the amount and scope of research focusing on these populations. This can lead to the situation described by Markham et al. (2018) where “systems of
32 Research Design
research and ethics governance do not facilitate and support social research that needs to
be done, because it has been traditionally been [sic] prohibited” (p. 3).
Another criticism is that the designation of vulnerable may take away the right of
individual participants in these groups to participate in research. They may be excluded
on the assumption that they are vulnerable (predetermined by others’ definition of that
term), and therefore not competent (predetermined by others’ definition of that term), to
give informed consent (also predetermined by others’ definitions of that term). The effect
of this is that large groups of people deemed as being vulnerable such as children or those
with cognitive disabilities “who all in various ways could stand to benefit from having
their living conditions elucidated by research” (Juritzen et al., 2011) are in effect excluded
and their rights and interests further marginalized.
All of this raises a series of questions around informed consent for which there are not
clear-cut answers. These questions include who decides whether a person is competent
to participate in a study. Particularly if the study is about something that the person has
knowledge of and wants to participate in, who makes that decision? Whose rights predominate? Who is able to speak for whom and when? Reflexively thinking about these
questions and their effect on the way that we design our research can help us avoid some
of the taken-for-granted assumptions that we might otherwise make about informed consent that on the surface may seem quite reasonable. It can help us shift the focus of our
thinking from whole of group vulnerability to vulnerability of individuals, or groups of
individuals, within that whole of group designation. In so doing, it can shift the emphasis
in our thinking to the idea of research participants as “capable and competent yet vulnerable
at the same time” (Lahman, 2018, p. 13). This can open up possibilities for the inclusion
of people in our research that otherwise may be excluded because of standardized and normalized understandings of what it means to be vulnerable.
PUTTING CONFIDENTIALITY AND ANONYMITY INTO PRACTICE
The ideas of confidentiality and anonymity are closely related. At times, you may even see
these terms used interchangeably. While there is no doubt that they are linked, they are
not the same.
Confidentiality is wider in scope than anonymity. Keeping a research participant’s identity anonymous is part of confidentiality. However, confidentiality is not
ensured by keeping the participant’s identity anonymous. For example, it is possible
to breach confidentiality if information from an anonymized participant is reported
or used in some way in the research when that participant specifically requested that it
not be used (Wiles, 2013). Confidentiality is more than just not disclosing the name,
identity, or identifying features of participants. It is also about the way that any data
that a participant has provided or is related to that participant is shared or not shared,
and with whom.
Anonymity is used by researchers to protect the identity of the participants in their
study. In research design, anonymity often involves using pseudonyms when referring to
participants or sites in the study. A pseudonym is a fictitious name given by researchers to
participants or sites in a research study. The pseudonym is used instead of their real name
when reporting and discussing the research. However, even such a seemingly benign and
simple process requires some careful thought when designing your research, and in itself
should not “be confused as equal to ethical research” (Lahman, 2018, p. 83).
Chapter 2
•
Ethical Issues in Research Design
33
For example, one thing we will need to think about is what pseudonyms will be used
and who will choose them and how. Will we ask participants to choose their pseudonym or
will we as researchers do this? If we choose the pseudonym, what impressions, intended or
otherwise, might be given about a participant by assigning them a particular pseudonym?
Being able to rename someone involves the use of power (Hurst, 2008; Lahman, 2018).
PUTTING IT INTO PRACTICE
WHAT DO YOU DO IF IT IS IMPOSSIBLE NOT TO IDENTIFY THE
SPECIFIC SITE OF A STUDY?
Sometimes, it may be impossible not to identify the specific site of a study. For example, a master’s student, Christine Moe Grav (Grav, 2015), that one of us supervised
wanted to study the effect on a group of middle managers in a specific government
department of having to implement a government-mandated departmental structural
reform that would result in them losing their position. In other words, these managers
would have to manage themselves out of their jobs since the result of the reform would
be that there was no more middle management. There were 15 of these middle managers in this specific department. Christine was interested in how these 15 managers
experienced this process and what those experiences could tell us about the process
of change management. The problem she faced was how to do this study in such a way
that it ensured confidentiality and anonymity for this group of 15 participants even
though it was impossible to do the study without identifying the actual reform and
therefore the department that the reform affected and therefore the 15 managers.
Christine protected the confidentiality and anonymity of each individual manager in
terms of what information they contributed to the study by making it impossible to identify which manager had said what. This was done by not linking any demographic data to
individual participants or to the information they provided for the research. If Christine
had linked the participant or the information they gave in their interview to demographic
data collected about them specifically, such as length of time in the position, it would not
have been difficult for a reader to cross-reference that information to the 15 managers
and figure out who had said what. By providing ranges of length of time in the position
rather than listing experience levels for each individual anonymized manager, Christine
was able to provide a demographic snapshot of this group while preventing specific
information being able to be linked to individual managers. When citing her participants,
Christine did not include any demographic data about the participant in question.
Sometimes we may need to consider what we will do if a participant does not want to
be given a pseudonym but wants their real name to be used or their identity to be linked to
the data collected about them. In these instances, it is important to make sure that participants understand what having their real name used will mean in terms of the way that the
information is presented in public arenas such as publications, presentations, and reports.
It means making it clear that it will be possible to identify them and link the information
that they gave, their data, to them.
The Use of a Pseudonym Does Not Necessarily Ensure Anonymity
We will also need to remember that in itself the use of a pseudonym for individual participants does not necessarily ensure anonymity for participants. For example, if you identify
34 Research Design
the specific region or town or organization or sector that the study is located in, it might
not be difficult for people to backtrack and identify who some of the participants might
have been in the study. This is particularly so if we collect demographic data about our
participants and directly link that data to other data collected in the study. An example of
this would be if we report excerpts from qualitative interviews in the following way: “managers here are not good at explaining things” (Julia, Female, 38 years, 2 years’ experience,
mid-level manager).
While Julia is not the person’s real name, the demographic data is real. Therefore, if
the site of the study can be identified, it will not be difficult to work out who “Julia” is.
This can also be the case even if the data are reported only using the pseudonym itself, for
example, “managers here are not good at explaining things” (Julia). This is because if in a
table of demographic data, each anonymized participant has been listed and their demographic data attached to their pseudonym, then it is a simple exercise to backtrack and
connect that demographic data to the participant.
Consequently, Morse (1998) suggests reporting ranges of the demographic data of
the entire participant group (e.g., age ranges), rather than specific demographic data of
each individual participant (e.g., age). She also suggests that we “do not attribute each
quotation to a particular participant, unless there is a compelling reason to do so” (p. 302),
arguing that the researcher will have selected particular quotes as illustrative exemplars of
their findings and so attaching a quote to an individual (even with a pseudonym) is not
necessary.
This does not mean that demographic data about the individual participants in a
study cannot be collected or used in a study. The key point is that specific demographic
data, no matter what methods are used, must not be linked to individual participants in
a way that it could identify those participants if the participants have been promised anonymity when participating in the study. It also protects the confidentiality of what that
anonymous person has said.
PUTTING IT INTO PRACTICE
Navigating Ethical Issues Is Not Always Straightforward
Our discussion of putting confidentiality and anonymity into practice has highlighted
that this is not always straightforward even if you have obtained informed consent and
taken steps to ensure the anonymity and confidentiality of the participants and sites
in your research. For example, some form of observation is often used by researchers as a way of obtaining information about a particular social setting of interest.6
Different sorts of issues arise when making those observations, depending on what
type of observations you make and where. Issues may also arise from your level of
involvement in what is being observed.
Therefore, issues that you will need to think about include
•
•
if you intend to make observations part of your study design you will need to
consider in what capacity you will make those observations. For example, will
you make those who are being observed aware that they are being observed or
what is being observed or why? How does your decision about this impact on
informed consent and also the privacy of those being observed?
if you are going to research people in public places such as shopping centers
or public streets, will you need to get consent from every person you observe?
Chapter 2
•
•
•
Ethical Issues in Research Design
35
Given that in a shopping center or public street it may not be possible to do
this, does this mean that it is not possible to do this type of observational
research?
if when you are making your observations you observe an interaction or behavior
that is concerning in some way— for example, bullying or sexual harassment—
what will you do? What about if you observe what you consider to be inadequate
or incompetent professional practice? Should you intervene or report that
behavior? What about anonymity and confidentiality considerations in all this?
if you are a student and then return to debrief your observations with others
and your analysis of those observations, how will you ensure the anonymity
and confidentiality of that observational data when you talk about it?
You will need to think through and take up your own ethical position in relation to
these types of dilemmas and justify what you choose to do as part of your research
design.
Although we have used observations as the vehicle for the discussion here, the
same sorts of considerations apply to other methods used to collect data from and
about people, such as interviews and surveying. There is no simple answer to these
types of questions. However, a good piece of advice is given to us by Ryen (2011):
There are no standard answers to these dilemmas. . . . We need to be prepared
for all these challenges, which demand that we put them on the agenda from
the very start of our projects and ask ourselves how they relate to our own
particular project.
But what should we do? A good piece of advice is always to invite experienced
researchers with particular knowledge in research ethics and in your field to
discuss matters with you. (pp. 418–419)
WHAT YOU NEED TO THINK ABOUT WHEN
REUSING, REPURPOSING, AND SHARING DATA
Some research designs involve “repurposing, reusing, combining, sharing and linking
data in new ways” (Ballantyne, 2019, p. 357). Indeed, there are increasing calls that
researchers should share their data (Meyer, 2018), resulting in pressure directly and
indirectly being placed on them to do so. Some journals and funding bodies are requiring (or at least strongly encouraging) some sort of data sharing as a prerequisite for
publication or funding. For example, the UK Economic and Social Research Council
Data Policy, last updated in 2021, states that “[p]ublicly-funded research data are a
public good, produced in the public interest, which shall be made openly available and
accessible with as few restrictions as possible . . . for future research” (Economic and
Social Research Council, 2021, principles 1 and 2).
One of the central questions that the reuse or sharing of data as part of a research
design raises is who decides that is it OK to share and re-use that data, and what can or
cannot be shared. As Flick (2015b) puts it,
If we start interviews, for example, by asking participants for their (informed)
consent—are such permissions for doing research or the informed consents
obtained valid for re-use of data for all purposes and for every other researcher?
What does this mean for clearances by ethics committees? (p. 604)
36 Research Design
To address these important questions requires us to consider what implications the
repurposing of data has for one of the keystones of ethical research, namely informed
consent. A central consideration is whether a participant’s consent for reusing, repurposing, and sharing their data can ever really be informed. When consenting to the reuse
of their data, research participants will not necessarily know exactly what their data will
be reused for, or by whom. This raises the possibility that participants’ data could be
used in a study that they would not have consented to their data being reused in, because
of, for example, who is doing the study or who is funding the study or what the aims of
that study are.
The potential difficulties of obtaining informed consent for the use of data from
previous studies have led some commentators to suggest that the primary question for
the reuse of such data is not “‘[C]an we get consent’, but rather ‘Does the public interest in using the data outweigh individual interests in controlling access to the data’?”
(Ballantyne, 2019, p. 358). However, this suggestion raises yet another set of questions
related to informed consent that will need to be thought through and about. Questions
such as, “Is there an upper limit to the risks that individuals should bear for the sake of the
public benefit of data use? What legitimizes decision-making processes? Who should be
held accountable for data misuse and how?” (p. 365).
To these questions posed by Ballantyne we can add even more related questions such
as, What does an upper limit to the risks those individuals should bear for the sake of the
public benefit of data use actually mean—is that harm OK in some instances? Who will
decide what this upper limit is, will it be applied equally to everyone, and will we know
that upper limit when consenting to our data being reused or repurposed? Similarly, who
will decide what the public benefit of data use is? Will this be a legal or regulative or ethical, individual, or collective decision? How will it be enforced? Moreover, again, who will
decide this, how, where, on a case-by-case basis or . . .?
Therefore, if when designing your research you are thinking about using data for
which the original consent did not explicitly include a clause or statement about data sharing or reuse, then you will have even more thinking to do. This relates to whether the more
“effective” and “efficient” use and reuse of data for some sort of greater public good should
outweigh the rights of each individual participant to determine the way that their data are
used and who has access to it.
How to Address These Types of Questions?
Beck (2019) suggests that using multiple layers of consent is one way of addressing, or
at least beginning to think about, how to address the ethical issues that arise around
informed consent when participants’ data collected for one purpose is used for another.
Such layered consent may make it possible for the participant to have more control in
relation to the extent of, and purpose for, the reuse of their data. This is because multiple
layers of consent would enable research participants giving informed consent for a specific
study (the primary study) also being explicitly asked whether they consent to some or all
of (1) participating in the primary study and only having their data used in that study,
(2) the researchers in the primary study being able to reuse that data in later studies, and
(3) the archiving/storage of that data which may then be accessed by other researchers and
reused in other studies.
However, even if multiple layers of consent are used, the issue that still arises is how
much information we need to give about each of these layers before we can say that any
Chapter 2
•
Ethical Issues in Research Design
37
“consent for reuse” (Bishop, 2009, p. 262) that is given really was informed? How explicit
does information about each of these layers need to be about what data will (and won’t be)
shared, when and how, and with whom, as decided by whom, in order to be able to claim
having informed consent for the reuse of that data and/or use of that data in a specific
secondary study?
There are no easy or “correct” answers to questions such as these. However, if you are
considering using a research design that involves data sharing and/or reuse and/or the
archiving of your study’s data for future researchers to access and use, you will need to
think through these questions. You will also need to declare how you have designed your
research with these questions in mind. This is not easy to do as “[b]alancing the rights
and responsibilities of the primary researcher and the research team, secondary researchers who want to make use of the data, the data archivist, research participants, research
funders and the general public may present significant challenges” (Wiles, 2013, p. 88).
Thus when designing research that involves some form of data sharing or repurposing, it
is important to think reflexively, not simply linearly, about the use of that data after it is
collected. This includes after your research is concluded.
WHAT YOU NEED TO THINK ABOUT WHEN USING
INFORMATION ON THE INTERNET AS DATA
The advent of the internet and the rise of digital data has refocused researchers’ attention
on concepts like “privacy” and “informed consent.” It raises questions such as, “When
should someone expect confidentiality on the internet? When should a researcher seek a
participant’s permission to conduct research on the internet?” (Lahman, 2018, p. 209). For
example, if information on social media or social networking sites is “harvested,” “mined,”
or “collected” in some way and used as data in a research study, does that mean that consent needs to be given for the collection and use of that data? If so, how might this be done?
For example, what if someone collects digital traces of our everyday activities, again
without our knowledge or consent, and aggregates them into massive sets of digital traces
known as Big Data?7 Such digital traces can be in the form of text, videos, and images
taken, for example, from social media (such as Facebook or Twitter), personal blogs,
chatroom conversations, and other forms of internet communities. The digital traces also
include data transmitted by microchips embedded in “clothing, products, credit cards,
and passports” (Mills, 2018, p. 596), Internet of Things,8 as well as records of some form
of activity generated by activity bracelets and individual health monitoring devices. Then,
what if that Big Data set is used as the basis for algorithmic analyses that are used to predict aspects of our behavior such as our political leanings or likely consumption patterns?9
Who regulates or makes decisions about this? Who actually owns that data, and who can,
and cannot, give consent for digital traces to be used for research and other purposes? If we
are planning to use this type of data, these questions need to be thought through carefully.
Blurring the Boundary Between Public and Private
The accessibility of digital data, such as digital traces of our daily activities, forces us to
consider what is public information and what is not? In the digital age, this is not so easy
to answer given that “much of Internet behavior is both private (person in their home
residence) and public (person is active in the Internet) simultaneously” (Lahman, 2018,
38 Research Design
p. 200). For example, in a study designed to reveal the potential of large datasets, Danish
researchers Emil Kirkegaard and Julius Bjerrekær (2016a, 2016b) obtained access to a
large set of data by pretending to be looking for partners on a dating site. In fact, their
intention in joining the dating site was to scrape10 the data in the user profiles of members
of the dating site OKCupid.com. From the information they obtained, they developed a
publicly available dataset.
The researchers justified not obtaining informed consent to access this information
by claiming they were merely presenting already publicly available data and hoped that
other researchers would use the dataset to address other research problems. They stated
that a dataset collected by researchers is too often “not used to its full extent” (Kirkegaard
& Bjerrekær, 2016b, p. 1), and this slows down “the progress of science immensely because
other researchers would use the data if they could” (p. 1).
The ensuing controversy was massive. OKCupid users, academics, online commenters,
and others accused the researchers of making confidential and sensitive private-user information public. However, the researchers continued to argue that the dataset was presenting already publicly available data.
This case raises the question of whether an internet-related activity, such as becoming
a member of OKCupid, is private or public or both private and public simultaneously. It
also raises the question of even if something is public, does that mean that it is OK to use
private information made public as research data.
The increasing use of data from the internet by researchers gives rise to many questions about what is public and what is private. For example,
• when someone posts information about themselves or others on social media such
as in tweets or blogs or conducts a conversation in a social networking site such as
Facebook, who can use that information and what for?
• when people post information about themselves and aspects of their lives on
public domains on the internet, where the information is accessible by all, does
that mean that this information can be used as data in our research studies
without asking the people who posted it if they consent to this? In other words,
does the fact that this information is being used formally as research data change
the thinking about ethics we need to do?
• what about “public” data in a closed or private internet community, such as
OKCupid?
• how do we make that decision?
• who makes that decision?
• given that digital traces have no geographical boundaries, what formal regulatory
requirements will need to be met?
Further complicating matters when trying to address the issues that the points raise
is the rapid rate of growth and change in both the development of technology related to
digitization11 and the use to which that digitization can be put. This includes commercialization of data such as the buying and selling of sets of digital traces.12 Who should
profit from that sale? Is this exploiting the people from whose activities the digital traces
were harvested? Who owns the data? During the time that has elapsed between us writing
Chapter 2
•
Ethical Issues in Research Design
39
this chapter and you reading it, there is no doubt that many new and different questions
will have emerged around thinking about ethics in relation to digitally based research and
informed consent. It is not possible to predict where discussions of the issues already being
raised by some researchers and scholars about the possible need for changes in our thinking about consent and privacy in a digital age will take us.
None of this means that the principles about the importance and centrality of the idea
of informed consent and privacy (or any other ethically related matter) we have discussed
previously in this chapter are outdated or no longer relevant in light of a fourth industrial
revolution and increasing digitization more generally. Quite the opposite! Rather, all of
this reminds us of, and returns our thinking to, a point made earlier: “Ethics is not a
static event but a continual process” (Sparkes & Smith, 2014, p. 206). New and different
issues constantly arise related to informed consent, anonymity, and confidentiality. This
includes research that is driven by various forms and aggregations of digitized data.
WORKING WITH ETHICS COMMITTEES
In many countries, formally constituted ethics committees also known as Institutional
Review Boards, or IRBs, regulate ethical matters relating to research design and conduct. When designing your research, you will need to find out what these formal regulatory requirements are, consider them when designing your research, and obtain the
necessary ethics committee approval to proceed with your research. This should happen
before you contact potential participants or commence collecting any data. In the case of
multiple-site research studies, you will need to ascertain if you are required to submit your
research design to several ethics committees if each site has its own ethics committee. You
will also need to find out what form your submission should take. Is there a template that
you are required to follow? Does the template vary from committee to committee?
Ethics committees, and their underpinning understandings of research, are often
influenced by the traditions of medicine. This is because “the original focus of IRBs and
the context from which they emerged was that of medicine and the scientific discourse
that underpins medicine” (Cheek, 2008, p. 57). For example, in the United Kingdom,
the Royal College of Physicians in 1967 recommended that all medical research be subject
to ethical review. By 1991, every health district was required to have a Local Research
Ethics Committee (LREC). In addition, Multi-centre Research Ethics Committees
(MRECs) emerged as a means of helping streamline proposals that otherwise would have
to go through numerous LRECs (Ramcharan & Cutcliffe, 2001).
Similarly, in the United States, the impetus for the initial development of IRBs
also came from medicine and the science underpinning medicine. Increasing public
and government concern over unethical research studies, such as the Tuskegee Syphilis
Study,13 ultimately resulted in the Belmont Report from the National Commission for
the Protection of Human Subjects of Biomedical and Behavioral Research in 1979. This
report detailed fundamental ethical principles for the conduct of human research: beneficence, respect, and justice. These principles are what shape IRB standards in the United
States today.
Ethics committees, such as IRBs, consider how a proposed research design puts principles such as beneficence, respect, and justice into practice. For example, they consider
whether the research is worth doing in terms of producing useful and significant findings and outcomes. They also examine whether the proposed research design will enable
40 Research Design
trustworthy results. This means that the methods to be used to collect data from participants need to have been clearly explained. This includes recognizing potential issues that
may arise from those methods about informed consent, anonymity, and confidentiality
and how they will be addressed.
Particular attention is often focused on information sheets or sufficient information
in some form that participants in the research will be given about the research before they
sign a consent form or give consent in some other acceptable way to participate in the
research. Ethics committees are also interested in how data and information related to the
study will be stored and who will have access to that data. This is important because it can
be a seemingly “ordinary” thing such as storing data on a laptop that destroys the entire
confidentiality and anonymity of a research project and compromises its ethical integrity
if, for example, the laptop is stolen during the research.
Focusing on the Principles, Not the Requirements
When thinking about where research ethics “fits” into the research design process, many
researchers immediately focus on what they must do in order to gain “ethics approval”
from an ethics committee or review board. As a result of this sort of focus, a form of
shorthand terminology such as “getting ethics” has emerged as part of the research design
process. The danger in the consolidation of this type of shorthand terminology is that
thinking about ethics is reduced to how to meet the requirements for ethics committees’
approval. “What are the requirements, and have they been met?” becomes the focus rather
than the ethical matters themselves. This type of thinking can lead to “checklist” type
thinking about ethics. In checklist thinking, ethical thinking becomes focused on ticking
the boxes by “passing” or “meeting” those specific requirements. Often the thinking that
sits behind those requirements is lost, or at best becomes a secondary focus with the main
goal being to get the approval.
Ethical matters and thinking through ethics throughout the research design process
cannot, and must not, be confined or reduced to a static list of standardized steps to be
followed that can be made into a checklist and then ticked off as complete one by one. For
example, do I have a consent form, information sheet, and so on. In such thinking, the
often unstated assumption is that if you address all the things on the list and get a tick next
to each of them on some sort of literal or mental checklist, then your research design is
“ethical,” ethics “approval” has been gained, and your thinking about “ethics” is done. In
fact, what actually has been gained is approval to proceed ethically. If the thinking about
ethics when designing research is reduced to a focus on meeting the requirements of an
ethics committee or checklist, the danger is that the focus on the ethical issues themselves,
or ethics in relation to the particular research being designed and conducted, can be lost
once the approval is gained.
Just complying with predetermined procedures, such as the format that an information sheet must take, may not necessarily ensure that the design and conduct of the research
is ethical. Research design, the research process, and therefore the thinking through ethics
that influences every part of that research process are not static but continually unfolding
and developing—this is why your thinking about ethics is not finished when you have met
the requirements of an ethics committee—in many ways it has just begun. New ethical
issues may arise, or change during the life cycle of your research, forcing you to revise your
initial research design.
Chapter 2
•
Ethical Issues in Research Design
41
The key point that emerges from all this is that there is much more to research ethics and ethics committees than regulation and procedures. To focus our thinking about
ethical matters related to research design solely on meeting or “passing” the procedural
requirements of an ethics committee runs the risk of standardizing and limiting thinking
about both ethics and ethics committees, reducing them to matters of procedure—where
ethics is “got” or the requirements “met.” Such thinking reduces the role of ethics committees to an administrative focus and loses the important contribution that they can
make as educational and scholarly bodies concerned with advancing conversations and
understandings of ethical research itself, thereby developing researchers’ “ethical literacy”
(Wiles, 2013, p. 1) and inner ethical poise.
Activity
Finding Out About Professional Ethical Guidelines
Thinking with ethics with respect to your research design will also need to take into
account relevant “[p]rofessional ethical guidelines and codes [containing] disciplinary
norms of ethical behavior,” but which “are not legally enforceable” (Wiles, 2013, both
quotes from p. 6). Such a code identifies standards of behavior and conduct based on
core ethical principles and values of a profession, organizations related to that profession, and those working in that profession.
Examples of such guidelines include the ICN Code of Ethics for Nurses (International
Council of Nurses, 2021), the Code of Ethics of the Education Profession (National
Education Association, 2021), and the Code of Ethics for Social Workers (National
Association of Social Workers, 2021).
Find out if there are professional ethical guidelines or codes related to the area that
your research will be conducted in and think about if, and if so how, these codes or
guidelines may affect your research design and the way it is put into practice.
CONCLUSIONS
Thinking about putting ethics into practice is a reflexive part of the research design process and part of being a responsible researcher. It is irresponsible to design research without
constant consideration of ethical matters from the initial idea until the conclusion of that
research and sometimes even beyond that, for example, when we are considering reusing a
study’s data. When unexpected issues arise during the research, reflexive thinking provides
a way of thinking through the issues and seeing them for what they are, making decisions
related to that thinking, and being able to justify both to yourself, and others, why you made
the decisions you did. These others include the participants in your study, your supervisors if
you are a student, the readers of your research, and formal bodies such as ethics committees.
Thus, thinking about ethics cannot be reduced to a series of predetermined decisions
about whether or not we meet certain predetermined criteria, and therefore can say that
something is ethical. Such reductionist thinking reduces research ethics to techniques or
points to be checked off on an “ethics to-do checklist.” Checklist thinking loses sight of
the fact that “to-do” requirements such as having a consent form or an information sheet,
42 Research Design
storing data appropriately, and ensuring confidentiality and anonymity are outworkings
of the ethical thinking that sits behind them. In themselves, they are not the ethics of the
design. The thinking behind the requirements is lost.
Issues and uncertainties can arise when thinking about how to put research related ethical principles and issues into practice. For example, what happens if there is disagreement
among researchers about what “ought to” or “ought not to” be done in relation to a proposed research design or what the “right thing to do” is, such as whether a dating profile
site should be scraped or not. How do we decide, based on what, if our research design
avoids “harm” or is “promoting good”? Grappling with such questions requires reflexive thinking on our part about ideas such as “ought,” “ought not,” “harm,” “promoting
good,” and “the right thing” themselves. Why do we think the way we do about each of
them; how does this affect the way we think about, design, and actually do our research,
and what, and why, we say and report about that research.
Thinking about ethics in this way opens up to scrutiny all of the decisions and actions that
we take when both designing and implementing our research. Such thinking extends the
focus of our thinking through ethics beyond an individual study or research design, or meeting the requirements of a specific ethics committee. It can “push scholars to question the
ethical stakes of what is not studied, the questions that are not asked, and the social groups
and communities that are not the subject of research” (Blee & Currier, 2011, p. 404).
As Macfarlane points out, “[R]esearch ethics is rarely about headline-grabbing incidents of scandal and drama. There is an ‘ordinariness’ about the day-to-day decisions we
face which is rarely recognized” (p. 24). Examples of such ordinariness include whether
to exaggerate the significance of our research in order to attract funding (Chubb &
Watermeyer, 2017).14 Or it might be about being tempted to cut the odd corner, perhaps
about the extent of our data collection or the detail we provide in the write-up of our
research, or adapting the way we report our research to meet the requirements of a journal. Or it might be any or all of
keeping the audio recording going for a few minutes after completing a
formal interview in the hope that the interviewee might say something
more interesting; promising to send someone their interview transcript to
check and never doing so in the (almost) sure knowledge that there will
be no consequences; referencing to sources that we may have found in the
bibliographies of others but never actually read ourselves; or taking more
authorial credit than we should do when working with other, perhaps less
powerful or experienced, researchers. (Macfarlane, 2010, p. 24)
This ordinariness can also include seemingly benign practices such as cultivating trust, or
even friendship, with those participating in our research. What is the motivation for cultivating such trust or friendships? Could it be to get people to participate in our research
and give us the information we want from them?
We have raised many issues in this chapter that you will need to think about when developing your research design, when putting that design into practice, and after you have
completed your research. Therefore, this chapter is not the last word on ethics in relation
to research design. Rather it is a first word in an ongoing discussion that aims to provide a platform for the thinking that will need to be done about ethics at all stages of the
Chapter 2
•
Ethical Issues in Research Design
43
development of your research design. It provides the foundations for you to develop your
own “inner ethical poise” (van den Hoonaard & van den Hoonaard, 2013, p. 13).
In this way, the chapter has set the scene for discussions in later chapters of the book
about putting ethical principles into practice in the various areas that make up what we
call a research design such as choices about research questions, methodology and methods, and what we write, and how we talk, about our research.
SUMMARY OF KEY POINTS
• Research ethics are concerned with moral behavior in research contexts.
• Ethics, and thinking about ethics, is a process that impacts on all aspects of designing
research.
• Ethical matters related to research design are multifaceted and dynamic.
• Three core principles of research ethics are the concepts of informed consent,
confidentiality, and anonymity.
• Repurposing, reuse, and sharing of research data pose particular issues about
informed consent and who has the right to share, or use shared, data.
• Emerging forms of digital data create new and different ethical challenges when
designing research.
• Ethics committees have an important contribution to make in advancing
conversations and understandings of ethical research.
• Getting ethics approval from ethics committees is not the same as designing and
conducting ethical research.
• Reflexivity is central to developing a researcher’s “inner ethical poise” (van den
Hoonaard & van den Hoonaard, 2013, p. 13).
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
anonymity
confidentiality
ethics committees
informed consent
Institutional Review Boards (IRBs)
layers of consent – repurposed data
pseudonym
qualitative research approaches
repurposing of data
research ethics
vulnerable populations
SUPPLEMENTAL ACTIVITIES
1.
Doing your homework and being prepared for interacting with, and learning
from, Ethics Committees
Find out which IRB or ethics committee(s) you may need to receive formal
ethics approval from before you could begin your research. When doing so, take
into account that this may vary depending on from where, and in what capacity,
you will submit your research proposal. For example, will it be as a student in a
44 Research Design
higher education institution, or will it be as an employee of an organization or both?
Remember, if your proposed study involves multiple sites, you may be required to
obtain approval from each site.
Next, obtain a copy of the guidelines and requirements of those ethics
committees. Think about what kind of information they request from you, and what
design decisions will you need to have made to address those requests.
• Why do you think that this information is requested?
• Do the forms all request the same information?
• If there are differences between them in terms of the information they
request, what are they and why might this be?
If you are in the process of developing your research design, begin drafting your
application for the relevant ethics committee(s) as you develop your proposal.
2.
Thinking about if the means justify the ends when we design our research
Thinking about whether the means justify the ends is a central ethical
consideration when designing our research. We will explore this point by looking
at the example of researchers exaggerating the “impact” of their proposed research
when writing an application for research funding in order to improve their chances
of attracting funding.
Chubb and Watermeyer (2017) studied how researchers wrote the section of
their application for funding that required them to identify and demonstrate the
“impact” of their proposed research by stating “how they will ensure economic
and/or societal returns from their research” (p. 3). They interviewed researchers
about what they did and why when writing this section about impact. Answers they
received included
• “. . . telling a good story as to how this might fit into the bigger picture. That’s
•
•
•
•
what I’m talking about. It might require a bit of imagination, it’s not telling
lies. It’s just maybe being imaginative.” (p. 8)
“If I want to do basic science I have to tell you lies.” (p. 5)
“It’s virtually impossible to write one of these grants and be fully frank and
honest in what it is you’re writing about.” (p. 5)
“I don’t think we can be too worried about it. It’s survival. . . . People write
fiction all the time, it’s just a bit worse.” (p. 6)
“People might, well not lie but I think they’d push the boundaries a bit and
maybe exaggerate!” (p. 9)
Here we see academics justifying what is at best exaggerating, and at worst
lying, when writing this impact statement. Justifications such as they were just being
imaginative or creative, or this is just part of what one has to do to get a grant.
This raises a dilemma for researchers when funding is needed for the research
to be able to proceed. If the funding is not gained, then the research cannot
proceed. Therefore, how far are researchers prepared to go to get that funding?
Does gaining the funding and therefore being able to actually do the research
outweigh exaggerating the impact of the research? This is an ethical matter and
consideration.
Chapter 2
•
Ethical Issues in Research Design
45
Think about and discuss the following dilemma that arises from this situation:
• As a researcher, do you write the impact statement (or any other part of
the proposal for that matter) “to fit the requirements of funders . . . on
the surface a seemingly commonsensical or ordinary position to adopt
if winning the funding is the goal” (Cheek, 2017, p. 31)? After all, this is
something everyone else is doing anyway.
• Or do you choose not to write your proposal in line with those requirements
and therefore effectively make yourself and your research uncompetitive in
the funding competition? Therefore, your project which may have significant
benefits to the population of people in it, will not proceed.
• Is there a middle course of action in all this?
FURTHER READINGS
Cheek, J. (2010). Human rights, social justice, and qualitative research: Questions and hesitations about what we say about what we do. In N. K. Denzin & M. D. Giardia (Eds.), Qualitative
inquiry and human rights (pp. 100–111). Left Coast Press.
Lahman, M. K. E. (2018). Ethics in social science research: Becoming culturally responsive. SAGE.
Wiles, R. (2013). What are qualitative research ethics? Bloomsbury Academic.
NOTES
1. See Chapter 1 for an extended discussion of reflexivity.
2. For example, when participants are consenting to be part of a research project testing the effects of a certain drug, one group of participants will be given the drug being
tested and the other group a placebo. Informed consent requires us to tell participants
that the design of this research involves some of the participants in the study being randomly allocated to the group being given the placebo and not the active drug. This might
be them. In this way, potential participants are not deceived into thinking that by participating in this study there will necessarily be the chance of any benefit to their individual
health. This is especially important if the participants are in poor health and looking to
participate as a chance of improving their health status. Further, it is equally important
to make clear that even if they are randomly allocated to the active group, and receive
the drug, the drug may have no, or even adverse, effects.
3. See Lahman 2018, pp. 79–81, for a good discussion of this.
4. See Chapter 5 for more about flexible and emergent study designs associated with qualitative research approaches.
5. The Declaration of Helsinki is “a statement of ethical principles for medical research
involving human subjects, including research on identifiable human material and data”
(World Medical Association, 2018, item 1).
6. In Chapters 6, 7, 8, and 9 we take a closer look at what observations are and can be used
for in your research design.
7. Defining exactly what Big Data is, is not easy as there is much confusion around the
use of this term. Most often Big Data is associated with the combination of many digital
traces. It is this combination of digital data traces which is “big”—in terms of the amount
and diversity of sources of those digital traces.
46 Research Design
8. “The internet of things, or IoT, is a system of interrelated computing devices, mechanical
and digital machines, objects, animals or people that are provided with unique identifiers (UIDs) and the ability to transfer data over a network without requiring human-tohuman or human-to-computer interaction” (IoT Agenda 2016).
9. See for example Tubbs (2018, paragraphs 6–7) for a discussion on big data and
marketing:
Publishers are gaining more data on their visitors and this allows them to provide more relevant advertising. Google and Facebook are already doing it with
their amazing targeting options but third party vendors will soon have the same
array of choices. You could target people based on their recent searches, articles they read, lookalike audience and so on. There is no limit to the impact big
data is making in the marketing world.
10. “Web scraping, also known as web data extraction, is the process of retrieving or
‘scraping’ data from a website. Unlike the mundane, mind-numbing process of manually extracting data, web scraping uses intelligent automation to retrieve hundreds,
millions, or even billions of data points from the internet’s seemingly endless frontier.”
Furthermore, “A web scraper is a specialized tool designed to accurately and quickly
extract data from a web page” (both quotes from scrapinghub, 2020).
11. Digitization is the process of converting information from a physical, analog format (for
example, paper based medical records, or scientific articles in printed editions of a journal) into a digital format (Gartner IT, 2018).
12. See the discussion of this in relation to the 2016 U.S. presidential election and the use of
big data (specifically, Facebook profiles) by the company Cambridge Analytica to individualize and better target political advertisements (Cadwalladr & Graham-Harrison,
2018; Detrow, 2018 ).
13. In 1932, the U.S. Public Health Service, working with the Tuskegee Institute, began a
study to record the natural history of syphilis in hopes of justifying treatment programs
for syphilis. Initially, 600 African American men were put in the study—399 with syphilis,
201 who did not have the disease. They were not aware that they were in the study or
that the study was about tracking the progression of untreated syphilis. The men with
syphilis received no treatment needed to cure their illness. Instead, they were told that
they were being treated for “bad blood” and received free medical exams, meals, and
burial insurance. The untreated men died, went blind, developed mental illness and
other severe health conditions arising from having syphilis. The duration of the study
was originally intended to be for a period of 6 months, but actually continued for some 40
years (https://www.cdc.gov/tuskegee/timeline.htm).
14. This example is explored further in supplemental Activity 2 at the end of this chapter.
3
DEVELOPING YOUR
RESEARCH QUESTIONS1
PURPOSES AND GOALS OF THE CHAPTER
This chapter focuses on the iterative process of developing the research questions that
underpin your research design. We emphasize the importance of constantly focusing, and
refocusing, your thinking about what your research problem actually is. Only when you
have worked that out can you begin to think about the specific aspects of that problem
that you will focus on in your study. These specific aspects will provide the basis for your
research questions.
We examine how the type of contribution to existing knowledge that you want your
research to make affects what form those research questions take. The role of hypotheses,
and the question of whether all research questions need to have hypotheses associated with
them, is discussed.
We highlight the importance of considering the feasibility of your proposed research
questions given the resources you have available to do the study. There is little point in
developing an excellent research question, the answer to which requires more resources
than you have available to you.
To demonstrate the type of thinking that sits in, and behind, the development of
research problems and questions, we include an invited contribution written by Maxi
Miciak and Christine Daum. This is an excellent reflexive account of how these researchers used iterative cycles of thinking to think big, plan big, but ultimately work out what it
was that they really wanted to focus their research on, as well as what was feasible in terms
of the resources available to them.
The goals of the chapter are to
• Establish what a research question is and the part that it plays in the research
design process.
• Explore different forms that research questions take, and why they do so.
• Explore the development of research questions as an iterative reflexive process
central to research design.
• Provide practical examples of such research question development.
• Explain how relevant research literature can be used when developing research
questions.
• Discuss the importance of thinking about the feasibility of a research question
when designing research.
47
48 Research Design
• Consider different forms that research questions can take.
• Highlight that the form that a research question takes reflects the type of
knowledge needed to address the purpose of the research.
• Introduce the idea of inductive and deductive reasoning and how that impacts
the way research questions are developed.
• Establish that hypotheses are related to specific forms of research questions.
BRINGING RESEARCH QUESTIONS INTO FOCUS
All research is designed to address some sort of problem. No matter what type of problem
your research focuses on, the overall goal of designing your research is to make an original
contribution of some sort to the existing body of knowledge, or ways of thinking, about
that problem.
The initial idea for what your research problem might be about can come from a number of places. These include “observing and asking questions about your everyday activities . . . social and political issues . . . the literature on a topic, or from theory. These areas of
course intersect” (Merriam & Tisdell, 2016, p. 76). There is “something” related to those
everyday activities, social or political issues, literature, or theory that you have observed or
read about that has caught your attention and that you want to know more about. This is
the area, or broad topic, that your research will focus on.
For example, the area, or broad initial topic, that your research will focus on might be
unemployment and its effects on people, or it might be recruitment and retention of employees in organizations. You will notice that these topics are very wide and therefore will need to
be focused more before it will be possible to decide on what you will research about them and
how. This means that having decided on an area of interest such as unemployment, you will
then need to think through what you want to know about the area and why. This is because
although you have identified the area of interest that will be the focus of your research
(unemployment), you do not yet have a research, or “researchable,” problem related to that
area. There is a difference between a researchable problem and problems that affect your
daily life. As Silverman (2001) notes, while social issues “like unemployment, homelessness
and racism, are important; by themselves they cannot provide a researchable topic” (p. 5).
Of course, this does not mean that a research problem cannot be related to a social
issue such as unemployment. What it does mean, however, is that the social issue of unemployment is not in itself a researchable problem. Rather, the social issue of unemployment
is a broad area of focus in which there are many possible research problems, each of which
has many possible research questions attached. To decide which research problem and
which research questions your research design will be developed to address, you will need
to think through what aspect(s) of that social issue you want to know about, as well as why
you want to know about that particular aspect or aspects.
For example, a particular aspect about unemployment that you might want to know
about is whether or not there is a link between where a person lives and whether they are
unemployed. In this case, you might want to know how many people are unemployed
in different socioeconomic areas of a city. Or is the aspect that you want to know about
related to the social issue of unemployment and how long people are unemployed for
after losing their jobs? Or why migrants are overrepresented in unemployment figures?
Chapter 3
•
Developing Your Research Questions
49
Or people’s perspectives about, and experiences of, being unemployed? Once you have
decided which aspect(s) of the topic you want to know more about, you can then think and
make decisions about what type of information you will need to enable you to address the
aspect of the topic you want to know more about. It is only at this point that you are ready
to begin to think about the type of methods you will need to use in your research design to
enable you to obtain that information.
In the following box, we model the type of thinking needed to put this process of
focusing your research topic on aspects of that topic and then developing research questions related to those aspects into practice.
PUTTING IT INTO PRACTICE
EXAMPLES OF THINKING THROUGH WHAT ASPECT(S) OF
THE TOPIC THAT YOU WANT TO KNOW ABOUT, AS WELL AS
WHY YOU WANT TO KNOW ABOUT THAT ASPECT/THOSE
ASPECTS
You have decided that you are interested in the area of long-term unemployment.
For example, your interactions with people who are long-term unemployed may have
highlighted that many people who cannot find employment are in fact both capable
of working and want to work. Consequently, you decide that you want to know more
about why people who want to work are not able to get jobs. In particular, you are
interested in how and why this group of people is excluded from the employment marketplace. This exclusion becomes the focus of your research problem. You then think
more about what exactly it is that you want to know about related to the problem of this
exclusion. For example, is it that in a competitive labor market, experience is required
for many jobs, so in effect people need to have had a job before they can get a job? Or is
it that people become increasingly disillusioned by their experiences of being rejected
when applying for jobs they are capable of doing? Once you have decided what it is that
you want to know more about related to the problem of exclusion, then you can develop
specific research questions designed to enable you to obtain a better understanding of
what it is that you want to know about.
Alternatively, you may be interested in the area of long-term unemployment
because you have read studies that show that long-term unemployment causes serious health issues for both the unemployed person and their families. Therefore, you
identify that what you want to know more about is health issues related to long-term
unemployment. You will then need to think more about what exactly it is that you want
to know more about related to these issues. Do you want to explore one or more specific issues that other studies have already identified such as depression or alcohol
abuse, for example? Or do you want to better understand what long-term unemployed
people in a particular situation of interest to you identify as health issues arising from
that situation? Do you want to focus on the person who is unemployed? Their partners? Or children in families where all adults are long-term unemployed? Or all of
the above?
Once you have decided this, you will go on to develop specific research questions related to the specific research problem or area of interest you have identified
related to your broad topic of interest. A research question is a question that a specific
research study sets out to answer. The answer to a research question will provide a
better understanding of an aspect of the research problem to which that research
question is attached.
50 Research Design
In addition to deciding upon what aspect(s) of your area or broad topic of interest (the
social issue of unemployment) and what type of knowledge you want your research to
contribute to understanding that area or broad topic, you need to ask yourself how your
research will relate to what is already known about the area of interest.
For example, will it build on what is known, will it replicate studies done in one context in another, will it pick up on recommendations for further research in the area, will it
challenge existing knowledge in some way, or will it contribute new ways of looking at an
aspect of unemployment that has been overlooked? All of these are possible contributions
to what is known, but it likely will not be possible to address all of them in a single research
study especially if you are a single researcher.
What the discussion so far in this section shows is that thinking through and about
your area of interest (e.g., the social issue of unemployment) enables you to identify what
aspects of that area you want to know more about and why. Those aspects (e.g., unemployment and people capable of and willing to work not getting jobs, or long-term unemployment and its effects on those affected and their families) give rise to your research problem.
Once you have worked out your research problem, then you can begin to think about what
specific aspects of that problem you will focus on in your study. Those aspects will be the
basis for the questions that your research is designed to address.
Consequently, to bring your research question(s) into focus, you narrow the scope of
your thinking by moving from a broad area to a problem related to that area and finally
developing research questions related to that problem. The terms research area, research
problem, and research question differ in terms of their level and scope of focus related to
what will be studied. Although not all researchers use such a clear-cut distinction between
these terms (for example, you will sometimes find research problem and research question
used interchangeably), we think it is useful to distinguish between them in terms of the
level and scope of focus that each relates to.
Therefore, in this chapter we use
• Research area to refer to the broad initial topic(s) or wide area(s) of interest giving
rise to the research.
• Research problem to refer to the part of that topic or area that we are interested in
and will focus our research on. It is a problem because there is something about
this issue or topic that we need to know more about.
• Research question to refer to the specific aspects of that part of the topic or area
that we are interested in. A research question is posed in such a way that it can be
researched, and when addressed, provides information able to contribute to the
body of knowledge relevant to the research problem and thereby provides better
understanding of the research problem that the question is related to.
TIP
THE NEED FOR CLARITY
When designing your research, you will need to make sure that you establish, and are
consistent in the way that you are using, terms related to that design such as research
area, research problem, and research questions.
Chapter 3
•
Developing Your Research Questions
51
Feasibility Considerations
Thinking about the specific research questions that will form the focus of your study will
also require you to think about whether you have the resources to enable you to address
these questions. Do you, for example, have sufficient time, expertise, and access to information or equipment or participants to be able to complete this research well? It is pointless coming up with what you consider a “perfect” research problem and set of associated
research questions if addressing those questions is not feasible. For example, your research
design is not feasible if you need 2,000 people in the study cohort to get enough statistical
power in your study, but you have no funding or way to recruit so many people.
The feasibility of a particular research design, that is, whether the research can be
done within the time and resources that are available, must be taken into account from the
outset of the research design process—including the development of the research questions. Not taking this into account may mean that the research may be limited by how
much of the information needed to address those questions you actually are able to obtain.
Such a limitation will mean that the significance2 of the research is reduced, or at worst,
the entire research project must be abandoned and cannot proceed at all.
Thinking, and being honest about the resources that you have available and whether
they are sufficient for what you are proposing to do, is an important part of designing your
research. Remember, if you are a student, then the resources that you usually will have are
yourself and your time, the guidance from your advisor, and perhaps a small amount of
funding. You need to take this into account when considering the scope of your research
problem and questions. This may mean reducing the scope of your research problem and
questions in some way. Consequently, Stake (2010) advises us to “[t]hink big, plan big, but
do a small, well-contained study” (p. 78).
How might you go about putting Stake’s advice into practice? We explore this question using the example of the thinking that was required to bring a student’s research
problem and associated research questions into focus.
PUTTING THE IDEA OF “THINK BIG, PLAN BIG, BUT DO A
SMALL, WELL-CONTAINED STUDY”3 INTO PRACTICE
A post-graduate student was interested in the broad area of the impact of higher education reforms on those working in the higher education sector. Her specific area of interest
was the effect of the 2015 higher education reforms in Norway on employees in the Norwegian
higher education sector.4
Here there is no doubt that the student has identified an area of interest and a draft
research problem related to that area. This is the higher education area and reforms in that
area and specifically those in Norway in 2015 and the effect that those reforms have had
on employees in the higher education sector. There is also no doubt that some focusing of
the research problem has already happened: the ands reflect that focusing. However, as the
proposed research problem stands now, the student would not be able to proceed with the
actual research. There is much more thinking work to do and decisions to be made before
sufficiently focused and feasible research questions about that problem can emerge.
A place for her to begin this focusing work is to ask herself what the key words are in
the statement of her research problem. For example, she identifies one of the key words
as effect. What has been the effect of the reforms? Thinking about this word it becomes
52 Research Design
apparent to her that more thinking is needed about what this word actually means. What
aspect of effect is she interested in, and why? Is she interested in a specific effect such as
economic effect? Effect on morale? Effect on behavior? Effect on outcomes, and if so, what
sort of outcomes? Or finding out what the people in the study say the effects are? Or are we
talking about all of the above? Asking these questions of the draft research problem indicates that there needs to be more thinking about what exactly it is that she wants to know
about in terms of “effect.” The thinking needed to unpack the word effect helps bring into
focus what it is that she actually wants to know something about at the end of her study.
This type of systematic and logical questioning will help focus and develop the
research problem. In addition, it will help to make sure that the research is feasible—that
is, that it can be done within the time and resources that are available. When producing
the first version of the research problem, namely How did the 2015 higher education reforms
in Norway affect employees in the Norwegian higher education sector in practice, the student
had “thought big” and “planned big.” However, to study the effects of the entire reform
on all employees in higher education, in the whole of Norway, is not feasible. By continuing to ask herself questions about all the key words in this draft research problem, she was
able to develop a feasible and significant research problem, ask feasible and significant
research questions about that problem, and do a small, well-defined study.
In the following box, we capture this reflexive process.
PUTTING IT INTO PRACTICE
HOW IDENTIFYING KEY WORDS HELPED A STUDENT FOCUS
AN INITIAL PROPOSED RESEARCH PROBLEM INTO A SMALL,
WELL-DEFINED STUDY
A student initially identifies her research problem to be
How did the 2015 higher education reforms in Norway affect employees in the
Norwegian higher education sector in practice?
She identifies key words in that initial research problem statement in order to
focus it further. She identifies the key words to be effects, employees, and the 2015
higher education reforms in Norway. She then asks herself questions about those key
words:
•
•
•
What effect(s) is she interested in and why? For example, economic effects?
Effects on morale? On behavior? On performance outcomes? Or all of the
above?
Which group of employees is she interested in knowing more about? For
example administrators, managers, academics, or all employees?
Which part of the reforms is she interested in knowing more about? One
specific aspect of the reforms, or if more than one aspect, which ones?
When these decisions about the focus and scope related to each key term of the
initial research problem statement have been made, she continues to ask questions
of those decisions:
•
•
Why has she chosen those specific effects, employees, and aspects to study?
What is it that she wants this research to contribute to the body of knowledge
about this problem? In other words, what is the significance of her research?
Chapter 3
•
Developing Your Research Questions
53
These decisions give rise to her research questions.
Such iterative thinking through and about the topic of interest enables the student to develop a feasible and significant problem and associated research questions. It enabled her to capture a small part of a very large area and do a small,
well-contained study.
USING THE LITERATURE WHEN DEVELOPING
RESEARCH QUESTIONS
The process of iteratively thinking with, and about, the research problem, and the questions associated with it, that you wish your research to focus on, does not occur in isolation. Throughout this process of thinking, you will need to be aware of, and interact with,
what is already known about the problem and how it is understood. In other words, you
will need to join conversations in the body of knowledge that has been built up by the
work of others in this area.
Richards likens entering these ongoing conversations to “soaring over the landscape”
(Richards, 2015, p. 20) in order to get an overview of what is “out there,” or already known
about, the area or issue that has caught your attention and that you want your research to
contribute to. Once you have got a sense of this landscape, you can make decisions about what
parts of that landscape warrant a closer look and might form the basis for the problem that your
research will be designed to address. Richards (2015) describes this process as “locating something small that can be captured” (p. 20) in that landscape. This locating of “something small”
is the beginning of bringing your research problem into focus. Of course, as we have pointed
out earlier, there is still much thinking to do about what aspects of this “something small” you
are particularly interested in knowing more about and why. These aspects are likely to become
the aims and research questions for your study, providing guidance and focus for your study.
TIP
SMALLER IN SCOPE AND FOCUS IS NOT NECESSARILY A BAD
THING WHEN DESIGNING RESEARCH
You may be worrying that focusing your research on something small will not result
in a worthwhile, or “as good as,” contribution to the existing body of knowledge about
your problem, as researching something big will. This is because you think that your
research will be less significant. The flaw in this type of thinking is that “something
big” and “something small” do not refer to the significance of the research. In fact,
smaller in scope and focus actually means that the research has a much better chance
of being completed and completed well, thereby making the findings of the smaller
study more significant than a larger in scope study not able to be completed well.
What Is Missing in the Existing Body of Knowledge in the Literature
Related to Your Problem?
One way to locate “something small that can be captured” (Richards, 2015, p. 20) is by
exposing gaps in the existing body of knowledge about the particular area we are interested in researching. These gaps may then form the basis for our research problem and
associated research questions. In such a scenario “[w]e read the literature, identify what is
missing or mis-stated or even under-emphasized, and then formulate a question that links
54 Research Design
these ‘gaps’ to our research interest” (Swaminathan & Mulvihill, 2017, p. 14). This is a
strategy used by many researchers when developing their research questions.
When developing research questions, building on or challenging existing knowledge
in an area is important. As Wentzel (2018, pp. 29–30) reminds us, “there will always be
areas where our understanding is no longer adequate, and the existing literature insufficient. . . . research always plays catch-up to the existing reality, and thus, new contributions are always needed.” He points out that research questions can arise when previous
research is inadequate or insufficient because
• Something has never been researched before, theoretically or empirically
• Something has been researched before, but the body of knowledge is incomplete
• Disagreement exists between scholars in the field of interest about what is going on
• There is inconsistency in the findings in the area
Caution: Mind the Gap
Whether or not filling a gap is a useful and worthwhile strategy to undertake is related to
the significance of the research. Will addressing a particular gap you have identified make
a substantial contribution to the existing body of knowledge? Or could it simply be considered addressing a gap for the sake of addressing a gap? This type of reflexive questioning
will force you to think about why a particular gap is something that needs to be known
about and how that knowledge will be used. In other words, it forces you to think about
the significance of addressing that gap.
Further, there is a danger that only developing research problems and questions by
addressing gaps in the knowledge that we already have about an area can result in a somewhat static, or bounded, view of what knowledge in that area is, or can be. This is because
the purpose of the research becomes filling those gaps in that existing knowledge—not
necessarily challenging or expanding the wider body of knowledge itself.
Limiting possible problems and questions to such gaps can presuppose that the total body
of existing knowledge is sound, true or a given (Wentzel, 2018). However, is that necessarily
the case? Is everything in the body of knowledge related to the research area you are interested
in “sound,” “true,” or a “given”? Asking this question can result in a different way of thinking
about connecting your research problem and questions to the existing body of knowledge. For
example, is this the only way that that area can be understood? Are there other ways of thinking with theory about the area? Can the area be studied in different ways methodologically?
Beyond the Gap
Rethinking the existing literature in this way opens up the possibility for different research
problems and questions about your area of interest, which in turn prompts very different
research questions than those focused on filling gaps in existing and dominant ways of
thinking about that area (Alvesson & Sandberg, 2011, 2013). Consequently, the contribution of your research to the existing body of knowledge in the area might be to challenge,
alter, or supplement that existing knowledge and thereby introduce new theoretical or
empirical understandings related to that area of interest.
Uncovering the often unspoken assumptions in existing bodies of knowledge about
a problem or concept related to that problem involves thinking about and identifying in
Chapter 3
•
Developing Your Research Questions
55
what way that problem or concept has been studied previously, and why researchers have
chosen to study the problem in that way and not another. The way other researchers have
investigated the problem you are interested in has been shaped by their assumptions about
what that problem is, what can be known about that problem, and how we can gain or
develop such knowledge. These assumptions have imposed limitations on what kind of
knowledge that can emerge from their research. Perhaps other ways of investigating a subject matter could lead to a kind of knowledge that is currently missing in the area? Could
it be that this missing knowledge will make an important contribution to the way that the
problem is thought about and strategies are developed to address that problem?
Activity
Putting It All Together in a Sentence
If you are in the process of designing research, try to tell a fellow student or a family
member what it is that you are interested in learning more about by conducting the
research using this sentence structure:
I am studying [fill in your research area or broad topic of interest] to investigate [fill in your research questions] in order to [fill in what it is that you
want the research to contribute to the body of knowledge about the research
problem, i.e., what the significance of your research is].
If you find this hard to do, then you know there is more thinking to do before you
have developed your research questions sufficiently enough so that you can make
informed decisions about other aspects of your research design. These are aspects such
as which method(s) to use to answer those research questions and why, and who you
will collect data from (or about) in your research study and why.
DIFFERENT FORMS OF REASONING AND HOW THEY
SHAPE THE FORM THAT RESEARCH QUESTIONS TAKE
Research questions can take different forms. The form that a research question takes is
related to the goal or purposes of the research. The goal of some research is to test existing
theories or ideas such as the older a long-term unemployed person is, the harder it is for
that person to get back into the workforce. To test this theory will require us to employ a
different type of reasoning than if our goal is to understand what the experience of older
long-term unemployed people is when they interact with the labor workforce marketplace.
In the first instance, we will need to employ deductive reasoning whereas in the second
instance we will predominantly employ a type of inductive reasoning.
TIP
REMEMBER
Different types of reasoning, and therefore different types of research designs, and
the different forms of research questions that underpin those designs reflect different
goals for doing the research.
56 Research Design
Deductive Reasoning
In research based on deductive reasoning, empirical evidence is collected and used to test
an existing theory or a theoretically based assertion. The researcher begins by thinking
about that theory, or aspects of that theory, and predicts what the empirical evidence,
that is, the data collected, should show if that theory, or those aspects, are correct. The
theory being tested guides the researcher’s thinking about what kinds of empirical evidence are relevant to look for. Thus, the research is deductive as it is designed to test
empirically those predictions and therefore that theory. The goal of both the research,
and the theory that it is designed to test in some way, is to generalize, using statistically based procedures, about “all instances or cases of a particular type” (Maxwell &
Mittapalli, 2008, p. 877).
For example, if you are studying effects of long-term unemployment, you might want
to test the theory or empirical findings you have come across in the literature that suggest that the longer a person is unemployed, then the harder it is for them to reenter the
workforce. In particular, you might have decided that what you are specifically interested
in is whether that theory or finding applies or is correct across all age groups and all
occupations. The form that your research questions take will reflect that decision. For
example, if you want to know whether it is harder for long-term unemployed people in
some age groups compared to other age groups to get back into the workforce, then you
will develop a hypothesis related to your research question that will enable you to answer
that question.
A hypothesis is a testable statement about what we predict empirical evidence to reveal
about a problem of interest in a specific situation in a specific context (and in the social sciences often for a specific group of people). For example, “It is harder for long-term unemployed people 50 years and over to get back into the workforce than it is for people aged
49 years and less.” This hypothesis focuses on the age of long-term unemployed and their
ability to reenter the workforce, and tests if there is a relationship between them. “Age of
long-term unemployed” and “ability to reenter the workforce” are examples of what are
known as variables.
A variable refers to whatever it is that your research is designed to measure. For example, a variable may be a characteristic of an individual such as age, weight, or height. Or,
a variable might be level of achievement, attitude toward vaccine programs, or amount
of exercise per week. In other words, a variable is something of interest that varies in relation to either the people or the places that you are studying. For example, if the variable of
interest is “age,” the value for a specific individual in the study in relation to that variable
could be “34.” For a different individual, the value of the variable “age” can, for example,
be “50.” On the other hand, if the variable of interest in the study is “area of work,” this
variable could have values such as “administration and business,” “health and well-being,”
“technology,” “education,” and “hotel/tourism industry.”
That a hypothesis is “testable” means that there must be statistical procedures
available to analyze the empirical evidence a researcher obtains in order to demonstrate support or not support for the statement made in the hypothesis about those
variables. For example, does the evidence support that being 50 or over and long-term
unemployed makes it harder to reenter the workforce? Since the focus is on establishing the relation between those specific variables of interest, answers will be in the form
of whether or not there is a relation (determined statistically) between those variables
Chapter 3
•
Developing Your Research Questions
57
and if there is such a relation, what it is, and how significant it is (in terms of statistical
significance).
Thus, “[d]eveloping hypotheses involves . . . moving from the theory back to the sort
of evidence supporting it” (Lewins, 1992, p. 48). This way of thinking and the reasoning
that underpins it is deductive.5
Inductive Reasoning
In research based on inductive reasoning, the goal of the research is to build theoretical and
empirical concepts and understandings. Thus, inquiry based on inductive reasoning does
not set out with a predetermined or a priori set of theoretical concepts to test. Instead, the
researcher poses questions such as, What is going on here? What do people understand
this to mean? How did it come to be perceived or understood in that way? Why that way
and not another? Such “questions address the content of meaning, as articulated through
social interaction and as mediated by culture” (Gubrium & Holstein, 1997, p. 14).
For example, if you are studying effects of long-term unemployment, you might want
to find out more about what it is like to be 50 and above, long-term unemployed, and finding it difficult to get a job. To do so you might ask this group of people who are long-term
unemployed about their experience of trying to find employment. What is their perception
of what happens when they apply for jobs? In their view why does this happen? What is the
effect on them? Once again, the form that your research questions take should reflect the
decision that you have made about what you want to find out about. Therefore, it is likely
that you will use some form of open-ended research questions that enable the emergence
of rich and qualitative descriptions and interpretations of the perceptions or experiences of
people about a specific aspect(s) of the everyday context(s) in which they exist.
Such an approach does not focus on specific variables identified in advance by the
researcher as being the only ones of interest. Instead, the form that the research question(s)
takes is designed to enable deeper and richer understandings about something that is happening in a particular social context to emerge.
Open research questions are designed to build understandings of concepts and theories related to the focus of the research. Therefore, research questions will need to be
formed in such a way that enables the answers to them to provide the type of knowledge
needed to build those concepts and theories.
This type of research question enables answers in the form of “thick interpretation”
(Denzin, 2001, p. 52)6 of the in-depth descriptions or constructions of the understandings or experiences of people in their everyday social context(s) about what it is that the
research is focused on. Thinking with and through this thick interpretation can develop
existing, or provide possibilities for different, theoretical concepts and understandings
related to the area and problem that the research is focused on. This way of thinking and
the reasoning that underpins it is inductive. Therefore, such research questions will not be
hypothesis driven. This is because the aim of the research is to “build, rather than to test
concepts, hypotheses, and theories” (Merriam & Tisdell, 2016, p. 84).
We will return to take a closer look at, and develop further, many of the points raised in
this section in parts of Chapters 4, 5, 6, 7, 8, 9, and 10. There is still a lot more thinking to
do about these matters and connections to make. The ideas that we have introduced here
provide you with a good foundation for that discussion. For example, in Chapters 8 and 9,
we include a discussion of how hypotheses can be developed from research questions and
58 Research Design
then tested. In Chapters 6 and 7, we discuss how to develop open-ended questions and
analyze the information you receive from them.
TIP
TO SUM UP
The form that a research question takes arises from the goal of the research. Does the
researcher aim to test existing theory or concepts derived from it that are related to
the research problem that the researcher is interested in knowing more about (deductive)? Or is the researcher more interested in trying to build theoretical concepts and
understandings about the area (inductive)?
PUTTING ITERATIVE AND REFLEXIVE RESEARCH QUESTION
DEVELOPMENT INTO PRACTICE—LEARNING FROM OTHERS
At this point, you may feel a little overwhelmed at the prospect of developing your research
problem and questions. How can you actually put the principles we have discussed into
practice? This type of discussion often remains hidden in most textbooks and descriptions of the final form research designs take. Therefore, we miss information about what
others have learnt when developing their research problem and questions. How did they
transform their thinking about their initial research problem into research questions that
guided their study? Such information can provide us with pathways to follow, dead ends to
avoid, and experiences to learn from when we are designing our research and the questions
that will underpin that research. Yet we miss insights into this process and hardly ever get
the full story of what happened from which we can learn. Therefore, we decided to ask two
researchers to share this information with us.
At the time of writing this book we knew that researchers Maxi Miciak and Christine
Daum had relatively recently completed excellent PhD studies that their principal advisor
spoke very highly of. Therefore, we decided to ask them if they would be willing to tell
their respective stories about how they undertook the process of developing and focusing
their research questions and what really happened along the way.
We made two key requests of Maxi and Christine. The first was that it must be the
real story, not the sanitized version of the story where everything seems to fall into place
effortlessly and the questions just seem, almost miraculously, to have appeared. The second request was that, if possible, they include a discussion of the various versions of their
research questions and the reasons for the changes that they made to those questions
along the way. We wanted them to expose the thinking that they did with, and about,
those questions. Such thinking usually is not visible in our accounts of our research or our
research design.
They agreed to tell their stories. Each of them addressed our requests in their own way.
The result is two unique analyses and overviews that enable us to get inside of their thinking when they were developing their research questions. This reflexive writing is a treasure
trove of tips and inspiration for how to navigate this difficult and at times frustrating
process. The next section of the chapter, “Scratching the Underbelly of Research Design:
Developing Clear Research Question(s),” is their account of putting Stake’s (2010) advice
to “[t]hink big, plan big, but, do a small, well-contained study” (p. 78) into practice.
Chapter 3
•
Developing Your Research Questions
59
SCRATCHING THE UNDERBELLY OF RESEARCH
DESIGN: DEVELOPING CLEAR RESEARCH QUESTION(S)
Reflections by Maxi Miciak and Christine Daum7
A Bit About Us, Our Projects, and What Makes Us “Qualified” to
Write This
Research always occurs within context, and the researcher is a part of that context. Given
that we are writing about research, we feel the same rule applies. Therefore, before diving
into speaking about research questions, we, Maxi Miciak and Christine Daum, thought it
appropriate to introduce ourselves.
Maxi is a physical therapist who spent a good portion of her clinical career working with people managing chronic pain. After almost 14 years in clinical practice, she
began doctoral studies in rehabilitation science to research a significantly understudied
phenomenon in physiotherapy—the therapeutic relationship between the therapist and
patient. She did a qualitative study using interpretive description as her methodological
orientation8,9 because she wanted to provide clinicians practical guidance when navigating
relationships with patients. To do this, the foundational components of the therapeutic
relationship specific to physiotherapy needed to be identified and described. Her final
research question was What are the conceptual descriptions regarding the components of the
therapeutic relationship in physiotherapy? Maxi is currently a postdoctoral fellow studying
research impact assessment, that is, the assessment of the influence research has on policies and practices and further downstream to health outcomes and social and economic
impacts. Her research focuses on operationalizing patient-centered care by developing,
implementing, and evaluating care models that impact the patient–practitioner therapeutic relationship, including how health services and policies support this relationship.
Christine is an occupational therapist with an interest in aging. She has spent most
of her clinical career working with older adults in community and institutional settings.
During this time, she observed the impact of environments, including neighborhoods, on
older adults’ function. Christine pursued a master of science in health promotion and a PhD
in rehabilitation science to better understand the relationships between the environments
in which older adults are situated and their performance of daily activities—those activities that people engage in to look after themselves, enjoy life, and contribute to society. The
purpose of her research project was to explore the daily activities of older women living in
inner city (disadvantaged) neighborhoods. Her research question was How do the neighorhoods of inner city-dwelling older women influence their daily activities? Like Maxi, she used
qualitative methods and interpretive description to explore her research question. Christine
is currently the coordinator of a transdisciplinary research program focused on creating,
deploying, and evaluating technologies to support older adults’ mental and cognitive health.
Drawing on our experiences, we understand that the simplicity and neatness of a
well-crafted research question often mask a process that can be daunting, exhilarating,
and at times, downright depressing! We often felt unworthy going through this process;
the term fraud was not far from our thoughts. However, we both feel qualified to write
about the “underbelly” of developing research questions because we have lived, learned,
and supported one another along the way. Also, we now mentor students and vicariously
relive the experience with more wisdom, patience, and practical suggestions.
It is important to note that we had different experiences coming to our questions,
as we describe below, but we each eventually got to a question that set the stage for our
60 Research Design
projects. And although different, we both experienced the process of exploring, generating, and landing on the research question as a necessary struggle deserving of more time
and thought than is often acknowledged. We also recognize that even though we each
landed on a single research question, you may have more than one question depending on
the complexity of the phenomenon you are studying. Respecting this, we will use the term
question(s) to represent the breadth of research possibilities.
In the Beginning There Was . . .
You have a brilliant idea that is important for advancing your field—everyone has told
you so. You have been talking about it, maybe for years. You may have even had visions
of future fame, jet-setting to conferences as the keynote and preeminent mind that was
brilliant enough to see and meaningfully address this issue. Then, you are rudely awakened from your daydream by your committee asking, “What is your research question(s)?”
You open your mouth. And then, the nightmare—words don’t come out—they are not
even formed in your brain let alone making it to your throat and out of your mouth. You
know it is there, but it’s a jumbled mess that you can’t seem to untangle. It is like grasping
smoke—you see it but it is not solid, and you can’t hold onto it. And you are confused and
even a bit ashamed that you assumed it was formed and right there to be plucked. You
painfully stumble through articulations that don’t quite fit, stabbing at air, as the committee members look at you, unblinking, waiting.
If you are like us, after experiences such as the one described above, you will go
through days of doubt, wondering whether you actually ever really knew the research issue
and if it is, in fact, relevant. Because if you did know the issue and its importance, then
surely the question would slide off of your tongue smoothly and articulately. However, we
want to share the hard lesson we learned: It does not work this way. Much more thinking
and often frustrating work is required to get the clarity you need.
Generating the Question(s)
Shocked into reality, you now must begin the process (and we repeat, process) of generating
and regenerating your research question(s). And as you might have guessed, there is no one
way to begin or complete a process influenced by various factors such as previous research
experience, existing knowledge, and even your supervisor’s methodological leanings. The
question(s) exist in a context. For instance, Maxi needed to consider the current state of the
physiotherapy profession, including practice models within physiotherapy and across different health care professions, as well as the potential to implement the findings of her study
in clinical research and practice. An example of a choice made from considering her context
came in the language she used in the question. The term components was used versus a term
such as dimensions. While the latter term may have resonated with researchers, components would likely ring truer with clinicians and translate better into practice. In addition,
Maxi’s process was affected by the fact that she had no previous research experience, so was
unaware of the challenges that lay ahead. Her supervisor had not used qualitative methods,
having exclusively used quantitative methods, so naturally they both assumed she would be
doing a quantitative study. However, when her doctoral committee pressed her to articulate
her interests, and specifically, what really needed to be known, it was clear that a qualitative, not a quantitative, question was needed. Maxi learned a key lesson—methodology and
methods follow the question(s), not the other way around. The series of iterations of the
questions and the reason for the change is captured in part in Table 3.1.
•
What is the construct and content validity of
a self-report measurement tool specifically
designed to assess the quality of therapeutic
relationship in physical therapy?
What are the common themes regarding the
definition of positive therapeutic relationship
in Edmonton private practice clinics that arise
from (a) adult clients’ descriptions of receiving
physical therapy treatment for musculoskeletal
injuries and (b) physical therapists’ descriptions
of providing treatment to clients with
musculoskeletal injuries?
What are the common themes regarding the
components of positive therapeutic relationship
in Edmonton private practice clinics that arise
from (a) adult clients’ descriptions of receiving
physical therapy treatment for musculoskeletal
injuries and (b) physical therapists’ descriptions
of providing treatment to clients with
musculoskeletal injuries?
•
•
•
A preliminary literature search that revealed minimal and disjointed research on the conceptual
understanding of the therapeutic relationship in physical therapy (i.e., What is it?)
My insights from clinical experience as a physiotherapist
This quantitative question(s) was added. However, feasibility considerations limited my ability
to fully assess the psychometric properties of a tool because I was now answering qualitative
questions as well. I decided to limit the assessment to construct and content validity. I had not
decided which tools I would be assessing.
I chose the term components over terms such as dimension and elements because it captured a
practical perspective important for the overarching purpose of the study—to provide clinicians with
knowledge for informing decision-making.
◦
◦
Qualitative methodology was required to explore and answer my questions surrounding the
knowledge base of the therapeutic relationship. My committee members and supervisor probed
me about “what needs to be known?” in the discipline. My responses to their probes stemmed from
the following:
My supervisor’s primary methodological lens was quantitative.
We assumed the therapeutic relationship construct in psychotherapy is transferrable to physical
therapy.
•
•
•
Mid-stage
of proposal
development
•
•
Assessing clinical outcomes requires valid and reliable measurement tools, and there was no tool
in physical therapy to assess the quality of the therapeutic relationship.
•
•
Pre-formal
proposal writing
No question(s) had been formally constructed
but preliminary thoughts were that it would
address assessing the psychometric properties
of established therapeutic relationship
measurement tools from psychotherapy in the
physical therapy context.
Changes and the Rationale
Question(s)
The Evolution of Maxi’s Research Question(s)
Timeline
TABLE 3.1 ■
Chapter 3
Developing Your Research Questions
61
•
Dissertation
What are the conceptual descriptions regarding
the components of the therapeutic relationship?
•
•
•
My supervisors and I agreed that I could keep the question broad so as not to impose limitations on
the nature of the findings and their future transferability to other populations and settings.
The objective of this research project was to identify and provide in-depth descriptions
of the key components of the therapeutic relationship relevant to physical therapists and
adult patients managing musculoskeletal conditions in private practice clinics.
The question became very focused, with the population and setting (physical therapists and
patients managing musculoskeletal conditions in private practice clinics) written into the study
objective:
I dropped the quantitative question for two reasons:
1. My committee members and I became increasingly aware that the qualitative study would
be quite involved. Therefore, it would not be feasible within a doctoral project to address all
questions.
2. We secured a qualitative methodologist as a co-supervisor, given my supervisor had no
experience with qualitative methods. This, combined with feasibility considerations, allowed us
to “let go” of the quantitative question(s).
My language when formulating the questions began to change. By this point, I had decided
interpretive description would be the appropriate methodological approach to respond to the
qualitative questions, so I began integrating language consistent with the method (i.e., “conceptual
descriptions” versus “common themes”). I also revised language around patient characteristics to
make it broader and to hold it open (e.g., “conditions” versus “injuries”).
•
•
Final proposal
What are the conceptual descriptions regarding
the components of the therapeutic relationship
in physical therapy arising from (a) adult clients’
descriptions of receiving physical therapy
treatment for functional limitations due to
musculoskeletal conditions and (b) physical
therapists’ descriptions of providing treatment
to clients with musculoskeletal conditions?
Changes and the Rationale
Question(s)
Timeline
62 Research Design
Chapter 3
•
Developing Your Research Questions
63
Christine, on the other hand, worked as a research assistant on several projects and
teams that used qualitative and mixed methods. She witnessed the teams untangle the web
of decisions that underpinned the development of research questions. As such, Christine
expected that she might encounter similar struggles and thus devoted 3 weeks to interrogating assumptions that surrounded the research issue she was exploring. As the area of interest
(that is, the influence of inner-city neighborhoods on older women’s daily activities) could be
explored from several disciplines and subdisciplines (occupational therapy, occupational science, environmental gerontology, geographical gerontology, human geography, to name just
a few), she first needed to understand the assumptions embedded in each discipline and how
these would shape the research question. She filled several whiteboards with potential questions (and components of questions), brainstorming words and ideas to try to get at what
she wanted to study. Christine then disassembled the questions into their components and
substituted words and concepts with others. She examined each of these, eliminating those
that truly were not what she desired to study and rewriting others that seemed closer to her
intent. For Christine, articulating a question was a lengthy process that involved being judicious about what she did not propose to study just as much as it was about generating a clear
research question. The series of iterations of the question is captured in part in Figure 3.1.
FIGURE 3.1 ■
The Evolution of Christine’s Research Question
What are the realities of daily life of older women who live in disadvantaged
neighborhoods?
What physical and social features of disadvantaged neighborhoods affect older
women's community activities?
How do the physical and social features of inner-city neighborhoods shape older
women's everyday activities?
How do the neighborhoods of inner-city-dwelling older women impact their daily
activities?
Embracing Rather Than Running From Critique
Central to the processes we have described in the development of our question(s) was a process of reflexively critiquing our own question(s). Asking question(s) derived from different
viewpoints or emphases will challenge you to consider your question(s) from different angles:
• Who will want to know about your study?
• What are the assumptions that (will) underlie the question?
• What question would your key stakeholder(s) ask?
• How would the question sound to X (e.g., patients) versus Y (e.g., therapists)?
• What do your key stakeholders need to know?
64 Research Design
• Can this question be answered in this context?
• Who (e.g., target population, discipline) would be the harshest critics of your
question? What would they say? Would it matter what they say (i.e., would you
revise your question in light of their feedback)?
These questions can help greatly in the refinement of your research question(s) and
address the issue that we call illusions of clarity, meaning you believe you have generated
clear question(s), until you unveil it to different audiences. For example, the instructor in
your research design class, who critiques your question(s) with more questions, may be the
first to reveal the illusion. Revising the question(s) from this feedback, you believe you
are set; no more holes could possibly be poked in the redeveloped question(s). So now you
are ready for the “big reveal” to your supervisor. The result? You again doubt whether you
know what you are studying and realize that there are many more questions that could
be asked and answered. Humbled yet resilient, you integrate your supervisor’s feedback
to finalize what you are sure are “the question(s)” (your supervisor is on board so who is
going to object?), only to hand off to a supervisory committee that has content and context
knowledge that likely influences the way the question you have come up with is viewed.
We could go on, but we think you get the point. Different audiences will provide critical
feedback that, although devastating in the moment, will ultimately result in a question(s)
that is more potent than it was to start.
Landing on a Question(s)
Ultimately, you will have to eventually “land” on a question(s). There is no such thing as a
perfect or finite question(s). However, your question(s) must be clear enough to move forward, in the right direction, to inform the rest of the design planning. This carefully considered question(s) must be justifiable. You must be able to articulate why the question(s)
is worthwhile answering, that it can indeed be answered, how it differs than what has
already been explored in the discipline, and how it potentially fills a knowledge gap. The
study required to answer the question(s) must also be feasible. In other words, it must be
within your capabilities, financial resources, and time span available. The nature of the
question(s) may bring up other issues of feasibility such as relationships with communities
and buy-in from stakeholders. For example, although conducting research within a specific marginalized community for the purpose of changing policy is admirable, it may not
be realistic for a researcher who has no existing relationships with the community nor with
policy influencers and has only one year to complete the study.
Landing on a question(s) does not mean it is written in stone. There is a certain degree
of arrogance in assuming this process is sterile, requiring only one pass through the gate.
As you hash out the rest of the design, you will circle back to see if your question(s) accurately reflects your design thinking, and vice versa. It is this iterative process in both
question formation and research design that is key to a well-designed, coherent research
project.
Key Messages
• Never underestimate the importance of a clear question(s). A clear, congruent
research question(s) is essential to the integrity of the research project and sets the
direction for the rest of the design planning.
Chapter 3
•
Developing Your Research Questions
65
• There are no perfect or finite questions. Developing a research question is a
dynamic iterative process that involves interrogating assumptions and positions.
• Researchers should expect and welcome the struggles that come with landing on
a feasible and justifiable question.
• When you get lost and forget what you are studying (and you will), your research
question is the touchstone, the North Star, that will get you back on track.
End of invited contribution.
CONCLUSIONS
Like the development of the overall research design itself, developing research problems
and questions is an iterative process requiring you to think reflexively. This reflexive process helps you refine your level of focus as you think through your research area and work
out what the problem in this area is that your research will focus on, and why. Once you
have decided this you are able to develop research questions, which will provide you with
the type of knowledge that you will need to find out what you hope to know something
about at the end of the research undertaking.
To arrive at a good research question, you have to make a lot of decisions about your
research design and the various iterations of the research questions which form part of
that design. There is no recipe to follow, or one size fits all, for “how to” develop a research
problem or formulate research questions related to it. It is impossible to give hard and
fast rules about, for example, how many research questions are needed, what form they
should take, and how they should be written. This is because research questions, as well as
any “rules” about them, make no sense if they are detached from the thinking about what
the study is trying to find out about and why.
Nevertheless, although there is no “recipe” to follow, or one-size-fits-all research question
template to fill in, there are some key questions to ask yourself, and think about, that can
help you focus your research problem and related questions. They include the following:
1.
What is the broad area (could be empirical, substantive, theoretical, or all of the
above) that your research is located in?
2.
What is it about this area that you are interested in—what is the problem?
3.
What is it that you are specifically interested in in relation to this problem and why?
4.
What contribution do you want your research to make, and to whom, at the end of
your study?
5.
Do you have the time and resources to be able to do this study? In other words, is it
feasible?
6.
Is this a problem that should be studied at all? If so, then are these the questions that
should be asked about it? In other words, have you asked questions of your research
questions in terms of the ethical considerations that we discussed in Chapter 2 and
which need to be thought through, in all areas of the research design, including the
questions asked about any problem?
66 Research Design
Asking questions such as these is a key part of developing a well thought-out, focused,
ethical, feasible, and responsible research design. It forces you to ask yourself about the
assumptions that you bring with you to the research design table. This includes assumptions about what a “research” problem is, or the form that a “good” research question
must take or be in. Thinking through, and asking questions of those assumptions, in turn
forces you to think about how and why they affect what type of problems you will focus
your research on, and the type of questions you will ask. What do these assumptions
enable you to ask questions about? What other questions, and types of questions, might
be asked? What do you gain or lose from choosing the question(s) you have decided on?
How does your choice of a certain type of research problem or questions impact on the
way that you design your research—for example, the type of methods that you will use to
obtain information to address those questions?
Choices about what the problem and associated questions that your research is being
designed to address will affect the way you think about, and design, all parts of your
research. For example, what methodological approach and associated methods you
choose to use in your research design will be based on what type of knowledge, and in
what form, will be needed to answer or address those questions. We return to this point to
discuss it more fully in the next chapter.
SUMMARY OF KEY POINTS
• Developing your research problem and questions when designing research is an
iterative, nonlinear process.
• You will use empirical and theoretical literature to assist in the identification or
development of your research problem and questions.
• Theoretical, methodological, and ethical considerations shape the nature, focus, and
form of the questions related to that problem.
• A research question is posed in such a way that it can be researched, and when
addressed provides information able to contribute to the body of knowledge relevant
to the research problem that the question is related to.
• The form that a research question takes affects other parts of the research design such
as what methods are proposed to be used in that design and why.
• Developing research questions requires reflexivity on the part of the researcher to
recognize and acknowledge the impact of the underlying assumptions that shape
their perspectives and understandings of what a research question is, and the form
that that question must take.
• We can learn much from the experience of others—how exactly did they develop
their research questions and why?
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
deductive reasoning
feasibility
hypothesis
inductive reasoning
Chapter 3
research area
research problem
•
Developing Your Research Questions
67
research question
variable
SUPPLEMENTAL ACTIVITIES
1.
If you are in the process of designing your own research, think through the following
questions that will enable you to develop a small well-defined and feasible study:
• What is the broad area (could be empirical, substantive, theoretical, or all of
the above) that your research is located in?
• What is it about this area that you are interested in—what is the problem?
• What is it that you are specifically interested in in relation to this problem
and why?
• What contribution do you want your research to make, and to whom, at the
end of your study?
• Do you have the time and resources to be able to do this study? In other
words, is it feasible?
• Is this a problem that should be studied at all? If so, then are these the
questions that should be asked about it? In other words, have you asked
questions of your research questions in terms of the ethical considerations that
we discussed in Chapter 2 and which need to be thought through, in all areas
of the research design, including the questions asked about any problem?
If you manage to address all of these questions, then you are well on your way to
being able to develop a well thought through research design. You are also ready to
make decisions about how you will do the research.
2.
After reading this chapter, discuss with your fellow students or researchers why it is
important that, when designing research, the research question(s) drives the choice of
methods rather than your choice of methods driving the research question(s) you ask.
FURTHER READINGS
Creswell, J. W., & Creswell, J. D. (2018) Research questions and hypotheses. In J. W. Creswell
& J. D. Creswell, Research design. Qualitative, Quantitative and Mixed Methods Approaches (5th
ed., pp. 133-146). SAGE.
Flick, U. (2020) From Research Idea to Research Question. In U. Flick, Introducing Research
Methodology (3rd ed., pp. 61-77). SAGE.
Merriam, S. B., & Tisdell, E. J. (2016) Designing Your Study and Selecting a Sample. In S. B.
Merriam & E. J. Tisdell, Qualitative Research: A Guide to Design and Implementation (4th ed., pp.
73-103). Jossey-Bass.
NOTES
1. This chapter includes the following invited contribution: Miciak, M., & Daum, C. (2023).
Scratching the underbelly of research design: Developing clear research question(s).
In Cheek, J., & Øby, E. (2023). Research Design: Why Thinking About Design Matters
(pp. 59-65). SAGE.
68 Research Design
2. Depending on the type of research problem, questions, and methodology used in the
research, significance might mean statistical significance, or significance in terms of
the contribution that the study makes to the existing knowledge in the research area, or
both. See the discussion in Chapter 5.
3. From Stake (2010, p. 78).
4. This student was one of the authors (Elise Øby) who was returning to postgraduate study
and undertaking a master’s study in social sciences after having completed her PhD in
mathematics some years prior.
5. Chapter 8 returns to the idea of hypotheses to explore in more detail how to develop a
hypothesis and then test it.
6. Clifford Geertz (1973) is largely credited with the introduction of the term thick description into qualitative research. His work builds on Ryle’s (1971) philosophical term.
Geertz’s original conception of thick description, which was anthropologically based,
was more descriptive in orientation than contemporary uses of the term, which have
emphasized an interpretive dimension to this description—for example Denzin’s (1989a)
idea of thick interpretation (see Ponterotto, 2006).
7. This is an invited contribution: Miciak, M., & Daum, C. (2023). Scratching the underbelly of research design: Developing clear research question(s). In Cheek, J., & Øby, E.,
Research design: Why thinking about design matters (pp. 59–65). SAGE.
8. Thorne, S. (2008) Interpretive description. Left Coast Press.
9. Thorne, S., Oliffe, J., Kimsing, C., Hislop, T. G., Stajduhar, K., Harris, S. R., Armstrong,
E. A., & Oglov, V. (2010). Helpful communications during the diagnostic period: An interpretive description of patient preferences. European Journal of Cancer Care, 19, 746–754.
https://doi.org/10.1111/j.1365-2354.2009.01125.x
4
WHY METHODOLOGY
MATTERS WHEN
DESIGNING RESEARCH
PURPOSES AND GOALS OF THE CHAPTER
In Chapter 1, we introduced the idea of methodological thinking as shaping the form that
a research design takes. Methodology was defined as “the strategy, plan of action, process or
design lying behind the choice and use of particular methods and linking the choice and
use of methods to the desired outcomes” (Crotty, 1998, p. 3) of a research design or study.
From the discussion that we have had in Chapters 2 and 3, you will now understand
that the desired outcomes of any research design or study relate to answering the research
questions we have identified as central to that study and doing so in a way that is both
credible and ethical.
Consequently, choosing which methods to use in our study involves thinking about
what type of data our chosen methods will enable us to obtain, and whether or not that
data will enable us to address our research questions. It also involves thinking about how
we will put those methods into practice. When we think in this way, we are thinking
methodologically.
When we design our research, we will need to think methodologically and make justifiable decisions, about what methods will form part of our design. Part of the credibility
of our research lies in us being able to justify how the methods we have chosen to use will
enable us to obtain the knowledge that we need to answer our research questions.
Given this, the purpose of the chapter is to explore how methodological thinking
guides, and provides the rationale for, the decisions we make when designing our research.
Decisions such as what type of knowledge we will need to answer our research questions,
how we can develop that knowledge, the form of data we will need to do so, which methods will enable us to obtain that form of data, and how we will analyze and interpret that
data.
We explore ways that our thinking about methodology and methods is influenced by
different onto-epistemological assumptions derived from different schools of philosophical thought. This includes the way that we think about what type of methods, and the
knowledge produced by them, can be considered scientific.
The idea of inquiry paradigms is used as the organizing construct for much of this
exploration. We demonstrate how the paradigmatic stance adopted by a researcher guides
that researcher’s thinking about matters such as whether the social world can be studied
in the same way that the natural and physical sciences are, and what valid or trustworthy
research findings are.
As you are reading the chapter, we encourage you to think reflexively about the methodological assumptions you make when thinking about the design of your research. What
view of what science and research are, are they based on? How do these assumptions,
declared or otherwise, affect methods related decisions you make about that design?
69
70 Research Design
The goals of this chapter are to
• Establish that the way we design our research is shaped by methodological
thinking which provides the rationale for the decisions we make related to the
methods used in our research.
• Highlight that our view of what data is, and is for, reflects our methodological
thinking and our beliefs about what science and research are, and are for.
• Introduce the idea of paradigms and paradigmatic stances as sets of basic beliefs
that guide our thinking when designing research.
• Highlight the role that ontology and epistemology play when thinking about
paradigms and paradigmatic stance.
• Explore the idea of an inquiry paradigm as providing an organizing construct for
understanding how onto-epistemological derived thinking impacts how research
is thought about and therefore designed.
• Illustrate the effect that onto-epistemological assumptions embedded in three
different inquiry paradigms—positivism, post-positivism, and constructivism—
have on our methodological thinking when we are designing our research.
• Demonstrate how our thinking about, and therefore understandings of, quality
and credibility of research are affected by the inquiry paradigm in which our
research is embedded.
• Emphasize the importance of thinking about whether the choices we make in
our research design related to methodology and methods are consistent with each
other, as well as with the research questions we are seeking to address.
• Provide examples of how to think through, and therefore become aware of, the
methodological assumptions we bring with us to the research design table and
why this matters.
THINKING METHODOLOGICALLY
Once we have identified what we want to know more about and why, we are ready to focus
our thinking on the way that we will do our research. This will require us to think about
what type of information we will need to address the issues, questions, or hypotheses that
our research is focused on, and how we might obtain it.
For example, if our general area of interest is how campus lockdown affects students
during the COVID-19 pandemic, we might choose to study how that lockdown has
impacted on those students’ motivation. When doing so, we are interested in finding out
if there is any quantifiable link between having only digital classes, a lack of social interaction as the result of having only digital classes, and students’ motivation. To do this, we
will need to collect numerical information in a form and amount that will enable us to
make those links in a credible way when analyzing that numerical information. Having
decided this (a methodological decision), we are now able to decide what methods we
can include in our research design that will enable us to collect that form and amount of
numerical information in this quantitative study.
Chapter 4
•
Why Methodology Matters When Designing Research
71
On the other hand, if we choose to study what it was like to be a student during a
period of campus lockdown, then we will be interested in finding out how students experienced campus lockdown and why. To do this, we will need to collect information about
that experience by, for example, interviewing students whose campus has been locked
down. When doing so, we will need to consider whether the words that make up these
interviews enable us to develop in-depth descriptions of these students’ perceptions and
understandings of not being allowed to attend classes or campus physically. Whether or
not these words can do this in a credible way in this qualitative study depends on how the
interviews are designed, conducted, and analyzed.
What these examples illustrate is that we need to have decided what form of data
will address the issues, questions, or hypotheses that our research is focused on, and why.
Until we decide that, we are not ready to begin thinking about how we can collect our
data. For example, will our research need to take a qualitative or quantitative approach?
Once we have decided that, we can then begin to think about which specific methods
within a quantitative or a qualitative approach will enable us to gain the form of data we
need.
When we think in this way, we are thinking methodologically. Methodology is “the
strategy, plan of action, process or design lying behind the choice of particular methods
and linking the choice and use of methods to the desired outcomes” (Crotty, 1998, p. 3).
The shape that a research design takes is the result of these methodological decisions.
Neither our choice of methods, or the data they produce, can be understood detached
from the methodological thinking that underpins them.
We will begin our exploration of how methodological thinking guides, and shapes,
the form that our research design takes by taking a closer look at the idea of data.1 To
do so we pose a series of questions about data that will need to be thought through,
and decisions made about, when designing your research. These include the following:
What is data? What types of data are there? What type of data will I need to enable me
to address the problems or questions that my research is designed to answer? How do I
decide this?
DATA: A CONCEPT SHAPED BY
METHODOLOGICAL ASSUMPTIONS
Despite data being a word that appears in most discussions of research design, and reports
of research, it is a word that we tend to use without much thought (Koro-Ljungberg,
2016). Data is also a word that, while used a lot, is not often clearly defined. It is as if everyone shares a common understanding of what data is and is for. However, this is not the
case. There is considerable variation in the way that data is thought about, and therefore
defined. Table 4.1 provides examples of this variation.
What does the variation in the definitions in Table 4.1 reveal about the way data is
thought about and viewed? One thing evident from these definitions is that data is not
just about the numbers or the words collected by a researcher. Data is much more than
that. It is about the way that parts of reality are reduced or “chunked” into manageable
units (Bernard et al., 2017) able to be analyzed and interpreted to produce the findings
or results of our study. What is “chunked” (i.e., what is considered data and why), and
how it is “chunked” (i.e., what methods are used or not used to collect and analyze that
72 Research Design
TABLE 4.1 ■
Different Definitions and Views of What Data Is
Author(s)
Definition of Data
Black (1999, p. 25)
Appears in many forms—“[f]or example, some components
may be numerical, while others may be in the form of
transcripts of interviews.”
Bernard et al. (2017, p. 5)
Data is created “by chunking experiences into recordable
units.”
Schostak & Schostak (2008, p. 91)
“a matter of seeing”
Sirkin (2006, p. 590)
“All the information we use to verify a hypothesis”
Polkinghorne (2005, p. 138)
“accounts gathered by qualitative researchers”
Black (1999);
Synonymous with “set of scores” from “measuring
instruments such as tests, questionnaires, and interview or
observation schedules” (Black, 1999, p. 24)
Bors (2018)
data, and why) depends on the way that the idea of data itself is viewed and thought
about. For example, as a set of numerical scores or a form of transcripts of interviews
(Black, 1999).
It also depends on why the researcher is collecting the data in the first place—the
purpose of the research. For example, will the data be used to verify a hypothesis, or will
it be used to build understandings of the experiences and perceptions of individuals in
their research about a topic or issue of interest? The numbers or words that our research
produces are a form of raw data that in itself has “little meaning and must be turned into
understandable information” (Black, 1999, p. 25). This is an important point. It highlights that just generating long lists of numbers or hundreds of pages of words of the transcripts of interviews will not in themselves provide the answers to our research questions.
Whether or not these words and numbers can provide the basis for addressing the issue or
problem that our research is being designed to address depends on a number of interconnected methodological considerations. Understanding these connections will also enable
the production of appropriate forms of data to address our research questions.
For example, we will need to consider the type of data needed in order to be able to
address the questions that guide our research and why. As we saw in Table 4.1, data can be
viewed as both “[a]ll the information we use to verify a hypothesis” and “accounts gathered
by qualitative researchers.” Which of those forms that data takes in a specific research
study is a methodological decision based on what we will need to know about in order to
address our research questions and what methods we can use to generate, collect, and analyze that specific type of data. Thus, data is about the way that meaning is produced and
verified (Crotty, 1998; Black 1999)—methodological considerations.
The Importance of Bringing Methodological Considerations Related to
Data Into Focus
The methodological thinking that underpins the way that we view data itself—what it is,
and why we consider one form of data more appropriate than another—remains invisible
in most discussions of research design or the research produced by that design. Instead,
Chapter 4
•
Why Methodology Matters When Designing Research
73
the data related focus of those discussions tends to be on “getting” data. For example, how
to “get it,” and what to do when we have “got” it. This is data as a self-contained “thing”to-be-got type of thinking.
The danger with fast forwarding our thinking to focus on how to “get data,” and
therefore skipping over what we understand data to be, is that we do not see data for
what it is. Data is not just about the numbers or the words collected by a researcher. Nor
is it a thing. Rather it is a concept that contains many (usually undeclared) assumptions.
These are assumptions related to, for example, the form that data can (even must) take in
order to be considered scientific, or the type of data that is “stronger” or “more” scientific.
Therefore, it is important to be aware of our assumptions about what data is, and is for,
when we design our research. The Activity box below provides a way of testing your own
assumptions and beliefs about what data is and is for.
Activity
Self-Test: Are You Taking the Idea of Data Itself for Granted?
Is data a word that you tend to use without much thought? Taking this self-test will
help you answer this question. Reflexively thinking about the following questions
provides you with insights into the beliefs you bring with you when designing your
research.
1. When you read the term data in a research report, or think about what type of
data you want your research design to produce, what do you understand data
to mean?
2. What do you base that understanding on?
3. Do you think about if there are other ways that data can be thought about
besides the way you think about it, or the way that it is presented in a particular research design or report?
4. If you did allow for the fact that different people may have different ideas
about what data is (and is not), then did you consider how different ways of
thinking about what data is affect the way we think about, design, and report
research?
5. Have you been assuming that everyone else views data in the same way as you
do? If so, why?
6. Or perhaps you didn’t really think about any of this at all, or consider it a part
of the thinking that you will need to do when designing your research. If so,
why not?
To sum up, no matter how data are collected, even in what might seem to be the
most objective forms of research, data does not stand alone completely removed from the
researcher. In any type of research design, it is the researcher who decides what type of
numbers or words will be collected and called data, how that collection will take place,
and how that data will be analyzed. These are subjective decisions made by all researchers based on what they believe “science,” “research,” and “data” are. These decisions
74 Research Design
are based on what Guba (1990) referred to as the “basic set of beliefs” (p. 17) held by a
researcher.
This set of beliefs provides what is known as a worldview or a paradigm that guides
the thinking of, and therefore the decisions made by, researchers when they are designing
research (Guba & Lincoln, 1994). This includes what a researcher might decide to study,
how they will study it, and the way that they interpret and make conclusions about what
they have studied. These are methodological decisions.
In the next section of the chapter, we take a closer look at the idea of paradigms and
how they affect the methodological decisions you make when you are designing your
research— whether you realize it or not.
PARADIGMS: SETS OF BASIC BELIEFS THAT
GUIDE METHODOLOGICAL THINKING
What is a paradigm? A paradigm is “a set of basic beliefs. . . . It represents a worldview that
defines, for its holder, the nature of the ‘world,’ the individual’s place in it, and the range
of possible relationships to that world and its parts” (Guba & Lincoln, 1994, p. 107). This
worldview shapes the cascade of methodological decisions that organize and give shape to
all parts of your research design.
For instance, a researcher might hold the view that the principal point of difference
between gaining knowledge in scientific and non-scientific ways is
the alleged objectivity of scientific knowledge. It is unlike the subjective understandings we come to hold. Those subjective understandings may be of very great
importance in our lives but they constitute an essentially different kind of knowledge from scientifically established facts. (Crotty, 1998, p. 27)
This set of basic beliefs, or paradigmatic stance, of the researcher about what scientific
knowledge is affects what that researcher can view as valid ways of producing that scientific research—their research methods.
For example, if subjective understandings are not considered to be scientific knowledge, then findings of research about people’s perceptions or experiences of a topic of
interest, that is, their subjective understandings, will not be able to be considered scientific
knowledge. Moreover, neither the methods that are used to elicit and explore those subjective understandings, nor the data collected by applying those methods, nor the analysis and interpretation of that data, will be considered appropriate to produce scientific
knowledge. Therefore, when designing research, the researcher’s choice of methods will
be limited to those methods able to produce the type of data able to produce the objective
knowledge believed by this researcher to be scientific.
This paradigmatic stance will also affect the way that this researcher judges the
research design of others, and the results produced by those designs. For example, this
researcher will be able to dismiss as invalid research findings based on the interpretation
of data gained from, and about, the subjective experiences and perceptions of individuals.
This dismissal is premised on, and made possible by, this researcher’s view of, and basic
set of beliefs about, the objectivity of scientific knowledge. This researcher considers subjective understandings as constituting “an essentially different kind of knowledge from
scientifically established facts” (Crotty, 1998, p. 27).
Chapter 4
•
Why Methodology Matters When Designing Research
75
PUTTING IT INTO PRACTICE
HOW PARADIGMATIC STANCE AFFECTS WHAT YOU STUDY
AND HOW
You are interested in turnover of staff in an organization. By turnover we mean staff
leaving their positions in that organization. Your paradigmatic stance will influence
what you will study about turnover and how.
If your basic belief is that the principal point of difference between gaining knowledge in scientific and non-scientific ways is the objectivity of scientific knowledge,
then you will design and conduct your research in a way that is in keeping with that
belief. Therefore, you will likely use some sort of measurement of variables of interest you have identified as affecting turnover such as pay rates, flexibility of work time,
and so on. You will then collect numerical data about these variables that is able to
be obtained and analyzed in what you consider to be an objective way drawing on the
principles of probability and statistics.
Further, in keeping with your basic belief that the principal point of difference
between gaining knowledge in scientific and non-scientific ways is the objectivity
of scientific knowledge, you will not consider research designs that seek to find out
about the perceptions and experiences of people working in that organization about
why people either stay or leave their positions. This is because this type of subjective
knowledge, while important, is not scientific. This means that you will not consider
employing a more qualitative approach in your study and methods such as in-depth
interviews associated with this approach. This is because you consider that this type
of data, and the knowledge that it produces, is not scientific.
The set of basic beliefs held by a researcher about, for example, what data, science, and
the purpose of research is will be influenced by their disciplinary background. The dominant methodological schools of thought and method-related traditions, in that discipline,
will affect how that researcher thinks when designing their research. Therefore, when you
are designing research, you need to think about how your “disciplinary socialization experiences . . . may reduce . . . [your] methodological flexibility and adaptability” (Patton,
2015, p. 92). Recognizing, examining, and understanding how your “social background
and assumptions can intervene in the research process” (Hesse-Biber & Leavy, 2006,
p. 141) is an important part of the reflexive process of designing your research.
Before we leave this introductory discussion of paradigms as sets of beliefs or worldviews that guide our thinking when designing our research, it is important to point
out that there are various ways used when referring to the philosophical influences and
assumptions that impact our thinking about research design. Merriam and Tisdell (2016)
point out that these include
traditions and theoretical underpinnings (Bogdan & Biklen, 2011), theoretical traditions and orientations (Patton, 2015) . . . paradigms and perspectives (Denzin &
Lincoln, 2011), philosophical assumptions and interpretive frameworks (Creswell,
2013) . . . epistemology and theoretical perspectives (Crotty, 1998, p. 8).
Consequently, there is “no definitive way to categorize the various philosophical and
theoretical perspectives” (Patton, 2015, p. 85) that have shaped and defined ways of thinking about and designing research. Therefore, it is important that when designing your
76 Research Design
research, you make clear what paradigm(s) or perspective(s) you are drawing on in your
research design and are consistent in the terminology you use to refer to them.
ONTO-EPISTEMOLOGICAL DERIVED ASSUMPTIONS
UNDERPIN METHODOLOGICAL THINKING
Different paradigmatic stances draw on different ontological and epistemological assumptions. Ontology and epistemology are philosophically derived concepts about the nature
of reality and how we know what we know about that reality (Crotty, 1998). Simplifying
to the extreme, ontology is about “philosophical assumptions about the nature of truth and
reality and whether it is external or constructed ” (Creamer, 2018, p. 43). It is about what
exists and what can be considered real. There are different views or ontological positions
about the nature of reality.
Viewing the world as having an external objective reality independent of human perceptions of it gives rise to an ontological position known as realism. In this ontological
view, the world exists independently of those living in that world (Schwandt, 2015). On
the other hand, viewing the world we live in as constructed by our interactions with that
world and our perceptions and understandings of it gives rise to an ontological position
known as relativism. In this ontological position, the world is understood as made up
of socially constructed meanings. These everyday social interactions and constructions
make up the way that the world is. The world does not exist independently of those in it.
Epistemological considerations focus on questions such as, What do we mean when
we say that we know something? and What enables us to claim that we know that? How
does knowledge differ from opinion or belief? What “constitutes credible or warranted conclusions or inferences” (Creamer, 2018, p. 43)? Therefore, epistemological considerations
when designing our research focus on the type of knowledge we will need to produce from
our research in order to be able to produce credible or warranted conclusions or inferences.
In turn, this will require us to think about how we will produce that particular type of
knowledge—methodological and methods related considerations. This will include considerations about the role of the researcher in the generation of that knowledge.2
What ontological viewpoint we adopt affects our epistemological considerations when
designing research. For example, a researcher may view, or understand, the world in a way
that is congruent with the ontological position called realism. Therefore, this researcher
believes that the way that the world actually is must be distinguished from the way that
individuals in that world interpret it (O’Reilly & Kiyimba, 2015). The goal of research then
is to study this objective reality in order to ascertain scientific facts about it. Put another way,
“[T]he facts of the world are essentially there for study. They exist independently of us as
observers, and if we are rational we will come to know the facts as they are” (Gergen, 1991,
p. 91). It is these facts that are real. The role of the researcher is to discover them.
On the other hand, a researcher may view the world in a way that is congruent with
the understandings of the ontological position called relativism. In this ontological position, the world can be understood as made up of socially constructed meanings. It is these
everyday social interactions and constructions that make up the way the world is. The
world does not exist independently of those in it. Such an ontological view does not consider that the only things that can be studied are the objective facts already “out there”
waiting for us to discover them. Rather it allows for the possibility of multiple realities
Chapter 4
•
Why Methodology Matters When Designing Research
77
comprised of the everyday social interactions and constructions that make up those realities. Consequently, human understandings and social constructions are legitimate areas
for research and scientific study. It is possible to make credible or warranted conclusions or
inferences (Creamer, 2018) from the study of socially constructed meanings.
PUTTING IT INTO PRACTICE
WHAT CAN WE KNOW ABOUT MUSIC? MUSIC VIEWED
FROM DIFFERENT ONTOLOGICAL AND EPISTEMOLOGICAL
POSITIONS
Music provides an example of how we can understand aspects of the world we live in
in different ways and how those understandings impact what we might seek to know
about those aspects. What music is can be understood in different ways. In the natural
sciences, music is understood as being comprised of sound waves that are produced
by the vibration of the particles of the medium through which the sound waves are
moving. Based on this understanding of music, you might design research to measure the wavelengths and frequencies of sound waves produced by different types of
music. Or you might be interested in the effect of these sound waves on the anatomy of
peoples’ ears as they listen to music at different volumes and the sound waves travel
down the auditory canal and strike the eardrum, causing vibrations in tiny bones in
the middle ear. Your role as researcher will be as an objective collector of information
that you will then use to support or disprove hypotheses based on the collection and
measurement of quantitative data.
However, if you understand music as a subjective art form derived from the interaction of sound and how the person hearing it experiences it, then it is possible to know
other things about music. For example, you might be interested in knowing about the
thoughts and feelings that people associate with different types of music and why they
do. Why is it that certain types of music (usually in a minor key and with a slow tempo,
such as for example “Ave Maria” by Schubert) may provoke intense feelings of sorrow?
Why do other types of music (usually in a major key and with a moderate to fast tempo,
such as club songs, national anthems, or coronations) provoke feelings of happiness
or even exhilaration on the part of those listening? Why is it that listening to a piece of
music can enable people living with dementia to relive positive memories triggered
by that specific music? 3 What do people subjectively experience when listening to that
music and how might this knowledge be used to develop, for example, therapeutic
interventions for people living with dementia? Your role as a researcher is to be an
interpreter of information designed to build understandings of the way that perceptions of music are socially constructed.
This example highlights that different views about what music is (ontology) affect
how, and what, you can know about music (epistemology), and hence the way that you
will research it (methodology and methods).
To reflect the way that ontological and epistemological assumptions interface when
designing research, you will see scholars referring to the onto-epistemological assumptions that underpin methodological thinking. What onto-epistemological view we
adopt affects the methodological decisions we make when we design our research.
Onto-epistemological assumptions underpin and shape our research design whether
we realize it or not. This is because they affect the methodological decisions we make
78 Research Design
when we design our research. While it may be possible to design and conduct research
without overtly acknowledging this, or even really thinking about our own assumptions
about reality and what can be considered knowledge about that reality, the effect of such
assumptions on the research in question is still there.
Therefore, one of the first things that you will have to do when designing your
research, after you have worked out what your research questions are, is to establish “which
paradigm and subsequently which methodology or strategy can best answer the research
question” (Welford et al., 2012, p. 29). In the next section of the chapter, we will explore
how understanding what an inquiry paradigm is can help you make appropriate methodological choices, including the methods you will use, when designing your research.
INQUIRY PARADIGMS AND HOW THEY CONNECT
TO METHODOLOGICAL THINKING
Guba and Lincoln (1994) developed the idea of inquiry paradigms to overtly emphasize,
and capture, the idea of ontology, epistemology, methodology, and methods as closely
connected concepts. The intersections between ontology, epistemology, methodology,
and methods affect what is studied, how it is studied, and the role that the researcher plays
in that study. They define “for inquirers what it is they are about, and what falls within and
outside the limits of legitimate inquiry” (Guba & Lincoln, 1994, p. 108).
There are different types of inquiry paradigms and each one comprises “a set of philosophical assumptions that are inherently coherent about the nature of reality and the researcher’s
role in constructing it that is agreed upon by a community of scholars” (Creamer, 2018, p. 43).
These assumptions guide the methodological decisions that we make. Inquiry paradigms
provide us with an organizing construct for thinking about the philosophical influences
and assumptions that impact on how research is thought about, and therefore designed.
In the discussion to follow, we take a brief look at three different inquiry paradigms—positivism, post-positivism, and constructivism. When doing so, we demonstrate
the effect that onto-epistemological assumptions that are embedded in them have on a
researcher’s methodological thinking when designing their research. Our aim is to unpack
the point made by Lather (2006) that “science is not the same in all paradigms in terms of
ontology, epistemology and methodology” (p. 37).
Positivism
Positivism has been a dominant inquiry paradigm in western scientific thought for many
hundreds of years. Positivism is based on a “conviction that scientific knowledge is both
accurate and certain” (Crotty, 1998, p. 27). The task of the researcher is to study the reality that already exists “out there” (Lincoln & Guba, 2013, p. 38). Consequently, positivism is premised on a realist ontological position. The task of the researcher is to discover
and establish objective facts about the aspect of that reality being studied. Therefore, the
methods used in scientific research to produce scientific knowledge must enable the production of that objective knowledge. The belief that inquiry
must be objectively carried out to discover “how things really are and really work” . . .
in turn suggests a methodology which is essentially experimental and manipulative,
in an effort to sort out the various influences (“variables”) that determine the true
state of affairs, and eliminate the confounding ones. (Lincoln & Guba, 2013, p. 38)
Chapter 4
•
Why Methodology Matters When Designing Research
79
In this way of thinking, what can be considered science is reduced to, and only
includes, what we can observe, measure, and then explain using those measurements. In
turn, this reduces ways of doing research, and producing what can be called science, to
scientific methods designed to observe phenomena—“the observation in question being
scientific observation carried out by way of the scientific method” (Crotty, 1998, p. 20).
The scientific method is a way of thinking about, and doing research derived from the way
research has been done, and science produced, in the physical and natural sciences.
Research designed according to the scientific method should
• have a measurable/testable hypothesis (or testable propositions within the
hypothesis),
• have steps to test the hypothesis that must be able to be repeated,
• be conducted as objectively as possible,
• outline the research method/design in enough detail that it can be replicated by
other researchers and the findings either supported or disproved. (Adapted from
Cutcliffe & Harder, 2012, p. 3)
Other views of research, science, methodology, and methods are deemed invalid on
the basis that they are not scientific as they do not conform to “the” scientific method.
TIP
KEY FEATURES OF POSITIVIST THOUGHT
A useful discussion of the key features of both positivistic thought itself and the
research that results from the assumptions on which this thought is founded is provided by Guba and Lincoln (1994, pp. 109–110). In Table 4.2, we present a summary of
their key points.
TABLE 4.2 ■
Ontology:
Realism
Epistemology:
Dualist and
objectivist
Key Features of Positivist Thought
•
An apprehendable reality is assumed to exist, driven by immutable
natural laws and mechanisms.
•
Knowledge of the “way things are” is conventionally summarized in
the form of time- and context-free generalizations, some of which
take the form of cause-effect laws.
•
Research can, in principle, converge on the “true” state of affairs.
•
The investigator and the investigated “object” are assumed to be
independent entities, and the investigator to be capable of studying
the object without influencing it or being influenced by it. When
influence in either direction (threats to validity) is recognized, or even
suspected, various strategies are followed to reduce or eliminate it.
•
•
Inquiry takes place as through a one-way mirror.
•
Replicable findings are, in fact, “true.”
Values and biases are prevented from influencing outcomes, so long
as the prescribed procedures are rigorously followed.
(Continued)
80 Research Design
TABLE 4.2 ■
Methodology:
Experimental/
manipulative
Key Features of Positivist Thought (Continued)
•
Questions or hypotheses are stated in propositional form and
subjected to empirical test to verify them.
•
Possible confounding conditions must be carefully controlled
(manipulated) to prevent outcomes from being improperly influenced.
Source: From Guba and Lincoln (1994, pp. 109-110).
Critiques of Positivism
There is no doubt that in the physical and natural sciences, hypothesis driven deductive studies are appropriate ways to study that physical reality. However, debates and disagreement
arise among researchers when this view of science and research is used to reject outright other
ways that research and science might be thought about. Such outright rejection restricts possibilities for thinking about research to the same type of thinking and research designs used
to study the natural sciences. It assumes that “the natural and social sciences should and can
apply the same principles to collecting and analyzing data” (Flick, 2015a, p. 20).
Not surprisingly, many scholars in the social sciences have critiqued this assumption.
These scholars question if it is possible to limit study of social settings, and the people in them,
to the type of research designs that have developed from the natural sciences. This is because
the social sciences are strongest where the natural sciences are weakest; just as the
social sciences have not contributed much to explanatory and predictive theory, neither have the natural sciences contributed to the reflexive analysis and discussion of
values and interests, which is the prerequisite for an enlightened political, economic,
and cultural development in any society. . . . [S]ocial science never has been, and
probably never will be, able to develop the type of explanatory and predictive theory
that is the ideal and hallmark of natural science. (Flyvbjerg, 2001, pp. 3–4)
You will note that Flyvbjerg’s critique is not about the types of methods, data, and
knowledge associated with the positivist inquiry paradigm per se. Rather the critique is
about only allowing one view of what science is, and how scientific research must be “done.”
As Crotty (1998) reminds us, “If we want to quarrel with the positivist view, our quarrel . . .
will have to do with the status positivism ascribes to scientific findings. Articulating scientific knowledge is one thing; claiming that scientific knowledge is utterly objective and that only
scientific knowledge is valid, certain and accurate is another” (p. 29, italics added).
Post-Positivism
As a result of the sort of critiques of positivism discussed above, largely from the social sciences, different inflections and forms of positivism emerged.4 For example, post-positivism5
emerged as a reaction to the basic beliefs underpinning positivism that “real knowledge
(as opposed to mere beliefs) was limited to what could be logically deduced from theory,
operationally measured, and empirically replicated” (Patton, 2002, p. 92). Instead,
post-positivist thought recognizes that no matter what methodological approach and associated methods form part of a research design, it is impossible to entirely remove the influence
of the researcher on both that design, and when that research is put into action. Therefore,
Chapter 4
•
Why Methodology Matters When Designing Research
81
“[N]o matter how faithfully the scientist adheres to scientific method, research outcomes are
neither totally objective nor unquestionably certain” (Crotty, 1998, p. 40). In other words,
there is always a subjective element in research and in the results that arise from that research.
Consequently, from a post-positivist perspective, research is designed to distinguish
between beliefs and valid beliefs, rather than to produce absolute truths (Campbell & Russo,
1999). Whether or not a belief is valid is based on judgments about the validity of the empirical evidence that the belief is based on, and how that evidence was obtained and analyzed.
Therefore, the task of the researcher when designing research is to develop a scientific strategy of
“managing subjectivity as tenaciously as possible, to come as near to ‘truth’ as human frailty permits. Experimentation and manipulation are retained as the basic methodological strategy, even
while it is conceded that they cannot produce ultimately infallible results” (Lincoln & Guba,
2013, p. 38). Put another way, post-positivism is a “humbler version of the scientific approach . . .
proponents of which . . . remain in the positivist camp but temper very significantly the status
they ascribe to their findings, the claims they make about them” (Crotty, 1998, p. 40).
Positivism and post-positivism are not the only possible inquiry paradigms and ways of
thinking about research, science, and research design. There are other inquiry paradigms
and related approaches to research that take a different view of what falls both within, and
outside of, the limits of legitimate inquiry. One such inquiry paradigm is constructivism.
Constructivism
Compared to positivist and post-positivist thought, constructivist thought takes a very
different view of reality (ontology), what can be known about that reality, and how we
can know that (epistemology), and consequently what is, and is not, legitimate inquiry
(methodology and associated methods). Unlike positivism and post-positivism, constructivism does not presuppose a reality removed from those who interact with that reality.
Rather it begins with “the premise that the human world is different from the natural,
physical world and therefore must be studied differently” (Patton, 2015, p. 121, italics
added). Truth is not waiting “out there” in an object to be discovered, but rather meaning
is constructed by the interaction of humans with that object and each other. Consequently,
constructivism allows for the idea of multiple realities which exist “in the form of multiple
mental constructions, socially and experientially based, local and specific, dependent for
their form and content on the persons who hold them” (Guba, 1990, p. 27).
Therefore, working within a constructivist paradigmatic stance, the researcher does not
seek to eliminate the subjective thoughts, feelings, and opinions of those being researched to
concentrate only on specific objective “facts” and variables that must be controlled. Instead,
the researcher actively seeks to understand “the complex world of lived experience from the
point of view of those who live it” (Schwandt, 1994, p. 118). This is reflected by the fact that
people in the study are referred to as participants, rather than subjects of the research.
If we adopt this paradigmatic view when we are designing our research, then,
• we will work with participants in our research as “a co-constructor of knowledge,
of understanding and interpretation of the meaning of lived experiences”
(Lincoln et al., 2011, p. 110).
• our research will be designed in such a way to enable us to seek out, obtain
and understand our research participants’ individual views, perceptions and
understandings about the aspect(s) of their reality that we are interested in
knowing something more about.6
82 Research Design
• choosing to use methods such as individual in-depth interviews, observations in
social settings7 as part of our research design would be methodologically appropriate.
• our research will be primarily inductive as the information obtained from
research participants is used to build understandings of the aspects of the social
constructions that we are focusing on in our research.
This is a very different view of research and the role of the researcher than in positivist
or post-positivist thinking, where the role of the researcher is to obtain information about
specific variables of interest from the people who are the subjects of, rather than participants in,8 that research. These variables of interest have been determined in advance of the
research by the researcher as the ones that can explain or enable the researcher to understand what is going on in a particular situation. The role of the researcher is to, as objectively as possible, obtain from these research subjects specific piece(s) of information in
the form of data about variables of interest that the researcher can then use to deductively
test theories. In these inquiry paradigms, the researcher deliberatively maintains distance
from the subjects (as opposed to participants) of that research.
Activity
Exploring How Inquiry Paradigms Affect Our Thinking of
About “What Falls Within and Outside the Limits of Legitimate
Inquiry”
You want to study why children succeed at school.
1. Think about the way that you might go about studying this topic if you draw
on post-positivist thinking when doing so. How will this affect what your
research questions are (and are not)? In turn, how will this affect the methods
that you will use in your study and the type of knowledge you will produce
about success at school?
2. Now repeat the above exercise but this time drawing on constructivist thinking. How will this affect what your research questions are (and are not)? In
turn, how will this affect the methods that you will use in your study and the
type of knowledge you will produce about success at school?
Differences in your answers to 1 and 2 above highlight the effect of the sets of basic
beliefs or paradigmatic assumptions that you bring to your research have on the way
that you design and think about that research, and “what falls within and outside the
limits of legitimate inquiry” (Guba & Lincoln, 1994, p. 108).
INQUIRY PARADIGMS AFFECT THINKING ABOUT
WHETHER RESEARCH IS CREDIBLE
The basic beliefs that make up inquiry paradigms affect whether or not a research
design, and the findings produced by it, can be considered credible. This is because the
onto-epistemological assumptions embedded in those basic beliefs shape understandings
of what is legitimate inquiry in a scientific study. This includes the methods chosen in that
study and the type of data they produce.
Chapter 4
•
Why Methodology Matters When Designing Research
83
For example, research drawing on a constructivist inquiry paradigm is designed to produce trustworthy “reconstructed understandings of the social world” (Denzin & Lincoln,
2018c, p. 98). It is not necessarily designed to identify patterns or consistencies in these findings which would be the case if the goal of the research is to produce findings that are objective, and hence generalizable across groups of people. Instead, “Each person who participates
in the study provides a different view on the topic being investigated. . . . Dissonant points of
view are acceptable [illustrating] the complexity of multiple realities” (Norum, 2008, p. 737).
Therefore, different criteria will be used to judge the credibility of research if the
researcher draws on a post-positivist or positivist paradigmatic stance than if they draw
on a constructivist paradigmatic stance. In a constructivist inquiry paradigm, positivist
derived criteria of validity 9 “are replaced by such terms as trustworthiness and authenticity”
(Denzin & Lincoln, 2018c, p. 98). The trustworthiness and authenticity of the reconstructed accounts of aspects of the social world that form the findings is what makes the
research, and the conclusions reached from research drawing on the set of basic beliefs
underpinning a constructivist inquiry paradigm, credible.10
This highlights that the way that credibility of research is understood arises from the
paradigmatic stance that a researcher brings to that research. Whether or not research is
considered credible (or even research at all) will be based on the basic beliefs that make up
that paradigmatic stance. Therefore, judgments about the credibility of research make
no sense unless they are in relation to a particular worldview, or basic set of beliefs, about
research. Criteria for making judgments about the credibility of research do not automatically transfer from one paradigmatic stance to another.
Patton (2015) provides a useful summary of the different ways in which quality and
credibility of research can be thought about, depending on the paradigmatic view adopted
by a researcher. He does this by identifying alternative sets of criteria for doing so. This
summary can be found in Table 4.3.
TIP
ALTERNATIVE SETS OF CRITERIA FOR JUDGING THE QUALITY
AND CREDIBILITY OF RESEARCH ARISING FROM DIFFERENT
PARADIGMATIC STANCES11
TABLE 4.3 ■
Traditional
Scientific
Research
Criteria
1. Objectivity of the inquirer (minimize bias)
2. Hypothesis generation and testing
3. Validity of the data
4. Interrater reliability of codings and pattern analyses
5. Conclusions about the correspondence of findings to reality
6. Generalizability (external validity)
7. Strength of causal explanations (attribution analysis)
8. Contributions to theory
9. Independence of conclusions and judgments
10. Credibility to knowledgeable disciplinary researchers (peer reviews)
(Continued)
84 Research Design
TABLE 4.3 ■
Social
Construction
and
Constructivist
Criteria
(Continued)
1. Subjectivity acknowledged (discuss and take into account inquirer
perspective)
2. Trustworthiness and authenticity
3. Interdependence: relationship based (intersubjectivity)
4. Triangulation (capturing and respecting multiple perspectives)
5. Reflexivity
6. Particularity (doing justice to the integrity of unique cases)
7. Enhanced and deepened understanding (verstehen)
8. Contributions to dialogue
9. Extrapolation and transferability
10. Credible to and deemed accurate by those who have shared their
stories and perspectives
From Patton, M. Q. (2015). Qualitative Research & Evaluation Methods (4th ed.). SAGE. (Part of Exhibit 9.7,
p. 680.)
Why Is Thinking About Your Paradigmatic Stance Important?
Our discussion of how inquiry paradigms affect thinking about whether research is credible demonstrates why thinking about what your paradigmatic stance is when you are
designing your research is important. It forces you to overtly focus on the intersections
between your methodological choices in your research design and the set of basic beliefs
you hold that resulted in you making those choices. This is a focus which has almost disappeared in discussions in many research methods courses and textbooks where students
are
more often than not taught particular “methods of data collection” (such as interviews, case studies, focus groups, ethnography, other basic research design
techniques, etc.). . . . [I]t is few and far between that philosophy of science and
philosophy of inquiry seminars are required of graduate students—and even fewer
still . . . that call into question or contest the very notion of data or evidence itself
(Denzin & Giardina, 2016, p. 6).
The loss of this focus has resulted in many of these courses, and the methods
textbooks they use, presenting methods exclusively as ready-made, stand-alone, and
self-contained techniques or procedures. These techniques and procedures can then be
selected, copied, and pasted into a research design. This masks the fact that when you
select, copy, and paste a ready-made method into your research design, you are copying
and pasting much more than just a technique or procedure. You are also copying and
pasting the methodological and onto-epistemological understandings from which this
method was derived.
Therefore, it is important that you know what these understandings are, to make
sure that they are congruent with obtaining the type of knowledge that you will need to
address your research questions in order for your research findings to be credible.
Chapter 4
•
Why Methodology Matters When Designing Research
85
THE IMPORTANCE OF ASKING METHODOLOGICAL
QUESTIONS OF YOUR RESEARCH DESIGN
Returning to a point we made in Chapter 1, between us, we have many years of experience both doing research ourselves and acting as advisors for students’ research. We have
noticed that the researchers and students who navigate the challenges of the process of
designing research well are those who take the time to think through, ask questions of,
and then make explicit the methodological decisions that they have made related to the
design of their studies. They are constantly asking themselves a series of interrelated questions about the methodology of that emergent design. Questions such as, What type of
knowledge will I need to address the problem or questions I want to ask? What is an appropriate way to obtain that knowledge? Appropriate in what sense?
They are also the researchers and students who reflexively think through the effect
that the way they answer the above types of questions has on their overall research design.
They ask themselves questions such as these:
• What happens to this part of the design if I make this methodological decision
and not another?
• How might this affect decisions that I have already made about other parts of my
research design?
• Why am I thinking about these questions, and adopting a stance in relation to
them, in the way I am?
Asking these types of methodological questions of your developing research design
enables you to reveal and understand your own paradigmatic viewpoint—the set of basic
beliefs you have about what research and science are. It also enables you to recognize and
think through the effects of those standpoints on the choices you have made in that design
process. This type of reflexive thinking makes you accountable, and thereby forces you
to accept responsibility, for the methodological decisions that you make when designing
your research. This is because this thinking enables you to expose, and therefore become
aware of, assumptions that you, or others, may be making about research, methods, data,
and what valid scientific knowledge is, or even can be.
Activity
Thinking Through These Types of Key Questions When Designing
Your Research
Think of a topic or research questions that you are interested in studying.
Now write down what you are thinking of studying about that topic and how.
Next write down why you are thinking about studying that topic in this way.
Think about what assumptions you might be making about what research,
methods, data, and what valid scientific knowledge is, or even can be.
Try to work out why you are making those assumptions. What are you basing
them on?
86 Research Design
Avoiding the Misuse of Methods
Understanding the methodological and onto-epistemological understandings from which
a method is derived can also help avoid the misuse of methods. This is because “abuses and
misinterpretations/misapplications of a method” (Cutcliffe & Harder, 2012, p. 13) are
more likely to arise if a method is detached from these understandings.12 Neither methods
nor the data produced by them can be understood apart from the methodological considerations and choices in the overall research design in which they are embedded. If removed
from these understandings, methods are reduced to being merely techniques—sets of procedural rules to follow.
This way of thinking also helps us to understand, and provides new ways of looking at,
why students (and even more experienced researchers) constantly struggle with methodological related questions such as these:
• Do I need to use a hypothesis in my research design?
• Is there one type of method that is more scientific than another?
• Is this really research or just opinion?
• What is the use of this approach if I can’t predict and generalize at the end of the
study?
• Why won’t my supervisor or thesis committee allow me to use a particular
methodological approach?
• Am I more likely to get my research published or funded if I use this method
rather than another?
CONCLUSIONS
The way that any research design is thought about and developed is informed by
onto-epistemological, methodological, and method related assumptions that shape and
inform our research design at all parts of the process—sometimes without us even being
aware of it. This is why we have emphasized the importance of remembering “science is
not the same in all paradigms in terms of ontology, epistemology and methodology” (Lather,
2006, p. 37).
The intersections between our ontological, epistemological, and methodological assumptions must be acknowledged, and their effects understood, when thinking about our
research design. This is because these intersections profoundly affect all areas and aspects
of that design. Therefore, it is important when designing research to think about, recognize, and acknowledge these intersections and how they impact the decisions we make
when we design our research.
How will, do, or have they influence(d) our thinking and choices at all parts of the
research design process—including our choice of methodology and associated
methods?
This is a type of reflexive thinking that enables you to focus your thinking on the connections between the parts of your research design, rather than focusing on each part of
Chapter 4
•
Why Methodology Matters When Designing Research
87
that design independently and in isolation. These key methodological related questions
include the following:
1.
What type of knowledge will I need to answer the questions that my research is
designed to answer?
2.
What methods will I use in my research design to generate data, the analysis and
interpretation of which can produce this knowledge?
3.
What methodological considerations govern the choice and use of those methods?
4.
Is my choice of methods consistent with those methodological considerations, and if
so, how?
5.
What onto-epistemological assumptions does that methodology, and therefore the
methods derived from that methodological thinking, draw on?
Such iterative thinking makes it possible for you to ensure that all the considerations,
assumptions, and choices that you make in your research design related to matters of
data, method, methodology, and onto-epistemology are consistent with each other, and
also with what you need to know or find out in order to address the research question(s)
you are asking.
When thinking about how to answer these types of questions, it is useful to keep in mind
Denzin’s (2009, p. 318) reminder about what is “good work” when making methodological and method related decisions in our research design. He notes that any “criteria of good work” apply only to work within a particular paradigm or understanding of
research and science, and not necessarily within other paradigms. This is why thinking
about research design and what makes for “good,” “better,” “best” research and science
is methodologically specific and cannot be generalized across different understandings
of research and science. “Better” methods will be those that enable the collection of a
form of data, the analysis and interpretation of which can provide the type of knowledge
needed to address the purpose of the research. This is why methodology matters when we
are designing research.
In the next chapter, we continue our discussion of why methodology matters. We do this
by exploring the methodological thinking that underpins qualitative and quantitative
approaches to research and how they are designed.
SUMMARY OF KEY POINTS
• Thinking methodologically enables us to understand the choices that researchers make
related to the methods they use in their research. It is a central plank of any research
design.
• Decisions about what form data can take, how it can be obtained and analyzed, and
what it can be used for are methodological ones.
• Paradigms are sets of basic beliefs that guide the choices and decisions of researchers
such as what can be studied, how, and why.
• Paradigms differ from one another in terms of the onto-epistemological assumptions
that they draw on.
88 Research Design
• Ontology and epistemology are philosophically derived concepts about the nature of
reality (ontology) and how we know what we know (epistemology) about that reality.
• Inquiry paradigms provide a way of thinking about how onto-epistemological
considerations shape and intersect with the methodological decisions we make when
designing our research.
• This includes decisions about what can be studied, how it can be studied, and what
form the findings must take in order for the research to be considered credible.
• When you decide to apply a specific method for data collection in your research, you
adopt the methodological and onto-epistemological understandings from which this
method was derived.
• Therefore, you must know what these understandings are in order to make sure that
they are congruent with the type of knowledge that you will need to address your
research questions.
• It is important to remember that “science is not the same in all paradigms in terms of
ontology, epistemology and methodology” (Lather, 2006, p. 37).
• Consequently, neither methods nor the data produced by them can be understood
apart from the methodological considerations and choices in the overall research
design in which they are embedded.
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
constructivism
credibility of research
data
epistemology
inquiry paradigms
onto-epistemological assumptions
ontology
paradigm
paradigmatic stance
positivism
post-positivism
realism
relativism
scientific method
SUPPLEMENTAL ACTIVITIES
1.
What are your “paradigmatic inclinations” (Nagel et al., 2015, p. 367)? Read
the following article:
Nagel, D. A., Burns, V. F., Tilley, C., & Aubin, D. (2015). When novice
researchers adopt constructivist grounded theory: Navigating less travelled
paradigmatic and methodological paths in PhD dissertation work. International
Journal of Doctoral Studies, 10, 365–383.
This article provides an excellent example of how the authors reflexively
navigated a range of choices when deciding on “a methodological approach that
(a) was appropriate to answering our research question, (b) resonated with the
philosophical values for knowledge development within our disciplines, and
(c) fit our personal beliefs, values, and goals” (p. 336). The authors conclude
Chapter 4
•
Why Methodology Matters When Designing Research
89
that anyone doing research “must be able to rationalize and defend each choice
made in the research design” (p. 380), which includes “gaining insight to one’s
own understanding of the epistemological, ontological, and methodological
underpinnings” (p. 380) of the chosen research design.
Now, ask yourself the following questions:
• What are your “paradigmatic inclinations” (Nagel et al., 2015, p. 367)?
• How are you orienting yourself toward a methodological approach to answer
your research questions?
• What are your beliefs about how those questions can be answered?
• Are there other possible ways of addressing your research questions?
• If there are, will those other ways produce different knowledge than your
preferred approach?
• Does it matter?
2.
What might you be taking for granted when you design your research?
Go back to the activity box Self-test: Are you taking the idea of data itself
for granted? in this chapter and read it one more time. Now, answer the questions
below:
A. When you read the term scientific in a research report, or think about how
to design research to make sure that your research is scientific, what do you
understand scientific to mean?
B. When you read the terms credible or trustworthy in a research report, or
think about how to design research to make sure that the findings of that
research is credible and trustworthy, what do you understand credible and
trustworthy to mean?
C. When you read the term knowledge in a research report, or when you,
while in the process of designing research, join conversations in the body
of knowledge built up by the work of others (and which is relevant to the
various areas that make up your research design), what do you understand
knowledge to mean?
• What do you base your answers to A, B, and C on?
• Do you think about if there are other ways that the terms scientific, credible,
trustworthy, and knowledge can be thought about besides the way you think
about them, or the way that they are presented in a particular research design
or report?
• If you did allow for the fact that different people may have different ideas
about what scientific, credible, trustworthy, and knowledge means, then did you
consider how different ways of thinking about these terms affect the way we
think about, design, and report research?
• Have you been assuming that everyone else thinks about these terms in the
same way as you do? If so, why?
• Or perhaps you didn’t really think about any of this at all or consider it a part
of the thinking that you will need to do when designing your research? If so,
why not?
Reflexively thinking about this series of questions (and others like them) will provide
you with insights into assumptions you have about what research is and therefore
how it might be designed.
90 Research Design
FURTHER READINGS
Crotty, M. (1998) The foundations of social science research. Meaning and perspective in the
research process. Allen & Unwin.
Greener, I. (2011) Designing social research. A guide for the bewildered. SAGE.
NOTES
1. You will notice that, depending on the context, at times we refer to data as plural and
therefore use “data are.” Other times we use “data is.” Many researchers, often those
from areas such as mathematics, statistics, and computer science, understand data to
be a plural noun. They think of data as a set of results related to a specific study. They
therefore write “data are” and not “data is” when discussing that data set—the specific
numbers or words collected by a researcher. However, you will also see data referred
to as a concept not necessarily related to the specific findings of a specific study. In
these instances, the term data is used in the same way as terms such as information.
Therefore, we write “data is” just as we write “information is.”
2. We return to and develop this point in later parts of this chapter as well as in the discussion of qualitative and quantitative approaches to research in the next chapter.
3. See https://www.youtube.com/watch?v=OT8AdwV0Vkw
4. For a detailed discussion of this, see Crotty (1998), pp. 18–41.
5. You will see this term written in several different forms: post-positivist, postpositivist,
post positivist being some of them. We refer to the term as post-positivist but when citing
others retain the form of the term that they have used.
6. See also Chapter 5.
7. These qualitative methods are discussed in Chapters 6 and 7.
8. We return to this idea and explain it more fully in Chapter 5
9. The idea of validity is discussed in more detail in Chapters 8 and 9.
10. We return to the idea of trustworthiness and how to make decisions about it in Chapters
6 and 7.
11. Patton (2015) actually called this Alternative Sets of Criteria for Judging the Quality and
Credibility of Qualitative Inquiry as he was writing a textbook about qualitative research
and evaluation methods. However, the points he makes can be applied more widely than
just to qualitative inquiry.
12. For example, see our discussion in Chapter 7 of how the logic of inquiry underpinning
representative sampling in quantitative approaches is misapplied to the idea of maximum variation sampling in qualitative approaches.
5
QUALITATIVE AND
QUANTITATIVE APPROACHES
TO DESIGNING RESEARCH
PURPOSES AND GOALS OF CHAPTER
As the first of a series of five chapters about qualitative and quantitative approaches to
designing research, this chapter explores the methodological thinking associated with,
and underpinning, qualitative and quantitative research approaches. Our focus is the
effect that this methodologically derived thinking has on the way that qualitative and
quantitative research is designed. We demonstrate that the different logic of inquiry that
underpins qualitative and quantitative research affects what form qualitative and quantitative data takes, how that data is collected and analyzed, and therefore what type of
knowledge can be produced by the interpretation of that data.
Throughout the discussion, we highlight that there is variation between types of
qualitative approaches and types of quantitative approaches to research. Both
approaches are diverse. Despite this, it is still possible to identify some common features
within each approach. These features reflect the methodologically derived principles and
understandings from which the design of qualitative research or quantitative research
approaches emerge.
These common features and principles provide a conceptual framework for the
discussion about qualitative and quantitative approaches both in this chapter and in the
chapters that follow it. In all of them, our level of focus is how the thinking that gave
rise to these common features and principles affects the shape that your research design
takes. This includes when you are making decisions about how to collect, analyze, and
interpret data using either qualitative (Chapters 6 and 7) or quantitative (Chapters 8 and
9) methods.
The goals of this chapter are to
• Highlight that the terms qualitative and quantitative when applied to research,
data, analyses, and methods are umbrella terms, neither of which can be reduced
to a single set of understandings.
• Emphasize that qualitative and quantitative approaches reflect different research
purposes as well as different logics of inquiry.
• Identify common features associated with qualitative and quantitative
approaches, respectively.
• Explore the variation between different types of qualitative, and different types of
quantitative, approaches.
91
92 Research Design
• Demonstrate that any understanding of qualitative and quantitative approaches
to research cannot be detached from the methodological thinking and
assumptions that shape those approaches.
• Illustrate that therefore what makes a research design, or an aspect of that design,
“qualitative” or “quantitative” is much more than simple dichotomies such as
the data produced being nonnumerical as in the form of words (qualitative) or
numbers (quantitative).
• Emphasize that the answer to the question of whether a qualitative or
quantitative approach is “better” or “best” is that “it depends.” This is because
this question can only be answered in relation to a specific research design and a
specific research problem and related research questions.
QUALITATIVE AND QUANTITATIVE RESEARCH STRATEGIES
It will not take you long when reading about ways of designing research to come across the
terms qualitative and quantitative. For example, you will read reports of research described
as using qualitative or quantitative approaches, or employing qualitative or quantitative
methodology, or using qualitative or quantitative methods, or producing qualitative or
quantitative data. You will also see entire research designs being described as qualitative or
quantitative designs.
Qualitative and Quantitative Approaches Reflect Different Research
Purposes
Qualitative research approaches are appropriate when the purpose of the research is to
gain qualitative and deep insights about, and understandings of, people’s lives or social
settings. Such insights “can help explain crises, injustices, and everyday life in new and
intriguing ways. It may push you into uncomfortable spaces whereby you see things differently” (Mayan, 2009, p. 10).
Consequently, qualitative research approaches, as their name suggests, will always
produce nonnumerical data in the form of texts or words of some kind.1 The research
will be designed in such a way as to enable qualitative interpretations of the perceptions
or experiences of people about a specific aspect(s) of the everyday context(s) in which they
exist. For example, people’s experiences, and perceptions of, voting (e.g., Hammond et al.,
2021), or being made redundant (e.g., Karsavuran, 2021). Although procedures for the
research will be identified beforehand, qualitative research designs are characterized by
“built-in flexibility, to account for new and unexpected empirical materials and growing
sophistication” (Denzin & Lincoln, 2005b, p. 376).
On the other hand, quantitative approaches are appropriate when the purpose of the
research is to quantify in some way something that you want to know about. Researchers
who employ a quantitative approach typically aim to identify, classify, and quantitatively describe characteristics of, and tendencies in, a given (large) group of people (i.e., a
study population). For example, how voting intentions are related to socioeconomic features such as educational attainment and employment status in a given population (e.g.,
Alabrese et al., 2019). To employ this sort of approach to inquiry you will need to follow
set mathematical and statistically based procedures. Consequently, quantitative research
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
93
approaches, as their name suggests, will always produce numerical data from measuring
something of interest. This is so that the data can be used to make valid probabilistic and
statistically based interpretations about those specific characteristics.2
Consequently, whether we use a qualitative or quantitative approach in our research
design depends on what the purpose of the study is. What are we trying to say something
about at the end of the study?
A Word of Caution
A word of caution is needed at this point. Often, the different forms of the data produced
by each approach (e.g., nonnumerical vs. numerical) is used to differentiate between
qualitative and quantitative approaches and even define what these approaches are. For
example, the “most basic definition of qualitative research is that it uses words as data . . .
collected and analyzed in all sorts of ways. Quantitative research, in contrast, uses numbers
as data and analyzes them using statistical techniques” (Braun & Clarke, 2013, pp. 3–4).
While the use of the terms qualitative and quantitative to name and describe these
approaches to research may reflect the type of data that these approaches generally produce, it is too simplistic to define what these approaches are on the basis of the form of data
they produce. Rather, the form that data takes reflects the purpose of the research, as well
as what type of information is needed to achieve that purpose. It also reflects the different
logic of inquiry underpinning each approach.
Qualitative and Quantitative Approaches Reflect Different Logic of
Inquiry
Inductive and deductive thinking, as we discussed in Chapter 3, “reflect different ways of
shifting between data and concepts. Inductive approaches tend to let the data lead to the
emergence of concepts; deductive approaches tend to let the concepts . . . lead to the definition of the relevant data that needs to be collected” (Yin, 2016, p. 100). Put another way,
inductive and deductive thinking differ in the logic of inquiry that they employ.
Quantitative Approaches Employ Deductive Reasoning
Quantitative approaches, and quantitatively driven research designs, are often described
as theory or concept testing and therefore deductive. This is because when designing their
research, the researcher employing a quantitative approach begins with a defined set of
variables or concepts related to the area of interest that they want to know something specific about. They then design their research to measure those variables and concepts or test
assumptions about them.
For example, if a researcher is interested in the broad area of media coverage of
vaccination, there are a number of possible focuses related to that area. One possibility is that a researcher is interested in how the uptake of a certain vaccine is affected
by media coverage about that vaccine.3,4 Specifically, they might want to know about
whether any effect of media coverage identified is different across age groups, ethnic groups, or geographical locations, and they might want to test whether a specific
hypothesis about such differences is supported by empirical evidence. For example,
the hypothesis that the number of media reports questioning the safety of a specific
vaccine influences the decline in vaccine uptake more among people born after 1995
compared to people born in 1995 or earlier.
94 Research Design
Given this purpose, the research strategies employed will draw on quantitative
research approaches. The researcher will identify in advance what aspects of that phenomenon (effect of media on vaccine uptake) they are interested in. They will work deductively
when doing so. The goal of the research is to quantify the presence of, and links between,
objective measures (such as age and vaccination uptake, ethnicity and vaccination uptake,
or geographical location and vaccination uptake). Therefore, the researcher will develop a
research design that enables them to produce the numerical data that they need to conduct
the statistical analysis procedures they will need to do in order to establish those links.
This type of thinking is called hypothetico-deductive thinking.5 This way of thinking may be summarized in the following way:
1. You start from one or more concept(s), or variable(s) (e.g., age and vaccination
uptake, ethnicity and vaccination uptake, or geographical location and
vaccination uptake), and the aspects you understand as being key elements of that
or those concept(s). These aspects are theoretically or empirically derived at the
outset of your study.
2. You formulate a hypothesis, that is, a testable statement that spells out specifically
what it is you assume is true about the variables in your study (e.g., teenagers
are less likely to make decisions about whether to have a vaccine based on media
coverage about that vaccine compared to people over 20 years of age).
3. You collect data using a fixed measurement instrument (e.g., a questionnaire)
designed to produce exactly the data you need to assess the plausibility of your
hypothesis.
4. By paying careful attention to the design of the measurement instrument, the
selection of participants, and the way you go about collecting your data, you can
assess the degree to which your design and conduct of the study meet statistically
derived criteria for arriving at credible research findings that can tell you whether
or not your hypothesis is supported.
Qualitative Approaches Predominantly Draw on Inductive Reasoning
On the other hand, qualitative research designs and approaches are most often associated with inductively based reasoning where building theoretical and empirical concepts
and understandings is the end point, rather than the starting point, for the research. As
we discussed in Chapter 3, inquiry based on inductive reasoning does not set out with a
predetermined set of specific variables to statistically assess in some way. Rather it sets out
to obtain data that will enable the emergence of rich and thick description6 about, and
in-depth understandings of, the problem or issue that the research is being designed to
address. Theory can then be generated inductively from this “bottom-up” data.
For example, within the broad area the of media coverage of vaccination, a researcher
might be interested in focusing on how people experience and make sense of that media coverage about a particular vaccine. Does this coverage affect their choice to either be, or not be,
vaccinated, and why? Is there anything else that affects their choice, and if so, what and why?
In this example, the researcher has not set out to deductively test specific aspects of the
link between media coverage and vaccination uptake, such as age, that they have hypothesized in advance. Instead, they are interested in exploring participants’ viewpoints about
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
95
that media coverage, that vaccine, and if this affected their decision about whether to have
the vaccination. Why or why not? If so, how did it affect that decision? Therefore, they
work inductively to find out about what is going on in the situation. In this case, they will
draw on strategies of inquiry derived from qualitative approaches.
Another Word of Caution
Another word of caution is needed at this point. It is too simplistic to define what quantitative and qualitative approaches are on the basis of the logic of inquiry on which they draw—
for example, that qualitative approaches are entirely inductive in nature versus quantitative
approaches which are entirely deductive in nature. While it is the case that most qualitative research approaches are predominantly inductive, it is important to note that at times
a researcher doing a qualitative study will follow up on issues or ideas that arise from the
analysis of interviews or observations to find out more about that issue or idea. Therefore,
the study will involve both inductive and deductive thinking throughout various points in
that research design. The Putting It Into Practice box below has an example of this.
We also point out that “nothing is wrong with taking a deductive approach” (Yin, 2016,
p. 100) in a qualitative study from the outset if the issue being studied or the research questions
require it. However, it is an unusual point of departure for a qualitatively driven study. There
is, however, definitely something wrong with saying that all qualitative approaches must take
a deductive approach and that the questions guiding that study must be in hypothesis form.
PUTTING IT INTO PRACTICE
AN EXAMPLE OF A RESEARCH DESIGN DESCRIBED AS
“QUALITATIVE” THAT MOVES BETWEEN INDUCTIVE AND
DEDUCTIVE THINKING THROUGHOUT VARIOUS PARTS OF
THAT RESEARCH DESIGN
We might begin a qualitative interview study about older people’s experience of interacting with technology. We begin inductively with no specific pre-research theoretically derived concepts to be focused on. The aim is to let the older person relate their
experiences and the aspects of that experience that they consider important, rather
than asking them to tell us about aspects of that experience that we have identified as
being important or the focus of our study.
During the analysis of our interviews, certain technologies might emerge as
more problematic than others in terms of the older person’s experiences of interacting with them. We then might want to know more about why these technologies are
more problematic than others for older people when interacting with them. Therefore,
we might choose to conduct follow-up interviews specifically about this with those
already interviewed, or we might identify and probe7 this issue when interviewing new
participants.
When we do this, we are acting deductively. However, we still allow participants
to tell us what they find problematic, and thereby inductively explore what makes or
why some technologies have been identified as being more problematic than others.
We also might search the research literature to see if other studies related to older
persons’ experiences of interacting with technology have identified this as a factor. If
they have, then we can compare these studies’ findings to our own. In doing so, we are
again acting deductively.8
96 Research Design
Rounding Off Our Introductory Discussion of Qualitative and
Quantitative Research Strategies
What makes a research approach or design or aspects of that design “qualitative”? What
makes them “quantitative”? As we have seen, these questions have no short or easy answers.
This is because what makes an approach to research either qualitative or quantitative lies
in the methodological thinking underpinning each of these approaches, and the assumptions about research and data embedded in this thinking.
Both qualitative and quantitative research approaches are strategies of inquiry
(Denzin & Lincoln, 2018a). These qualitative and quantitative research strategies “connect researchers to specific approaches and methods for collecting and analyzing empirical
materials” (Denzin & Lincoln, 2018a, p. 313). What specific qualitative or quantitative
approaches and methods you will include in your research design will depend on what
type of knowledge you will need to address the research question that underpins your
research.
This is why Guba and Lincoln (1994) point out that concepts and terms such as qualitative and quantitative research are “secondary to questions of paradigm, which we define
as the basic belief system or worldview that guides the investigator, not only in choices
of method but in ontologically and epistemologically fundamental ways” (p. 105). It is
these basic belief systems or worldviews (paradigms of some sort), and the logic of inquiry
associated with them,9 that result in different forms of data (e.g., numbers and words and
what type of numbers and words) being produced by the use of different methods that use
different logics of inquiry to answer different types of research questions.
With this in mind, this chapter aims to get beyond simplistic nonnumerical (e.g.,
words) versus numerical definitions of qualitative and quantitative research approaches to
explore the thinking that sits in each of these approaches, and which gives rise to common
features associated with qualitative and quantitative research approaches, respectively. In
the next section, we discuss what these common features are.
COMMON FEATURES ASSOCIATED WITH
QUANTITATIVE AND QUALITATIVE APPROACHES
It is possible to identify some common features of the ways of thinking that underpin
quantitative, and qualitative, research strategies. It is important to remember that these
features cannot be understood in isolation from each other. It is their interconnections
that shape what we understand quantitative and qualitative approaches to research to be.
Common Features Associated With Quantitative Ways of Thinking
When Designing Research
Common features of a quantitative way of thinking about research and ways to design and
enact that research include the following.
1. Quantitative approaches to research are employed when the answer to our
research question(s) (i.e., our methodological considerations) requires the
production of some form of numerical data, which can then be analyzed using
numerically based statistically derived strategies. As Greener (2011) puts it,
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
97
“Quantitative research is primarily concerned with techniques that analyze
numbers. If we are calculating descriptive statistics (calculating averages,
probabilities or exploring numerical relationships), then we are doing quantitative
research” (pp. 2–3).
2. These approaches use a predominantly deductive way of thinking when deciding
which questions to ask, what data to collect, and how to analyze that data. You
will see this type of research referred to hypothetico-deductive.
3. Consequently, the methodology underpinning quantitative research aims at
deriving estimates and deducing properties about a population supported by
statistical analyses of data from a subset of that population. In other words,
statistics are used as a tool to transform numerical data to research findings that
can be generalized to a large group of people.10
4. This requires the use of standardized instruments for data collection. Such
a standardized instrument may, for example, be a questionnaire as part of a
quantitative survey. This questionnaire takes the form of a predefined list of
questions, often with predefined response options. Requesting people’s answers
to the specific list of questions makes it possible to fit “the varying perspectives
and experiences of people . . . into a limited number of predetermined response
categories to which numbers are assigned” (Patton, 2015, p. 22). The interest of
the researcher “is in the aggregates rather than particular individuals” (Presser,
1985, p. 95).
5. Quantitative research designs, and methodologies and the methods that arise
from those methodologies, are associated with, and derived from, positivist or
post-positivist thought.
6. Given an emphasis on objectivity in positivist and post-positivist thought,
quantitatively driven research approaches are designed to eliminate any form of
subjective bias on the part of the researcher. This means that the ideal role of the
researcher is that of the distant impassive observer.11
7. Therefore, quantitative approaches typically aim to arrive at objective, conclusive
answers able to be applied to large populations of people by generalizing
the results of statistically analyzed data from a relatively small subset of that
population.
8. The number of people that we collect this variable specific data from, as well as
the strategy for selecting those people, is determined by the rules of statistics
and probability. You will often see sampling in quantitative research described
as being probabilistic or aiming for statistical representativeness. This point is
developed further in Chapter 8 of this book.
9. In quantitative approaches, the people in our research, our sample, are chosen
because they exhibit or “fit” the parameters of the variables of interest in our
study. For example, they might be of a certain age, living in a certain area, and of
a particular income group. They are of interest to the study only in terms of these
specific parameters.
98 Research Design
10. The credibility of quantitative research is judged on the basis of objective
measures related to issues such as (a) the process used for selecting participants,
(b) how many participate in the study, (c) how well the instrument you are
using to measure the variables of the study actually does measure those variables,
(d) how well the data produced by the measurement instrument fits with the
statistical analyses performed on that data, as well as (e) the interpretation of the
outcomes of those analyses. We return to unpack these points (a–e) further in
Chapters 8 and 9.
Common Features Associated With Qualitative Ways of Thinking
When Designing Research
While qualitative approaches to research vary, it is possible to identify some common features and/or principles of the type of thinking involved in qualitative research
approaches:
1. Qualitative research, and the type of data that arises from it, is situated in the
everyday reality or social world of the participants and sites that makes up the
context for a specific study. Mayan (2009) calls this trying “to make sense of life
as it unfolds” (p. 11).
2. The researcher observes and explores aspects of the world of the participants in
the study in order to understand the way that they perceive, experience, and make
sense of that world. “[A]ll qualitative research is interested in how meaning is
constructed, how people make sense of their lives and their worlds. The primary
goal of a basic qualitative study is to uncover and interpret these meanings”
(Merriam & Tisdell, 2016, p. 25).
3. Consequently, the way that the data is collected is termed “naturalistic” in that it
is directed toward understanding what is going on in a specific context or setting
under everyday conditions. There is no experimental, or researcher, manipulation
of that setting. Thus, “People will be performing in their everyday roles or will
have expressed themselves through their own diaries, journals, writing and
photography – entirely independent of any research inquiry” (Yin, 2016, p. 9).
4. Information about the context and setting of the research (the everyday of
people’s lives) is as much a part of the data and findings of a study as are any data
produced by, for example, an interview or observation conducted in that context.
5. The aim is for in-depth information or data from participants in the research on
which to develop and base thick interpretations of “what is going on” from the
point of view of people in a particular situation or context.
6. Hence, qualitative research is primarily inductive as the information obtained
from research participants is used to build understandings of the aspects of the
social constructions that we are focusing on in our research.
7. Qualitative inquiry is premised on the ontological position that the world can
be understood as made up of socially constructed meanings. It is these everyday
social interactions and constructions that make up the way the world is. The
world does not exist independently of those in it.
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
99
8. Consequently, qualitative inquiry is associated with non-positivist (such as
constructivist) inquiry paradigms that allow for the idea of multiple realities
which exist “in the form of multiple mental constructions, socially and
experientially based, local and specific, dependent for their form and content on
the persons who hold them” (Guba, 1990, p. 27).
9. Done well, qualitative inquiry generates large amounts of in-depth and rich
data about the participants and sites that the research is designed to find
out something about. Thus, you will see qualitative data and approaches
being described as providing multiple sources of rich or thick information or
description about a situation or experience.
10. The researcher is present and actively involved in the research. They are “the
primary instrument for data collection and analysis” (Merriam & Tisdell, 2016,
p. 16). Most often they will collect the data and interact with the participants
and study sites. For as Denzin (2010) reminds us, “The qualitative researcher
is not an objective, politically neutral observer who stands outside and above
the study of the social world. . . . A gendered, historical self is brought to this
process” (p. 23).
11. Since the aim is for in-depth information, the researcher deliberately and
carefully selects information-rich participants or sites for study. This is why
you will often see sampling in qualitative inquiry described as purposeful or
purposive as the researcher purposefully seeks to find people to interview who
are information rich about what it is that we are trying to find out about—the
purpose of the study.
12. Since purposeful sampling is based on logic and criteria derived from qualitative
thinking, and not on logic and criteria derived from the laws of probability, it is
referred to as non-probabilistic sampling.
13. The number of participants and/or sites in a qualitative study is usually smaller
than in quantitative inquiry where the sample size is driven by statistical
considerations. However, the numerically smaller sample in a qualitative study
will contain more in-depth data about each person or site in the study. Such
in-depth and rich information will enable the emergence of rich and thick
description.
14. In qualitative research approaches, data analysis employs an “art of
interpretation” (Denzin & Lincoln, 2000, p. xii) and aims for “thick
interpretations” (Denzin, 2001, p. 52).12 This means that analysis of qualitative
data does not simply describe what people said or did, but also interprets it.
15. During the analytical process, our thinking about, and working with, the
data continually moves forwards and backwards examining, re-examining,
thinking more about, collecting more data when necessary, and interpreting and
re-interpreting our data.13
16. The idea of trustworthiness is used to make decisions about the veracity of the
research. The trustworthiness and authenticity of the reconstructed accounts of
aspects of the social world that inform the findings of the qualitative research is
what makes that research credible.14
100 Research Design
TIP
DO NOT REDUCE YOUR THINKING ABOUT QUANTITATIVE
AND QUALITATIVE RESEARCH APPROACHES TO A SERIES OF
COMPARATIVE DOT POINTS
At this point you may be expecting that we will tidy up and summarize the discussion
so far by producing a table that compares and contrasts quantitative and qualitative
research across a range of selected features captured in comparative dot points (such
as numbers vs. words). By default, such a table then become the “definition” of each
approach—a form of what Koro-Ljungberg (2016) refers to as “easily digestible overviews” (p. 6) of complex matters.
In such an easily digestible overview, quantitative research usually becomes
defined as a research approach (or very often simply as a method) according to a set of
dot points such as it produces data in the form of numbers, takes a deductive approach
to research, uses some form of survey or experiment to collect data, is objective in
nature, and draws on a positivist/post-positivist inquiry paradigm.
Qualitative research is then defined as a research approach (or very often simply
as a method) according to a set of dot points set up in contrast to the corresponding dot points for quantitative research. For example, data is in the form of words
rather than numbers, takes an inductive approach to research, uses some form of
interviews, observations, or texts to collect data, is subjective in nature, and draws on
non-positivist inquiry paradigms such as constructivism.
An example of such a table is the one we have developed below (Table 5.1) and
called An Example of the Definition by Dot Point of Qualitative and Quantitative Research.
Note that in this type of “definition by dot point,” qualitative and quantitative research
approaches are effectively set up as “polar opposites” (Crotty, 1998, p. 15). Each
approach is defined, whether intended or not, in terms of how it differs from the other
related to a series of criteria.
TABLE 5.1 ■ An Example of the Definition by Dot Point of Qualitative
and Quantitative Research
Criteria
Qualitative Research
Quantitative Research
Form data takes
Words
Numbers
Data collection
Interviews, observations,
textual analysis
Surveys, experiments
Type of logic
Inductive
Deductive
Type of paradigm
Non-positivist (such as
constructivist)
Post-positivist
Typical questions
addressing
Individual experiences,
feelings
How many, how much
Analytic Focus
Individuals—their subjective
experiences, perceptions,
understandings
Large populations—objective
measurements and predictions
about how many or how much
Note: If you are thinking of only using this table to define qualitative and quantitative research, please
read the text in this section to find out why maybe this is not such a good idea. The text reveals that such
dot points only make sense in light of the thinking that gave rise to them in the first place.
However, simply using a series of comparative dot points such as the ones in Table
5.1 to base your understandings of quantitative and qualitative research approaches
on is inadequate and limited. The thinking that sits behind, and which produced, those
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
101
dot points is missing. Further, such dot points risk losing sight of the fact that there is
not just one way of thinking about, describing, designing, or doing either quantitative or
qualitative research, just as there is not one form of the numbers or words that make
up quantitative or qualitative data. Consequently, what we can, and often do, end up
with are sets of fairly normative, somewhat empty, simplified dot points that do not
reflect either the complexity or the diversity that is embedded in, and makes up, the
terms qualitative and quantitative.
Therefore, while dot points might be a suitable device to sum up key points of an
extended discussion about the thinking that makes up qualitative and quantitative
approaches, detached from that extended discussion they are not. It is important to
have had the extended discussion before summarizing it in the form of dot points. This
is because the dot points cannot be understood removed from the thinking that gave
rise to them—the assumptions and understandings that sit behind each dot point.
VARIATION WITHIN QUANTITATIVE AND
QUALITATIVE RESEARCH APPROACHES
Although it is possible to identify features common to quantitative research approaches
and features common to qualitative research approaches, it is important to keep in mind
that specific quantitative and specific qualitative approaches may vary widely. Not all
approaches to quantitative and qualitative research are the same; they use different quantitative or different qualitative strategies of inquiry.
Consequently, both quantitative and qualitative research approaches can be thought of as
fields of study in their own right. In the same way that applied fields of study such as education, business, and nursing are made up of different schools of thought both theoretically and
methodologically, so are the fields of quantitative research and qualitative inquiry. Therefore,
the terms quantitative and qualitative when applied to research, data, analyses, and methods
are umbrella terms, neither of which can be reduced to a single set of understandings.
Just because two studies use quantitative research approaches, it does not necessarily
mean that they will use those approaches in the same way. For example, they may use different
methods to collect their quantitative data, or they may collect different types of quantitative
data, or use different types of procedures to analyze that quantitative data. However, what the
two quantitative studies will have in common is that they will be designed in such a way that
they are in keeping with agreed principles or features common to all quantitative studies.
Similarly, just because two studies use qualitative research approaches, this does not
mean they will necessarily be designed in the same way. For example, they may use different
methods to obtain their qualitative data, or they may obtain different types of qualitative
data, or use different types of processes to analyze that qualitative data. However, what the
two qualitative studies will have in common is that they will be designed in such a way that
they are in keeping with agreed principles or features common to all qualitative studies.
Quantitative Inquiry as a Diverse Approach
There is variation between research approaches described as “quantitative.” A broad distinction can be made between descriptive approaches suitable for identifying and describing population characteristics, correlational approaches suitable for establishing the
existence of, and interpreting, relationships between such population characteristics, and
experimental/quasi-experimental approaches that aim to establish cause–effect relationships between the characteristics. What distinguishes these approaches from each other
102 Research Design
is what they enable the researcher to say something about at the end of that quantitative
study. Or, put another way, their potential, and therefore use, to resolve different types of
quantitatively focused research questions. Descriptive and correlational approaches have
the potential to address research questions about what is going on in a study population,
while experimental/quasi-experimental approaches have the potential to address research
questions about why something happens.15
For example, if your research question is about identifying phenomena in a population, or to compare the occurrence of phenomena across subgroups within that population, you will apply a descriptive quantitative approach. This is because a descriptive
quantitative approach will enable you to map the state of things in that population, such as
vaccination uptake across age, ethnicity, and geographical areas
However, if your research question is about identifying trends and patterns related to what
you are interested in, you will apply a correlational approach allowing for the detection and
description of such patterns. For example, a correlational approach will allow you to investigate
whether there is a pattern such as lower age corresponding to lower vaccination uptake, or
less distance from the nearest vaccination center corresponding to higher vaccination uptake.
Hence, a correlational approach allows you to say something about “what goes with what”
(Oppenheim, 1992, p. 21) in a population. However, it is important to be aware that a correlational approach will not enable you to say anything about why x goes with y. To explain why
x goes with y (for example, why lower age is connected to lower vaccination uptake), you need
to apply an experimental/quasi-experimental approach. Such approaches enable you to make
credible claims about cause and effect or causality. Or put another way, why x goes with y.
In experimental or quasi-experimental research approaches, you develop a highly
structured research design, making sure all variables under consideration are taken into
account, while at the same time making sure that they are not affected by any other factors
that might influence the outcome. An example of this type of highly structured quantitative research design is a Randomized Controlled Trial (RCT).
A Randomized Controlled Trial (RCT) is a type of experiment where you randomly
assign people to an experimental group and a control group to study the effect of a drug,
treatment, or other intervention on a variable of interest. The intervention being tested is
applied to the experimental group, but not to the control group. Such random assignment
underpins the assumption that the groups cannot be distinguished by any common factor other than the application of the intervention to the experimental group. The study
is conducted to find out whether there is a statistically significant difference related to
the variable in question between the group having had the experimental intervention and
the control group. If there is, the study provides support for a claim that there may be
a causal connection between the intervention being tested and the variable in question.
This approach attempts to explain why something is as it is by employing methods and
research designs that resemble those of laboratory experiments.
PUTTING IT INTO PRACTICE
PUTTING THE THINKING BEHIND THE RCT INTO PRACTICE
To determine how a new type of shoe insole affects the development of foot corns and
calluses, 100 trial participants can be randomly separated into equal groups of 50—
the experimental group and the control group. All participant’s feet are then examined
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
103
to map the pretrial prevalence of corns and calluses in both groups. During the course
of the experiment, the experimental group is treated in one way (they start wearing the
new type of insole), while the control group remains as it was prior to the experiment
(they make no changes to their shoes).
After the experiment, you compare the experimental group to the control group
to see if there are differences between the two groups related to the characteristics
in question (i.e., you compare the prevalence of corns and calluses across the two
groups). If the groups are different after the experiment (i.e., the prevalence of corns
and calluses has changed more in one group than the other), you know that the new
type of insole is in fact related to the development of foot corns and calluses. Moreover,
assume your research design includes making sure that during the experiment the
participants act in exactly the same way as they did prior to participating, for example,
use the same socks, walk the same distances, in the same type of shoes. Then, you
may argue that the observed differences in prevalence of corns and calluses across
the two groups after the experiment can be attributed to the effect of the treatment
(i.e., wearing the new type of insoles), and not something else. In other words, you can
conclude that in all probability, the treatment caused the characteristics in question to
become different between the two groups.
Quantitative Approaches Vary in the Methods That They Use
Within quantitative approaches, there is a variety of specific methods that can be used to
collect the data, as well as a variety of statistical procedures that can be used when analyzing that data. Which methods and which statistical procedures you choose to incorporate
in your research design will depend on your research questions or hypotheses.
For example, quantitative approaches often involve the use of some sort of quantitative survey as a data collection tool. However, not all surveys are the same. Therefore,
even when you have decided that you will use some sort of survey, you will still have to
decide which type of survey you will use. If you want to make inferences about a specific
characteristic to a broader population, such as the population of all individuals eligible for
a vaccine during a specific time period, you will need a survey design reflecting that aim.
Alternatively, if you want to know about how often a particular income related characteristic (e.g., high or low income) occurs together with opting out of an offered vaccine program, then your survey must be designed to enable such findings, that is, what goes with
what or the correlations between the variables of interest (income and nonparticipation in
the vaccine program) in your study population.
Moreover, a survey conducted to test whether empirical data supports a hypothesis
is different from a survey conducted to describe the state of things in a population (with
no a priori assumptions about the state of things). The difference between all these types
of surveys is what the data obtained by them will enable you to say something about after
that data has been analyzed. A survey conducted to test whether empirical data supports a
hypothesis will be constructed around the variables that that hypothesis is about. On the
other hand, a survey designed to describe the characteristics of a population will collect
a wider set of data than just data related to variables making up a hypothesis. Therefore,
which type of survey you use depends on the aims of the study.
Capturing the Variety in Quantitative Approaches
We have tried to capture the idea of the diversity in quantitative approaches, at the level
of what the different approaches enable you to say something about, in Figure 5.1 below.
We are not suggesting that these are all possible approaches to quantitative research, but
FIGURE 5.1 ■
Examples of Diversity in Quantitative Approaches
104 Research Design
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
105
simply using the diagram to demonstrate the diversity of research approaches that fits
under the umbrella term quantitative approaches.
What Figure 5.1 illustrates is that there are cascading layers of decisions to be thought
through, and then made, when thinking about designing research using quantitative
approaches. All of these decisions connect to the purpose of your research—your research
questions and the issue your research is being designed to address. You will need to think
through these layers when developing your research design. The box below provides you
with some tips for how you might begin to do this.
TIP
NAVIGATING THE DIVERSITY OF QUANTITATIVE APPROACHES
WHEN DESIGNING YOUR RESEARCH
Given the variation that exists between, and within, quantitative research approaches,
if you are employing some sort of quantitative approach in your research, then you will
need to think through and be able to justify all the decisions you make related to choosing that approach. For example,
•
•
Why use a quantitative approach as the organizing construct for your study?
•
•
What do you base that decision on?
•
How will your answers to these questions shape your research design?
What type of quantitative research will be appropriate for addressing what you want
to know about, and be able to say something about, at the end of the study?
If you are using a survey, what type of survey will you use to collect your quantitative
data and why?
Being able to answer these questions, and justify your answers, is an important
part of ensuring the credibility of your quantitative research. Just stating that your
research is a quantitative study, or a particular type of quantitative research such as a
survey study, is not enough.
Qualitative Research Approaches as Diverse Strategies of Inquiry
There is a great deal of diversity among qualitative research approaches and the strategies
of inquiry that they employ. This makes it difficult to come up with a simple definition
of the field of qualitative inquiry, or an overview of the strategies of inquiries that qualitative approaches can use, given the breadth of the field. One way to capture, convey, and
get a sense of this diversity is the idea of different territories (Tracy, 2020) within the
field or landscape of qualitative research. There is a common territory that all qualitative
approaches share. This is a territory demarcated by the common features of qualitative
research approaches that we mapped out earlier in this chapter.
Studies that draw on these common features are referred to as types of a “basic” form
of qualitative research (Merriam & Tisdell, 2016, p. 23) or “a kind of generalized qualitative research” (Yin, 2016, p. 66). Such studies are not basic because in some way they simplify the principles of qualitative inquiry. Rather they are basic or generalized because the
research design of the studies draws on basic, or general, methodologically derived understandings and principles germane to all qualitative research approaches. Examples of such
studies would be designing a research study using basic qualitative principles to inform the
106 Research Design
use of interviews or observations “to answer concrete program and organizational questions” (Patton, 2002, p. 145).
For example, the study by Cheek and Ballantyne (2001) used basic qualitative principles to inform the use of interviews to answer questions about how older people and their
family members perceived, and experienced, the move of the older person from home to
acute care, and then to an aged care facility:
This exploratory, descriptive study examined the search and selection process for
an aged care facility following discharge of a family member from an acute setting.
Few studies have examined this process and its effects on families. Individuals from
25 families where a family member had been recently admitted to an aged care
facility following discharge from an acute setting were interviewed. This article
reports participants’ perceptions of the search and selection process and its effect on
the family. Five major themes emerged from the data: good fortune, wear and tear
on the sponsor, dealing with the system, urgency, and adjusting. The results can be
used to inform and assist families and health professionals working with families in
this situation. (Cheek & Ballantyne, 2001, p. 221)
More Specialized Forms of Qualitative Research
However, there are also approaches within the field of qualitative research that draw on more
specialized (Yin, 2016) strategies of inquiry. You may find it useful to think of each of these
specialized approaches to qualitative inquiry as specific or specialized territories within the
general territory or field of qualitative inquiry. Each specific territory, or specialized type of
approach, has its own theoretical and methods related traditions. Each is also “connected
to a complex literature; each has a separate history, exemplary works, and preferred ways
for putting the strategy into motion” (Denzin & Lincoln, 2018b, p. 21). This means that
different specialized approaches to qualitative research will be designed differently to reflect
those theoretical and methods related traditions. It also means that different qualitative
approaches will be more suited to some types of research questions than to others.
Ethnography as an Example of a Specialist Qualitative Approach
For example, consider the specialist qualitative approach known as ethnography. This is
an approach that has been central in many disciplines and fields of study—most notably
the fields of anthropology and sociology, but more recently fields and disciplines as diverse
as business, organizational studies, nursing, and education. It is one of the foundational
methods of qualitative inquiry.
The aim of an ethnographic study is for the researcher to immerse themselves in the setting being studied in order to understand the culture of that setting from the point of view,
understandings, or perspectives of those being studied. You will see this referred to as taking
an emic view. The idea of an emic view is captured well by Spradley (1979) when he wrote,
I want to understand the world from your point of view. I want to know what you
know in the way that you know it. I want to understand the meaning of your experience, to walk in your shoes, to feel things as you feel them, to explain things as
you explain them. Will you become my teacher and help me understand? (p. 34)
It is the focus on the theoretical idea of culture that makes a study ethnographic. Put
another way, “to be an ethnographic study, the lens of culture must be used to understand
the phenomenon” (Merriam & Tisdell, 2016, p. 31) that is being studied. Culture can be
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
107
understood as “the various ways different groups go about their lives and . . . the belief
systems associated with that behavior” (Wolcott, 2008, p. 22). When designing an ethnographic research study, this lens must be added to the basic or general methodological
considerations that underpin and make up any form of qualitative inquiry.
Ethnographic research involves the researcher becoming immersed in the culture of
interest (sometimes referred to as undertaking fieldwork) over a period of time. During
this time the researcher uses data derived from observations, interviews, documents, and
other texts such as pictures and image, to build a “thick description” (Geertz, 1973) of the
culture that the research is designed to provide in-depth understandings of. For example,
the culture of an institution/organization such as a prison (Cashin et al., 2010; Wacquant,
2002), or a group within society such as a sporting club (Bolin & Granskog, 2003), or a
particular ward or setting in a hospital (Bjerknes & Bjørk, 2012).
Just as there is a great deal of variety among qualitative approaches to inquiry generally, there is also a great deal of variety in research approaches that call themselves
ethnographies. For example, you may come across studies described as critical ethnography, feminist ethnography, institutional ethnography, focused ethnography, and
auto-ethnography. Each type of ethnography varies in the theoretical understandings that
underpin and shape the overall ethnographic study design.
Activity
Taking a Look at Ethnography in Action
Obtain a copy of the study by Bjerknes and Bjørk (2012), Entry into Nursing: An
Ethnographic Study of Newly Qualified Nurses Taking on the Nursing Role in a Hospital
Setting.
Read it and then answer the following questions:
1. Why was this study suited to an ethnographic qualitative research approach?
2. What culture was being studied?
3. What types of data were collected and why were they appropriate?
4. Did the study take an emic view—what do you base your answer on?
Remember, even though your field of study may not be nursing, you should be able to
make decisions about this study in terms of its claims to be an ethnography.
Discourse Analysis—Another Form of Specialized Qualitative Inquiry
However, not all research questions lending themselves to some sort of qualitative strategy of
inquiry will require an ethnographic approach. If, for example, you want to know something
about how print-based and digital media have influenced and shaped societal attitudes to
understandings of risk related to vaccines during the COVID-19 pandemic, you may choose
to employ a different form of specialist qualitative inquiry known as discourse analysis.
Discourse analysis is premised on the understanding of “language as a meaning constituting system which is both historically and socially situated” (Cheek & Rudge, 1994,
p. 59). Therefore, the focus of analysis is the meaning constituting systems in texts generated from, for example, interviews, news articles, or visual texts such as, pictures and
films (Taylor, 2013). How did these texts come to be the way that they are in terms of the
meanings that they convey, for example, about risk and vaccines, and what sustains the
understanding that they produce about that risk? When designing qualitative discourse
108 Research Design
analysis studies, this focus or lens must be added to the basic or general methodological
considerations that underpin and make up any form of qualitative inquiry.
Like other qualitative approaches, “discourse analysis is not a unified, unitary approach”
(Cheek, 2004, p. 1144). It “does not refer to a single approach or method . . . [rather it]
. . . refers to a range of approaches in several disciplines and theoretical traditions” (Taylor,
2013, p. 1). As a strategy of inquiry, discourse analysis is understood in different ways by
different groups of researchers. This is influenced by the theoretical and methodological
lens a researcher uses when thinking about what discourse analysis is and how it might be
put into practice. Which approach or method is used by a researcher depends on the theoretical assumptions made by a researcher about what discourse is and what a discourse analysis might involve. Thus, it is possible to design discourse analysis in very different ways.16
For example, discourse analysis drawing on traditions drawn from the area of linguistics will vary from discourse analysis drawing on Foucauldian theory both in focus (i.e.,
the types of questions about discourse that guide the study) and design (i.e., how that
study will be carried out). Thus, “Discourse analysis will always be, because of its interdisciplinary origins, a multiperspective approach with different emphases and understandings in use depending on the position adopted by the researcher employing the approach.
There cannot be “the” set of rules for discourse analysis” (Cheek, 2004, p. 1148).
TIP
WHERE TO START NAVIGATING THE FIELD OF DISCOURSE
ANALYSIS
A good starting point to see what a qualitative study using a discourse analysis approach
involves, and how it can be designed, is Chapter 3 in Stephanie Taylor’s book What Is
Discourse Analysis? (2013). In this chapter, she discusses four examples of research
published that has used some form of discourse analysis as the organizing lens for the
study. At the end of each example, she provides a summary about what we can learn
from this example about discourse analysis and designing research using this strategy
of inquiry.
Another starting point is the book Doing Discourse Research: An Introduction for
Social Scientists by Reiner Keller (2013), which provides a thorough introduction to the
basic principles of discourse research and provides practical strategies for doing discourse analysis. Part of Chapter 2 in the book, pages 13 to 32, is devoted to a discussion
of different approaches to discourse analysis.
How Many Specialist Types of Qualitative Inquiry Are There and What
Are They?
There is no agreement about how many types of specialist qualitative research there are.
For example, Yin (2016) identifies 12 frequently cited specialized types or variants of qualitative research that can be distinguished from one another in terms of the differences in
the theoretical and methodological lenses that shape the strategies of inquiry that make
up their design.
These are action research, arts-based research, autoethnography, case study, critical
theory, discourse analysis, ethnography, ethnomethodology, grounded theory, narrative
inquiry and life history, oral history, and phenomenology (Yin, 2016, pp. 68–70). Each
reflects a specific way of thinking about what will be studied and how.
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
109
What this highlights is that the theoretical position of a qualitative researcher undertaking a study using any specialized type or variant of qualitative research will affect the
shape that the specialized strategy takes. This means that if you do choose to use one of the
specialist types of qualitative inquiry, or call your study an ethnographic study, discourse
analysis, grounded theory, or any other specialist name, you will need to read up and learn
more about that specialist approach before you design your research. This means that “you
should have carefully reviewed the specialized literature, incorporated key concepts, and
emulated the specialist methods in doing your study” (Yin, 2016, p. 67).
TIP
APPLYING A QUALITATIVE LENS TO ONLINE SOCIAL CONTEXTS
It is also possible to view the internet, and the various online environments related to it,
as a place in and of itself to be studied, a specific type of virtual social context to which
a qualitative lens can be applied (Markham, 2018). For example, some qualitative study
designs are forms of ethnographies of the spaces and contexts created in this virtual
world (see Boellstorff et al., 2012). In this type of research, you will choose to observe
online environments, sites, or groups that are information rich in terms of the information you seek to address your research questions. New and different questions arise
about your role of observer and the extent to which you participate in those sites.
Capturing the Variety of Qualitative Approaches
Denzin (2008, 2010) uses the metaphor of the big tent to capture the idea of the diversity
of qualitative approaches. The tent is not a fixed size that limits the possibilities for what
can be known as qualitative research. Rather the tent can be made bigger as qualitative
inquiry grows and develops. There is room for many different approaches to qualitative
inquiry in this tent.
In Figure 5.2, we have tried to capture the idea of such variation within qualitative approaches by building on and adapting Merriam and Tisdell’s (2016, p. 42) figure
of Types of Qualitative Research. Our additions, drawing on Yin 2016, are in gray.
When reading this diagram, keep in mind that we are not suggesting that these are the
14 approaches to qualitative research, or the only way we might think about types of
qualitative approaches. We are simply using the diagram to demonstrate the diversity of
research approaches that fits under the umbrella term qualitative approaches.
It is important to remember that every one of the approaches in Figure 5.2 has its own
theoretical understandings and focus. Further, each approach to qualitative research has
accepted strategies of inquiry and ways of putting those strategies into practice. Therefore,
how you actually design and put that research design into practice if you are using
grounded theory will be different than if you are using discourse analysis or ethnography.
It is beyond the scope of the focus of this book, which is research design, to discuss each of these approaches. Why we mention them is to point out that you will come
across these terms, and if you are going to name your study as “a [. . .] specialist research
design” (e.g., a ethnographic study, or a discourse analysis, or a grounded theory, or any
other type of specialist qualitative research), then you have another whole series of design
considerations to make. These considerations are related to the effect that the theoretical
and methodological understandings the particular specialist approach has on the way you
will design your research.
110 Research Design
As we emphasized previously, this means that you will need to read a lot about the
specific type of qualitative research approach you have chosen to use in your research and
make sure that your study is in keeping with the accepted principles of that approach.
Like your initial choice of a qualitative approach, the type of qualitative approach that you
choose to use will need to be justified by demonstrating why it is an appropriate choice in
light of the purpose of your research and how using that specific approach will enable you
to achieve credible research findings. This is particularly so given the variety that exists
within each type of qualitative inquiry.
FIGURE 5.2 ■ Diversity in Qualitative Approaches
Activity
Mayan’s Example of the Coffee Shop
Mayan (2009) points out that “[e]thnographers, grounded theorists, phenomenologists, and discourse analysts, for example, can observe the same phenomenon but will
naturally ask different questions of it” (p. 35). She captures this idea well using the
everyday example of qualitative researchers in a coffee shop that focus on different
things when they are in that coffee shop. For example, the ethnographer asks, “What
is coffee shop culture?” (p. 35). To address this question, the ethnographer watches
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
111
how people behave in the coffee shop, for example, what they do when they order
their coffee, what type of coffee they order, and where they drink that coffee and who
with. Are there certain rules or norms of behavior in the coffee shop that regulate
behavior in that space? For example, how people serving coffee act, how people ordering coffee act, and what is acceptable behavior and what is not.
On the other hand, the qualitative researcher who is a discourse analysist focuses on the
language being used in the coffee shop context and its effects. For example, they may
“likely look for the power embedded in more elite or obscure coffee language—‘I’ll have
an el grande, decaf, mocciachino, no fat, no foam’—versus the more plain language (‘I’ll
have a regular coffee’)” (Mayan, 2009, p. 35). Who does the elite or obscure language
include and who does it exclude? How might this work to produce and sustain types of
customers in that coffee shop? How might it create an impression of who belongs and
who does not?
Now, choose another form of specialist qualitative approach from Figure 5.2, find
out about it, and try to work out what they might look for and ask about related to
the social setting called “a coffee shop.” If you are finding this particularly difficult,
Mayan provides two more examples you can refer to.
Summing Up: Design Considerations in Light of the Variety Within
Qualitative Research
What you will need to think through, and make decisions about, when you are
designing qualitative research is not whether there are 12 or 14 types, or variants
of, qualitative inquiry, or more or less. Rather, being aware that there is a great deal
of difference possible between and within research designs and approaches that are
described as qualitative, what you will need to consider, and make decisions about, is
which qualitative approach might be most appropriate for addressing your research
questions and why.
When doing so, it will be important to keep in mind that the fact that there is
this diversity of approaches does not mean that “anything goes” when designing your
qualitative inquiry. Each approach to qualitative research has accepted strategies of
inquiry and ways of putting those strategies into practice. There are some common
understandings that all qualitative approaches draw on, and in order to be credible,
your research design will need to be in keeping with them. There are also some common understandings pertaining to types of specialist types of approaches to qualitative research. Your research design will need to reflect those understandings and show
evidence of you putting them into action in the methods you use, and the way that you
analyze your data.
For now, what we want to highlight is that given the great deal of variety that characterizes the field of inquiry known as qualitative research, there are cascading layers of methodological decisions to be thought about, and then made, when designing research using
qualitative approaches. All of these decisions connect to the purpose of your research—
your research questions and the issue your research is being designed to address. You will
need to think through these layers when developing your research design. The box below
provides you with some tips for how you might begin to do this.
112 Research Design
TIP
NAVIGATING THE DIVERSITY OF QUALITATIVE APPROACHES
If you are in the process of designing, or intend to design, your research using a qualitative approach of some kind, ask the following questions of that design:
•
•
Why use a qualitative approach in the first place?
•
•
What do you base that decision on?
•
How will your answers to these questions shape your research design?
What type or variant of qualitative research will be appropriate for addressing what
you want to know about, and be able to say something about, at the end of the study?
What tradition or school of thought will you draw on within that type or variant of
qualitative research and why?
You will need to be able to justify your answers to these questions when writing
about the way that you designed your research. This will contribute to the trustworthiness and credibility of your research. Just stating that you are doing, or that your
research is, a qualitative study is not enough. What type of approach and strategy of
inquiry are you using and why?
WHICH ARE BETTER: QUALITATIVE OR
QUANTITATIVE RESEARCH APPROACHES?
With such variation and fluidity in research approaches and the type of data that they
produce, it should not surprise you that the relative worth of different types of data, and
the methods used to obtain it, can become a very debated and contested area. For example,
each year when we are teaching a research design class, inevitably questions arise from
students in the class related to which type of data or research approach, quantitative or
qualitative, is the “best” or the “most scientific.”
These questions are a version of the long-standing debates about what science is, and
the onto-epistemological understandings on which that view is premised. As we saw in
Chapter 4 when discussing inquiry paradigms, in turn this leads to debates about ways
that science can (even must) be done. This in turn raises the question of which methodologies and associated methods are scientific and which are not. Often this debate is reduced
to a discussion of “the relative merits of quantitative/experimental methods versus qualitative /naturalistic methods” (Patton, 2015, p. 88). The Activity box below provides a good
example of this. Working through this example provides insights into what this debate
about which approach is better is all about, and how it plays out in practice.
Activity
Exploring a Recent Example of the Quarrel of “Which Are
Better—Qualitative or Quantitative Approaches to Research?”
In 2015, the British Medical Journal (BMJ), a well-respected high impact factor peer
reviewed journal, stated that publishing qualitative studies was an “extremely low priority” for the BMJ. The rationale given for this decision given by BMJ editors was that
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
113
our research shows that they are not as widely accessed, downloaded or cited
as other research. We receive over 8000 submissions a year and accept less
than 4%. We do therefore have to make hard decisions on just how interesting an article will be to our general clinical readers, how much it adds,
and how much practical value it will be. (from “Excerpt from rejection letter tweeted by McGill Qualitative Health Research Group (@MQHRG),
September 30, 2015” in Greenhalgh et al., 2016)
What this in fact was saying was that “no matter how good the qualitative inquiry
might be, its methodology, in effect, automatically precluded it from being published
in The BMJ” (Cheek, 2018a, pp. 51–52). In other words, it was a decision made about
what science is, can and cannot be, and what evidence is, can and cannot be, based on
methodological and onto-epistemological assumptions.
The effect of that decision was to constrain the way in which research could be
thought about and designed methodologically. In turn, this effectively meant that
certain types of research questions and problems were not able to be reported in the
BMJ since the way of addressing them was deemed not scientific. They should be
published elsewhere.
Find out more about the BMJ example in Cheek (2018a) “The BMJ debate and what
it tells us about who says what, when and where, about our qualitative inquiry.” When
doing so, think about how this debate could emerge in the first place, and why it is
important to understand this.
The example in the Activity box reflects the contestation that arises from questions
and views about whether a particular type of method, and the data produced by using it,
can be considered “scientific” and able to provide credible research evidence. It is a version
of while “subjective understandings may be of very great importance in our lives . . . they
constitute an essentially different kind of knowledge from scientifically established facts”
type of thinking (Crotty, 1998, p. 27). As Guba and Lincoln (1994) point out,
Historically there has been a heavy emphasis on quantification in science.
Mathematics is often termed the “queen of sciences,” and those sciences, such as
physics and chemistry, that lend themselves especially well to quantification are
known as “hard.” Less quantifiable arenas . . . particularly the social sciences, are
referred to as “soft.” . . . Scientific maturity is commonly believed to emerge as the
degree of quantification found within a given field increases. (pp. 105–106)
Therefore, it comes as no surprise that some researchers, and editors of journals, reject
“qualitative” or nonnumerical data on the basis of it not being scientific or producing
“hard” data.
For example, MacInnes (2019) writes “Numerical data is an essential way to present
clear and concise evidence for any argument. Be skeptical of arguments made without
numbers!” (p. 117). However, what this statement ignores is that just having numbers as
data does not make something science, or better science than nonnumerical or qualitative
data. Why numbers have been used as data, what numbers have been used as data, how
those numbers have been determined and then analyzed in relation to the context of the
114 Research Design
research design and what the research question is, is what gives a quantitative study validity
and rigor, not the fact that the data is in the form of numbers.
For as MacInnes himself (2019) points out, just using numbers does not make data
stronger or better because, “like any other kind of information, numerical data can be excellent or poor quality, and can be used well or badly” (p. 11). Perhaps then, it would be more
accurate to state that “[n]umerical data is can be an essential way to present clear and concise
evidence for any argument specific research problems or questions. Be skeptical of arguments
made without with numbers until you have ascertained the validity of what those numbers are
being used for to claim!” (MacInnes, 2019, p. 117—our strikethroughs and our additions
in italics). The point here is that what makes numbers “good” data is if they are produced
according to accepted principles for quantitative approaches to research and they provide the
kind of information needed to answer the research problem or questions.17
The same point applies equally well to qualitative data—namely that this data “can be
excellent or poor quality, and can be used well or badly” (MacInnes, 2019, p. 11). Whether
it is excellent or poor depends on if this data is produced according to accepted principles
for this type of research and data generation and provide the kind of information needed
to answer the research problem or questions.18
The question then is not what the “best” form or type of data is—quantitative numbers or qualitative words. Rather the question to ask yourself, and think through, is what
type of data will enable you to make the interpretations will you need to make in order to
answer the research questions or problem that you want to be able to say something about
at the end of the study. Equally importantly, you will also need to ask yourself what won’t
you be able to say something about, and does this matter? Why or why not?
Therefore, the answer to the question of which is “better”—qualitative or quantitative
research approaches—is “it depends,” because this question can only be answered in relation to a specific research design and a specific research problem and questions. We agree
with the renowned quantitative researcher Bram Oppenheim (1992) when he states that
“choosing the best design or best method is a matter of appropriateness. No single approach
is always or necessarily superior; it all depends on . . . the type of question to which we seek
an answer” (p. 12).
CONCLUSIONS
Quantitative and qualitative approaches differ from one another because of the different methodological thinking that they draw on and which affects the way that these
approaches are put into practice when designing and conducting research. This includes
the way we think about the purpose of the research + the type of data we will need to
achieve that purpose + the way we can obtain that data + how we make judgments about
the credibility of the research + if the type of information that the research produces can
be considered scientific.
Therefore, it is important not to detach quantitative and qualitative approaches
to research from the methodological thinking and assumptions that shape those
approaches. For example, it is not the fact that data is in the form of numbers in the
case of quantitative research, or in the form of words in the case of qualitative research,
that makes a study quantitative or qualitative in approach. What does, is why the data
takes that form, what numbers or words are collected from whom, how and why, and
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
115
how these numbers and words are analyzed and interpreted. Therefore, defining qualitative and quantitative approaches cannot be reduced to collections of simplistic dichotomies, often in the form of comparative dot points such as words vs numbers or inductive
versus deductive.
In all of this, it is important to remember that there is a great deal of diversity in
approaches to research that fit under the umbrella terms of qualitative research or quantitative research. There are various approaches within qualitative or within quantitative
research that differ in terms of the type of questions they focus on, and therefore the
methods that they use in their research design. Therefore, deciding that a qualitative or
quantitative approach to research is an appropriate approach for the research that you are
designing is just the first of a cascade of interrelated decisions that you will need to make
about that design.
These are decisions related to putting the approach that you have chosen into practice and will have to be thought through regardless of whether you are using qualitative or quantitative approaches. Making these decisions will require you to focus your
thinking at the level of the methods that you will use in your research design. Such
thinking will require reflexive conceptual and intellectual work at the level of how
quantitative or qualitative data will actually be collected and what you will need to
think about when doing so.
For example, what method(s) will you use to collect qualitative or quantitative data and
why? Once you have decided on those methods, what shape will those methods take?
What will you need to decide about them to be able to put them into practice? For
example, who will you collect data from and why? How will you organize and analyze
that data? While answers to these questions will be different in qualitative and quantitative approaches, such incremental, iterative, and connected thinking needs to be done
no matter which of these research approaches, or methods associated with them, you are
using in your research design.
In Chapters 6 to 9, we build on the discussion that we have begun here about qualitative
and quantitative approaches to research. In these chapters, the focus of our discussion
is at the level of methods and the way they are put into practice in qualitative research
approaches (Chapters 6 and 7) and quantitative research approaches (Chapters 8 and 9)
when collecting and analyzing data.
SUMMARY OF KEY POINTS
• Qualitative and quantitative are umbrella terms, each comprising a variety of
approaches to research.
• Quantitative and qualitative approaches differ from one another because of the
different methodological thinking they draw on.
• This methodological thinking affects the way these approaches are put into practice
when designing and conducting research.
• Defining qualitative and quantitative approaches cannot be reduced to collections of
simplistic dichotomies such as words versus numbers, inductive versus deductive.
116 Research Design
• What makes research qualitative or quantitative includes why the data takes a specific
form, what numbers or words are collected from whom, how, and why, and how these
numbers and words are analyzed and interpreted.
• There are different types of quantitative research. They vary in the type of research
question they are able to answer.
• For example, some quantitative approaches enable us to describe what is going on in
a group of people in a situation (descriptive quantitative approach), others enable
us to make links between aspects of what is going on (correlational quantitative
approach), and yet others enable us to say why those links exist (experimental/
quasi-experimental approach).
• There are different types of qualitative research. Some qualitative studies use more
specialized approaches where a particular methodological or theoretical approach
provides a lens for the entire study, for example, ethnography or discourse analysis.
• Which type of qualitative or quantitative approach you choose to underpin your
research design depends on the type of knowledge that you will need to address what
it is that your research is designed to find out more about.
• Neither qualitative nor quantitative approaches are innately better than the other.
Which approach is better depends on what type of data will enable you to make the
interpretations you will need to make in order to answer the research questions that
you want to be able to say something about at the end of your study.
• Deciding that a qualitative or quantitative approach to research is an appropriate
approach for the research that you are designing is just the first of a cascade of
interrelated decisions that you will need to make about that design.
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
correlational approach
descriptive quantitative approach
discourse analysis
ethnography
experimental/quasi-experimental approach
hypothetico-deductive thinking
qualitative approach
quantitative approach
SUPPLEMENTAL ACTIVITIES
1.
Find two published reports of research, one of which reports using qualitative
approaches or describes itself as a qualitative study of some kind and one that uses
quantitative approaches or describes itself as a quantitative study of some kind.
And ask these questions of each article. Do the authors make it clear
• Why they used a qualitative or quantitative approach?
• What types or variants of qualitative and quantitative approaches were used
in the respective reports?
• What tradition or school of thought the authors have drawn on within that
type or variant of the chosen approach, and why?
Chapter 5
•
Qualitative and Quantitative Approaches to Designing Research
117
• Why those approaches were considered appropriate for addressing what the
researchers wanted to be able to say something about at the end of the study?
The answers to these questions are part of establishing the credibility of that reported
research.
2.
If you are in the process of designing research and have chosen either a qualitative or
a quantitative approach, you will need to think about how to put that approach into
practice.
Try to write answers to the following questions:
• Why have I chosen a qualitative or quantitative approach for this study?
• What type of qualitative or quantitative approach am I using, and why?
• What method(s) will I use to collect the qualitative or quantitative data, and
why?
• Once I have decided that, what else will I need to decide in order to be able to
put those methods into practice?
It may be that you are not able to write much about dot points 3 and 4 at this stage.
Therefore, keep what you have written and progressively add to, or modify, it as
you read Chapters 6 through 9. When doing so, also write down why you made
the additions or changes you did. At the end of this reflexive process, you will have
developed a large part of the methodology and methods sections of your research
proposal in terms of what you will do, how, and why.
FURTHER READINGS
Cheek, J., Onslow, M., & Cream, A. (2004). Beyond the divide: Comparing and contrasting
aspects of qualitative and quantitative research approaches. Advances in Speech-Language
Pathology, 6(3), 147–152.
Gorard, S. (2003). Quantitative methods in social science. Continuum.
Tracy, S. J. (2020). Qualitative research methods: Collecting evidence, crafting analysis, communicating impact (2nd ed.). John Wiley . See especially Chapters 1, 2, and 3.
NOTES
1. This point is picked up and developed in Chapters 6 and 7.
2. This point is picked up and developed in Chapters 8 and 9.
3. Uptake: The proportion of the eligible population who accepted a vaccine offered during
a specific time period.
4. At the time of writing this chapter, this is happening in relation to media reporting about
the COVID-19 vaccine known as Astra Zeneca.
5. We already have discussed aspects of this thinking and the way it affects the development and form of research questions in Chapter 3—see the section “Deductive
Reasoning.” Chapters 8 and 9 have an extended discussion of this type of thinking.
6. Clifford Geertz (1973), who drew on Ryle’s (1971) earlier work, is largely credited with
the introduction of the term thick description into qualitative research. Geertz’s original conception of thick description, which was anthropologically based, was more
118 Research Design
descriptive in orientation than contemporary uses of the term, which have emphasized
an interpretive dimension to this description.
7. See Chapter 6 on probes and interview guides.
8. We return to take a closer look at this analytical process in Chapter 7.
9. See Chapter 4 for a discussion of inquiry paradigms.
10. See Chapter 8 for a detailed discussion of this.
11. However, the idea of the researcher as a distant impassive observer is a simplification
of the reality of research. For example, the questions in a questionnaire are not a priori given; neither are they the only possible questions to ask. Rather, they are a result
of qualitative choices made by the researcher when developing their theoretically or
empirically derived understanding of the variables included in the study. This point will
be developed in Chapter 9 of this book.
12. See Chapter 7.
13. We take a close look at this process in Chapter 7.
14. We return to the idea of trustworthiness, and how to make decisions about it, when discussing qualitative analytic strategies in Chapter 7.
15. In Chapter 8, we provide a more extensive discussion about different types of quantitative approaches and their potential to address different types of research questions.
16. Keller (2013) and Taylor (2013) provide excellent overviews of the wide variation in qualitative discourse analysis approaches arising from the different types of theoretical perspectives about discourse used in them.
17. This point is developed in Chapters 8 and 9 where we take a closer look at designing, and
using, methods associated with what is referred to as quantitative research.
18. This point is developed in Chapters 6 and 7 where we will take a closer look at designing, and putting into practice, methods associated with what is referred to as qualitative
research.
6
OBTAINING DATA USING
QUALITATIVE APPROACHES
PURPOSE AND GOALS OF THE CHAPTER
The purpose of this chapter is to explore the series of research design–related decisions
you will need to make when collecting data using qualitative methods. It is the first of
four interconnected chapters (Chapters 6, 7, 8, and 9) which explore how to put the methodological thinking associated with, and underpinning, qualitative and quantitative
research approaches into practice.
In this chapter, our focus is what you will need to think about when making choices
about the type, and form, of methods that you will use to collect qualitative data. After
reading it, you will know a lot more about what you will need to find out about, think
through, and decide upon before you are able to put any qualitative method into practice
when collecting data.
There are a number of ways that data can be collected or obtained using qualitative methods. These include interviewing people about what you want to find out
more about; observing what is happening in a social context of interest related to your
research problem; and analyzing written and visual texts such as documents, diaries,
minutes of meetings, photographs, and other images. When designing your research,
you will need to think about, and decide, which of these methods will form part of
your design. You will need to be able to justify your decision for your research to be
credible.
However, even when you have decided which of these methods might enable you
to obtain the data you need to address your research problem, your method-related
decision-making is not yet finished. This is because there is a great deal of variety within
each of these types of qualitative methods. Qualitative interviews do not all take a standard form. Neither do qualitative observations and textual analyses. Therefore, a key decision you will need to make when designing your research is which form of these qualitative
methods you will include in your research design, and why.
Throughout the discussion we demonstrate this type of thinking and decision-making
in action. When doing so we use the example of qualitative interviews as the primary vehicle for the discussion. We structure the discussion around four questions, the answers to
which will affect the form that the qualitative interviews that you use in your study take,
as well as how those interviews will be put into practice:
• How structured will your qualitative interviews be?
• Will you interview your participants individually, or in some form of group?
119
120 Research Design
• What will you ask your participants in the interview?
• Who will you interview?
We highlight how these questions, and the decisions you make about each of them,
shape how a qualitative interview is designed and put into practice. There are a range of
possible answers to these questions. How you answer each question will be influenced by
the theoretical and methodological assumptions you are making in your study.
Therefore, it is important to mentally add an and why at the end of each of these questions. This is because thinking about why you have answered that question in a particular
way exposes the assumptions, both methodological and theoretical, you are making when
you decide to design your qualitative methods in one way and not another.
Throughout the discussion, we demonstrate that these types of considerations also
shape the form that other qualitative methods take. For example, deciding how standardized or structured observations will be, and why, involves the same type of considerations
as deciding how structured qualitative interviews will be. Likewise, deciding which texts
you will analyze involves the same type of considerations as deciding who you will interview and/or what you will observe.
The goals of the chapter are to
• Establish that the choices that you make about the form that any type of qualitative
method takes are part of your research design and therefore need to be made explicit.
• Emphasize that qualitative methods are about methodical and justifiable ways of
both collecting, and analyzing, data. Therefore, qualitative methods are much
more than data collection techniques. They cannot be viewed as stand-alone
procedures able to be inserted into your research design.
• Explain that qualitative methods, like qualitative approaches, are diverse. They
cannot be standardized. There is a range of possibilities for how to put any
qualitative method into practice when collecting qualitative data.
• Highlight that making decisions about what type of qualitative method you will
use to collect data requires you to have thought about, and decided on, the type of
information you will need to address your research problem or questions.
• Illustrate that the form that a specific qualitative method takes results from a
series of interconnected considerations about how to obtain the type of data you
will need to address your research problem or questions.
• Use the example of qualitative research interviews to explore this series of
interrelated choices needing to be made when putting qualitative methods into
practice.
• Demonstrate that, and how, making decisions about those choices is influenced
by the theoretical and methodological assumptions you are making in your study.
• Act as a guide for what you will need to think through, find out more about, and
then decide on, before including some sort of qualitative method to collect data as
part of your research design.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
121
QUALITATIVE METHODS ARE NOT STAND-ALONE
DATA COLLECTION TECHNIQUES
We saw in Chapter 5 that there is a great deal of diversity in approaches to research that fit
under the umbrella term of qualitative research. Each is “connected to a complex literature;
each has a separate history, exemplary works, and preferred ways for putting the strategy
into motion” (Denzin & Lincoln, 2018b, p. 21). Given this diversity, it should come as no
surprise that there are also a number of ways that data can be collected or obtained using
qualitative methods.
Decisions about which qualitative method you will use in your research design, and
the form that that method will take, results from a series of interconnected considerations
about how to obtain the type of data that, when analyzed, can provide the type of information you will need to address your research problem or questions. There is no point in
collecting data if you do not have a plan for analyzing it.
In qualitative research, data collection and analysis occur iteratively and simultaneously throughout the study.1 Therefore, qualitative methods are about methodical and
justifiable ways of both collecting and analyzing data. This is an important part of establishing the trustworthiness and credibility of a qualitative research design and the conclusions that are based on that data collection and analysis.
The key point in all this is that qualitative methods are much more than data collection techniques. They cannot be viewed as stand-alone procedures able to be inserted into
your research design. The overall research design strategy that you are employing provides
the context for, and justification of, why particular methods are part of that design, as well
as what form those methods take. Therefore, any qualitative method, and the way that it
is put into practice, cannot be understood apart from all the other choices that collectively
make up your research design.
Consequently, this chapter about research design considerations related to collecting
data using qualitative methods must be read in context. This context is the thinking that
you have done so far about your overall research design which has led you to conclude that
the type of data you will need to address your research problem can be obtained using
some form of qualitative method. This means that you
a. know what the purpose of your research is, and have a clearly defined research
area or problem or question (Chapter 3).
b. have thought about what type of information you will need to address the
problem that your research is being designed to address (Chapter 4).
c. are aware that addressing your research question or problem requires the
exploration of one or more aspect(s) of individuals’ perceptions, experiences,
and understandings of the matters related to that problem, as well as the social
settings that give rise to those understandings (Chapters 4 and 5).
d. have decided that the type of information needed can be provided by analyzing
and interpreting the type of data produced by a qualitative research approach
(Chapter 5).
e. have considered the ethical implications of each of the above points and decided
that the research can be done in a responsible and ethical manner (Chapter 2).
122 Research Design
Having considered, and thought through points (a) through (e) above, you are now
ready to focus your thinking on what type of qualitative approaches, and the methods
associated with them, might be able to provide you with the information you need to
address the problem that your research is being designed to address.
TIP
USE THE TERM DATA COLLECTION WITH CARE
It is important to recognize that there are different points of view about the use of
terms such as collecting data or data collection among qualitative researchers. Not all
qualitative researchers would be comfortable using these terms. Instead, they might
choose to use terms such as generating data, strategies for obtaining data, or even
making data. This reflects their view that terms such as data collection can give the
impression that data is a “thing” to be “got.”
You will recall from Chapter 4 that data is much more than that. It is about the way
that parts of reality are reduced or “chunked” into manageable units (Bernard et al.,
2017) able to be analyzed and interpreted to produce the findings or results of our study.
Therefore, although throughout the chapter we have chosen to use the terms collecting data or data collection, this does not mean that we see data as a thing waiting
there for us to find or collect.
DIFFERENT QUALITATIVE METHODS USE
DIFFERENT STRATEGIES OF INQUIRY
There are a range of methods used in qualitative research approaches. Common methods often used to collect data in qualitative inquiry include observations, interviews, and
analyses of some form of written or visual texts. Such texts can range from documents
such as minutes of meetings, organizational policies, government regulations, through to
personal diaries, poetry, photos, or films.
There is also a lot of diversity within each of these types of methods. There are different types of interviews, observations, and textual analyses. Each type of interview, observation, or textual analysis employs different strategies of inquiry. Such different strategies
of inquiry arise from different disciplinary, inquiry, and theoretical traditions that shape
the way that the same method (e.g., an interview or an observation or a textual analysis) is
understood and enacted. There is not a standardized form that all interviews or observations or textual analyses take.
However, despite their diversity, what all qualitative methods, and the strategies of
inquiry they employ, have in common is that they will be designed and used in keeping
with the basic principles and features of qualitative research outlined in the previous chapter—principles such as collecting data in a way that enables you to explore aspects of the
world of the participants in the study in order to understand the way that they perceive,
experience, and make sense of that world. There is no overt experimental or researcher
manipulation of that setting. Thus, “People will be performing in their everyday roles, or
will have expressed themselves through their own diaries, journals, writing and photography—entirely independent of any research inquiry” (Yin, 2016, p. 9).
Therefore, when making any decisions about the methods you will use, and the
form that they will take in your qualitative research design, you will also need to think
about how the form that those methods will take is in keeping with the basic principles of
Chapter 6
•
Obtaining Data Using Qualitative Approaches
123
qualitative research. Such thinking will require reflexive conceptual and intellectual work
at the level of how qualitative data will actually be collected, and what you will need to
think about when doing so.
TIP
SOMETHING TO KEEP IN MIND
It is possible to conduct interviews, make observations, and analyze texts as part of
quantitative studies. However, these will be types of interviews, observations, and textual analyses designed to produce numerical data.
Therefore, it is important that you think about what form your method for collecting data takes, what type of data it will produce, and if this is the type of data that
enables you to make the types of analyses and interpretations you will need to be able
to address your research questions.
It is not enough to say that your research design will use interviews or observations
or textual analyses as a data collection method. You will need to outline and justify all the
choices you make about what type of interviews, observations, or textual analyses that you
will use and why. This is an important part of establishing the credibility of your research.
Key Questions to Ask Yourself When Choosing Types of Qualitative
Methods or Strategies of Inquiry
The diversity between and within qualitative methods means that you will need to think
through which qualitative method(s), and which form(s) of that method, is the best fit
with your research design. These are key questions you will need to ask yourself and think
reflexively about:
• What method(s) will you use to collect qualitative data and why?
• Once you have decided on the methods you will use, what shape or form will
those methods take and why?
• What will you need to decide about them to be able to put them into practice and
why?
• For example, who will you collect data from and why?
• How will you organize and analyze that data and why?
You will notice that after each of these questions we have explicitly added “and why”
as part of that question. This is because, as we indicated previously, thinking about why
you have answered that question in a particular way exposes the assumptions, both methodological and theoretical, you are making when you decide to design your qualitative
methods in one way and not another.
Navigating the Diversity Between and Within Qualitative Strategies of
Inquiry When Designing Your Research
The rest of the chapter explores the thinking that you will need to do about each of these
key questions when choosing the methods, and the form that those methods will take,
for your research design. It takes the form of an extended discussion about what you will
124 Research Design
need to think about when making, and justifying, the decisions about how you will go
about putting a qualitative research method (such as observations, interviews, or textual
analyses) into practice in order to obtain the type of data that will enable you to answer
your research problem or questions.
We use the example of collecting data using some form of qualitative interview as the
vehicle to drive, and around which to organize, the discussion. We explore what you will
need to think through, and make choices about, both before and during collecting qualitative data using some type of interview. Specifically, choices and decisions about
• how structured your qualitative interviews will be;
• whether you will interview your participants individually, or in some form of
group;
• what you will ask your participants in the interview;
• who you will interview.
While these are not the only choices to be made when using some type of interview
to collect qualitative data, exploring these choices provides you with good examples of the
reflexive thinking and questioning that you will need to do about all the choices you make
about the qualitative research interviews you will use in your study design. You will identify further questions to ask yourself as you systematically, and iteratively, think through
what you will need to know, and decide on, related to the form your interviews will take.
The same type of reflexive thinking applies to the decisions that you will make about
the form that any other qualitative method will take. For example, if you have decided
to use qualitative observations of some kind in your research design, here are some of the
choices you will have to make about those observations:
• how structured your qualitative observations will be;
• whether you will observe individuals or groups or both;
• what you will observe and where;
• which individuals or groups or settings will you observe, where, and for how long.
Similarly, if you decide to analyze texts in some way, you will need to think reflexively
through the same sorts of choices:
• What form will your textual analysis take, for example, a more structured content
analysis or an unstructured discursive analysis?
• Will you analyze one type of text or several different types of texts?
• Which specific texts of that type(s) will you choose?
Therefore, the discussion to follow will be of use to anyone collecting data using some
form of qualitative methods, not just those using interviews.
Having established the purpose and context for the discussion to follow, we will begin
our exploration of this reflexive thinking in action by exploring the choices you will need to
make related to the question of how structured your qualitative interviews will be and why.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
125
TIP
ANOTHER KEY ISSUE TO THINK ABOUT: HOW MIGHT CONDUCTING
EMOTIONALLY LADEN RESEARCH IMPACT YOU AS A
RESEARCHER?
Collecting qualitative data can have a powerful impact on the researcher. This is particularly so when people are talking about difficult and emotive issues such as losing
a child, coping with cancer, or being abused. In her article called “Compassion Stress
and the Qualitative Researcher,” Kathleen B. Rager (2005) explores her experience
in relation to the effect that researching the self-directed learning of breast cancer
patients had on her as a researcher:
The participants in my qualitative study of the self-directed learning of breast
cancer patients were protected carefully. . . . The question that was not asked
as I began my study but that I wish to raise at this time is whether researchers
should not also attend to themselves in the process. No thought was given to
ensuring that I would take steps to address the impact on me as I conducted
such emotionally laden research. (p. 423)
It is important to be aware of and not underestimate the impact that entering and
interacting with people in their context can have on you when you are researching
those contexts.
Therefore, if you are a student who is about to collect their first qualitative data,
it will be helpful to talk this through with your supervisor and work out a strategy for
dealing with any distress that arises.
HOW STRUCTURED WILL YOUR QUALITATIVE
INTERVIEWS BE AND WHY?
Among other things, qualitative interviews vary in terms of how structured the interview is. Therefore, when using qualitative interviews as part of your research design,
you will need to make decisions about how structured your interviews will be. Will your
research design use fully structured standardized or closed interviews? Or will you use
semi-structured or open/unstructured/informal interviews? What will you base this
choice on?
In what are referred to as structured or standardized or closed interviews, an interview
schedule is used and followed “as if it were a theatrical script to be followed in a standardized and straightforward manner” (Fontana & Frey, 2005, p. 702). Therefore, each
research participant being interviewed is asked the same predetermined interview questions, using the same wording, in the same order. Usually there is a limited set of response
categories where “the interviewer records the responses according to a coding scheme that
has already been established by the project director or research supervisor” (Fontana &
Prokos, 2007, p. 19).2 This is because the aim in using a closed interview structure is to
obtain information from each participant in your research about the same specific aspects
of the research problem you are designing your research to address. Your motivation will
be to obtain some form of standardized data that can then be compared across large numbers of participants.
Closed interviews are common in quantitative survey designs. This is because they are,
in many ways, an “oral form of a written survey” (Merriam & Tisdell, 2016, p. 110), and
enable you to statistically generalize the answers given to the wider population of interest.3
126 Research Design
Not surprisingly, closed interviews are uncommon in qualitative research approaches. The
type of data this type of interview enables is not congruent with the rich and in-depth
information that most qualitative research is designed to obtain. If used at all in qualitative research designs, closed interviews usually collect some form of demographic data
about the participant(s) in the study.
Unstructured or open qualitative interviews have a much more flexible and nonstandardized structure. You will see this type of interview referred to as an “unstructured,”
“informal,” or “open” interview. In this type of interview, the interviewer “attempts to
understand the complex behavior of members of society without imposing any a priori
categorization that may limit the field of inquiry” (Fontana & Prokos, 2007, p. 40). The
“questions are broad and open-ended in ways that let the participants discuss their own
thoughts on the topic” (Morgan, 2016, p. 61). The interview is more a conversation with
the researcher picking up on, and probing in more depth, areas that the participant has
chosen to talk about when answering those questions.
For example, Morgan’s study (1989) of how spouses adjusted to becoming widows had
only one question: “What sorts of things made it easier for you to deal with your widowhood and what sorts of things made it harder for you?” (p. 102). He notes, when reflecting
on the study 30 years later, that
[t]his design made it possible to hear about participants’ experiences in their own
terms, and even though there was a clear underlying research question, the point
was to learn about how this topic fit into the lives of these women without directing them to discuss the things that interested the research team (Morgan, 2019,
p. 8).
Choices About Structure Are Choices About the Degree of Control You
Have Over the Interview
The higher the level of structure of an interview, the more control a researcher exercises
over that interview such as what is asked and in what order. If you conduct a completely
structured or closed interview (or standardized interview), you, as the researcher, are controlling that interview by directing the person you are interviewing to discuss things that
interest you (Morgan, 2019). You are controlling what will be asked, and equally importantly, what won’t be asked or followed up in the interview. This is because you are going
to ask each person being interviewed the questions that you have decided on before the
interview has begun. No matter what the person being interviewed talks about or the
issues they raise, the only questions that you will ask are the ones that you came up with
originally. Therefore, the only information you will obtain is about specific aspects of the
specific areas that you have deemed important before you began the interview.
A less structured interview enables you to obtain in-depth and rich information about
the experiences and understandings about the situation from the point of view of the
person being interviewed. This type of interview uses an interview guide rather than an
interview schedule. An interview guide is “a flexible list of questions to be asked during
the interview, which are meant to stimulate the discussion rather than dictate it” (Tracy,
2020, p. 178). In other words, an interview guide is just that, a guide. It gives you as the
researcher flexibility to respond and engage with what the person has said. This makes it
possible for you to find out about things which you may not have thought about before
that interview and would now like to know more about.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
127
In this way, the more open or unstructured interview can enable an active exchange of
ideas that produces a depth and richness of data in a way that a closed interview structure
cannot. It thereby can enable much richer in-depth information to be obtained. In this
type of interview, it is more of an interview where both the researcher and the person being
interviewed shape the form that the interview takes. The researcher does not control the
entire interview.
In reality, most interviews fall somewhere on a continuum between fully closed or
structured and fully unstructured or open interviews (Morgan, 2016; Merriam & Tisdell,
2016). Therefore, they are described as semistructured. A semistructured interview gives
some structure to what is to be talked about but does not dictate in what order, form, or
how it must be talked about as a more closed interview structure does. The semistructure
does not preclude the possibility of exploring and asking questions about what was said.
This enables you to follow up and probe unanticipated and interesting directions and
areas that may arise during the interview. Therefore, the semistructured interview guide
“is meant to stimulate discussion rather than dictate it” (Tracy, 2020, p. 158). There is
flexibility in the way that the interview guide can be used and the interview conducted.
Both semistructured and open interviews enable rich, in-depth information to
emerge. Participants’ “own words can be captured by the method and hence the researcher
can focus on issues that are important to the participants. . . . It provides opportunities for
the researchers to probe and explore in great depth, and to follow up clarification immediately” (Liamputtong, 2013, p. 71). Therefore, you will find both semistructured and open
interview formats used extensively in qualitative research.
When you are designing your research, you will need to think about how structured
your qualitative interview will be and justify the decisions that you make in this regard.
This will require you to ask yourself questions such as, Will your research design use
fully structured, standardized, or closed interviews? Or will you use semistructured, or
open, unstructured, or informal interviews? What will you base this choice on? And is
that choice methodologically consistent with the qualitative approach framing your study,
and in turn the inquiry paradigm and theoretical assumptions framing that qualitative
approach?
Reflexively thinking through questions such as these will require you to consider the
purpose of the interview, that is, what you want to be able to use that interview data for.
You will need to make sure that there is congruency between this purpose and the amount
of structure that your interview has.
Using the Same Reflexive Thinking When Collecting Data Using Other
Qualitative Methods
The same type of thinking regarding how open or closed the parameters for collecting
your qualitative data will be applies when using other qualitative methods. For example,
observations vary in terms of the degree to which they are standardized reflecting that the
“nature of observations vary along a wide continuum of possibilities” (Patton, 2002, p.
267). This variation arises from the different theoretical influences, inquiry paradigms,
and disciplinary traditions that shape the form that those observations take.
For example, there is much variation between types of observations in terms of how
those observations are made and what those observations are for. Standardized observations are where you, as the researcher, have decided in advance what will be observed,
when, and how, just as in a closed interview structure, the researcher has decided in
128 Research Design
advance what will be asked, how, and in what order. In other words, you control what
will be observed, and how it will be observed. The form that you record data about those
observations will also be very structured—often in a form of an observation schedule.
Standardized observations are often used to gain data about the frequency of particular designated activities, events, or actions that a researcher has decided are of interest. For
example, your observations might be about how many times, when and by whom, medications were crushed when being administered to older people in residential aged care. Or
they might be about counting how many times drivers of cars, and which categories of
drivers, leave the car park of a large shopping mall without wearing a seat belt. Or counting the number of cars passing through a specific roundabout during a 24-hour period in
order to map the times of day there is likely to be a traffic jam at that roundabout.
In all these examples your observations will take the form of some sort of observational log of the frequency of the event of interest occurring. The data will be primarily
numerical and used to describe the frequency of an aspect of interest or predict something
related to that frequency. Therefore, standardized observations are most often associated
with hypothesis testing and quantitative research approaches. In this type of observation,
the researcher is a spectator (Patton, 2002) with a high degree of separation from the setting being observed. Observing what people do related to the area of interest from behind
a two-way mirror would be an example of this type of observation (observer as spectator
with complete separation from the setting). So would sitting at the back of a classroom and
making observations without interacting with those being observed (observer as spectator,
but making those observations from within the setting).
However, in keeping with the underlying theoretical influences, inquiry paradigms,
and disciplinary traditions on which most qualitative inquiry draws, qualitative research
studies employ a type of observation associated with some kind of field work in a naturalistic setting. This way of observing is often referred to as entering the field. The researcher
enters into, and observes, the everyday social settings and contexts in which people or
activities of interest are located. This social setting is both where the researcher makes
their observations, as well as part of what those observations are about. The researcher
may be a full participant in the setting in which they are making their observations, or
some mix of part participant and part observer (see page 277 in Patton, 2002).
In this type of observation, the researcher is an important and central part of data
collection. Observations occur in situ. They are embedded in, and arise from, the context in which they are made. Understandings and knowledge related to the problem the
researcher is addressing are generated “by interacting, watching, listening, asking questions, collecting documents, making audio or video recordings, and reflecting after the
fact” (Tracy, 2020, p. 76, drawing on Lofland & Lofland, 1995).
The goal of such observation is to understand people and their actions in context. To
gain such in-depth understanding requires a much longer period of immersion in that
context than if the goal of the observations is to obtain information about the frequency
or intensity of a specific action. It requires building rapport with those in the context
being observed, as well as getting a sense of how interactions and actions related to what a
researcher is interested in knowing more about “work” in that context.
Consequently, in qualitative research studies using some form of observation to collect
qualitative data (not just numerical data), observations are less standardized, and therefore
less structured. The focus of the observations is therefore broader than focusing on a single
element (Patton, 2002) in a social setting—such as whether a seat belt is worn or not. As
Chapter 6
•
Obtaining Data Using Qualitative Approaches
129
the researcher, you do not set out to only observe specific predetermined events or count
instances of them. Rather, you enter the field and collect observational data using “a logic
and process of inquiry that is open-ended, flexible, opportunistic, and requires constant
redefinition of what is problematic, based on facts gathered in concrete settings of human
existence” (Jorgensen, 1989, p. 14). Here your goal is to understand what is going on in
that context related to your research problem. Therefore, you exercise less control over
exactly what you will observe.
PUTTING IT INTO PRACTICE
AN EXAMPLE OF PUTTING NONSTANDARDIZED
OBSERVATION INTO PRACTICE
You may be interested in understanding family mealtimes as a social event—how
families share or do not share mealtimes and what role mealtimes play in how family relations develop. Therefore, you choose not to standardize, and therefore limit,
in advance what aspects of family mealtimes you will make observations about (e.g.,
duration, timing). Instead, you remain open in terms of what you will observe related
to family mealtimes. You are interested in getting a more holistic picture of what is
going on (Patton, 2002) in the social setting called a family mealtime. What did the
people you observe understand a family and mealtime to be? What happened at the
mealtimes? What role do family mealtimes as a social event play? What roles do different family members play at mealtimes?
Your data will take the form of detailed field notes about what happened at the
mealtimes you observed. Once you have made some initial observations that have
indicated possible areas of interest, you may then make further observations about
those specific areas of interest—in some ways a bit like the use of probes in interviews.
As was the case when making decisions about the amount of structure interviews will
have in a qualitative research design, decisions about how standardized your observations
will be depends on the purpose of your research. The key consideration is whether the way
you are designing your observations is methodologically consistent with the qualitative
approach framing your study, and in turn the inquiry paradigm and theoretical assumptions framing that qualitative approach.
WILL YOU INTERVIEW YOUR PARTICIPANTS INDIVIDUALLY
OR IN SOME FORM OF GROUP AND WHY?
In this section, we take a closer look at the choices you will need to navigate and decide
on related to whether you will use an individual or some form of group interview when
collecting your data. Each option has advantages and disadvantages. You will have to
make decisions about which of these ways of interviewing your participants you will use in
your research design. This means that you will need to read and think about their relative
strengths and weaknesses in the context of your overall research design. What can each of
these ways of interviewing participants contribute to your study that other ways cannot?
At the same time, what does that mean that you will miss out on, and does that matter?
130 Research Design
Using a form of individual interview is one of the, if not the most, common ways
that data is collected in qualitative research approaches. As its name suggests, this form
of interview involves a one-to-one dialogue between an individual participant and the
researcher. This is why you will see an individual interview sometimes referred to as an
in-depth individual interview. On the other hand, group interviews, as their name suggests, involve interviewing more than one person at the same time.
Focus Groups—A Specific Type of Interview
A specific type of interview involving a group of people is a focus group. “Focus groups are
a research method that collects qualitative data through group discussions. This definition contains two components: first, the goal of generating data, and second, the reliance
on interaction” (Morgan, 2019, p. 4).4 It is this second aspect—the reliance on the group
interaction—that sets focus groups apart from individual interviews. This is because
while a focus group is an interview, the “twist is that, unlike a series of one-on-one interviews, in a focus group participants get to hear each other’s responses and to make additional comments beyond their own original responses as they hear what other people have
to say” (Patton, 2002, p. 386).
There is a range of views about how many people make up a focus group. This ranges
anywhere from 3 to up to 12 people, although most views seem to land somewhere
between 6 and 10 or 6 to 8 as “ideal” (see Liamputtong, 2011; Morgan, 2019). Carey and
Asbury (2012) point out that having a smaller number of participants in a focus group
“usually leads to greater depth of data, and small size is especially important for sensitive,
complex topics. . . . With a small group, the facilitator can more easily manage the group
dynamics, process the information, and attend to each member” (p. 45).
The emphasis on, and importance of, facilitating group interaction when conducting focus groups is why the person who conducts the focus group interview (usually the
researcher) is referred to as a moderator. The role of the interviewer is to moderate the
group in such a way as to enable active, relevant, and respectful discussion of the topics of
interest—the focus of the group.
You will also note when reading about focus groups that it is often recommended that
a notetaker be present as well. The role of the notetaker is to record things that happen
during the group, such as nonverbal reactions to comments, as well as recording their
thoughts and impressions of the information emerging from the discussion in the group.
It is harder for the moderator to do this when they are concentrating on keeping the interactions in the group focused while ensuring that all members of the focus group have the
opportunity to participate. However, in reality, often the notetaker and moderator are the
same person largely because of resource constraints, particularly in the case of students
doing their research.
Which to Choose?
In many ways, choosing between whether to use individual in-depth interviews or focus
groups in your study design is really a matter of choosing between a series of trade-offs. For
example, one trade-off in using focus groups as a form of collecting data as compared to
using individual interviews is the loss of depth of the information that can be gained from
each individual participant. On the other hand, a trade-off that you will need to make
when using individual interviews instead of focus groups is the loss of data that could be
Chapter 6
•
Obtaining Data Using Qualitative Approaches
131
gained from the group interaction of the participants that you are interviewing individually. Therefore, you will need to think about what is more important for you in terms of
meeting the goals of your study. Is it the information and insights that can be gained from
the discussion and interactions of the participants in the focus group, or is it the in-depth
information that you can gain from each individual? What do you base your answer on?
There are also factors related to feasibility5 that you may have to consider, and make
tradeoffs about, as you must do when designing any form of research. For example, a feasibility design related consideration is the time and resources that you have available for
doing the research. Students doing a time delimited study with little, if any, monetary support may not be able to conduct individual interviews that each require travel (a financial
and time cost), and then transcription6 (a time cost). However, conducting and transcribing fewer focus groups may be feasible cost and timewise.
On the other hand, if for some reason it is difficult to get potential participants
together for a focus group in the same place at the same time (such as health-related
issues preclude them from traveling), then it may not be possible or desirable to use focus
groups in your study design. This is because the richness of the data obtained from a focus
group relies on the interaction between the participants. Therefore, if “not enough people
attend, this is a more severe problem than if one person misses her or his individual interview” (Morgan, 2019, p. 21). Hence, Morgan advises “if individual interviews and focus
groups are likely to be equally productive for a given purpose, then sheer practicality may
well favor individual interviews” (p. 21).
Sometimes, depending on the time and resources available, it may not be necessary
to make these trade-offs. It may be possible to incorporate both individual interviews and
focus groups into your study design. When designing research, “[f]ocus groups and individual interviews are often complementary rather than competing methods” (Morgan,
2019, p. 21). The individual interviews in the study identify perspectives about what it is
you are studying at the level of the individual. You might then use focus groups to probe
further into those perspectives and, for example, their implications for a practice area of
interest.
However, when you are designing your research, it is important to remember that just
having more data, or more interviews, or interviews with both individuals and groups in
your research design, does not necessarily make that design stronger or even sound. It
might, but this depends on whether having this extra data and these additional interviews
in some way enables you to better answer or understand the problem that your research is
designed to address.7
Therefore, the answer to the question of whether using some form of individual interview is “better” to use in your research design than some form of group interview such
as a focus group, is that this depends. What it depends on is what you are hoping to be
able to say something about after having completed your data collection and analysis.
Put another way, what do you want your data to be able to be used for? Focus groups and
individual interviews are only “better” than each other (or any other possible method of
qualitative data collection) if they can “better” produce the type of data or information
needed to “better” address the issues that a specific research study is being designed to
illuminate.
Such thinking about what better means is a conversation that you will need to have
with yourself when thinking about, and using, any method that forms part of your
research design.
132 Research Design
WHAT WILL YOU ASK YOUR PARTICIPANTS
IN THE INTERVIEW AND WHY?
In the courses we teach about research design and qualitative inquiry, students who have
decided that some sort of qualitative interview will be their method of choice often present
us with what they have decided will be their interview guide. They seem quite surprised, at
times even defensive, when we ask them why they have chosen those particular interview
questions and not others. This is because in their mind these are the obvious and important questions to ask.
The reason why we ask our students how they decided on this interview guide, and
the questions that make it up, is because we want them to fold back on the thinking that
has led them to come up with these questions. When doing so we want them to consider
whether these are the optimal (or only) interview questions to ask in order to be able to
obtain the in-depth information needed to address their research problem, and thereby
achieve the purpose of their research.
Put another way, we want to encourage students to get into a reflexive dialogue with
themselves about why they have chosen these particular interview questions and not others. In so doing, we want this reflexive dialogue to highlight the thinking that students
still might have to do before finalizing their interview guide. We want to remind them
that “[t]he key to getting good data from interviewing is to ask good questions; asking
good questions takes practice” (Merriam & Tisdell, 2016, p. 117). Therefore, before, during, and after that practice, they will need to do a lot of thinking both about each question
and about the way that each question connects to the other questions in their interview
guide.
Thinking about the connections between the questions, not just each individual question, when designing your interview guide is important. Collectively the questions that
make up that guide will provide the multifaceted and rich data that will enable your research
questions to be addressed. While each question must be able to be justified in terms of what
it will add to meeting the purposes of the study, it must also be able to be justified in terms of
how it complements the other interview questions in your interview guide. Therefore, you
will need to consider how the information obtained from a particular question adds to, or
extends, the information obtained from the other questions in that guide.
Not thinking this through well enough when you are designing your interview guide
can lead to problems when you analyze your interview data. This is because it may not be
clear how the data obtained from the various questions are related, and therefore how that
set of not clearly related data enables you to address your overall research problem.8
Developing Lines of Inquiry
Given these considerations, how might you go about designing a qualitative interview
guide made up of a coherent and connected set of questions? One way to begin is to identify possible lines of inquiry (Patton, 2002) to guide the interview. Initial lines, or areas, of
inquiry emerge from your thinking about what it is important for you to know more about
to get an in-depth picture of what is going on in the situation that your research is focused
on. Once you have identified your lines of inquiry, you can then develop questions related
to each of them.
How do you come up with initial possible lines, or areas, of inquiry when designing
your interview guide? A good place to begin is to think about your research problem and
Chapter 6
•
Obtaining Data Using Qualitative Approaches
133
questions—the purpose for doing your study. Within this broad area of interest, ask yourself what it is that you want to know more about, and why. Are there specific foci or areas
of interest within the broad problem area that you want to know about? What is it about
these foci and areas of interest that has triggered your interest? In addition, the findings of
other researchers working in related areas to your proposed study might provide starting
points for developing your lines of inquiry. You could also read critiques of these studies and use that critique as a starting point for thinking about what your lines of inquiry
might be. Or, you might draw on your own or others’ experiences of, or hunches about, the
situation or issue that is of interest.
PUTTING IT INTO PRACTICE
USING PHOTOGRAPHS BOTH AS A DATA SOURCE AND TO
DEVELOP LINES OF INQUIRY
In their study of body image in middle-to-older age women with and without multiple
sclerosis, Bailey et al. (2021) asked seven women to provide up to 10 photographs
“that represented their body image” (p. 1542) as well as to participate in a one-on-one
interview. In the interview, the photographs provided the impetus for four of the six
major areas or lines of inquiry. For example, participants were asked why they had
selected a particular photograph, and how it represented their body image including
how they felt, saw, thought, and acted toward their body. You can find the full interview guide in Table 1 on page 1545 in Bailey et al. (2021). This study provides a good
example of how visual texts and methods can be used as data sources in qualitative
research.
Using lines of inquiry as an initial organizing construct for your qualitative interview rather than a list of discrete, standardized, and fixed questions opens up, rather
than closes down, the interview. Lines of inquiry do not determine how that interview
will proceed in terms of what will be asked and how answers can be given. This is not
the case in interviews that have a closed structure. In a closed interview, the person being
interviewed may be required to give what is known as a forced response by choosing an
answer from a list of predetermined responses such as agree, strongly agree, disagree,
strongly disagree.
Further, in line with the flexible and emergent nature of qualitative research design,9
additional lines of inquiry may emerge once you begin to collect data obtained from your
interviews. This occurs when participants raise issues or areas of interest that you may
not have anticipated, or even thought about, prior to the interview. In addition, the use
of probes during an interview will enable you to pursue new and interesting leads as the
interview progresses or to clarify aspects of the answers given to the questions you have
asked.
For example, when being interviewed, a participant in a study about what it is like to
move from home to a residential aged care facility says, “Well, when I moved it was chaotic. It was really difficult.”10 A probe to an answer like that might be “You describe this
situation as chaotic. What was chaotic about it?” Here you are probing and unpacking the
134 Research Design
information that they have given you. The interview probe helps ensure the richness of the
information gained. Just reporting that the situation was chaotic does not necessarily tell
us in depth what about the situation was chaotic. Similarly, you could also probe what the
participant meant by “it was really difficult.”
Thoughtful use of probes when interviewing is important to enable rich and in-depth
interview data that can move beyond description of events (e.g., they were chaotic) to
provide insights into participants’ understandings of those events (why they were chaotic) and the contexts in which those understandings are derived. Seeking more details, or
such elaboration, is a key part of the analysis and interpretation of the interview data that
begins during the interview itself.11
It is important that you document and justify any changes to the interview guide or
questions that occur as each interview or your overall study progresses. This is “an important part of documenting the process of the study” (Carey & Asbury, 2012, p. 50). It is also
an important part of ensuring the trustworthiness of your qualitative research design, and
therefore the findings that emerge from it.
How Many Lines of Inquiry and Associated Questions Are Ideal for an
Interview Guide?
It is important not to have too many lines of inquiry, and questions associated with each
of them. If you have too many lines of inquiry, then the interview guide will probably end
up taking more the form of some sort of structured interview. The large number of lines
of inquiry, and questions related to them, limits the opportunities that you will have to
explore, and thereby gain in-depth information about each of those lines of inquiry or the
questions asked about each of them.
For example, if you had 10 lines of inquiry and three questions related to each of
them, you would have 10 areas of inquiry to explore and some 30 questions to ask in
the interview. This may not leave much time, or no time at all, to follow up on what the
respondents say when addressing those areas or questions. This will at best result in the
loss of some depth, and therefore richness, of the interview data. At worst, it will result in
the production of shallow and thin data or “superficial snippets” (Carey & Asbury, 2012,
p. 50) that range across a long list of areas that you have identified as being important to
explore. This data will be of limited use in addressing a research problem for which you are
seeking in-depth, rich qualitatively derived information.
So how many lines of inquiry and associated questions are ideal for an interview guide?
Drawing on years of experience of doing and supervising qualitative research, Merriam
and Tisdell (2016), suggest that
the fewer, more open-ended your questions are, the better. Having fewer broader
questions unhooks you from the interview guide and enables you to really listen
to what your participant has to share, which in turn enables you to better follow
avenues of inquiry that will yield potentially rich contributions. (p. 126)
Fewer, broader questions also ensures that you are not focusing more on how to get
through all the questions making up your ambitious interview guide than on what your
participant is saying. Hence, Carey and Asbury (2012) suggest limiting “guideline questions to three or four, with subquestions used to further explore each question” (p. 51).
You can then use probes to explore what is being said about each question during the
interview.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
135
TIP
DIFFERENT QUESTIONS FOR DIFFERENT PURPOSES
What the discussion highlights is that just using your broad research questions as
interview questions is not likely to enable you to obtain the in-depth and rich data
you were hoping to obtain by using interviews as your method. Your research questions need to be thought through carefully and broken down into initial lines of inquiry.
This is because research questions are at a much higher level than what your interview questions will be. They are also likely to be framed in very different ways than
well-developed interview questions. In short, they are different sets of questions
developed for different purposes.
Designing Good Interview Questions
However, just having fewer lines of inquiry, and questions related to them, in itself is not
enough to ensure depth, or richness, of the information that each interview can give us.
When designing our interview guide, we will also need to take a very close look at the way
that the questions associated with those lines of inquiry are structured. This includes the
specific wording used when asking a question. The way that the questions are worded,
and structured, affects whether or not it is possible for in-depth, rich information to
emerge from the interview. Therefore, when you are designing your research, as much
care needs to go into the design of each question as it does into the design of the overall
interview guide, and before that, the choice of interviews as a method in your overall
research design.
Some of the things that you will need to think about are asking one question at a time,
avoiding asking dichotomous questions, and not “leading” participants to respond in the
way that they did.
Ask One Question at a Time
When developing your interview guide, you will need to think about how to design interview questions that are, “at a minimum, . . . open ended, neutral, singular, and clear”
(Patton, 2002, p. 353). To ask clear singular questions requires that “no more than one
idea should be contained in any given question” (Patton, 2002, p. 358). This is to avoid
the 100 questions in 1 question syndrome! An example of this syndrome is the following
question, which is in fact a bundle of four different questions: “You recently moved from
home to hospital to residential aged care facility—correct? (Question 1) What went well
in the move (Question 2) and what did not (Question 3) and why do you think that this
was? (Question 4).”
If you try to remember that question now after just reading it, maybe you can recall
the first question asked, or the last one. However, it is very likely you will have forgotten
the other questions in between them. This makes it likely that you would forget to provide
information about what went well, and what did not. Even if you did, it is unlikely that
you would remember to respond to the question “Why do you think that this was?” in
relation to both of the moves being asked about—the move from home to hospital, and
then the move from hospital to residential aged care facility. All these questions need to be
untangled and made into singular questions related to a clear line of inquiry.
136 Research Design
Avoid Asking Dichotomous and Therefore Redundant or Limiting Questions
When developing your interview questions, it is also important to avoid asking dichotomous questions. These are questions “with a grammatical structure suggesting a ‘yes’ or ‘no’
answer” (Patton, 2002, p. 354). For example, “Did you feel sad when you had to leave your
home?” You will have to follow up with a number of other questions to get any depth in the
answer given (e.g., yes/no), which in effect makes the dichotomous question redundant.
Asking dichotomous questions also has a limiting effect on the interview. For example, the question “Did you feel sad when you had to leave your home?” means that participants are limited to answering something related to feeling about sad (yes, no, maybe,
don’t know). Even if you ask them why they did or did not feel sad, they are limited to
answering in relation to whether they were sad or not—which is the idea that you asked
them about. In effect this is a form of leading—see the discussion in the next section about
what a leading question is.
Don’t Ask Leading Questions or Make Leading Comments, When Interviewing
A leading question is when your question leads the person to talk about or agree with
something when answering a specific question. Or you make a comment in response to
an answer that the participant has given you during the interview that leads them to reply
in a certain way. For example, they tell you when being interviewed that they don’t know
anyone in the residential aged care facility. You then comment, “Oh, given that you don’t
know anyone in the residential aged care facility, I bet you miss your neighbors.” The
participant agrees that they do miss their neighbors. You then claim that your research
“found” that this participant said they miss their neighbors.
However, what in fact you found was that this participant agreed with your suggestion
that they must be missing their neighbors. You have no way of knowing if “missing their
neighbors” was an answer that they would have given, or a subject they would have talked
about, if asked a more open question about how they felt about the move.
Try Out Your Draft Lines of Inquiry and Questions Before You Do Your
Interviews
Once you have an idea of what your initial lines of inquiry are, and what the questions
associated with each line of inquiry might be, it is important to try that draft interview
guide out in practice. This is to sharpen, polish, and rehearse that interview guide. The
trial interview can be done with a volunteer who knows something about the area you are
asking about, but who is not going to be part of your actual study. If possible, it is important that they also know something about qualitative inquiry and interviews. If you are
unable to find a person fitting both criteria, then you may think about rehearsing your
interview guide twice—initially with a person familiar with the substantive focus of the
interview, and then with a person familiar with the methods you are using.
In the trial interview, you can try all or some of the ways suggested in the following activity box for getting feedback about how the interview guide works or could be
improved. The purpose of the trial interview is getting feedback about how the overall
interview guide, as well as each individual question in it, works or does not work. It is
also to get feedback about how you did when interviewing—did you rush things, did you
make any leading comments, did you miss opportunities to probe something that was
said, and so on.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
137
Activity
Ideas for How to Polish and Rehearse Your Draft Interview
Guide
Find someone (two people if necessary) who knows something about the area of your
study, or qualitative interviews. Ask them if they are willing to volunteer to give you
honest feedback so that the outcome of the test interview can provide you with information enabling you to modify your draft interview guide, and way of interviewing,
if needed.
• When you ask the volunteer your questions, instead of them answering them,
•
•
•
•
•
first ask them to tell you what they think you are asking. This will tell you
if your questions are clear enough (remembering that clear is not the same as
directive).
Then ask them to answer the questions to see if those questions are open
enough to obtain in-depth information.
When you are doing so, note how long it takes you to ask them about each of
the questions you have in your draft interview guide. This will help you decide
whether the demands on the person being interviewed in terms of time needed
to attend the interview are reasonable. As we have discussed earlier in this
book, this is in fact an ethical issue.12 It will also help avoid interview related
fatigue during the interview caused by having to think through, process, and
respond to so many different questions.
Think about who is doing most of the talking. Is it you asking the questions
or the person trying to think about, and then answer them? How can you
optimize this interaction in the interview?
Be aware of whether you feel under pressure to get through the questions in
the time allocated for the interview and if this affected the way that you are
able to respond to, probe, and follow up what the person being interviewed
said.
Ask the volunteer for feedback during and after the interview. Remember, it is
this feedback that is the main purpose of this type of trial interview.
Applying the Same Type of Thinking to Other Types of Qualitative
Methods
When collecting qualitative data using interviews, you make choices about what you will
ask participants about. Your goal is to obtain data, the analysis of which can enable you
to address your research problems or questions. The choices that you make have that goal
in mind. In the same way, when collecting qualitative data by making observations, you
make choices about what you will observe. These choices reflect your goal of obtaining
data, the analysis of which can enable you to address your research problems or questions.
When interviewing, lines of inquiry provide you with a way of guiding and organizing
what you will ask your participants about, while not constraining the interview to a list
of preset questions. When making observations in social settings, “sensitizing concepts”
(Blumer, 1954, p. 7) can play a similar role to that played by lines of inquiry in qualitative
interviews. They provide initial “jumping-off points or lenses” (Tracy, 2020, p. 29) when
138 Research Design
collecting data using observations. Sensitizing concepts for observations in qualitative
inquiry also provide a way of organizing “the complex stimuli experienced so that observing that becomes and remains manageable” (Patton, 2002, p. 278).
Examples of sensitizing concepts include rituals (such as how people dress, how they
meet each other, where they sit), incentives (such as how people are encouraged to behave
in one way and not another), codes both spoken and unspoken (such as sitting in a certain
place or speaking in a certain order at a business meeting), and routines (when and why
things happen regularly in particular social settings or parts of that social setting).13
What sensitizing concepts you choose to use when making observations will emerge
from your thinking about the areas or aspects in the social setting or context you are
observing that are important for you to know more about. This is because those areas or
aspects will enable you to develop an in-depth picture of what is going on related to the
issue or problem that is the focus of your research. In this way, sensitizing concepts provide
an initial organizing construct for what you observe. You watch for “incidents, interactions, and conversations that illuminate these sensitizing concepts” (Patton, 2002, p. 279)
in a particular social setting.
However, like lines of inquiry in an interview, sensitizing concepts act as guides. They
do not prescribe what you observe, when, and how. Nor do they remain unaffected by
what is observed. As observational data is collected, and analyzed, new or different sensitizing concepts may emerge, or your original sensitizing concepts may be modified in
some way. In such an iterative and reflexive process, the observer “moves from sensitizing
concepts to the immediate world of social experience and permits that world to shape and
modify his [sic] conceptual framework” (Denzin, 1971, p. 168).
Just as too many lines of inquiry in an interview guide can constrain the opportunities
that you will have to explore, and thereby gain in-depth information about, each of those
lines of inquiry, so too can having too many sensitizing concepts in an observational study.
Overusing sensitizing concepts when collecting qualitative data using observations can in
fact “become desensitizing” (Patton, 2015, p. 359). This is because if you have too many
things you are trying to observe, this can in fact desensitize you to what else you might be
seeing that could be relevant to your research problem. Like lines of inquiry in interview
guides, sensitizing concepts related to observations are there to guide, not constrain.
As with the draft lines of inquiry in an interview, it is important to try out your draft
sensitizing concepts in practice. Reflexively thinking through what worked, and what did
not, when you did so will help you refine and develop the sensitizing concepts which will
form the organizing construct for your observations. At the same time, you can use this
experience to refine and develop your observational skills.
PUTTING IT INTO PRACTICE
LEARNING FROM OTHERS ABOUT HOW SENSITIZING
CONCEPTS MAY BE PUT INTO ACTION
Helene Aksøy’s (2009) study of issues for front-line nurse managers arising from a
change in home care service in Norway provides a good example of reflexive thinking
when putting the idea of sensitizing concepts into practice. Her study comprised three
case studies designed to “explore what goes on in a nurse manager’s professional life”
Chapter 6
•
Obtaining Data Using Qualitative Approaches
139
(Aksøy, 2009, p. iii) as they deal with constant change. Qualitative data was collected
using observations, focused conversations related to the observations, and analysis of
texts (such as minutes of meetings or official communications) related to the change.
Aksøy noted that, as she was a master’s student and therefore “a novice researcher
entering into a study that by definition had no set formula” to follow, she “felt a strong
need for guidelines when designing the study and deciding how my data was to be
collected” (both quotes from Aksøy, 2009, p. 43). This led her to use the theoretical threads that shaped and informed the study to develop a list of sensitizing concepts that could “help guide observations while out in the field and the organizing of
field-notes later” (p. 43). She reports that this “list later proved vital when trying to get
a sense of the whole by looking at parts when I started my observations. It helped me
focus my observations during field-work” (p. 44).
She notes that while her focus was narrow in the sense that she wanted to observe
how issues arising from the change affected the nurse managers’ daily work,
my observations were wide. I was observing and looking for my topic of interest
in a number of different places and situations. Silverman (2006) stresses the
need to both funnel and overview at the same time while observing, and as hard
as it might sound, I really tried to do just that, by observing an array of different
situations, but focusing on the nurse managers the whole time. (pp. 47–48)14
She also used focused conversations (Street, 1995) “to probe and deepen what I
had already observed during field observations. I made the decision to use focused
conversations based on the fact that I did not yet know what I was going to observe
during the field observations” (Aksøy, 2009, p. 49).
The sensitizing concepts used by Aksøy were
• Geographical setting. Where are the offices placed? How does the staff
access their patients?
• Working environment: Are the offices crowded? Does the nurse manager
have her own office, or does she share her workspace with others?
• Social environment. Who are present? What is going on? When is this
happening?
• Historical perspectives. The history of the home care nursing services in
each municipality. Number, nature, and result of organizational changes
• History of recruiting staff. Is it easy or difficult to recruit staff? Do people stay
in their jobs when recruited or is the turnover of staff high?
• Description of planned activities and structured interaction. What kind of
formal meetings are held? Who are present? What is happening?
• Observing and describing unplanned activities and informal and impromptu
interaction. Describe situations that arise during the day. What does the nurse
manager do?
• Observe and describe interactions with others. Who does the nurse manager
meet during a day? Are there drop-in visitors? When does she meet her staff?
When does she meet senior management?
• Comment on non-occurences—what does not happen? “What does not
happen?” or rather, what was not talked about, turned out to be a vital finding
of this study.
• Record special language. Identify insider language—explain terms used.
• Observe nonverbal communication. Observe NM [nurse managers—our
addition] in interaction with others in both planned and unplanned activities.
• Analyze documents. (pp. 43–44)
Aksøy’s study is an excellent example of what you will need to think about and justify when putting observations into practice. She does not gloss over the struggles that
she had when doing the research, thereby providing useful and important insights into
putting research design into practice. She highlights that this is a very challenging and
messy process at times. In so doing, she provides a frank and honest account of reflexive thinking in practice as a research study unfolds—sometimes in unexpected ways.
140 Research Design
WHO WILL YOU INTERVIEW AND WHY?
A qualitative research interview, in keeping with the methodological assumptions that it
is embedded in, seeks people’s perceptions or understandings or constructions or experiences of situations or contexts related to the research problem that your research is being
designed to address. The aim of the interview is to obtain in-depth, layered, and richly
nuanced information about that situation or event. Therefore, the people we select to
interview in our study, our study sample, will be chosen because they are information
rich in some way about that situation or event or the context in which it occurred. When
selecting our study sample, we purposefully seek to find people to interview who can tell
us something about what we want to know about—the purpose of the study. This type of
sampling is referred to as purposeful sampling.
Likewise, if our research design includes using observations or texts as our
source of qualitative data, we will need to decide who, and what, we will observe or
which texts we will include in our sample. These sampling considerations will also
be based on the underlying principle of purposefully seeking to find sites or people
to observe, or texts to analyze, that are information rich about whatever it is that is
the focus and purpose of the study. This is because the logic and power of purposeful sampling lies in selecting information-rich sites and examples for study in depth
(Patton, 2002).
Choosing Between Different Types of Purposeful Sampling Plans in
Your Study Design
There are many ways that purposeful sampling is put into practice. While there is not
agreement among qualitative researchers about how many types of purposeful sampling
there are, or what they are called, Tracy (2020) provides a useful overview of what she
terms eight purposeful sampling plans. They are included in Table 6.1 below.
Each of these purposeful sampling plans is underpinned by a different logic that
“serves a particular purpose” (Patton, 2002 p. 230). It is this logic that needs to be thought
TABLE 6.1 ■ Commonly Used Purposeful Sampling Plans. Taken from Tracy
(2020, p. 86, Tips and Tools 4.1). Reproduced With Permission
Sampling plans
Type of sample
Purpose
Random
Creates an equal opportunity for all the members of a certain population
to be chosen
Convenience/
Opportunistic
Appropriate when time and money are scarce, but may indicate laziness
Maximum variation
Includes the entire rainbow of possible data. Helps to ensure the
inclusion of usually marginalized data
Snowball
Expands in size as the researcher askes study participants to
recommend other participants
Theoretical construct
Helpful for testing and finding gaps in existing theory
Typical instance
Focuses on routine, the average and the typical
Chapter 6
•
Obtaining Data Using Qualitative Approaches
141
Sampling plans, continued
Type of sample
Purpose
Extreme instance
The most/least/best/worst of a certain category. Can be valuable and
interesting, but also time-consuming
Critical instance
Focuses on data that are rare, under-studied, or strategically bounded to
the argument at hand; can help create logical deductions that show how
findings are transferable to other populations
through and used to decide which (or which combination) of these purposeful sampling
strategies will enable you to meet the goals of your research.15
For example, consider the purposeful sampling strategy known as maximum variation. “A maximum variation sample is one in which researchers access a range of data or
participants who will represent wide variations of the phenomena under study” (Tracy,
2020, p. 83). If you choose to use some form of maximum variation purposeful sampling
in your study, there must be a reason why you need this variation in order to be able to
address what it is that you are focused on finding out more about in your study. This reason must be congruent with the logic behind maximum variation purposeful sampling.
This logic is that “[a]ny common patterns that emerge from great variation are of a particular interest and value in capturing the core experiences and central, shared dimensions of
a setting or phenomenon” (Patton, 2002, p. 235). It also might reflect your desire to help
“ensure the inclusion of usually marginalized data” (Tracy, 2020, p. 86).
Most qualitative research approaches, and designs, draw on non-positivistic inquiry
paradigms. In these non-positivistic qualitative approaches, maximizing variation is about
maximizing the variation among the information-rich sites, texts, or people that comprise
our sample. The sites, texts, or participants in our study sample are all information rich,
but they vary in some way. For example, the geographical location of the information- rich
sites you choose to make observations about may vary, as might the level of experience of
the participants you choose to interview. However, what all the sites and all the participants have in common is that they are all information-rich in terms of what they can tell
us about the problem or questions that our research is being designed to address.
Your reason for maximizing the variation in your purposeful sample of sites, texts, or
participants must be able to be justified in terms of such variation adding complexity and
breadth (Tracy, 2020), and thereby depth and richness, to your data about your research
problem or questions.
TIP
HOW TO ANSWER THE QUESTION OF HOW MUCH VARIATION IS
NEEDED TO BE CONSIDERED MAXIMUM VARIATION
Our students often ask us, “Do I need one of everything?” when they are thinking about
employing maximum variation purposeful sampling in their study. They ask this question
because they are worried that maximum variation purposeful sampling means they need
to have every possible type of variation that exists between the information-rich sites, or
participants, present in the sample for the study. For example, if they are studying residential aged care facilities, do they need to have one city, country, large, small, public, private,
established, new, and so on residential aged care facility in their study? And then within
142 Research Design
each of the sites, one of every type of potential information-rich participant, for example,
each possible level of experience, age, gender, ethnicity, educational and professional
training, and so on?
We answer this question with a question. We ask them why they think that they
need to have one of everything. What does that enable them to say that is important
to answering their research questions? As they think about this, we remind them
that it is important to remember that the logic underpinning the choice of maximum
variation sampling is not the same as the logic underpinning representative sampling. Representative sampling draws on understandings derived from positivist and
post-positivist inquiry paradigms. Within such inquiry paradigms, “a representative
sample is arguably needed, involving representatives of each of the sub-segments of
the total population to be researched” (Boddy, 2016, p. 431) in order to enable statistical generalization to the broad population from which the sample is drawn.
In a qualitative research design that does not draw on positivist or post-positivist
inquiry paradigms, statistical generalization to the broad population from which the
sample is drawn (e.g., all types of residential aged care facilities) is not the purpose of
the study. Rather the purpose is to develop in-depth and thick interpretations of what
is going on related to the aspects of the residential aged care facility as a social setting
that your research is focused on. Therefore, as we stated previously, your reason for
maximizing the variation in your purposeful sample of sites, texts, or participants must
be able to be justified in terms of adding complexity and breadth (Tracy, 2020), and
thereby depth and richness, to your data about your research problem or questions.
The take home point here is that choices between different types of sampling strategies when collecting qualitative data must be congruent with both the purpose of the
research and the methodological assumptions underpinning the research design.
Putting Purposeful Sampling Into Practice When Designing Your
Research
How do you actually put purposeful sampling into practice when designing your research?
A useful way to address this question is to share an example of how we have thought about
and used the logic of purposeful sampling when designing our research. In this example,
which can be found in the Putting It Into Practice box below, we have cut and pasted in
sections from parts of the research design that was the basis for the application for funding
for our study “Digitalization, Organisations and People: Issues and Challenges. Exploring
the Digitalization Imperative in Higher Education Institutions in Norway.” Using these
excerpts, we demonstrate the logic behind the decisions we made about the sampling we
were proposing to use in that study.
PUTTING IT INTO PRACTICE
PUTTING THE LOGIC OF PURPOSEFUL SAMPLING INTO
PRACTICE WHEN DESIGNING RESEARCH
We wanted to study the effects of the imperative to “go digital” in higher education
institutions in Norway. Therefore, we wrote a proposal to seek funding for a research
study that we called “Digitalization, Organisations and People: Issues and Challenges.
Exploring the Digitalization Imperative in Higher Education Institutions in Norway”
(Cheek & Øby, 2018).
Chapter 6
•
Obtaining Data Using Qualitative Approaches
143
The purpose of the study was to
1. Explore how digitalization is understood and enacted in higher education
institutions in Norway.
2. Identify and describe issues such digitalization poses for people in those
institutions.
3. Explore and identify implications of the findings from Aims 1 and 2 for how to
optimize the implementation of, and therefore potential offered by, digitalization
of higher education.
Sampling: Selection of Study Sites and Participants
In keeping with the principles of qualitative inquiry, the study used a purposeful sampling strategy.
The sampling in the study was purposeful at three levels:
1. Choice of higher education as the organizational context in which the study will be
based
2. The higher education institutions selected as the sites for the study
3. The participants selected at various stages of the study
Justification for the Three Levels of Purposeful Sampling
1. The higher education sector
The logic behind the purposeful choice of the higher education section as the site
for the study relates to the fact that the entire Norwegian sector of higher education is currently affected by the government’s overt emphasis on the need for this
sector to embrace digitalization. This overt emphasis is apparent in the recent
report Digitalization Strategy in Higher Education and Research 2017–2021 (our translation), published September 19, 2017.16 The report reminds readers that institutions in higher education are “subject to the ministry’s authority and instruction”
(Kunnskapsdepartementet, 2017, p. 27, our translation), in this case to digitalize,
and that the institutions themselves are responsible for implementing digitalization
initiatives.
Thus, Norwegian higher education provided a timely, and information-rich context,
for our study of the connections between people, organizations, and digitalization.
2. Specific higher education institutions as study sites
A total of four higher education institutions would be selected according to the following grid (see Table 6.2 below) which employs the principles of maximum variation
sampling (including opposites). The logic behind this type of sampling is that “common patterns that emerge from such variation are of particular interest and value in
TABLE 6.2 ■ Categories of Higher Education Institutions From Which
Study Sites Will Be Selected
Government
Nongovernment
Research
intensive
Government higher education
institution designated research
intensive—university or
specialized university status
Nongovernment higher education
institution designated research
intensive—university or specialized
university
Non- research
intensive
Government higher education
institution designated
non-research intensive—public
University College status
reflecting a tradition of focus on
teaching rather than research
Nongovernment higher education
institutions designated nonresearch intensive and/or teaching
intensive—have individual
study program accreditation
(e.g., bachelor studies), but not
organizational accreditation
144 Research Design
capturing the core experiences and central, shared dimensions of a setting or phenomenon” (Patton, 2002, p. 235)—in this study the complex intersections between
digitalization, organizations and people in them.
3. Selection of participants
Participants would be drawn from staff who are employed at the study sites and
would include
• Administrators of varying levels and experience at each site
• Academic staff of varying levels and experience at each site
• Managers at varying levels who contribute to or determine policy related to
how digitalization occurs at each site
Purposefully selected academics, administrators, and managers selected from
each of the four sites using critical case and “snowball” (Patton, 2002, p. 243) purposeful sampling would be invited to tell their “stories” of specific digitalization-related
incidents in which they have been involved.
Key points that the example in the box above highlights about using purposeful sampling to obtain the qualitative data needed for addressing your research problem are that
• Thinking needs to be done about every sampling choice that you make in your
research design.
• There are different levels of sampling choices to be made when designing
your research. For example, what instance, example, or case of what you
are interested in will be studied (e.g., digitalization in higher education);
where will it be studied (the specific higher education institutions); and who
will be asked about it (the specific participants in those higher education
institutions).
• It is possible to employ different strategies of purposeful sampling at different
levels or within levels of the study. For example, we chose higher education as
the overarching information-rich case or instance of digitalization to study. This
is because due to the government requirement that organizations in the higher
education sector in Norway digitalize, they are information-rich sites about issues
and challenges arising from the intersection of digitalization, organizations, and
people.
• Having decided this, we then chose to employ a number of purposeful
sampling strategies within this overarching choice. We used the strategy of
maximum variation sampling to select which specific higher education
institutions we would use as the actual sites for the study. Following this we
used “critical case” and “snowball” purposeful sampling strategies (Patton,
2002, pp. 236–238) to identify which people in those sites we would
interview.
• Each level of this cascade of choices related to sampling strategies to be used
needed to be justified and explained. This required us to add an “and why” to our
thinking about every decision that we made about our sampling. It was part of
ensuring the trustworthiness and credibility of our research design.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
145
PUTTING IT INTO PRACTICE
GOING DIGITAL—HOW DOES THIS AFFECT HOW QUALITATIVE
DATA IS COLLECTED?
The internet provides us with tools and strategies that can be used for collecting qualitative data online. For example, in-depth individual interviews or focus groups can
be conducted verbally and virtually over platforms such as Zoom, Teams, or Skype.
This can be very useful when it is not possible to meet in person for the interview, for
example, during the COVID-19 pandemic.
Alternatively, you may choose to include some form of online interviews in your
research design because of feasibility considerations. For example, the purposefully
selected participants for your focus group live in different countries (e.g., if your study
is about something to do with how CEOs of major companies in different countries
perceive good leadership post–COVID).
However, even when you have decided to interview online, you still have some decisions to make. One of them is whether choosing to interview online will affect the
ability of your information-rich participants to be able to participate in your study.
For example, depending on what you are studying and who makes up your purposeful
sample, some participants in your sample may not have access to computers or be
familiar with or comfortable using platforms such as Zoom.
You will also need to decide if you will conduct online interviews comprised of
verbal exchanges or use some form of textual exchanges such as using email, or
combinations of both, such as a verbal exchange over Zoom followed up by an email
exchange.
If you choose to interview using some form of email exchange with a participant,
you will also need to decide if that exchange will occur in a set time frame when both
you and your participant are online at the same time (known as synchronous interviews), or will you send and receive emails over a longer period of time when you both
may not be present online at the same time (known as asynchronous interviews).
If using focus groups, you may design the focus groups around asynchronous written exchanges on electronic bulletin boards or other types of online textually based
forums. In this case, you might place an initial question on that forum or bulletin board
after which the participants respond to your question and to each other’s responses
about that question. In this way, a series of discussion threads is developed for each of
the questions that you pose (see Morgan, 2019).
Alternatively, you may choose to have all participants in the virtual focus group
present at the same time. You can do this by either having them all on video links at
the same time (e.g., at a Zoom meeting) or you may have them all participating in a
text-based email exchange between a set period of time.
No matter what way you decide to design your online interviews or focus groups,
you will need to make sure that technical issues do not interfere with the interview
or focus group quality—for example, connection problems resulting in participants
dropping in and out of meetings.
The take-home point from all this is that collecting qualitative data online introduces another layer of issues that you will need to think through, make decisions
about and justify, when designing your qualitative research.
One of these is why using online methods to collect data is appropriate for your
qualitative research study. Another is why using the specific form of the online method
you have chosen to use is appropriate for the study. When thinking about this, “the
nature of the topic and the characteristics of the desired participants will be crucial in
deciding whether to use online groups [or interviews], and if so, which form” (Morgan,
2019, p. 123).
146 Research Design
CONCLUSIONS
In this chapter, we have explored some of the interrelated design decisions you will need
to make related to collecting qualitative data and the methods that you will use to do so.
Choosing to use a particular type of qualitative method to obtain your qualitative data is
just the beginning of the thinking that you will need to do about that method. As our discussion has highlighted, there is still a lot of thinking to be done after making that choice.
You will need to put as much effort into designing how the method(s) that you have chosen
to use will be put into practice, as you did into the choice of those methods in the first place.
For example, if your method of choice is some form of qualitative interview, you will need
to consider, and make choices about, how structured the interviews in your study will
be, whether they will be individual or group interviews of some kind, what you will ask
in those interviews, and who you will interview. The credibility of the method you use
to collect qualitative data, and therefore the trustworthiness of the research, depends on
how you justify these choices.
Such justification includes demonstrating that the form that the methods take, for
example, the type of sampling that they use or the amount of structure that they have,
is congruent with the methodological assumptions underpinning your research. It also
requires you to show how the methods you use, and the type of data you obtain by using
those methods in the way that you do, enables you to address your research problem or
questions. Such incremental, iterative, and connected thinking needs to be done no matter which qualitative research approaches, or the methods associated with them, you are
using in your research design.
In putting the emphasis on the thinking about methods and not simply methods as techniques throughout this chapter, we hope we have disrupted the way that methods are
very often presented and discussed. In these discussions, methods are removed from the
context of the research design that they are part of and presented as self-contained procedures—procedures that in turn are made up of stand-alone sets of techniques that can
simply be picked up and inserted into a research design.
In the next chapter, we continue our discussion of how to put the methodological thinking associated with, and underpinning, qualitative research approaches into practice
when designing research. Our focus is the series of decisions you will need to make about
how you will analyze the qualitative data that you collect. Such analysis begins from the
moment you obtain your first qualitative data and continues throughout the qualitative
research process. It affects other design considerations such as sample size. Therefore, how
and when this analysis occurs in qualitative research, and why, are important considerations when you design your research.
SUMMARY OF KEY POINTS
• There is much variation between types of qualitative methods.
• Qualitative methods do not have a standard form; disciplinary and theoretical
paradigms and traditions shape the way that a qualitative method is understood and
put into practice.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
147
• You will need to think about what you want your data to be able to be used for,
and why, when choosing what type of qualitative method (e.g., an interview or an
observation) to employ in your research design.
• However, choosing a specific qualitative method to collect qualitative data as part of
your research design is just the beginning of the thinking you will have to do about
that method.
• For example, if using interviews to collect qualitative data, you will also need to think
and make choices about how structured the interview will be, who you will interview,
whether to use individual interviews or group interviews, as well as what questions to
ask.
• Similar principles apply when using other qualitative methods. For example, using
observations you will need to think, and make choices, about how structured your
observations will be, as well as who, what, and where you will observe.
• All choices you make when putting a qualitative method into practice must be
justified and explained. This is part of ensuring the trustworthiness of the research
and the credibility of your research design.
• Collectively, this thinking, and the choices and decisions associated with that
thinking, will determine the form that the method you use in your study will take,
and how that method will be put into practice.
• Therefore, before being able to use any qualitative method to collect data that will
enable you to answer your research question(s), there is a series of questions you need
to ask yourself, and decide upon, about that method.
• All decisions you make should also involve thinking about the feasibility of the study.
• Deciding to use some form of online method to collect data raises further design
considerations.
• It is important to remember that methods are not just about collecting data, they also
provide strategies for analyzing that data. The next chapter develops this key point.
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
focus groups
individual interviews
interview guide
interview probes
lines of inquiry
purposeful sampling
semistructured interviews
sensitizing concepts for observations in
qualitative inquiry
standardized observations
structured or closed interviews
unstructured or open interviews
SUPPLEMENTAL ACTIVITIES
1.
Tracy (2020) highlights eight purposeful sampling plans that may be used when
collecting qualitative data. You will find them in Table 6.1 in this chapter. For each
of these sampling plans, work out
148 Research Design
•
•
•
•
2.
A research question that might use this type of sampling strategy
What inquiry paradigm(s) might use that strategy, and which would not
The benefits or strengths that each type of sampling has
The disadvantage or limitations that each has
Find an article that is a published report of qualitative research that used interviews
to collect the qualitative data. When reading this report ask yourself these questions:
Do the authors make clear
•
•
•
•
what the purpose of the research was?
why they used a qualitative approach?
why they used interviews as their method of collecting qualitative data?
what disciplinary or inquiry traditions, or theoretical schools of thought, the
authors have drawn on related to the interviews and how this affected their
view of what type of interview to use?
• what choices they made related to
a. how structured their qualitative interviews were?
b. whether to interview participants individually, or in some form of
group?
c. what they asked participants in the interview?
d. who they interviewed?
The answers to these questions are part of establishing the credibility of that reported
research. How convinced a reader is of that credibility is based on what the authors
report about these types of choices.
If you struggle to find an article, ask your tutor or fellow students for
recommendations relevant to your area of study.
FURTHER READINGS
Denzin, N. K., & Lincoln, Y. S. (Eds.). (2018a). The SAGE handbook of qualitative research (5th
ed.). SAGE.
Tracy, S. J. (2020). Qualitative research methods: Collecting evidence, crafting analysis, communicating impact (2nd ed.). John Wiley.
Yin, R. K. (2016). Qualitative research from start to finish (2nd ed.). Guilford Press.
NOTES
1. This is a point that we will return to in the next chapter which focuses specifically on the
analysis of qualitative data.
2. In Chapter 9, we return to discuss closed questions.
3. See Chapter 8.
4. Morgan (2019, pp. 5–6) points out that one of the issues regarding the definition of
focus groups “is whether they are different from group interviews. Although some
early authors (e.g., Frey & Fontana, 1991) made a distinction between focus groups and
group interviews, the broad tendency since then has been to treat the two labels as synonyms. As a result, ‘focus groups’ is an umbrella term encompassing many alternative
formats.”
5. See the discussion of feasibility in Chapter 3.
Chapter 6
•
Obtaining Data Using Qualitative Approaches
149
6. See Chapter 7, where transcription is discussed.
7. We return to this point in Chapter 7 which explores sample size considerations when
analyzing qualitative data.
8. We return to this point, and develop it, in Chapter 7.
9. See Chapter 5.
10. See Cheek, J., & Ballantyne, A. (2001) Moving Them On and In: The process of searching
for and selecting an aged care facility. Qualitative Health Research, 11(2), 221–237.
11. See Chapter 7 where this iterative, ongoing analysis process, involving the simultaneous collection and analysis of qualitative data, is discussed in more detail.
12. See Chapter 2.
13. See Denzin (1971) for a very useful discussion of how sensitizing concepts guide naturalistic inquiry such as field work. Also see Patton’s excellent discussion of field work
strategies and observation methods—Chapter 4 in Patton (2002).
14. Silverman, D. (2006). Interpreting qualitative data: Methods for analyzing talk, text, and
interaction (3rd ed.). SAGE.
15. Activity 1 in the supplemental activities at the end of this chapter develops this point by
focusing on different purposeful sampling plans (Tracy, 2020) and asking you to work
out for each of these sampling plans: a research question that might use this type of
sampling strategy; what inquiry paradigm(s) might use that strategy, and which would
not; the benefits that each has; and the disadvantages that each has.
16. The report states that
[f]or higher education and research in Norway to make use of the potential of
technology, . . . using technology . . . [must be] lifted to the strategic level of
the institution and integrated into all academic and administrative activities. . . .
Development and use of technology in the sector must therefore be anchored
in strategies, both at national and institutional level (Kunnskapsdepartementet,
2017, p. 5, our translation, our emphasis).
7
ANALYZING AND
INTERPRETING
QUALITATIVE DATA
PURPOSES AND GOALS OF THE CHAPTER
This chapter is the second of four interconnected chapters (Chapters 6, 7, 8, and 9) exploring how to put the methodological thinking associated with, and underpinning, qualitative and quantitative research approaches into practice.
The previous chapter (Chapter 6) focused on what you will need to think through and
make decisions about when collecting data using qualitative methods. However, qualitative methods and the strategies of inquiry they employ are not just about collecting data.
They are also about analyzing and interpreting that data. In this chapter, our focus is the
thinking you will need to do and the decisions you will have to make related to how you
will analyze and interpret the qualitative data you collect.
Throughout the discussion we emphasize that the collection and analysis of qualitative data is a simultaneous and iterative strategy. Analysis of qualitative data begins when
that data is being collected. Data collection and analysis continues iteratively until trustworthy and credible interpretations of that qualitative data can be made.
Developing an iterative qualitatively driven analytic strategy involves making decisions about how you will organize the data you collect, as well as how you will justify and
keep track of the decisions you make when analyzing that data.
We explore how the decisions you make when designing your research affect the credibility or trustworthiness of the interpretations you make based on the data that you have
collected and analyzed. For example, how will you know that you have “enough” data, or
have done “enough” analysis, to be able to make credible and trustworthy interpretations
of that data?
Throughout the chapter, we establish some common features, and principles, of qualitative analysis and interpretation. However, at the same time, we highlight, given the
variation in types of qualitative approaches and methods, how these common features and
principles are put into practice varies.
The goals of the chapter are to
• Establish that analyzing qualitative data is an iterative and reflexive process that
is fundamentally inductive.
• Emphasize that data analysis and data collection occur simultaneously in
qualitative research approaches.
• Highlight that analysis of qualitative data is underpinned by the “art of
interpretation” (Denzin & Lincoln, 2000, p. xii).
151
152 Research Design
• Stress that analyzing and interpreting qualitative data are strategies, not
procedures.
• Illustrate why design considerations related to analyzing and interpreting
qualitative data cannot be reduced to, or captured by, a fixed linear series of
prescribed steps or phases.
• Establish some recurring features and principles common to many approaches to
analyzing qualitative data.
• Identify practical strategies for managing and organizing the large amounts of
data that you obtain such as transcription of interviews.
• Explore the role that strategies such as coding and writing memos play in many
qualitative analyses.
• Demonstrate how the decisions related to the collection and analysis of data that
you make when designing your research affect the credibility or trustworthiness
of the interpretations you make based on that data and those analyses.
• Explore how decisions about how many participants to include in a qualitatively
driven research study are guided by the principle of obtaining in-depth and rich
data that enable credible interpretations to be made, rather than the number of
participants as such.
ANALYSIS OF QUALITATIVE DATA: AN
ITERATIVE AND DYNAMIC STRATEGY
Analysis of interview, observational, or textual data does not simply describe what people
said in an interview, did when they were observed, or what a text depicts. Rather, analysis
is about making sense of what was said, observed, or depicted to address or answer your
research questions. Making sense of such data requires “thinking about what the data
mean” (Patton, 2012, p. 349).
Thinking about what the data mean requires you to become very familiar with, or
immersed in, the data you have collected. You will need to spend a lot of time with it—reading it, thinking carefully about what you have read and what it means, reading it again, and
so on. In this iterative process, you examine and reexamine your data, collect more data
when necessary, and describe, analyze, and reanalyze that data in order to make sense of it.
For example, if you use some form of qualitative interviews to obtain your data, then
you continually, or constantly, compare1 what you have found in one set of data (such as an
interview) to other sets of data (such as other interviews) + continually compare and refine
your hunches or tentative interpretations of that data and how they relate to what it is that
you want to know about (your research questions) + compare those tentative interpretations to other relevant theoretical and empirical work in the area (what have others found
and how does what you have found relate to this work).
As a result of this iterative and reflexive process, “the researcher’s understanding grows
so that she or he can begin to create models or diagrams of the relationships in the data,
make links with the literature, seek relationships among concepts or categories” (Mayan,
2009, p. 88). Such analytical work is a central part of the “art of interpretation” (Denzin
& Lincoln, 2000, p. xii), which is a central tenet of the craft of qualitative inquiry. The
analytical work that you do “creates the justification, provides the evidence for the
Chapter 7
•
Analyzing and Interpreting Qualitative Data
153
interpretations that result from that work, and sets the stage for the organization and style
of presentation for those interpretations” (Freeman, 2017, p. 123).
PUTTING IT INTO PRACTICE
PUTTING AN ITERATIVE AND REFLEXIVE ANALYSIS
STRATEGY INTO PRACTICE
Putting an iterative analysis strategy into practice is not straightforward. There are
no linear steps to follow. Rather, the strategy is comprised of reflexive cycles of constantly thinking about the data we are collecting and the interpretations we are making of that data. So how might you put this sort of strategy into action?
Agar’s (2008) description of how an iterative analytical process unfolds provides
you with a way forward when thinking about how to undertake such a nonlinear analytic process. We have broken his original quote into dot points to indicate the elements of such a process:
•
•
•
•
you learn something (“collect some data”),
then you try to make sense out of it (“analysis”),
then you go back and see if the interpretation makes sense in light of the new
experience (“collect more data”),
then you refine your interpretation (“more analysis”) and so on.
(text from Agar, 2008, p. 62, dot point structure added)
Such a process will require you to keep asking yourself a lot of questions, both
about the data itself and what you are saying about it and using that data for. Figure
7.1 below provides a visual illustration of how an iterative analytical process unfolds,
according to Agar (2008).
FIGURE 7.1 ■
How an Iterative Analytical Process Unfolds
You learn
something
(“collect
some data”)
You try to make sense
out of it (“analysis”)
You go back and see if
the interpretation makes
sense in light of the new
experience (“collect more
data”)
and so on
You refine your interpretation
(“more analysis”)
and so on
At this point, you also might find it useful to revisit the discussion in Chapter 1
about the ideas of iteration and reflexivity as they are central to any discussion of analyzing qualitative data, including the discussion in this chapter.
WHEN DOES ANALYSIS “BEGIN” WHEN DESIGNING
AND CONDUCTING QUALITATIVE RESEARCH?
Given our previous discussion (see box above) of how you might go about putting an
iterative analysis strategy into action, it should not surprise you that when using qualitative research approaches, data collection and analysis is a dynamic process that occurs
154 Research Design
simultaneously (Merriam & Tisdell, 2016). Therefore, data analysis “begins” as you are in
the process of collecting your first qualitative data.
For example, if you are collecting data using some form of qualitative interview, then
analyzing information obtained from that interview begins as the interview is taking
place. The use of probes2 such as, “What did you mean by that?” or “Am I right in understanding that what you are saying is . . .” or “How does what you just said fit with what you
said previously about X?” indicates that the interviewer is listening to, and at the same time
analyzing and trying to make sense of, what is being said as the interview unfolds. Such
probing and clarification of what was said, and why, continues throughout the interview.
When the interview is completed, you continue your analysis of it. For example, immediately after the interview ends, you may make notes of your reflections of what stood out to you
from what the participant said in terms of it being relevant to your research problem or questions. If you have conducted other interviews prior to that interview, you may also make notes
about areas or ideas that seemed to be the same as, or different from, other interviews, and make
a note or memo to yourself to check this out further. These notes capture your tentative hunches
about what might be going on in this interview in terms of what it is telling you about what it is
that you want to know more about and how this relates to what other interviews have told you.
In addition, as soon as possible after each interview you will usually produce some
form of written record or transcript of that interview, read it line for line, and highlight
parts of the interview that are information rich in terms of an aspect or aspects of the questions being asked. When doing so, you will make notes about your impressions of how
what you have just highlighted in the interview transcript relates to your research questions. You will also make notes about how what you have highlighted relates to sections
that you have highlighted in earlier interviews.
You will undertake a similar process of analysis for each interview you conduct. At the
same time, you will continually reflect on, and compare, the information gained from each
specific interview to all the other interviews that make up the data for the study. When doing
so the focus is “making connections or asking questions about why something is the way it
is. We are entertaining theoretical notions about the phenomenon” (Mayan, 2009, p. 89).
Making such connections adds depth to your analysis and lifts it beyond a commonsense
description of what each person said in the interview. It is an important part of the “art of
interpretation” (Denzin & Lincoln, 2000, p. xii) that underpins qualitative analysis.
TIP
CODING
These notes, and highlighted parts of the transcript text, are the parts of the interview
that you identify as being potentially relevant and useful in some way for addressing
your research problem. This process is often referred to as coding. Coding is a process
used in many types of qualitative research approaches. We will return to explore the
idea of coding in more depth later in the chapter.
Using Memos to Capture Your Analytic Thinking and Hunches
Often you will see the notes you make about the interview, and your hunches about how
that interview might relate to other interviews you have conducted and begun analyzing,
called memos. Writing memos enables “thinking on paper” (Maxwell, 2013, p. 20) as you
Chapter 7
•
Analyzing and Interpreting Qualitative Data
155
try to capture the insights you have gained when describing and analyzing what people
have told you in an interview.
The idea of memos was introduced in a type of qualitative research known as
grounded theory by Glaser and Strauss in 1967. They observed that when “generating
theory it is often useful to write memos on . . . the copy of one’s field notes. Memo writing
on the field note provides an immediate illustration for an idea” (Glaser & Strauss, 1967,
p. 108). Later, Glaser defined memos as “the theorizing write-up of ideas about codes and
their relationships as they strike the analyst while coding” (Glaser, 1978, p. 83).
Since then, the idea of memoing and writing memos has come to be used by qualitative researchers more generally, not just those generating grounded theory. In this case, the
term memo is used to refer to the preliminary and developing analytical notes or hunches
written by a qualitative researcher during analysis of data. Keep in mind that when you
begin to write memos, the first memos you write might seem “awkward and simple. . . .
Later in the analysis, memos become longer and take on more depth” (Corbin & Strauss,
2015, p. 121). Early on, when you are in the beginning stages of your analysis, it is likely
that your memos will take more the form of descriptions and beginning hunches about
some aspect of the data collection process. As your iterative analysis proceeds, your memos
will change from being mainly descriptive to more analytic and interpretive in focus.
PUTTING IT INTO PRACTICE
WHAT DO YOU WRITE MEMOS ABOUT AND WHAT FORM
SHOULD THEY TAKE?
There is no set form that a memo must take or what it must be about. Memos might
be descriptions of what happened when you were collecting your data, your hunches
about what is going on related to your research problem or questions, or your thoughts
about what you will need to find out more about. Memos can also be used to highlight
consistencies or inconsistencies in the data collected that require further elucidation
or follow-up.
For example, if you are using some form of qualitative interview to obtain data
there might be consistencies or inconsistencies within an individual interview, or that
emerge across interviews, as you collect more interview data. You can use memos to
capture your thoughts about this as you read the transcript of, or reflect on, the sound
file of the interview.
An example of this is the memo written during one of the interviews with one of the
participants in the study one of us was involved in about the process of searching for
and selecting an aged care facility following the discharge of a family member from
an acute setting. The focus was participants’ perceptions of the search and selection
process and its effects (see Cheek & Ballantyne, 2001).
This participant, like every other person we have interviewed so far, mentioned
that they had to fill in different forms for each aged care home when applying
for a space for an older person and that this caused them great stress. Some
mentioned having to apply to 10 or more residential care homes to try to get
a place. Maybe we need to think more about the effect of something that can
seem so ordinary such as the filling in of an application form. How has this situation arisen? What is this an example of and what is causing and sustaining it?
Maybe need to check other studies to see if this has come up in them. Can also
156 Research Design
probe more in next interviews—perhaps even return to those already interviewed to seek more clarification?????
(memo related to lines 50–55 of the transcription of interview with participant
X, taken from private study notebook, Cheek & Ballantyne).
This memo is about a surprising consistency that we identified emerging across
the interviews and our thinking about what this might mean in terms of what else we
might need to find out about. Ultimately, after more data about this was obtained and
much more thinking was done, the series of actions that this initial hunch captured in
this memo led to the emergence of one of the key findings of the study. This was the
stress caused by dealing with an uncoordinated system, which “meant that each form
had to be collected or requested, filled out, and returned to each facility” (Cheek &
Ballantyne, 2001, p. 231).
The statement from the interview that this memo was made about when we read
it was
[h]alf of them wanted them to be signed by a justice of the peace and have a
doctor’s signature. Everyone wanted a doctor’s report, so you’d have to go back
to the doctor to get a doctor’s report and everyone was writing differently. . . .
Instead of treating them all the same, so that we could photocopy them and
send them out, they all wanted something different. (Cheek & Ballantyne, 2001,
p. 231)
Our initial code for this section of text was lack of system and coordination and it
became part of the overarching theme we called “Dealing With the System—Cutting
Through the Maze” (see Cheek & Ballantyne, 2001, pp. 228–231).
Why You Should Not Wait to Begin Analyzing Your Qualitative Data
Until All Your Data Is Collected
The previous discussion highlights that the analysis of qualitative data, for example a
single interview, begins as you are undertaking that interview, and continues throughout
the entire process of the research as more interviews are conducted. In this process, you
will return to each of your interviews and compare your initial analyses of them with the
analyses you make about each new interview that you have just done. Do the insights
gained from new interviews change in any way those gained from earlier ones? Are they
in keeping with them? Are there new or different insights emerging from a new interview?
This type of reflexive iterative thinking involves “an ongoing dialogue with and
between data and ideas” (Coffey, 2018, p. 25). For example, you may add questions to your
interview schedule because of responses (possibly unanticipated) that you have received
and now need to know more about. You cannot do this if you wait to begin the analysis of
your qualitative data until all your data is collected.
TIP
DIFFERENT WAYS OF THINKING ABOUT THE TIMING OF
COLLECTING AND ANALYZING DATA
This is a very different way of thinking about the timing of, and strategy for, the collection and analysis of data compared to quantitative research approaches. In quantitative research approaches, the collection of all the quantitative data occurs in a
Chapter 7
•
Analyzing and Interpreting Qualitative Data
157
standardized way and precedes the analysis of that data. This is because in quantitative
research approaches, all the data (i.e., all the numbers) need to be collected before
statistical rules and procedures can be used to analyze those numbers and determine
what type of conclusions can be drawn from that analysis. These statistically derived
rules and procedures are standardized and are comprised of very structured and
clearly articulated steps to be followed linearly.
DEVELOPING AN ITERATIVE QUALITATIVELY
DRIVEN ANALYTIC STRATEGY
Given the iterative nature of the analysis and interpretation of qualitative data, it is not
possible to give you a prescribed set of standardized procedural rules or instructions to
be followed in all instances when analyzing the qualitative data that you have collected.
Design considerations related to analyzing and interpreting qualitative data cannot be
reduced to, or captured by, a fixed linear series of prescribed steps or phases.
However, although qualitative analysis cannot be reduced to a linear series of steps, or
a normative or standardized checklist of what to do, when and how to do it, this does not
mean that analysis of qualitative data is in any way a “free for all” where anything goes.
There are “features that recur in many approaches to qualitative analysis” (Miles et al.,
2014, p. 10).
For example, no matter what type of qualitative data you are analyzing, or which specific qualitative approach you are using in your study design, you will simultaneously need
to develop strategies related to
• organizing the data you collect, and keeping track of your analytical thinking
about that data, and
• deciding on some sort of process for making analytical choices about what parts
of the large body of data you have collected are relevant for addressing your
research questions.
In this section of the chapter, we will explore some of the strategies developed and used
in many approaches to qualitative analyses to address the dot points above.
Strategies for Organizing the Data You Collect and Keeping Track of
Your Analytical Thinking About That Data
When you are collecting qualitative data, you will need to think about what form that
data will take and how it will be organized. At the same time, you will also need to think
about how you will organize and keep track of your iterative analysis of that data, including any memos that you make related to that data. In other words, you will need to make
choices about how you will sort and compile your data, including the record of the process
of its analysis. Such a compilation will provide you with a type of dynamic qualitatively
derived “database” (Yin, 2016, p. 186) of both your initial data and your progressive analytical thinking about that data.
For example, if you are using interviews to collect your data, you will need to think
about how you will organize that interview data. It is highly likely that during each interview you will make a record of what is said using some form of recording device. It makes
the interview data much easier to work with if the sound file of the interview is converted
158 Research Design
into written text as soon as possible after the interview. This is called transcribing the
interview.
On the surface, transcription may seem to be a simple exercise of converting spoken
words to written words to provide written texts that can then be analyzed—a prelude to
the analysis. But this is not the case. Transcribing an interview is more than a mechanical
process or a prelude to the analytical process. Rather, producing the transcript is part of
the analysis itself. During the process of transcribing an interview, you revisit what happened in that interview while becoming more familiar with both what was said and the
interactions that led to it being said. In other words, you immerse yourself in that interview data and begin to analyze it.
Listening to the sound recording of an interview and producing the written wordby-word transcript of it provides a way of reflecting on that interview. When listening to
the interview, you can think about aspects of what happened during the interview: Why
was that said? Why was there silence at that point of the interview? Why did this question
seem to be so important to the participant? These reflections can be captured in memos
you make to yourself as you are producing the transcription. When doing so, you begin
“making connections or asking questions about why something is the way it is” (Mayan,
2009, p. 89). In other words, you are in the process of analyzing that data.
On a more practical level, you will need to think about how you will organize your
transcribed data so that such a large amount of data is able to be navigated effectively—
both within, and across, interviews. One way of doing this is to number each interview
(e.g., Interview 1, 2, and so on) as well as numbering each line of the transcript so that
it is possible to readily find, and refer to, specific segments of the text (e.g., Interview 5,
line 250).
You will also find it useful to write a separate short description of who was interviewed,
including any relevant demographic data about that participant, and any other contextual details you think are relevant or useful. However, when doing so it is important that
you consider and think through the ethical issues we discussed previously in Chapter 2
that may be raised by linking demographic and interview data and thereby inadvertently
violating participant confidentiality and anonymity.3
TIP
ANOTHER THING TO THINK ABOUT
Even more issues arise if for some reason the researcher does not transcribe the
interview themselves and instead, for example, the interviews are transcribed by a
research assistant, another researcher in the team who did not conduct the interview,
or a paid transcription service. See Liamputtong (2013, pp. 66–68) for a good discussion of these issues and what you will need to think about when addressing them.
It is also important to transcribe (if necessary) or systemize the way that you compile
any memos you have made during or after interviews, or during the transcription process.
Each memo should be given numbers, labels, or descriptors that make it easy for you to
find both specific memos (e.g., Memo 4 made 5/1/2022 after Interview 2 in relation to . . .)
and include the specific parts of the interview that the memo relates to (e.g., Memo 4
related to Interview 2, line 2–7).
Chapter 7
•
Analyzing and Interpreting Qualitative Data
159
Strategies such as these ensure that as you develop your inventory, or type of data base,
of memos and interviews, you are building a rich depository of information that can be
navigated effectively because it is organized in a careful and systematic way. Without some
sort of system such as the ones outlined above, it will be easy to have a sense of drowning,
rather than being immersed, in your data!
Strategies for Deciding What Parts of the Data You Have Collected
Are Relevant for Addressing Your Research Problem
In any qualitative analytical strategy, you will make choices about what parts of the large
body of data you have collected are relevant for addressing your research questions. When
doing so, you will use some sort of strategy for data condensation. Data condensation
refers to “the process of selecting, focusing, simplifying, abstracting, and/or transforming
the data that appear in the full corpus (body) of written-up field notes, interview transcripts, documents, and other empirical materials” (Miles et al., 2014, p. 12).
TIP
CONDENSATION OR REDUCTION?
Sometimes you will see this process referred to as reduction of data. We use the term
condensation, rather than reduction. This is because it better captures the idea of putting data together, or condensing it, to create richer and more nuanced data. Reduction,
on the other hand, can give the impression of taking something away from that data, or
making it lesser in some way (Miles et al., 2014).
This process of data condensation begins the moment we start making research
design–related choices about our qualitative inquiry—not just when we are analyzing the
qualitative data we have collected. As Miles et al. (2014) point out,
Even before the data are actually collected, anticipatory data condensation is
occurring as the researcher decides (often without full awareness) which conceptual framework, which cases, which research questions, and which data collection
approaches to choose. As data collection proceeds, further episodes of data condensation occur. . . . The data condensing/transforming process continues after
the fieldwork is over, until a final report is completed. (p. 12)
Therefore, when you condense the data that you have collected, you are not just making that volume of data more manageable. You are also making analytic choices about what
parts of that data are relevant and can contribute to addressing your research question(s).
For as Wolcott (1994) reminds us, “Everything has the potential to be data, but nothing
becomes data [including what is said in interviews, done in observations, or written in
texts] without the intervention of a researcher who takes note—and often makes note—of
some things to the exclusion of others” (pp. 3–4).
The Process of Data Condensation
If, for example, you have collected qualitative data using some form of interview, then the
first thing you will need to do is to read the transcript of each interview and any memos or
160 Research Design
observations that you have made related to the collection of that data. This enables you to
get a “sense of the whole” (Patton, 2002, p. 440),4 or what is there, in terms of the data you
have collected. As you read and re-read those texts, you will simultaneously begin to think
about what things stand out, or seem important, with respect to the research questions
you are asking.
One way you might think of doing this is by highlighting sections of your transcribed
interview texts that relate to the particular lines of inquiry that you are interested in, and
around which your interview guide was constructed.5 In this way, your lines of inquiry
provide an organizing construct for what you highlight. For example, you might highlight
in green the sections of text on a printed, or electronic, version of the interview transcript
that seem to be related to line of Inquiry 1, orange for sections of text related to line of
Inquiry 2, and so on. The chunks, or sections, of text that you highlight are ones that you
have identified as meaningful in some way in relation to your lines of inquiry, and hence
research questions or aims. Put another way, the sections of highlighted text are units of
data that are “a potential answer or part of an answer to the question(s) you have asked in
this study” (Merriam & Tisdell, 2016, p. 203).
Instead of working with pen and paper or using text highlighting on a digital transcription on your preferred word processor software, you might choose to use one of the
computer software programs that have been developed to assist in selecting or presenting
sections of text that might be relevant to your lines of inquiry. However, if you do choose
to do this, it is critical to remember that such programs do not do the analytical thinking
and interpretation for you. Rather, the “basic idea behind these programs is that using the
computer is an efficient means for storing and locating qualitative data” (Creswell, 2014,
p. 195).6
It is important to remember that no matter which way you choose to work—“low
tech” (highlighter pen and paper) or “high tech” (software programs)—the outcome will
only be as good as the thinking behind why you highlighted certain chunks, or sections,
of text in the first place. These highlighted sections of text “should reveal information relevant to the study and stimulate the reader to think beyond the particular bit of information” (Merriam & Tisdell, 2016, p. 203).
Examples of thinking beyond the particular bit of information include asking yourself
questions such as these: How might this unit of text be explained? What could help me
explain it? Why was this said and not that? Does this mean that . . . ? Such a questioning
process “sharpens, sorts, focuses, discards, and organizes data in such a way that ‘final’
conclusions can be drawn and verified” (Miles et al., 2014, p. 12).
PUTTING IT INTO PRACTICE
USING COMPUTER-ASSISTED QUALITATIVE DATA ANALYSIS
SOFTWARE (CAQDAS)
There is a variety of software designed to assist qualitative data analysis. These
include ATLAS.ti, MAXQDA, NVivo, and many more (see page 48, Saldaña, 2021, for
a more comprehensive list of software to explore). Computer-assisted qualitative
data analysis software (CAQDAS) is specifically designed to assist researchers when
Chapter 7
•
Analyzing and Interpreting Qualitative Data
161
“organizing, managing, coding, sorting, and reconfiguring data—both transcribed textual documents, PDFs, photographs, online data, and digital audio/video files” (Tracy,
2020, p. 242). While such software may help you “find, categorize, and retrieve data
and texts more quickly than using a manual search” (Liamputtong, 2020, p. 269), in
themselves they will not analyze the data:
Just as word-processing programs like Microsoft Word do not write a paper and
presentation, programs like PowerPoint do not, by themselves, design a slide
show, CAQDAS does not analyze data on its own. Rather, CAQDAS facilitates
qualitative data analysis—just as word-processing software facilitates writing
and presentation software eases presentation design. (Tracy, 2020, p. 242)
In other words, the need for immersion in the data accompanied by iterative and
reflexive thinking and deep mental involvement in the total body of data is not reduced
just because you choose to use some form of software to assist you when analyzing
qualitative data.
For example, if your data include transcripts of qualitative research interviews,
your total body of data will include many(!) pages of text. While CAQDAS enables coding text while working on a computer (no need for manually writing codes in the margin of interview transcripts, for example), and retrieving and grouping all chunks of
text marked with the same code (no need for manually copying text associated with
the same code and then pasting them together in a separate document), you will still
need to read the transcripts, and work iteratively with the data, as well as come up
with the codes.
Deciding whether to use CAQDAS to assist your analysis is a matter of cost, preference, and how much time you will need, or have, to spend mastering new software.
Bryman (2016) reminds us that CAQDAS may
be too expensive for your personal purchase, but if you have a free access to the
software, you may like to try. If you plan to use it in future research, it may be
worthwhile taking the time to learn. (p. 603)
To sum up: The key point to remember is that any CAQDAS will not in itself analyze
the qualitative data. It may assist in organizing and keeping track of that analysis but
even then, this will only be as good as the parameters that you enter to guide the software’s functions. It is you who will need to do the hard thinking work that underpins
qualitative analysis. It is also you who will have to justify the way that the CAQDAS program has organized your data. Therefore if, when reporting your research, you simply
state that the data you collected were analyzed using some form of software, “reviewers will know that you lack knowledge about data analysis in qualitative research”
(Liamputtong, 2020, p. 269).
CODING—A STRATEGY TO CONDENSE YOUR DATA
In many forms of qualitative analysis, but not all,7 the process of sharpening, sorting,
focusing, discarding, and organizing data is associated with the strategy of coding your
data. Coding refers to “the active process of identifying, labelling and systemizing data
as belonging to or representing some type of phenomenon” (Tracy, 2020, p. 234). When
coding, you will identify a segment of data, such as a word or a series of words in an interview transcript, and give that segment of data an initial label or code that captures what
the segment of data is about or relates to. Thus,
When you assign a word to a part of your data, when you write something in the
margin of a transcript, when you underline a word in a document, when you focus
162 Research Design
on a specific part of a visual, you are coding. . . . Coding is the first step in being
able to say something about the data. . . . It is the first step in enabling you to make
comparisons among pieces of data (Mayan, 2009, pp. 88–89).
For example, in the study mentioned earlier in this chapter about the process of searching for and selecting an aged care facility following the discharge of a family member from
an acute setting, the focus was participants’ perceptions of the search and selection process
and its effects (see Cheek & Ballantyne, 2001). Therefore, when reading the transcripts of
those interviews, you would highlight the sections of text in an interview transcript where
the person being interviewed talks about something related to, or a specific aspect of,
that experience. For example, they might tell you, “I had to sell most of the furniture and
personal items I had for many years.” Thinking about what this is an example of, you may
assign it the initial code LOSS. Later in the interview you might highlight another section
of text where they told you, “I knew I could not stay and had to move.” You may assign this
the initial code RESIGNATION. You will continue to do this as you work through your
interview transcript. This phase of initial coding of your data is sometimes called first
cycle (Saldaña, 2021) or primary-cycle (Tracy, 2020) coding.
However, just having developed initial8 or tentative codes for segments of your data
does not mean that your thinking about those initial codes is finished. In fact, in many
ways, your thinking about those codes has just begun. Throughout the iterative process of
coding, you will collect more data, identify more codes within that new data, write memos
about those codes, and compare these additional codes and memos to those you already
have. For example, if using interviews to collect your data you will constantly revisit your
initial codes and memos in order to refine or confirm them by comparing them with the
codes and memos you develop when analyzing the next interview.
When doing so, you will look for patterns in the codes across the interviews and will
begin to undertake a process of what Yin (2016) refers to as a “[r]eassembling phase”
(p. 202) of iterative data analysis. As you become more familiar with, and think more
about, the codes that you are giving to the segments of text that form your units of analysis, you will begin to look for patterns or regularities in those codes in terms of what they
are instances or examples of. To do so, you will need to “organize, synthesize, and categorize these codes into interpretative frameworks” (Tracy, 2020, p. 234). This is second cycle
(Saldaña, 2021) or secondary-cycle (Tracy, 2020) coding.
This type of coding requires “such analytic skills as classifying, prioritizing, integrating, synthesizing, abstracting, conceptualizing, and theory building” (Saldaña, 2021, p.
89). It requires you to ask questions of yourself such as, Are there segments of text (i.e.,
codes or parts of memos) that all talk about or relate to a particular aspect of a line of
inquiry you are interested in finding out something about? Are there common understandings and assumptions in the segments of text that you are thinking of putting
together? Or are the segments of text you have coded different instances or examples of
the same idea? If they are, could the codes related to these segments of data be grouped
together, or reassembled, to form a category—a higher level of focus that captures what
the codes in that category are instances of?
The development of categories, like coding, is an iterative process. As you collect more
units of data, and new and different codes emerge, you also revisit your tentative categories
and modify them in light of the emergent codes. You compare the developing and tentative categories to each other as they are being developed. This can then lead to the emergence of major themes that capture your findings.
Chapter 7
•
Analyzing and Interpreting Qualitative Data
163
When developing categories, it is important to keep in mind that a category is not just
a collection of codes. It is also a collection of the ideas behind those codes. This is imperative if the formation of categories is to move beyond the mechanical clumping of codes
together, and if the description and understandings of themes or categories are to move
beyond the superficial description of each individual code in that category.
TIP
USING TERMS WISELY AND CONSISTENTLY WHEN DESCRIBING
YOUR ANALYTIC STRATEGY
You will soon notice that some researchers use the terms codes, categories, themes,
and findings interchangeably in discussions of qualitative data analysis or reports of
research that used some sort of coding strategy. For example, Merriam and Tisdell
(2016) declare that “in our view a category is the same as a theme, a pattern, a finding,
or an answer to a research question” (p. 204). On the other hand, Mayan (2009) asserts
that often there is confusion and a lack of understanding about the difference between
categories and themes:
Themes are thoughts or processes that weave throughout and tie the categories
together. Theming, then, is the process of determining the thread(s) that integrate and anchor all of the categories. To form themes, the researcher returns
to the “big-picture” level and determines how the categories are related. . . .
Through the categories and then the themes, the researcher can make overall
conclusions about the research (p. 97).
The point that we are making here is that terms such as codes, theme, category,
findings, and conclusions, while they may sound like everyday words, cannot be used in a
commonsense or everyday way when talking about and describing qualitative analysis.
In this context, they are research method derived terms related to specific (although,
as we have seen, at times different) ways of thinking about complex aspects of analyzing qualitative data. Therefore, they must be used thoughtfully, with care, and with
acknowledgment of the complexity that their seeming simplicity masks. This applies
whether or not you are using coding as your principle analytic strategy.
Given the different views about terms such as codes, theme, category, findings, and
conclusions, it is important that you think through these terms (like all others that you
use as part of describing your research design), decide on the way that you will use
them, declare this, give reasons for that choice, and then be consistent in that use.
MORE CHOICES AND DECISIONS TO MAKE
WHEN PUTTING CODING INTO PRACTICE
Having made the decision to include coding as part of your analytical strategy in your
research design, you still have more decisions to make. This is because there is considerable variation in the way qualitative researchers who choose to use coding as part of their
analytic strategy put coding into practice. How they put coding into practice is influenced
by what they are trying to find out about—their research questions.
Methodological Choices About Whether to Employ an Inductive or
Deductive Approach to Your Coding
One of the choices you will have to make is whether you will employ an inductive or
deductive approach when coding your data. This choice is related to the purpose of the
164 Research Design
research and what type of knowledge will be able to meet that purpose. Given the exploratory and interpretive nature of qualitative inquiry, most qualitative researchers, at least
initially, will employ inductive logic or thinking when coding. This is because working in
this way, they can remain “open to what the site [or the person or the text] has to say rather
than determined to force-fit the data into preexisting codes” (Miles et al., 2020, p. 74).
These initial codes are developed inductively from the researcher’s reading of, and
thinking about, the data. The goal is to get a sense of the whole or what is going on in
the situation of interest related to the problem or research questions that their research is
about. What is happening? Why? How? These are the types of questions the researcher
will ask themselves as they constantly think through their data, as well as the codes, categories or themes, and initial tentative interpretations that they have inductively developed.
Other qualitative researchers, from the outset of the coding process, may use a form
of deductive coding. They look for the presence of codes that are instances of the specific
aspects or characteristics of their research problem that they identify as being the ones of
interest. Therefore, they begin their coding strategy with a list of codes they have developed before reading the data or before any analysis has taken place. The rationale for this
development is that the “conceptual framework, research questions, and other matters of
research design suggest that certain codes . . . are most likely to appear in the data you collect” (Saldaña, 2021, p. 40).
One device that may be used as part of a deductive coding strategy is some form of
codebook or list of predefined codes that the researcher specifically looks for when reading the transcripts of interviews or field notes of observations. The codebook is “a compilation of the codes, sometimes accompanied with their content descriptions, and a brief
example for reference” (Saldaña, 2021, p. 41). The researcher looks specifically for examples of those codes in the data.
The difference between these two approaches to coding can be summed up in the
following way: “[T]he deductive researcher starts with a preliminary list of codes, and
the inductive researcher ends up with one” (Saldaña, 2021, p. 41). The researcher who has
developed their inductively derived list of preliminary codes can then use this set of preliminary codes to guide their second-cycle coding and thereby working more deductively.
They may even choose to use some form of tentative and evolving “codebook” developed
from their inductively derived codes when doing so.
PUTTING IT INTO PRACTICE
WHAT YOU WILL NEED TO THINK ABOUT IF YOU CHOOSE TO
DEVELOP A CODEBOOK OF SOME SORT
If you do decide to use some sort of codebook, what might this codebook look like and
how will you develop it? There is not just one way to develop a codebook. Codebooks
vary in terms of how much detail they contain for each code, and also how they are
used. They also vary, as we have seen, in terms of when and how the codes are
developed—“[T]he deductive researcher starts with a preliminary list of codes, and
the inductive researcher ends up with one” (Saldaña, 2021, p. 41).
Guest et al. (2006) provide a good description of how they developed a codebook
using what they describe as a standard iterative process. Each code in the codebook had
Chapter 7
•
Analyzing and Interpreting Qualitative Data
165
(1) a “brief definition” to jog the analyst’s memory; (2) a “full definition” that
more fully explains the code; (3) a “when to use” section that gives specific
instances, usually based on the data, in which the code should be applied; (4) a
“when not to use” section that gives instances in which the code might be considered but should not be applied (often because another code would be more
appropriate); and (5) an “example” section of quotes pulled from the data that
are good examples of the code. (Guest et al., 2006, p. 64)
An example of putting (1)–(5) above into practice is provided by Creswell (2016) using
the hypothetical example of a code called Safety used in a codebook. See Table 7.1.
TABLE 7.1 ■
Code
Safety
The Hypothetical Example of a Code Called Safety
Brief
Definition
Full
Definition
When to
Use
When Not
to Use
Sample
Questions
Whether
students felt
safe after
the incident
Safety as
identified
through
possibility of
the incident
occurring
again, safety
in their dorm
rooms or on
campus
When
students
actually
mention
the word
safety or a
synonym
When it
cannot be
reasonably
interpreted
as the
students’
talking about
safety or
their security
“I was scared
for my own
safety.”
“The campus
was no
longer a safe
environment
for me or my
friends.”
Source: Taken from Creswell (2016, p. 197, Table 23.1: A Sample Codebook Organization Based on
Guest, Bunce, and Johnson [2006]). Reproduced with permission.
If you would like to read more about the development of codebooks, useful places
to start are
•
•
the article by Jessica T. DeCuir-Gunby et al. (2011), “Developing and Using a
Codebook for the Analysis of Interview Data: An Example from a Professional
Development Research Project.” The authors discuss how they used
codebooks to analyze data from semistructured interviews including how
they developed the codes from the data (i.e., data-driven codes) and provide a
sample of such data-driven codes, including descriptions and examples.
Sarah Tracy’s (2020) discussion of focusing and creating a codebook provides
a useful overview of developing a formal codebook: “[A] data display that lists
key codes, definitions, and examples that are going to help you guide your
analysis” (p. 221). She includes an excerpt of a codebook (p. 222) that she used
in a study she was part of about “male executives’ viewpoints on gender, work
and life.” (p. 221)9
Choosing a Coding Strategy Congruent With the Theoretical Pillars of
Your Design
As we highlighted previously in Chapters 4, 5, and 6, the theoretical and disciplinary
assumptions that we bring with us to the research design table provide a lens through
which we make choices about our research design. This includes the choices we make
about what analytic strategies will form part of that design. Therefore, given the variation in the disciplinary and theoretical assumptions that underpin qualitative research
approaches, it is not surprising that there are differences between these approaches
166 Research Design
related to if, and if so how, codes and coding are used as part of a specific type of qualitative analytical strategy.
Coding in Grounded Theory
For example, if you are using grounded theory (Charmaz, 2006, 2014; Glaser & Strauss,
1967) as the whole-of-study theoretical and methodological approach underpinning
your research design, then your coding strategy will reflect the understanding of coding
that grounded theory is premised on. This understanding is that coding is part of an
analytic strategy designed to enable you to develop a theory emerging from that strategy
able to address your research problem or provide answers for your research questions.
The theory is grounded in, and arises from, the analysis and interpretation of the data
collected—hence the name grounded theory. Specific types of coding10 and specific
ways of memoing are part of the analytical strategy designed to enable that grounded
theory to emerge.
However, when designing an analytical strategy for a grounded theory study, you
will also need to keep in mind that there are different schools of thought with different emphases about what grounded theory is and is for (see Charmaz, 2014; Glaser &
Strauss, 1967; Strauss & Corbin, 1998). The way you develop your analytic strategies
within grounded theory, including which coding practices you employ (such as constant comparison), or how you use the literature,11 must be congruent with the understandings and emphases of the type of grounded theory that you have chosen to employ
in your study.
For example, if your grounded theory study is informed by constructivist or interpretative inquiry paradigms, then your analytical strategies will reflect the basic belief that
“subjectivity is inseparable from social existence” (Charmaz, 2014, p. 14). You will emphasize developing grounded theory based on interpretative understandings. Rather than
assuming the role as a neutral and objective observer, you will acknowledge and engage
reflexively with the subjectivity you bring to your study, and “examine rather than erase
how [your] . . . preconceptions may shape the analysis” (p. 13). This includes how your
preconceptions may shape the codes that you develop as part of that analysis. You will
actively seek out the social contexts that make up your study and reflect on, and make analytical memos about, that context as part of your data collection and analysis. Thus, the
form that your analytic strategy takes reflects your belief that “we construct our grounded
theories through our past and present involvements and interactions with people, perspectives and research practices” (p. 17).
Therefore, when justifying using some sort of coding as part of your overall analytic
strategy in your research design, it is not enough to simply say that you are “doing,” for
example, grounded theory or “using” a grounded theory approach and assume that this
makes it clear what you are “doing” or “using.” Rather, you will need to explain why your
study is informed by a specific approach to qualitative inquiry, such as grounded theory,
and how you have put that approach into practice. For example, what type of grounded
theory have you chosen to use and why? How is the strategy that you have chosen for
analyzing the qualitative data that you have collected congruent and consistent with that
choice? This includes how and what you code, what you do with those codes, and what
interpretations you make based on them. To be able to do this will require hard work, and
a lot of thinking on your part.
Chapter 7
•
Analyzing and Interpreting Qualitative Data
167
TIP
LEARNING FROM OTHERS
Nagel et al. (2015) provide an excellent account of what they needed to think about
when navigating the different ways that grounded theory is thought about and designed
when they were doing doctoral studies using grounded theory.
They describe such navigation as “weaving through a myriad of paths in a landscape of varied and divergent perspectives of ontological and epistemological philosophies on GT, requiring navigational skills we would not have anticipated at the outset of
our dissertation work” (p. 366).
The discussion that they provide about how they navigated this myriad of paths provides an excellent example of what is often missing from reports of research, namely
what they needed to think about when putting grounded theory into practice, including
the analytic strategy they would employ, and why.
Rounding Off Our Discussion of Coding
The key point that we have been making in this section is that there is variation in coding
strategies, just as there is variation in the overall analytic strategies employed in qualitative inquiry. This variation in coding strategies occurs both within specific qualitative
approaches (e.g., between different types of grounded theory) and between different types
of qualitative approaches.
To be able to employ a coding strategy “well” requires you to know a lot about the idea
of codes and coding themselves in order to be able to think through and make choices
about what employing them “well” in the context of your particular research design
means. This includes ensuring that the choice of using coding as part of your analytical
strategy at all is congruent with the methodological and theoretical traditions that your
research design is premised on.
Equally, employing a coding strategy “well” requires you to think about how you will
design that strategy. How will you put the idea of coding into practice in such a way that it
is congruent with the purpose of your research and the qualitative approach you are using?
For example, in other qualitative approaches you are not developing codes, or employing
coding strategies, to produce emergent grounded theories. Rather, both the type of codes
that you develop, and the strategy that you use when developing them, will reflect the
theoretical and disciplinary traditions of the qualitative research approach that you have
chosen to underpin your research design.
Finally, when designing your coding strategy, you will need to be aware of the critiques of using coding and codes (see tip box The Importance of Thinking About Why to Use
Coding, Not Just How to Do It below) and be able to justify the position that you take up
in relation to them in that coding strategy. It is important to remember that codes or categories or themes arising from the analysis of your data in themselves are not the findings
of the study. The “art of interpretation” (Denzin & Lincoln, 2000, p. xii) that is central to
qualitative analysis is not complete just because categories or themes are developed. There
is more thinking to do.
This thinking is about how all the parts of your analysis fit together, how they relate
to, and what they reveal about the problem, issue, or question that your research has been
168 Research Design
designed to address or illuminate in some way. Such reflexive thinking is central to all
qualitative analytical strategies, not just those employing coding as part of that strategy.
We develop this point in the next section of the chapter.
TIP
THE IMPORTANCE OF THINKING ABOUT WHY TO USE CODING, NOT
JUST HOW TO DO IT
Some scholars view the idea of coding and using coding strategies when analyzing
qualitative data with a great deal of skepticism and suspicion. Coding and coding strategies are viewed as reflecting the “[l]ong reach of logical positivism” (St. Pierre, 2016,
p. 19). This is because codes and coding strategies have become taken for granted
procedures with little or no thought given to what the use of the idea of codes and coding might assume. Assumptions such as when coding data that “words can contain and
close off meaning” in the form of codes, or assumptions that “practices of formalization and systematicity [such as reducing coding to some form of prescribed steps] . . .
guarantee rigor” (both quotes from St. Pierre, 2016, p. 26).
Related to this critique of emphasis on formalization of coding as a procedure,
some qualitative researchers are concerned that “coding routines can produce their
own distractions—for example, struggling with the mechanics of the coding process
rather than being able to think deeply about the data” (Yin, 2016, p. 200). In other words,
coding becomes a mechanistic procedure in itself, rather than part of an overall analytical strategy. The focus is then on following the “right” procedure rather than what
the codes and coding are for. This has the effect of distancing you from your data and
reducing coding to an instrumental technique. The result is unthoughtful and poor coding strategies.
However, as Saldaña (2021) wisely points out, critiques such as those above may be
more a critique of not doing “[c]oding well” (p. 22) than critiques of the idea and concept
of coding itself.
We take the view that coding done well can be an important and central part of the
overall analytical strategy used to analyze and interpret the qualitative data that you
have collected. We both have used forms of coding in our qualitative research studies.
This is because coding is a way that you can demonstrate that your work is rigorous
in that it “has adhered to some set of criteria that provides for systematicity, and for
public inspectability” (Lincoln, 2015, p. 204).
In this way, coding provides a way of showing how your interpretations were arrived
at, including the logic behind the choices you made to arrive at those interpretations.
Consequently, it provides a type of audit trail for readers of your research to follow
(Lincoln, 2015). This audit trail is part of establishing the credibility and authenticity
and therefore the dependability and trustworthiness (Lincoln et al., 2018) of your findings and any claims made on the basis of them.12 “If a report is credible, readers feel
confident in using its data and findings to act and make decisions” (Tracy, 2020, p 275).
THE ART OF INTERPRETATION
Using an iterative analytic strategy, you actually begin making tentative interpretations of
the data from the moment you begin to collect it. For example, you are making tentative
interpretations when you reflect on, analyze, and write a memo about why a person you
are interviewing said what they did about what you asked them about. As you simultaneously collect more data and analyze that data, the level of focus of your interpretation
shifts. For example, you begin to compare and reflect on the initial interpretations that
Chapter 7
•
Analyzing and Interpreting Qualitative Data
169
you have made as you begin to develop categories or themes. You also revisit decisions you
made about codes and categories or themes in the earlier phases of the analysis. The focus,
and purpose, of your interpretation
is not a narrow one, such as interpreting the data in a specific table [or for our
purposes here we could add in a single interview, observation, or text]. Rather,
the goal is to develop a comprehensive interpretation—still encompassing specific
data—whose main patterns and themes will become the basis for understanding
your entire study. (Yin, 2016, pp. 220–221)
Developing such a comprehensive interpretation requires thinking about how the
ongoing analysis or interpretations of the data that you have collected come together into a
higher level of abstraction. To do this requires “taking stock of what is being worked with”
(i.e., the data you have and any previous analytical decisions you may have made about it)
and “a process for making a statement about the topic of inquiry” (i.e., the analytical strategies that you are employing). In other words, developing a comprehensive interpretation
requires “some sort of transformation of what is identified, organized, selected, created,
recognized into a statement about the topic of the inquiry or ‘findings’” (all quotes from
Freeman, 2017, p. 3). It is this transformation which gives rise to the conclusions you make
related to your research problem/questions.
Such reflexive and demanding work comprises the art of interpretation. It is this intellectual work that enables the thick description13 and thick interpretation (Denzin, 1989a)
that qualitative inquiry aims for. Remember, thick description, and the thick interpretation it can give rise to, is not about how much description you have. Rather, thick description is defined by the “kind of intellectual effort” (Geertz, 1973, p. 6) it employs:
[T]o thickly describe social action is actually to begin to interpret it by recording
the circumstances, meanings, intentions, strategies, motivations, and so on that
characterize a particular episode. It is this interpretive characteristic of description
rather than detail per se that makes it thick. (Schwandt, 2015, p. 306)
Such iterative, careful, and interpretive intellectual work enables you to address questions such as
• Can the findings of my research help understand the problem that triggered the
research in the first place? Why or why not?
• How can the findings be used in practice?
• What theoretical concepts can be used to add richness and help explain what I
have found?
• How does what I have found add to or change our understandings of those
concepts or theories?
• What implications do the findings have for what else I, or others, might want or
need to know about related to this problem?
• What ideas might need to be explored in further research?
• Can the findings of this study, including any theoretical constructs developed,
be applied to other situations or instances besides the one studied using what Yin
(2014, 2016) calls a form of analytic generalization ?
170 Research Design
• Does, and if so how, the design and conduct of the research offer new or different
ways of studying theories, concepts, or issues—is it methodologically significant?
(Tracy, 2020)
Thinking about these types of questions is central to the systematic and reflexive work
needed to establish the credibility of your research and the significance of the findings of,
or conclusions reached by, that research.
Ways of Establishing the Credibility of the Interpretations You Make
and Therefore the Rigor and Trustworthiness of Your Research
Establishing the rigor and credibility of your research, including the interpretations you
make and conclusions you reach based on them, is closely linked to the idea of establishing
the trustworthiness (Lincoln & Guba, 1985) of the research.
“What is trustworthiness? The basic issue in relation to trustworthiness is simple:
How can an inquirer persuade his or her audiences (including self ) that the findings of
an inquiry are worth paying attention to, worth taking account of?” (Lincoln & Guba,
1985, p. 290). Or put another way, how can you persuade the audience of your research
that they can trust that your research, and the claims that it makes, are credible and
significant?
One way to do this is to persuade them of the rigor and credibility of the interpretations you have made about your data. When we talk about rigor in qualitative inquiry, we
are taking about “the care and effort taken to make sure that the research is carried out
in an appropriate manner” (Tracy, 2020, p. 271) such as, for example, “spending enough
time in the field to gain trust; practicing appropriate procedures in terms of writing fieldnotes, conducting interviews, and analyzing data; collecting enough data to support significant findings” (Tracy, 2020, p. 271, dot points in original removed)—a point we will
return to in the next section in this chapter.
TIP
SARAH J. TRACY’S EIGHT “BIG TENT” CRITERIA FOR EXCELLENT
QUALITATIVE RESEARCH
Tracy (2010, 2020) identifies what she terms eight “big tent” criteria or end
goals for excellent qualitative research. She reveals that she developed this
model by synthesizing “a number of practices across theoretical traditions and
paradigms into an expansive ‘big tent’ framework for high quality qualitative
research” (2020, p. 268). In this framework, she identifies eight criteria for the
end goal of achieving quality.
The eight big tent criteria for quality are worthy topic, rich rigor, sincerity, credibility, resonance, significant contribution, ethical, and meaningful
coherence. For each of these criteria Tracy identifies various means, practices, and methods through which to achieve those end goals. For example,
means, practices, and methods through which to achieve the end goal of a
worthy topic include that the topic of the research is relevant, timely, significant, and interesting (Tracy, 2020).
Chapter 7
•
Analyzing and Interpreting Qualitative Data
171
Although there are no hard and fast rules for what a comprehensive or “good” interpretation is, Yin (2016) suggests that one way to achieve a credible interpretation is to
consider striving for as many of the following attributes as possible:
1. Completeness (Does your interpretation have a beginning, middle, and
end?)
2. Fairness (Given your interpretative stance, would others with the same
stance arrive at the same interpretation?)
3. Empirical accuracy (Does your interpretation fairly represent your data?)
4. Value-added (Is the interpretation new, or is it mainly a repetition of your
topic’s literature?)
5. Credibility (Independent of its creativity, how would the most esteemed
peers in your field critique or accept your interpretation?) (p. 221)
TIP
TRIANGULATION
An approach that is used in some qualitative research designs to establish credibility
and therefore trustworthiness of any interpretations made is the idea of triangulation.
“[T]riangulation means that you take different perspectives on an issue in your study or
in answering your research questions” (Flick, 2020, p. 187). The idea is that by incorporating more than one data source or theory or method or researcher in a study, fuller,
or richer, or more nuanced descriptions, analyses, and therefore interpretations, of an
issue are possible. “The goal of multiple triangulation is a fully grounded interpretive
research approach. . . . In-depth understanding, not validity, is sought in any interpretive study” (Denzin, 1989b, p. 246).
Note that in the quote above, Denzin makes it clear that the purpose of using triangulation is to provide in-depth understanding, not validity. It is the enhanced in-depth
understanding provided by triangulation that can assist in establishing the credibility
of a qualitative study. This is an important point.
If you do claim that using different data or investigators or theories or methodologies (the four forms of triangulation identified by Denzin, 1989b). you will need to clearly
demonstrate and justify how employing different perspectives (e.g., methods, or theories) can assist in developing in-depth understandings of what you are interested in
knowing more about (e.g., a particular issue) + how these in-depth understandings
enhance the rigor, credibility, and hence the trustworthiness of the interpretations you
make, and on which you base your conclusions.
COLLECTING AND ANALYZING DATA—WHEN
DO YOU KNOW THAT YOU ARE “DONE”?
There are no hard and set rules for when to end the iterative cycles that comprise the process of qualitative data collection and analysis. However, in many (but not all) discussions
of qualitative analysis, you will see the idea of saturation used to signal the point at which
data collection, and analysis of new data, ceases. An example of a typical definition for
172 Research Design
what is meant by saturation is “when, in qualitative data collection, the researcher stops
collecting data because fresh data no longer sparks new insights or reveals new properties”
(Creswell, 2014, p. 248).
You will often see this understanding of the idea of saturation referred to as theoretical
saturation. The idea that underpins theoretical saturation is that when you conduct and
analyze more interviews, observations, or texts, no new “properties, dimensions, conditions, actions/interactions, or consequences” (Strauss & Corbin, 1998, p. 136) related to
your research problem or questions emerge. This does not mean that all participants in
your research are saying or doing the same thing. Rather, it means that what they are saying or doing are more examples of the same properties, dimensions, conditions, actions
or interactions, or consequences that you have identified from the iterative analysis of the
previous data you have collected.
Any conclusions about saturation can only be reached after a period of immersion in,
and careful thinking about, the qualitative data that you have collected. This includes
making sure that you have actively looked for data that does not support the properties,
dimensions, conditions, actions or interactions, or consequences that your analysis and
interpretation of that data has developed. In this way, you attempt “to understand what
. . . [you] have obtained” (Liamputtong, 2009, p. 277) by getting a “sense of the whole”
(Patton, 2002, p. 440, bold in original changed to plain text).
Connecting Analytical Considerations to Decisions About Sample Size
Keeping this discussion about saturation in mind gives us a clue to as how we might
answer a question that inevitably arises when we are designing qualitative research.
This is the question of how many people we will interview, or how many focus groups
we will need to conduct before reaching theoretical saturation. In other words, how
many interviews are sufficient to base our analysis on in order for the interpretations
arising from that analysis to be credible? Or putting it very simply, how do we know
that we have collected and analyzed “enough” data? How do we know when we are
“done”? The same sort of question applies to how many observations we will make, or
how many texts we will analyze. These types of questions are questions about what the
size of our sample will be.
At this point, you may be hoping that we will now provide you with the number of
interviews, observations, and texts that are sufficient to enable “enough” rich enough data
to ensure the trustworthiness of a qualitative study. Or that we can provide you with some
sort of mathematical calculation that you can do to tell you this. However, we are not
able to give you a number or a calculation that you can do to obtain that number. This
is because the inductive and iterative nature of the thinking that underpins a qualitative
research design allows for aspects of that design to emerge as the research progresses; for
example, to follow up inconsistencies in the data collected that require further elucidation,
or to add questions to an interview guide as a result of (possibly unanticipated) responses
that you have received and now need to know more about.
Such an emergent design mitigates against the idea of determining or calculating precisely how many interviews are “enough” to include in your research design at the outset
of the research. Considerations of how much and what type of data to collect do not stand
alone from the iterative analytical process of qualitative data. It is this analytic process
that guides how much data you will need or whether you will still need to collect more in
order for your interpretations of that data to be credible, and your analytic strategy to be
Chapter 7
•
Analyzing and Interpreting Qualitative Data
173
trustworthy. This is why while “[an] initial approximation of sample size is necessary for
planning, . . . the adequacy of the final sample size must be evaluated continuously during
the research process” (Malterud et al., 2016, p. 1759).
Activity
Navigating the Requirement to Give a Number When It Is Not
Possible to Do So
Research funding bodies, dissertation committees, and ethics committees often require
a number, or at least an indicative number, to be given in advance for the number of
interviews to be conducted, or observations to be made, when they are evaluating the
credibility and feasibility of your research design (see Cheek 2000a, 2018b).
Researchers who are new to qualitative inquiry often have problems with this question. They worry that if they give a number, will they have “enough” data or a “big
enough” sample? What happens if they want to change the number? Given all this
uncertainty, how can this requirement from these types of committees be navigated?
Baker and Edwards developed a National Centre for Research Methods (NCRM)
Review paper (2012) to address these types of questions. They collected and reviewed
responses from renowned social scientists and early career researchers whom they asked
how many participants is enough in a qualitative study. The answer they received
from most of them was “it depends” (Baker & Edwards, 2012, p. 2). Moreover,
In considering what “it depends upon” however, the responses offer guidance on the epistemological, methodological and practical issues to take
into account when conducting research projects. This includes advice about
assessing research aims and objectives, validity within epistemic communities and available time and resources. (p. 2)
Baker and Edwards go on to provide an excellent overview and summary of the 19
responses in their introduction to the Review paper.
We recommend that you take a close look at both this introduction and the actual
responses themselves. Think carefully about them, and then take up a position that
you are able to justify about the way you are going to answer the question of how
many [insert the method(s) you are using, e.g., interviews] are enough when developing your research design.
Principles to Guide Sample Size Considerations in Your Qualitative
Research Design
There are some principles that can guide us when thinking about sample size when designing qualitative research. A key one is that “[t]he validity, meaningfulness, and insights
generated from qualitative inquiry have more to do with the information richness of the
cases selected and the observational/analytical capabilities of the researcher than with
sample size” (Patton, 2002, p. 245). This means that while sample size may be numerically
smaller in qualitative inquiry than in quantitative inquiry, it may not be smaller in terms
of the amount of information that is obtained from that smaller in size sample.14
174 Research Design
Just including more participants, or making more observations, in your qualitative
study does not necessarily make that study more credible or more trustworthy. Indeed, the
opposite can be true. If a sample is too large, then it will not permit “the deep, case-oriented
analysis that is the raison d’ être of qualitative inquiry” (Boddy, 2016, p. 429, drawing on
Sandelowski, 1995). A numerically smaller sample in a qualitative study can provide much
more in-depth and rich data about each information-rich person or site in the study than
a lot of thin and shallow description or analysis or interpretation of each person or site in a
numerically larger sample.
Trying to determine an exact number, or insisting that one is given, at the outset of
the research, or working explicitly or implicitly on the assumption that having more participants or more sites in your study necessarily means “better” or “more in-depth” data,
is to apply positivist assumptions to qualitative research. This violates the methodological understandings underpinning the vast majority of approaches in qualitative research,
which are non-positivistic in orientation.
Therefore, it is not the number or length of interviews or observations that matter
when making decisions about when you are “done” with collecting your data. Rather, it
is how information rich the participants and sites in your sample are, and how much of
that information you were able to glean from those participants and those sites. The more
information rich each person or site in your sample is, and the more information rich each
interview, observation, or text that you base your analysis on is, then the lower the number
of sites, participants, and interviews, observations, or texts that are likely to be needed to
reach, for example, some form of saturation.
Therefore, questions about when and how you know if you have collected and analyzed “enough” qualitative data can be answered by asking yourself other types of questions that take the emphasis off the quantity of data or amount of analysis that you have
done and put it on what you are able to use that data or analysis for. For example, do you
have enough rich and in-depth information to drive your analytical strategy? Is the “information power” (Malterud et al., 2016, p. 1753) of your purposeful sample and the data
collected from that sample sufficient to enable you to make and justify credible interpretations of the data you have collected and analyzed?
You can only know the answer to these questions when, after a long period of reflexively working with the analysis of the data you have collected, you decide that you are
able to produce credible and trustworthy interpretations of what it is that your research
is focused on and that can address your research problem or questions. This is part of
ensuring the credibility of your research design and the conclusions you reach from the
qualitative research that you conduct using that design. Remember, “[a] systematic and
careful description of the whole process of interaction with the reality under study is what
indicates good quality when using a qualitative method” (Stenbacka, 2001, p. 555). It also
assists you to know when enough is enough.
TIP
AN EXCEPTION
We have argued above that, in qualitative research, it is the amount and richness of
relevant information gained from each person or site in your sample that is key, not the
sample size per se. One exception to this is when qualitative data is collected within a
post-positivist inquiry paradigm. For example, if the aim of collecting the qualitative
Chapter 7
•
Analyzing and Interpreting Qualitative Data
175
data is to inform the development of items for a questionnaire for a quantitative survey
study, then the sample of people or sites from which that qualitative data is obtained
will need to reflect that aim. Therefore, it would be appropriate to obtain qualitative
data from a representative sample of the intended population from which the survey
participants will be drawn in order to be able to prioritize what items or questions will
be included in the questionnaire. In this type of study design, the researcher applies
post-positivist derived thinking to “a qualitative element of research to set the parameters for a further positivist quantification. . . . [U]nder this approach, a criticism of
sample size because of smallness may well be justified” (Boddy, 2016, p. 429).
This is because the principle of representativeness underpins the generalization
of the finding from analyzing sample data to the larger population in question (see
Chapter 8 of this book). Therefore, when
qualitative research is being undertaken under a positivist approach . . . for
example with a view to developing a quantitative measurement instrument
such as a questionnaire . . . [it] would necessitate sampling a greater number of
respondents, and, in general, at least one representative of each segment of the
population under consideration in the wider research should be sampled in the
qualitative research (Boddy, 2016, p. 430).
CONCLUSIONS
Designing qualitative research involves thinking about, and making choices related to,
how you will obtain the type of qualitative data you need to be able to address your
research problem or questions. It also involves thinking about what you will do with that
data when you have collected it. How will you analyze that data, and why? To do so will
require you to reflexively think about, and work through, questions such as these:
• How will you organize the qualitative data you collect?
• What analytic strategy will you choose for working with, analyzing, and interpreting
that qualitative data?
• What form will aspects of that analytic strategy take?
• How will you convince readers of your research of the credibility of your analysis
and interpretation of that data, and therefore the conclusions you reach about the
questions your research is being designed to address?
These are crucial questions to think through, and make decisions about, when designing
qualitative research. This is because “[t]he strengths of qualitative data rest centrally on
the competence with which their analysis is carried out” (Miles et al., 2014, p. 12).
To answer each of the above questions you will need to add another question: Why did you
answer each of those questions in the way that you did? For example, why will you organize
your data in that way? Why will your analytic strategy take that form and not another?
Justifying the way that you analyze the qualitative data you collect is an important part of
establishing the trustworthiness and credibility of your qualitative research design, as well
as the trustworthiness and credibility of the conclusions you reach as a result of putting that
design into practice. This includes demonstrating that the way you choose to organize the
176 Research Design
qualitative data you collect, and the form that your analytic strategy takes, is congruent with
the theoretical and methodological assumptions underpinning your research.
For example, when designing your research, you will need to think about, and make
choices about, whether coding will play a part in the way you will analyze and interpret
your qualitative data. Why or why not? If you decide that the strategy of using coding
will be a part of your overall analytical strategy, then you will need to decide and justify
how you will put your coding strategy into practice. What type of coding will you choose
to use and why?
A set of transcribed interviews, a description of them, or a set of codes and categories
or themes derived from a coding strategy employed on those interviews, or a description of each of those categories or themes, in themselves are not the findings of your
research. This is because while such description is part of the analysis and interpretation
of your data, on its own it is not enough to make sense of the data. Even if participants in
your research have provided you with in-depth and rich information, unless you think
through, analyze, and interpret that information carefully, its richness and depth will be
lost. Thin and shallow analysis, predominantly made up of lengthy verbatim quotes of
what was said, or descriptions of what was done, with little, if any, analytical or interpretive depth, will be the result.
What are the findings of your research is the intellectual work, or art of interpretation,
related to those codes, categories, or themes. Therefore, throughout the discussion in this
chapter, we have emphasized the interconnections between data collection, analysis, and
interpretation. This interconnectedness is crucial because “inquiry conclusions, interpretations, or warranted assertions arise from the mind of the inquirer, not directly from the
output of a statistical or thematic analysis” (Greene, 2007, p. 142).
Regardless of what form your analytic strategy takes, it is important to keep in mind an
important point that many scholars of qualitative research have highlighted when discussing the analysis of the qualitative data. This is that the only way to really understand
how analysis works is by doing it (Freeman, 2017). A textbook can provide you with guiding (not prescribing) principles for putting a qualitative analytical strategy into practice.
However, at some point you are going to have to take the plunge and actually put the analytic strategy that you have developed into practice. “[R]eading theory—just like reading
cookbooks or studying sheet music—will only get you so far. . . . [A]t some point you need
to stop reading and start practicing” (Tracy, 2020, pp. 220–211). The discussion in this
chapter has highlighted what you will need to think about when taking that plunge.
In the next chapter, we continue our discussion of how to put the methodological thinking associated with, and underpinning, methods into practice when designing research.
Our focus shifts to the series of decisions you will need to make related to analyzing data
when designing research that employs quantitative research approaches.
SUMMARY OF KEY POINTS
• Analyzing qualitative data is an iterative and reflexive strategy.
• Design considerations related to analyzing and interpreting qualitative data cannot
be reduced to, or captured by, a fixed linear series of prescribed steps or phases.
Chapter 7
•
Analyzing and Interpreting Qualitative Data
177
• Therefore, ways of analyzing and interpreting qualitative data are strategies, not
procedures.
• When using qualitative research approaches, data collection and data analysis happen
simultaneously.
• Therefore, analysis “begins” as you are in the process of collecting your first
qualitative data.
• There is variation in the overall analytic strategies employed in qualitative inquiry,
both within specific qualitative approaches and between different qualitative
approaches.
• All aspects of the analytical strategy you choose in your research design must be
congruent with the methodological and theoretical traditions that the design is
premised on.
• For example, the decision of whether to include coding in your analytical strategy,
and if so, how you put the idea of coding into practice, must be congruent with the
purpose of your research and the methodological and theoretical traditions of the
qualitative approach you are using.
• Analysis can only end after a period of immersion in, and careful thinking about and
interpretation of, the qualitative data that you have collected. This includes making
sure that you have actively looked for data that does not support the properties,
dimensions, conditions, actions or interactions, or consequences that your analysis
and interpretation of that data has developed.
• Developing a comprehensive interpretation requires thinking about how the ongoing
analysis and interpretations of the data that you have collected come together into a
higher level of abstraction.
• This reflexive work comprises the art of interpretation.
• It is not the number of interviews/observations or amount of data that matter when
determining sample size and when you are “done” with collecting your data. What
does matter is how information rich the participants and/or sites in your study are,
and how rich the information is that you obtained from them
• You are done collecting your data when after a long period of working with your data
you decide that you are able to produce credible and trustworthy interpretations of
what it is that will enable you to be able to address your research questions.
• Becoming a good analyst of qualitative data takes practice and a lot of hard work.
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
analysis of qualitative data
audit trail
category
codebook
coding
computer-assisted qualitative data analysis
software (CAQDAS)
data condensation
grounded theory
memos
178 Research Design
rigor in qualitative research
sample size
transcription
triangulation
trustworthiness
SUPPLEMENTAL ACTIVITIES
1.
Many people make assumptions about how research should be done, including
how data collected in that research should be analyzed. Often this is based on
the different methodological and disciplinary traditions that they are part of. To
illustrate this point, try the following activity:
a.
b.
c.
2.
Find a fellow student or colleague who is studying or has studied
qualitative research. Explain to them the type of analytical strategy you are
intending to use in your study and why.
Take note of the questions that they ask you and reflect on why they
may have asked them and what implications this might have for the way
you design your research.
Find a student or colleague who is studying or has studied quantitative
research. Explain to them the type of analytical strategy you are intending
to use in your study and why.
Take note of the questions that they ask you and reflect on why they
may have asked them and what implications this might have for the way
you design your research.
Compare the type of questions asked in (a) and (b) above. If the questions
differed, work out why this might be so.
Read the article by Bailey et al. (Bailey, K. A., Dagenais, M., & Gammage, K. L.
(2021). Is a picture worth a thousand words? Using photo-elicitation to study
body image in middle-to-older age women with and without multiple sclerosis.
Qualitative Health Research, 31(8), 1542–1554. https://doi.org/10.1177/10497323
211014830.). This provides an excellent example of putting an iterative qualitative
analytical strategy into practice.
a.
b.
c.
While reading it, write memos about what might convince you of the
credibility and trustworthiness of the research being reported.
Share your memos with another person who has done the same exercise.
When doing so, discuss similarities in the memos you wrote as well as
differences.
Think about how you might be able to use these memos and this discussion
when designing the analytical strategies in your own research design in
order to convince readers of the trustworthiness of what you are reporting.
FURTHER READINGS
Saldaña, J. (2021) The coding manual for qualitative researchers (4th ed.). SAGE.
Tracy, S. J. (2020) Qualitative research methods: Collecting evidence, crafting analysis, communicating impact (2nd ed., Chap. 8–11). John Wiley.
Yin, R. K. (2016) Qualitative research from start to finish (2nd ed.). Guilford Press.
Chapter 7
•
Analyzing and Interpreting Qualitative Data
179
NOTES
1. We are using the term generally. There are more specific uses of the term, such as constant comparison in grounded theory. If you are designing a grounded theory study, you
will need to draw on the specific understanding of constant comparison associated with
a grounded theory research design.
2. See Chapter 6 for a discussion of the use of probes in interviews.
3. See Chapter 2, section Putting Confidentiality and Anonymity Into Practice.
4. Quoted text is in bold in Patton (2002).
5. See Chapter 6 for an extended discussion of the idea of lines of inquiry.
6. For accessible discussions of the uses of computer software in qualitative studies,
see Miles et al. (2014, pp. 46–50). Yin (2016, pp. 184–213) also has a good discussion of
what you will need to think about when using computer software as part of your analytic
strategy in qualitative research, as does Hesse-Biber (2017, pp. 329–333) and Merriam
and Tisdell (2016, pp. 221–226).
7. See Yin (2016), pages 199–201, where he discusses “[d]isassembling data without coding them” (p. 199).
8. You will also see this initial stage of coding referred to as open coding or Level 1 and 2
coding (see Yin, 2016).
9. The full reference for the study is Tracy, S. J., & Rivera, K. D. (2010). Endorsing equity
and applauding stay-at-home moms: How male voices on work-life reveal aversive sexism and flickers of transformation. Management Communication Quarterly, 24(1), 3–43.
doi: 10.1177/0893318909352248
10. See Saldaña (2021, p. 72) for a discussion of these types of coding, including where you
might go to read further about these types of coding and the memos arising from them.
11. One of the original tenets of grounded theory (see Glaser & Strauss, 1967) was that
researchers should delay reviewing the literature until after they had collected and analyzed their data. Glaser has later maintained this view:
Grounded theory’s very strong dicta are a) do not do a literature review in the
substantive area and related areas where the research is to be done, and b)
when the grounded theory is nearly completed during sorting and writing up, then
the literature search in the substantive area can be accomplished and woven into
the theory as more data for constant comparison. (Glaser, 1998, p. 67)
This was highly controversial and not all grounded theorists agreed with this. For example, Strauss and Corbin (1998) argue that the literature can be used to help formulate
questions acting “as a stepping off point” (p. 51) during interviews and can help understand “[w]hat is going on” (p. 51) during data analysis. For a thorough discussion of the
literature review debate among grounded theorists, see Giles et al. (2013).
12. You will recall from Chapter 4 that in a constructivist inquiry paradigm, positivist
derived criteria of validity “are replaced by such terms as trustworthiness and authenticity” (Denzin & Lincoln, 2018c, p. 98). The trustworthiness and authenticity of the reconstructed accounts of aspects of the social world that form the findings is what makes the
research, and the conclusions reached from research drawing on the set of basic beliefs
underpinning a constructivist inquiry paradigm, credible.
13. See Chapter 5 of this book.
14. Even in quantitative approaches such as quantitative surveys (see Chapters 8 and 9 of
this book), which are driven by probability-based considerations, the sampling strategy
and the overall conduct of the survey assessed in light of the goal of the study is just as
important as sample size. Increasing the sample size will in most cases strengthen the
validity of the findings of a quantitative study, but a large sample size is in itself not sufficient to ensure that the findings are considered valid.
8
FOUNDATIONAL DESIGN
ISSUES WHEN USING
QUANTITATIVE METHODS
PURPOSES AND GOALS OF THE CHAPTER
This chapter is the first of two chapters that focus on the series of research design decisions you
will need to make when putting quantitative research approaches into practice. In this chapter, our focus is the decisions you will need to make before you will be able to begin collecting
numerical data when using some form of quantitative approach as part of a research design.
Answering your research questions will involve you reaching some sort of credible
conclusions about those questions based on the analysis of the numerical data you have
collected. In quantitative research, statistical procedures are used to undertake that analysis and base judgments about the credibility of those conclusions on.
However, statistical procedures differ from one another in terms of what sort of conclusions they enable us to make. Some statistical analyses enable us to describe the frequency of
something we want to know more about related to a group of people that is of interest—for
example, how many people in that group eat fast food more than twice a week. Other types
of statistical analyses enable us to draw conclusions about the relationship between one or
more of the things we are interested in about the group of people—for example, is there a
link between age of people in that group and the frequency of the consumption of fast food?
Therefore, when designing your research, you will need to incorporate into that design
a type of statistical procedure(s) that will enable you to draw the type of conclusions you
will need to make to answer your research questions.
Just as you need to choose a procedure for statistically analyzing the research data that
will enable the sort of conclusion that can provide a credible answer to your research question, you will also need to make sure, before you collect any data, that your design includes
a plan for collecting enough of the “right” type of numerical data to base that statistical
analysis on. Otherwise, that procedure for statistically analyzing the research data will not
enable you to arrive at credible research findings.
However, even having enough data of the right type is not necessarily enough for you
to be able to arrive at credible research findings by using that data. This is because who will
provide the numerical data, that is, who will be the participants1 in your study, will enable
or impede reaching such credible research findings.
Therefore, there needs to be a clear and explicit link in your quantitative research
design between the numerical data you propose to collect (type and amount), the type of
statistical analyses that you propose to do, and the research questions that you are asking.
Making this link explicit lies at the heart of establishing a statistically reasonable quantitative research design. Consequently, that link will be part of establishing the credibility of
your research findings.
In addition, there needs to be an equally clear and explicit link in your quantitative
research design between the group of people that you want the answers to your research
181
182 Research Design
questions to apply to, and the (often smaller) group of people from which the numerical
data will be collected.
How to make these clear and explicit links, and what effect that will have on the way
you design your quantitative research, is what this chapter is all about. This sets the stage
for the next chapter, which is about the decisions you will have to make in relation to actually collecting this numerical data from that group of people.
Note: Discussing in-depth details from the field of statistics is outside the scope of
this book. Therefore, you will not find detailed discussions of any specific procedure for
statistically analyzing research data in this chapter. What you will find are discussions
about the thinking needed to be able to incorporate and use statistics well in your research
design. You will also find tip boxes containing advice about statistical concepts you may
need to learn more about, as well as providing tips about when it may be wise to seek help
from a statistician when developing your design.
The goals of the chapter are to
• Establish that although statistics are part of any research using a quantitative
approach, statistics are not what quantitative approaches are. Quantitative
approaches are about much more than the statistics involved in them.
• Highlight that using statistics well in a quantitatively driven research design
requires statistical procedures and probability-based principles to be incorporated
into that design, rather than seen as stand-alone parts of it.
• Provide you with enough knowledge related to data analysis, and selecting
participants, to be able to use statistics and probability-based principles well in
your quantitatively driven research design.
• Demonstrate that to be able to incorporate statistical procedures and
probability-based principles into a research design, you need adequate, but not
necessarily exhaustive, knowledge about the logics and requirements of such
procedures and principles.
• Highlight that there needs to be a clear and explicit link in your quantitative
research design between the group of people to whom you want the answers to
your research questions to apply, and the (often smaller) group of people from
which the numerical data will be collected.
• Emphasize that there also needs to be a clear and explicit link in your quantitative
research design between the numerical data you propose to collect (type and
amount), the type of statistical analyses that you propose to perform on those
data, and the research questions that you are asking.
• Provide direction and advice about what statistical concepts you will need to
know more about when designing quantitative research.
WHAT YOU NEED TO THINK ABOUT IN ORDER TO
DESIGN CREDIBLE QUANTITATIVE RESEARCH
The departure point for the discussion in this chapter is that you already have a clearly
defined research area, problem, or question and you have thought about what type of
information you will need to address the problem that your research is being designed to
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
183
address. Your thinking has led you to a point where you have decided that your research
question or problem is about identifying, and quantitatively describing tendencies in, and
characteristics of, a given (large) group of people in relation to their behaviors, preferences,
and experiences. This has led you to conclude that the type of information needed for you
to be able to do this can be provided by a quantitatively driven research approach.2
The purpose of all research is to arrive at credible answers to research questions by way
of applying an accepted method for collecting and analyzing research data. In quantitative
research, such accepted methods include the use of probability-based procedures to decide
who will provide the data for the study as well as procedures for statistically analyzing that
data. Moreover, these accepted methods include using some sort of device or tool to enable
the collection of relevant and credible data for the study, as well as accepted ways of putting that device or tool into practice to actually collect that data.
In other words, arriving at credible research findings using some form of quantitative
approach requires using statistical and probability-based procedures in such a way that they
enable answers to the research questions that are statistically reasonable. By statistically
reasonable, we mean that the statistical procedures incorporated into that design are used
in a way that is in keeping with the laws of statistics. When the research design is statistically reasonable, it can be justified and defended (within the rules of statistics). A statistically justifiable and defensible research design is necessary for the research to be credible.
However, arriving at credible research findings using some form of quantitative
approach also requires designing a way of obtaining (numerical) data that are relevant
to the issue in question, as well as a way of collecting those relevant data such that the
data obtained represent credible pieces of relevant information about the issue in question.
By credible, we mean that the data represent pieces of information that you have reason
to believe correspond well to the real-life behaviors, preferences, and experiences of the
people from whom those data have been collected
TIP
WHY STATISTICS ARE A MEANS, NOT AN END IN THEMSELVES
WHEN DESIGNING YOUR RESEARCH
The thinking you need to do in order to design quantitative research that is statistically
reasonable, includes thinking about statistical procedures at two levels:
1. the role of procedures derived from laws of statistics and probability when aiming
to learn something about the group of people that is the subject of the research, and
2. how different types of research questions require different types of statistical procedures and data to answer them.
The way we use the term statistically reasonable in this chapter is close to what textbooks about statistics and quantitative research methods would call statistical validity.
Conclusions drawn from a procedure for statistically analyzing the research data are
statistically valid if the analysis procedure is appropriate for drawing that specific type
of conclusion, and if the analysis is performed on a set of data that meets the requirements of that analysis procedure.
The idea of statistical reasonableness highlights that when designing quantitative
research, a researcher is not primarily interested in statistics per se. Rather they are
interested in what kind of answers to their research question(s) these statistical procedures may enable. Consequently, in a quantitative research design, the statistics are
a means, not an end in themselves.
184 Research Design
Key Questions to Ask Yourself When Designing Quantitative Research
In the discussion so far in this chapter, we have highlighted that when designing quantitative research, what you need to think through, and then find out more about, before you
can develop and put any quantitative approach into action includes asking yourself questions such as
• What exactly do I want to be able say something about, deduce, confirm, or
demonstrate in relation to my research questions?
• To whom do I want the answers to my research questions to apply?
• With this in mind, what type of statistical procedures will I need to include in my
research design to enable me to say or demonstrate this (i.e., answer my research
questions) on the basis of the analysis of my numerical data?
• What type of numerical data will I need to enable, and ensure, the credibility of
those specific statistical procedures?
• Once I know this, how will I actually go about collecting that type of numerical
data? For example, who will I collect the numerical data from? How will I decide
that?
• What type of measurement instrument will I use to do so?
Your research design arises from the decisions you systematically make about each of
the questions above.
Why You Need to Ask Yourself All These Key Questions
Simultaneously
Note that none of the above key questions can be thought about in isolation. If your research
questions are not well enough developed, then you will not be able to make decisions about
any of the other questions. If you don’t know what you need the analyses of your data to
be able to show you in order to be able to answer your research questions, then it will be
impossible for you to work out what procedures for statistically analyzing the research data
are reasonable to use in your study design. If you don’t know what you want your numerical
data for (i.e., enabling conducting the procedures for statistically analyzing the research
data that will underpin the answers to your research questions arising from the analyses of
your data), then it will be impossible to work out what form the numerical data you collect
will need to take. If you don’t know to whom you want the answers to your research questions to apply, it will be impossible for you to work out who to collect the data from.
However, even if you do work out what are appropriate statistical procedures to incorporate into your research design, in itself that is not enough to ensure the credibility of
your research. This is because no statistical analytical procedure, no matter how well it is
done, can make up for shortcomings in the credibility of the data on which such statistical
analyses are performed and conclusions are based. If the numerical data are not credible,
then neither is any analysis based on that data.
Therefore, the credibility of your quantitative research design and findings lies as
much in the way that the numerical data has been collected and the form that that data
takes, as it does in the statistical procedures used to analyze that data and as it does in the
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
185
probability-based principles used to decide who will provide those data. This is why you
need to ask yourself all these key questions simultaneously. The discussion in this chapter
and the following one will unpack the implications that asking yourself all these key questions simultaneously has when you are designing your research.
In the next section we focus on how procedures derived from laws of statistics and
probability can help us learn something we want to know about the group of people that is
the subject of the research.
TIP
HOW TO NAVIGATE OUR DISCUSSION ABOUT DESIGNING
CREDIBLE QUANTITATIVE RESEARCH IN CHAPTERS 8 AND 9
In this chapter (Chapter 8), the spotlight is on what you need to think, and make decisions, about to develop a quantitative research design that is statistically reasonable.
We will begin by looking at what you will need to think about when making decisions
about selecting your study population, or a subsample of it. We demonstrate that your
decisions about who makes up that population, or sample, must be statistically reasonable for your research design to enable credible answers to your research questions.
We then move on to look at differences in the types of statistical procedures you
might choose to use to analyze the data collected from that population or sample. Will
the procedures you choose to incorporate into your design enable answers to your
research questions that are statistically reasonable?
Finally, we look at different types of numerical data and what type of data you will
need to collect in order to be able to perform the statistical procedures required for
answering your research questions. We point out that simply having numerical data
(even lots of it) is not enough to ensure that your research is statistically reasonable.
The discussion in this chapter about what makes a research design statistically
reasonable sets the stage for the next chapter (Chapter 9). In that chapter, we shift the
focus of the discussion to a more operational level: When doing so we take a closer
look at the decisions you will need to make in relation to designing a way to go about
collecting relevant and credible numerical data.
WHERE TO BEGIN? DECIDING WHO YOU WILL
COLLECT NUMERICAL DATA FROM AND WHY
Somewhere in the process of designing quantitative research to enable you to answer your
research question(s) you must make it very clear who or what comprises the group that the
answers to those research questions will apply to and why. This group is called the study
population. In other words, the study population for your research design is the group that
you want to be able to say something at the end of your research. Therefore, they are the
group of people or objects you will need to obtain data from or about.
In social sciences, this study population is generally a (large) group of people, where
every member of that group meets a set of inclusion criteria relevant to the research
question(s), and where no one meeting those inclusion criteria is excluded from that group.
For example, a researcher in education may want to address the question, Does class
group size impact the learning outcomes in mathematics among first-year university students in California? In this case, the study population meeting your three inclusion criteria would be (1) all first-year students in California, (2) who are enrolled in a study
program, and (3) that includes classes in mathematics.
186 Research Design
TIP
OUR USE OF “WHO” AND “PEOPLE” WHEN WRITING ABOUT STUDY
POPULATIONS
When writing about study populations generally, we will refer to who comprises that
population, or the people in that population.
However, study populations do not always include people. For example, you might
want to know the density of algae (measured by the number of cells of algae 3 per cubic
meter of water) in Lake Michigan, or you might want to know about objects such as
surveillance cameras—how many there are and where they are placed in banks in
the United States. In these cases, your study population is made up of algae in Lake
Michigan and surveillance cameras in banks in the United States.
We talk about “who” to avoid having to write who/what each time and also people/
objects/things and so on each time we mention study populations.
If you choose to collect data from every member of the study population, the answers
to your research questions arising from analyzing those data will of course apply to the
study population as a whole. However, it is not always possible to include every member
of the study population in your research study, for example, if you are a single researcher
with limited resources (time and money) or the study population is very large. In this case,
your research design will include a strategy for selecting a subset, or sample, drawn from the
study population.
The study sample comprises the members of the study population who actually provide the information that comprises the data of the study. You can then use the information gained about a characteristic of interest from that sample to make an estimate for the
entire study population about that characteristic. In the box below we provide an example
of how this works.
PUTTING IT INTO PRACTICE
HOW ESTIMATION PROVIDES THE LINK BETWEEN SAMPLE
DATA AND POPULATION CHARACTERISTICS
Suppose you want to know how many people per square kilometer in Canada have red
hair. You have two options for finding this out:
1. Count the number of people in Canada that do have red hair (we call that number
x), and then divide by the total number of square kilometers making up Canada
(we call that number y). The accurate number of red-haired people per square
kilometer in Canada is then z = x ⁄ y.
2. Count the number of people in selected square kilometers of Canada with red hair
(we call that number ^x), and calculate the total number of square kilometers in
that sample (we call that number ^y). Then ˆ
z = ˆx ⁄ˆy is your estimate—or best guess—
for the unknown number z.
In other words, if you are unable to find the accurate numerical value of a population characteristic (how many people per square kilometer in Canada have red hair),
you can compute the numerical value of the corresponding characteristic of a sample
drawn from that population (number of people in selected square kilometers of Canada
with red hair). That computed numerical value of the sample characteristic is then the
estimate for the numerical value of the corresponding population characteristic.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
187
What You Will Need to Think About When Using a Sample in Your
Research Design
Whenever a sample is used as part of research using some form of quantitative approach,
keep in mind that the sample is not of interest in itself. Rather, the sample is “a tool to find
out about the population” (Henry, 1990, p. 11). Therefore, the key issue to think about when
using a sample is that the sample must enable learning something about the study population.
Given that the purpose of the research is to learn something about the study population—not just about the sample—you need to make sure that the sample of the study
enables you to make statistically reasonable claims about the study population. The degree
to which your findings about the sample hold for other persons in your study population is
called the external validity of those findings.
Arriving at conclusions about a study population based on data from a sample drawn
from that population is called generalizing in quantitative research.4 Such generalization
is enabled by establishing a relationship between what is going on in a sample and what is
going on in the population from which that sample is drawn.
The nature and strength of that relationship (i.e., how confident you can be in the
claims you make about the study population by way of generalizing what you have found
to be going on in the sample) is decided by laws of probability. Therefore, probability-based
principles are what inform your decisions about the size of the sample, as well as selecting
which members of the study population to include in the sample. These decisions make up
your sampling strategy.
The key characteristic of a sample that enables statistically reasonable generalizations of
the findings about the sample to the study population from which the sample is drawn is that
the sample adequately mirrors, or represents, the study population. A sample that represents the
study population well (often referred to as a representative sample) replicates the population
according to specified criteria, such as the distribution of age, gender, and level of education in
the population, as well as other criteria relevant to the research question(s) the study is designed
to address. Because of this, representative samples can enable answers to your research question
which are very close to those that would have been obtained had you not used a sample at all.
TIP
STOP AND THINK BEFORE YOU RELY ON INFORMATION SEEMINGLY
BASED ON STATISTICS
The need for thinking about sampling is not limited to the situation of designing
research—it is equally important when you read the work of someone else, and also
in your everyday life.
For example, there is no doubt you have at some point come across online polls.
For example, in an online newspaper reporting about an incident involving the local
police, readers may be asked to respond to the polling item:
Select the option that best represents your opinion about the statement
“My local police treat members of the general public with respect.”
1
2
3
4
5
6
Strongly
disagree
Moderately
disagree
Mildly
disagree
Mildly
agree
Moderately
agree
Strongly
agree
(Continued)
188 Research Design
(Continued)
As readers tick the box that best represents their opinion about the statement, the
result of the poll is constantly updated, for example as in the figure below.
If the newspaper on the basis of such a polling result claims that “Close to 80%
of readers feel their local police does not treat members of the general public with
respect,” you should stop and think about whether that claim is statistically reasonable, or just what is commonly called clickbait.
When thinking further about it, you will realize that you do not know if the sample
of readers who have chosen to participate in the poll is representative of the group of
readers of the newspaper (i.e., the total possible study population) as a whole.
Consequently, you have no way of assessing whether the claim applies to a wider
population of readers or if it only applies to the group of readers who have participated
in the poll.
Remember, the key issue to think about when using a sample is that the sample
must enable learning something about the study population. In this case, the only thing
that we can be sure of is that we have learned something about the sample, but not
necessarily about the study population.
So therefore, when reading any form of claim based on statistics, remember that it
is not the statistics themselves that matter, rather whether the statistics can be used
to make that claim.
How Do I Design a Sampling Strategy That Enables a Representative
Sample?
A random selection process is a probability-based process in which every member of the
study population has an equal probability of being selected. For example, when a sample
comprising 20 people from the population of 200 female + 200 male employees of a specific company is selected by drawing the names of 20 employees out of a hat containing
the names of all 400 employees, the selection process is random.
When employing probability sampling, that is, when using a random process to select
who will be in the sample, you can, purely by chance, end up with a sample comprising
names of only male employees, for example, from the population of 400 employees. This
would cause certain groups (males) to be overrepresented in the sample (i.e., only men are
represented in the sample, yet the total number of employees comprise 50% male employees and 50% female employees). Such overrepresentation is the opposite of what you are
trying to achieve, which is a representative sample.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
189
However, by increasing the number of random selections from the study population
(i.e., increasing the sample size), the chances of, purely by chance, ending up with a sample
in which certain groups of the study population are overrepresented, decreases.5 Therefore,
for large enough samples, “probability sampling will ensure approximate representation of
the population” (Nardi, 2018, p. 118).
Therefore, large probability samples enable generalizing, in a way that is statistically
reasonable. This makes it possible to generalize what you learn from statistically analyzing
the data collected from a sample to the study population from which that sample is drawn.
TIP
WHAT MAKES A SAMPLE “LARGE ENOUGH” TO ENSURE
APPROXIMATE REPRESENTATION OF THE POPULATION?
Unfortunately, there is not a magic number we can give you. Deciding on sample size
for a particular research study is usually rather complicated and depends on a number
of interrelated considerations. These include
•
the number of variables 6 included in your study—the more variables included, the
larger the sample needed to capture the variability in all of them;
•
the variability of the issue or phenomenon you are studying within the study population —a larger sample is needed to capture a large variation in a phenomenon;7
•
the type of statistical analysis you need to perform on your data.
In light of this, Gorard offers a piece of general advice when thinking about what
makes your sample large enough. This is to “have as large a sample as possible”
(Gorard, 2003, p. 60). Deciding on what is “possible” for you includes taking into account
the resources and time you have available.
Are There Other Sampling Strategies I Can Consider if Probability
Sampling Is Not Feasible?
Probability-based sampling is not always possible or feasible in practice. When this is the
case, it will mean that you may have to work with the sample you can get, even if that
sample is not a probability sample. For example, lack of resources (both time and money)
or lack of access (for many possible reasons) to every member of the study population may
force you to employ some form of non-probability sampling strategy.
Non-probability sampling, while having the advantage of making it easier for you to
obtain your sample, has the definite drawback of not ensuring the generalizability you are
after. This is because “the selected cases can be systematically different from the others in
the study population, and there is no means to adjust or estimate how similar or different
these cases selected through nonprobability sampling may be” (Henry, 2009, p. 79).
However, there is something you can do to adjust a non-probability sampling strategy
to assist in avoiding the produced sample being “atypical” of the study population. This is
to incorporate elements of probability sampling into a non-probability sampling strategy.
Vehovar et al. (2016) suggest two ways of doing this:
1. employing a combination of quota sampling8 and probability sampling; for
example, (a) choose a specific geographical area in which to conduct your
190 Research Design
research (the geographical area defines the subgroup of the study population from
which sampling will take place), and (b) employing probability sampling within
that geographical area, or
2. employing a combination of convenience sampling and probability sampling; for
example, (a) ask people on the street to take a survey, and (b) employ a strategy of
randomly selecting which street you do this in and/or which time of day you are
at that street.
Such incorporation of probability sampling into a non-probability sampling strategy,
may very well have the potential to underpin claims about that population that are statistically reasonable (Vehovar et al., 2016). What is important is how you describe the “correspondence between the sample and the population” (Henry, 1990, p. 12) to the audience
of your research. Or in other words, what is important is that the claims you make based
on incorporating probability sampling into a non-probability sampling strategy are statistically reasonable.
Rounding Off: Important Things to Keep in Mind if You Decide to Use a
Sample in Your Research Design
1. Data collected from a sample can (but does not always) enable making
statistically reasonable claims about the entire population from which that
sample is drawn (i.e., enable the purpose of the research to be met).
2. A claim made about the study population based on sample data is never 100%
certain. Using a sample to infer claims about the study population means
basing those claims on probabilistic expectations of what is going on in the
population rather than conclusive findings about that population. In other
words, any claim you make about the study population based on what you
learn about a sample (drawn from that population) is based on probability, not
certainty.
3. Consequently, if you decide to use a sample in your research design, you will not
be able to say that something is undoubtedly true, or an undisputable fact, about
the population from which that sample is drawn. Rather, your claims will always
take the form “it is highly likely that X (X being whatever it is that you have found
out from the sample) is true within the population in question.” There is always a
(small) chance that the claim you make about the population you set out to learn
something about is faulty.
4. A claim about a study population that is not 100% certain may still be
statistically reasonable. While you will never be able to find something that
is undoubtedly true about your study population, this does not necessarily
mean that the claims you make about the study population based on your
findings from your sample are not statistically reasonable. Rather, whether
or not those claims are statistically reasonable is related to who makes up that
sample.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
191
PUTTING IT INTO PRACTICE
DECISIONS ABOUT SAMPLING ARE NOT ALWAYS
STRAIGHTFORWARD
Example 1: Sometimes Non-Probability Sampling May
Underpin Credible Claims About a Study Population
As we have discussed in this section, large probability samples enable generalizing,
in a statistically reasonable way, what you learn from statistically analyzing the data
collected from a sample to the study population from which that sample is drawn.
Therefore, it may be tempting to default to random sampling in a research design
without giving it much thought.
However, you will need to think carefully about who comprise your study sample to
make sure that probability sampling is feasible for you. For example, if you are in the
process of designing research to learn something about the inhabitants of a specific
geographical area and you plan to use a sample, can you be sure that every individual
living in that area has an equal chance of being selected for the sample?
To ensure every individual living in that area has an equal chance of being selected
for the sample requires that you have access to everyone in that population. This is
because if there are individuals or groups of people that you for some reason do not
have access to, their chances of being selected for the sample is zero. Therefore, there
is not an equal chance for them to be selected in your sample. These are groups who
live their lives in the geographical area you are interested in but, for example, may not
have a permanent or official address there and therefore are unable to be contacted,
such as people who are homeless.
However, if you openly acknowledge that certain groups of people were inaccessible to you and discuss why this does not affect representativeness of your sample,
you still may be able to claim external validity based on your sample. Consequently,
the credibility of your research may be considered strong even though you have not
employed strict probability sampling.
For example, consider the research question “What is the average proportion of
the total household income that is spent on electrical bills by houseowners in the city
of Fairbanks, Alaska?” In this case, if you are unable to access groups of people who
live in Fairbanks but are homeless, then this will not necessarily weaken the external
validity of what you learn about the sample.
Example 2: Why, When You Do Use Probability Sampling, You
Still May Not Get a Representative Sample
If you draw a sample comprising 100 students from a population of 500 data science
students and 500 linguistics students, and you do not think carefully enough about
how to select these 100 students, your sample may, purely by chance, end up comprising 95 linguistics students, simply because of the way you selected students for
the sample. For example, you randomly recruited respondents on campus between
9 a.m. and 12 p.m. on Monday and Tuesday. However, when doing so you failed to
take into account that no data science classes were held on those days during those
times. This made it far less likely that you would recruit data science students for
your study.
Any pattern detected in the data collected from those 95 linguistics students + 5
data science students would therefore not necessarily reflect a pattern present in
192 Research Design
the study population of 500 students from each study program. Rather, it could be a
pattern arising from the disproportionate number of linguistics students in the study
sample. This disproportionate number of linguistics students in the sample actually
disqualifies what you learn about the sample from being generalized to the population
of 500 data science students and 500 linguistics students.
Consequently, it is who makes up your sample that either enables or impedes you
to be able to make statistically reasonable claims about the study population based
on what you learn from that sample. This is why you will need to think carefully
about who will be in the sample in your study in order for you to be able to generalize, in a statistically reasonable way, what you learn from that sample to the study
population.
CHOOSING AN ANALYSIS PROCEDURE SUITABLE
FOR ANSWERING YOUR RESEARCH QUESTION
We saw in the last section that applying probability-based principles when making decisions about selecting your study population, or a subsample of it, is part of establishing
that your research design is statistically reasonable. Another part of establishing that your
research design is statistically reasonable relates to your choice of statistical procedures to
analyze the data collected from that population or sample. This is because different types
of research questions require different types of analysis procedures to answer them.
A specific procedure for statistically analyzing your research data enables you to
answer a specific research question only if that analysis procedure provides the type of
information needed to answer that research question. For example, a procedure for statistically analyzing the research data designed to quantify a characteristic within one group
of people in your study population (e.g., teenagers) will not provide the type of information needed to answer a research question about comparisons across different groups of
people (all age groups) in that study population related to that characteristic. Therefore,
it is the research question—what you need to know about and why—that guides your
thinking when you are in the process of deciding what specific procedure for statistically
analyzing the research data to use in your design.
From our discussion in Chapter 3, you will remember that the variables related to your
research questions are “something of interest that varies in relation to either the people or
the places that you are studying.”9 Most often research question(s) will focus on at least
one variable. For example, consider this research question:
How does changing from instructor-centered teaching methods to more learnercentered teaching methods10 affect students’ learning outcomes?
The variables addressed by this research question are “instructor-centered teaching
methods,” “learner-centered teaching methods,” and “learning outcomes.” In addition,
you may have identified variables such as “study program,” “subject matter,” “age,” and
“level of education” as potentially relevant to answering your research question, even
though that research question does not specifically include those variables.
What type of procedure for statistically analyzing the research data will enable
answering your research question depends on what type of issue the research question
addresses in relation to one or more of these variables of the study.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
193
Research Questions About What Is Going On in a Study Population
If your question is about what is going on in a study population related to your variables
of interest, there are two groups of procedures for statistically analyzing the research data
that will enable answering that question:
1. Descriptive analysis procedures
2. Correlational analysis procedures
We discuss them separately in what follows.
Descriptive Procedures
Descriptive analysis procedures are used to investigate and then quantify the variables
of the study in order to describe trends in, and characteristics of, the study population.
They can also be used to compare such descriptions and trends across subgroups of a study
population.
For example, assume your area of interest is turnout at local elections and that your
research question is
Which factors affect turnout at local elections in constituency X?
To answer this question, you can approach members of constituency X and ask them
to tell you, for example, their age, level of education, how many children they have, and so
on, as well as how likely it is that they will decide to vote at the next local election. “Age,”
“level of education,” “number of children,” and “probability of deciding to vote at the next
local election” are the variables of the research.
From the data set you have obtained, you can calculate the average probability of
deciding to vote in the next local election among the members of that local community.
This is an example of quantifying a variable of the study in order to describe an aspect of
the study population of interest (in this case quantifying the variable “probability of deciding to vote in the next local election”).
You can also calculate the average probability of deciding to vote in the next local election across subgroups of the population (e.g., this could be age groups, groups defined by
how many children each person has, groups defined by level of education, and so on). The
computed average probability of deciding to vote in the next local election for each subgroup enables you to describe variation in expected turnout at the next local election across
subgroups in the population comprising all members of the local community in question.
You are likely to find that the average probability of deciding to vote in the next local
election varies across such subgroups. However, you will still need to statistically assess
that variation. In other words, you still need to decide whether the variation is due to
underlying differences across the corresponding subgroups of the population in question,
rather than being coincidental for the particular group of people you collected data from.
To do this you will need to include as part of your research design a procedure for
statistically analyzing the research data appropriate for comparing numerical values representing a population characteristic (in this case, the average probability of deciding to vote
in the next local election) across subgroups of that population, and which will also enable
you to decide whether any observed difference between those groups is likely to be coincidental or not. In the box below, we direct you to some of those procedures.
194 Research Design
PUTTING IT INTO PRACTICE
PUTTING COMPARING GROUPS INTO PRACTICE
To compare the average value of a population characteristic between two groups and
decide whether an observed difference between them is likely to be coincidental or
not, you can apply a procedure for statistically analyzing the research data called the
t-test. The t-test is one of many procedures for statistically analyzing data used for
this purpose. For example, for large samples, the z-test may be appropriate.
Other options for procedures for statistically analyzing data designed to compare
something across groups include ANOVA, chi-square (χ2), Kruskal-Wallis H, and many
more. Which one is most appropriate to use depends on the nature and amount of the
data you have, as well as how many variables you set out to compare.
You will need to find out more about how to do these procedures if you are thinking
to use them in your study design. However, no matter how well you do these procedures, the most important thing, as we have discussed, is to know why you are using
them.
Correlational Procedures
Rather than simply describe trends and characteristics related to the variables of interest
in your population of interest, your research question(s) may require you to look for the
existence of, and then interpret, relationships or correlations between those variables.
In such cases, correlational analysis procedures will need to be part of the research
design to enable you to establish the existence (or not) and nature of these correlations.
For example, consider this research question:
How does turnout at local elections relate to age in constituency X?
Calculating the average probability of deciding to vote at the next local election across
age groups in the population comprising members of constituency X, and then simply
comparing that probability across age groups, will not reveal the nature and strength of
a possible relationship between the variables “probability of deciding to vote at the next
local election” and “age.” For example, if you find that the probability of deciding to vote
at the next election is lowest among people 70+ years old, you still won’t know exactly
what the relationship between the variables “probability of deciding to vote at the next
local election” and “age” is.
For example, is it the case that as age increases, the probability of deciding to vote in
the next local election decreases (see the illustration to the left in Figure 8.1 below)? Or is
it maybe the case that once people turn 70, the probability of deciding to vote in the next
local election drops, but then stays the same as age increases (see the illustration to the
right in Figure 8.1 below)?
Therefore, you will need to use a procedure for statistically analyzing the research
data designed to identify and describe the relationship between the variables in question,
that is, some form of correlational approach. This will enable pattern detection, such as
the pattern related to the link between variables “age” and “probability of deciding to vote
in the next local election.” Such pattern detection will underpin claims about the population, such as the claim “as age increases, the less likely it is for members of this population
to decide to vote at the next local election.”
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
195
FIGURE 8.1 ■ Detecting Patterns
Probability of deciding
to vote in the next local
election, increasing
upward
Probability of deciding
to vote in the next local
election, increasing
upward
Illustration to the left: As age increases, the probability of deciding to vote in the next local election decreases. Illustration to the right:
For ages below 70 years old, the probability of deciding to vote in the next local election is constant across the age groups. However, once
people turn 70, the probability of deciding to vote in the next local election drops, but then stays the same as age increases.
18-40
Age,
increasing toward the right
41-69
70
Age,
increasing toward the right
To sum up, descriptive and correlational analysis procedures can be used to explore the
characteristics of the study population, that is, detect and describe characteristics of the
population. This would be the analytical strategy to use if the purpose of the research is
to answer the question, “Which factors affect turnout at local elections?” To answer this
question requires your study design to contain statistical procedures enabling exploration
of a data set across several variables to identify and describe factors that, according to the
data, affect the turnout, as well as any patterns related to links between those factors.
Research Questions About Why Something Happens in a Study
Population
Some research questions require you to do more than describe trends and patterns about
variables in a population. They require you to make statements about why those trends
and patterns occur in that population. For example, consider this research question:
What causes turnout at local elections to be less among people over 70 compared
to other age groups?
Detecting a pattern telling us that the expected turnout at the next local election in
constituency X is less among people over 70 compared to other age groups does not indicate that turning 70 is, in itself, a reason for not voting at local elections. Rather, there
could be a variety of reasons for the detected pattern. For example, do people over 70 feel
that the issues raised by the candidates are not relevant to them? Or are they tired of politicians not sticking to their promises after the election? Or maybe the detected pattern has
something to do with how the specific election procedure is organized, for example, where
the older person has to physically turn up to vote and the difficulty of getting there if they
no longer drive.
Therefore, to reveal why the expected turnout at the next local election in constituency
X is less among people over 70 compared to other age groups requires you to identify and
then select specific factors that you have reason to believe causes the detected pattern to
occur. Then you must design your research in such a way that the design enables detecting
any effect on turnout at elections of each of those specific factors. The design must also
include ways of making sure that the effects of those specific factors are not mixed up with
196 Research Design
the effect of any other possible factor. You will see such designs being called experimental
or quasi-experimental in textbooks about specific statistical procedures.11
TIP
A REMINDER
You will recall from Chapter 5 that in experimental or quasi-experimental research
approaches, you develop a highly structured research design, making sure all variables under consideration are taken into account, while at the same time making sure
that they are not affected by any other factors that might influence the outcome. You
will also recall that we included an extended discussion about an example of this type
of highly structured quantitative research design in Chapter 5, namely the Randomized
Controlled Trial (RCT).
Research designs enabling answering why questions are quite rigorous in that they
must enable explicit descriptions of how variables relate to each other, as well as being
transparent in terms of identifying possible effects of variables not included in the study.
For example, assume you make a claim about why the reading skills development
program currently used in schools in your country does not help students with dyslexia
improve their reading fluency. When doing this, you are also implicitly suggesting that
using a different program for dyslexic students or changing the factors that cause the
existing program not to work will improve the reading fluency of the dyslexic students.
This causes you to focus on variables related to this implicit assumption. In other words,
you focus on specific variables related to an assumption you have about the relationship
between a reading skills development program and reading fluency among dyslexic students. Moreover, you omit any other variable that could be related to issues in question,
for example, contextual peer factors such as level of trust in peers in the classroom or the
dyslexic students’ acceptance or rejection by classroom peers.
So how can you make sure that what you claim about the link between a specific reading skills development program and reading fluency among dyslexic students is credible,
given that a range of factors that possibly affect that link is not included in the study? That
this question is not easy to answer tells you that it is quite difficult to make credible claims
about why something is as it is when dealing with people in social settings.
Credibility of research findings arising from an experimental or quasi-experimental
design is as much a design issue as it is an issue of choosing an appropriate procedure for
statistically analyzing the research data. This is because no procedure for statistically analyzing the research data can in itself establish cause–effect relationships between variables,
that is, why something happens in a study population. To be credible, the results of these
procedures must be able to be replicated and therefore corroborated12 by other researchers.
Reproducing research results with the intention to confirm the findings of an original
study can be very difficult if the original study is not transparent about its design, including which variables are included, which are left out and all the assumptions made about
both sets of variables.
Consequently, the level of rigor required in an experimental/quasi-experimental
research design can only be achieved by developing a hypothesis about a proposed cause–
effect relationship between the variables of interest, and then designing the study to test
that hypothesis. In what follows, we explore how research questions that take the form of
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
197
hypotheses affect both your choice of procedure(s) for analyzing your research data as well
as what you will need data about.
When the Research Question Takes the Form of a Hypothesis
You will recall from Chapter 3 that a hypothesis is a statement about what we predict
empirical evidence to reveal about a problem of interest in a specific situation in a specific
context. It is a “hunch derived from an informed reading of the literature, a theory, or personal observations and experience . . . capable of being tested” (Nardi, 2018, p. 48).
For example, consider this research question:
Which factors affect turnout at local elections?
Now suppose that what you want to do is to test the following hypothesis developed
from this research question:
For ages over 70, an increase in age corresponds to a decrease in probability for
deciding to vote at the next local election.
In this case, you will collect data specifically about the variables involved in the hypothesis (“age” and “probability of deciding to vote at the next local election”) only. This is
because you are interested in these two specific variables only. Finally, you will apply a
procedure for statistically analyzing those data designed to test the specific hypothesis in
question. Testing the hypothesis means applying a procedure for statistically analyzing the
research data designed to detect any inconsistency between the data collected about one
or more variables and the hypothesis made about that (those) variable(s) if there is such an
inconsistency to detect. As was the case for research questions in general, so it is the case that
not all procedures for statistically testing hypotheses are designed to test all types of hypotheses. What is an appropriate procedure for statistically testing a hypothesis depends on what
type of issue the hypothesis addresses in relation to one or more of the variables of the study.
Aspects That Affect Whether or Not the Conclusions From Testing a
Hypothesis Are Statistically Reasonable
Several aspects of a research design, such as sample size and significance level, will affect
what is known as the power of a hypothesis test. Simply put, the power of a hypothesis
test is its ability to correctly claim support for the statement making up the hypothesis.
Research designs that enable high in power hypothesis tests will improve the probability
of the test detecting whatever the test is set up to look for in the data set, if there is something present to detect. Consequently, a research design that includes testing a hypothesis
should enable a hypothesis test with as much power as possible.
TIP
WHAT YOU WILL NEED TO LEARN MORE ABOUT RELATED TO
HYPOTHESIS TESTING
There are a lot of statistical concepts involved in deciding whether or not to claim support for your hypothesis. These include
•
•
p-value
Type I error
(Continued)
198 Research Design
(Continued)
•
Type II error
As well as
•
•
what happens to Type II error as Type I error decreases,
•
what the connection between the significance level α, power, and sample size has to
do with Type I and Type II errors.
what happens to the power of a hypothesis test when the size of the sample is
changed,
To help you navigate these concepts, we advise you to seek expert statistical help.
Further, it is important to keep in mind that not all procedures for statistically analyzing the research data are designed to test all types of hypotheses.13 Rather, it is
the nature of the hypothesis that guides your thinking about which specific hypothesis
testing procedure to apply when testing a specific hypothesis, for example, whether
the hypothesis is about testing assumptions about relationships between variables or
about comparing characteristics across subgroups.
We strongly recommend you seek expert statistical help when designing your
research to ensure that there is a match between the nature of the procedure for statistically testing hypothesis that you include in your design and the types of hypotheses
you are designing your research to test.
Still More Thinking to Do
At this point, we have established that designing quantitative research that is statistically
reasonable requires you to think about statistical procedures at two levels:
1. the role of procedures derived from laws of statistics and probability when aiming
to learn something about the group of people that is the subject of the research
2. how different types of research questions require different types of statistical
analysis procedures to answer them
Your research question guides your thinking when deciding what procedure for
statistically analyzing the research data is appropriate to use in your research design.
Remember, no matter how competently individual statistical procedures may be conducted, if they are not the correct ones in terms of enabling you to reach the type of
conclusions needed to answer your research questions, then they will be of little or no use
to your research.
However, even when you do find an appropriate procedure for statistically analyzing the research data in order to provide the kind of information needed to address your
research question, you still have more thinking to do. This is thinking related to what type
of data you will need to collect in order to be able to perform the statistical procedures you
need to in order to reach the type of conclusions needed to answer your research questions.
You will need to think about this before you begin collecting any data because once
you have your data set, if that data set does not enable conducting the procedure for statistically analyzing the research data you need to conduct to address your research question, then your research design will fail to achieve its intended purpose. Simply having
numerical data (even lots of it) is not enough to ensure that your research is statistically
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
199
reasonable. This means that when designing your research, you will need to think about
what types of data you have to choose from, and what will be the right type of data to collect in your study.
In the next section of the chapter, we take a closer look at what we mean by types of
data and which type of data will be the right type.
WHAT TYPES OF DATA ARE THERE?
Recall that the variables of a study are concepts of interest in that study, related in some
way to addressing your research question(s). For example, for a study population comprising teachers, the variables of interest in a study focusing on that population might be age,
level of formal education, level of education in which the teacher works, the teacher’s subject area(s), and so on.
For each variable, you need to define all possible values that the variable might take.
For example, the variable “age” might take values in the form of integers (i.e., whole numbers)14 between 18 and 75, and the variable “level of formal education” might take the
values “bachelor’s degree/professional graduate certificate in education,” “master’s degree/
postgraduate certificate in education,” or “PhD/specialist certificate.”
The data you collect about those variables are produced by assigning a numerical value
to a variable of interest for each participant included in the study. When doing this, you
are measuring the variable of interest. For example, if a respondent provides the answer
“51” to the question “What is your age?” then the value “51” is assigned to the variable
“age” (and thereby measuring the variable “age”).
Because you need to measure all the variables that are included in your research,
your research design must include the design of a measurement instrument, specifically designed to produce (1) data representing relevant information about/measures of
all the variables included in the study, and (2) data that is of the type that meets the
requirement(s) of the procedure for statistically analyzing the research data needed to
produce the kind of information that will underpin an answer to the research question.
Such a measurement instrument can, for example, take the form of a set of questions (the
complete set of questions is called a questionnaire) or it could be a structured observation
schedule.
What type of data will be produced by measuring a variable depends on what type of
values you assign to that variable. The type of values you assign to a variable is called the
level of measurement of that variable. When measuring a variable, the measurement can
produce either nominal, ordinal, or interval/ratio data.
You need to know what differentiates these types of data from each other, because not
all types of data will allow or enable all statistical procedures to be performed on those
data. Therefore, when designing your quantitative research, your design must enable you
to obtain the type of data that will allow you to conduct the statistical procedures that will
enable you to address your research questions.
Nominal Data
Nominal data is the type of data produced by assigning values to a variable that take the
form of labels. For example, a specific university could categorize their study programs
related to physics. In your research study, these “study programs related to physics”
200 Research Design
may be a variable of interest. Values related to this variable of interest can be assigned
using the way that the university categorizes these physics-related study programs as
being mainly about either astrophysics, geology, meteorology, material science, electrical engineering, or nuclear and particle physics. Hence the labels astrophysics, geology,
and so on are the possible values assigned to the variable “study programs related to
physics.”
In some studies, you may apply numerical-based coding to those categorizations. For
example, you might code astrophysics as “1,” geology as “2,” and so on. What the numeric
coding can be used for is, for example, applying count functions embedded in software
used for statistical analysis. For example, the university can collect data about what people
enter into the search field on the university web page. If the number “1” is assigned to any
search string15 containing the word astrophysics, statistical analysis software can count how
many times the word astrophysics is included in a search string on their web pages, and
thereby get a sense of the public interest in that specific academic discipline. By assigning
the number “2” to any search string containing the word geology and so on, such counting
provides data about the relative interest in each of the disciplines within the wider field of
physics.
However, even though nominal data can be coded as numbers, those numbers cannot
be ranked. For example, “astrophysics” is not smaller or less than “geology” just because
astrophysics is coded as 1 and geology is coded as 2. Moreover, the numbers cannot be
used to perform calculations. For example, there is no average value between astrophysics
and geology even though the average of 1 and 2 is 1.5.
Ordinal Data
Ordinal data are produced by defining the possible values of a variable in such a way that
the range of possible values of a variable can be ordered, or ranked, but where the difference between each of the ranked values is not defined. In other words, ordinal data has the
properties of nominal data, but in addition ordinal data offers the opportunity to order or
rank the values of the variable being measured to produce such data.
Lists of results are an example of ordinal data. For example, suppose Huey, Dewey,
and Louie enter a contest. Dewey wins first prize, Huey wins second prize, and Louie
ends up third. Then we can rank their performance as we know that Dewey performed
better than Huey in that competition, and Huey better than Louie, but we do not know
anything about how much better Dewey performed compared to Huey or Huey compared
to Louie.
Another example of ordinal data is the use of a device by the exit door of a shop with
the text “We want your feedback! How satisfied are you with your shopping experience?”
written on it, followed by three clickable buttons:
When customers click one of the buttons, ordinal data are produced.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
201
Interval/Ratio Data
Interval/Ratio data are produced by defining the possible values of a variable in such a way
that the range of possible values of a variable can be ordered, or ranked, and in such a way
that the difference between one value and the next on the ordered list of possible values
is fixed across the complete list of values. The fixed difference between one value and the
next across the total range of values enables doing calculations (involving addition, subtraction, multiplication, and division) on this type of data.
For example, when measuring the variable “age,” if you assign the values 20, 25, and
35 to participant A, B, and C respectively, then you know that participant B is 5 years
older than participant A, and that participant C is 10 years older than participant B. You
also know that the age difference between participants A and B (5 years) is half that of the
age difference between participants B and C (which is 10 years). It is only for interval/ratio
level of measurement that expressions like “twice as many,” “half as much,” or “three times
as often” make sense.
In other words, interval/ratio data has the properties of ordinal data, but in addition
interval/ratio data offers the opportunity to define a difference between each of the ranked
or ordered values of the variable being measured to produce such data. For example, when
counting the frequency of an event, registering test scores, measuring age or height, or collecting data about annual income, you get interval/ratio data.
PUTTING IT INTO PRACTICE
DECIDING ON LEVEL OF MEASUREMENT IN PRACTICE
Think of a situation where a sports journalist is asked to rank NBA (National Basketball
Association in the USA) players based on numerical data such as the number of assists
per game, points per game, steals per game, and blocks per game. The journalist is
asked to provide a list of the top 5 players.
If, for example, player X has an average of 32.0 points per game, player Y has an
average of 28.0, and player Z has an average of 26.0, the difference in average points
per game between player X and player Y is 4.0, and the difference in average points
per game between player Y and player Z is 2.0. Moreover, the difference between
average points per game between player X and player Y is twice that of players Y and
player Z.
The fact that we can quantify the difference between the number of assists,
points, steals, or blocks per game across the players in question, and that statements such as “the difference between average points per game between player X
and player Y is twice that of players Y and player Z” make sense, tells us that the
data used by the journalist to provide the list of the top 5 players are interval/ratio
data.
On the other hand, the data produced by the journalist’s list of top 5 players is ordinal. This is because you know that the player ranked as number 1 is considered better
than the player ranked as number 2, but you are not able to quantify how much better
the player on top of the list is compared to the player on second place.
What makes nominal, ordinal, and interval/ratio data different from each other is
summed up in Figure 8.2:
202 Research Design
FIGURE 8.2 ■ Three Types of Data and What Makes Them Different
From Each Other
Examples:
Eye color, subject
matter of study
program, marital
status
Examples:
Student letter
grade (A-F),
football team
rankings, customer
satisfaction ratings
Examples:
Years, days, age,
height, probability
of something
To sum up, we have established that the data you collect about the variables of your
research are produced by assigning a numerical value to each variable of interest in the
study for each participant included in the study. When doing this, you are measuring the
variable of interest. Moreover, the form that such measurement takes defines what type of
data will be produced by that measurement, and therefore defines which procedures for
statistically analyzing those data can be conducted in a statistically reasonable way. In the
next section, we take a look at two examples that illustrate this point.
Activity
Thinking About What Type of Data You Need Before Collecting
Any Data
Figure 8.2 above demonstrates that interval/ratio data can be treated as ordinal data,
and ordinal data can be treated as nominal data—but not the other way around.
Because different types of data allow different types of statistical analyses to be performed on those data, you need to think through the implications for your study
design related to different types of data before you collect any numerical data.
This is because once you have collected your data, there is no way of changing,
say, ordinal data to interval/ratio data, or nominal data to ordinal data. In other
words, you are facing a problem if your chosen procedure for statistically analyzing
the research data assumes you will have interval/ratio data, while you have collected
ordinal data.
This problem is that your research design, and the data and analyses arising from that
design, will not be credible in terms of you being able to reach statistically reasonable
conclusions about your research questions.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
203
With this in mind, think about what type of data you would need to collect to answer
the following questions and justify your choices:
1. What are the reasons given for absence from work among business executives?
2. Where do people of different age groups prefer to take overseas holidays?
3. Does the level of stress in patients about to undergo surgery depend on any
specific type of nurse intervention?
If you have already developed research questions of your own, use them instead.
HOW DIFFERENT TYPES OF DATA ENABLE
DIFFERENT TYPES OF KNOWLEDGE
Data differ in terms of what procedures for statistically analyzing research data, and therefore
what type of knowledge, they enable. We will illustrate what we mean by this by taking a
closer look at two examples inspired by Saris and Gallhofer (2014). The examples highlight
how types of data affect what you potentially can learn (as well as what you definitely won’t
be able to learn) about the research question(s) that you are designing your research to answer.
Consider this question:
How does the status of different occupations differ compared to the occupational
status of a schoolteacher?
In what follows, we provide examples of two different ways of collecting data that
might be relevant for answering this question. Version 1 produces interval/ratio data and
Version 2 produces ordinal data.
Version 1:
Occupations differ with respect to status. We ask you to estimate the status of a
series of occupations. If we give the status of the occupation of a schoolteacher a
score of 100, how would you evaluate the other occupations? For example, if an
occupation that, in your opinion, is twice as high in status compared to a schoolteacher, score it 2 × 100 (i.e., 200). If the status of the occupation is half that of a
schoolteacher, divide 100 by 2 (which gives 50).
What is the status of a
General physician?
Your answer: ____
Real estate agent?
Your answer: ____
Bus driver?
Your answer: ____
Carpenter?
Your answer: ____
Version 2:
Occupations differ with respect to status. We ask you to evaluate the status of a
series of occupations. The reference point for you to use when making your evaluation is the status of the occupation of a schoolteacher. For each occupation on the
list, we ask you to evaluate the status of that occupation as either lower than, the
same as, or higher than that of a schoolteacher.
204 Research Design
Lower
Same
Higher
General physician
Real estate agent
Bus driver
Carpenter
The way that status is measured in Version 1 produces interval/ratio data. Because interval/ratio data has the properties of ordinal data, you will be able to find how many respondents evaluate the status of a specific occupation as lower than or the same as or higher than
that of a schoolteacher simply by treating any value lower than 100 as having a lower rank
than the value 100, and any value higher than 100 as having a higher rank than the value 100.
Moreover, the way that status is measured in Version 1 allows performing calculations
involving addition, subtraction, multiplication, and division on the data, and therefore it
is possible to, for example, quantify the average opinion about the status of each occupation
compared to that of a schoolteacher, across the group of research participants. This facilitates, for example, developing a “status scale” reflecting the average opinion of the status
of each occupation compared to that of a schoolteacher.
The way that status is measured in Version 2 produces ordinal data. This is because
the values that the variable “status” can take in Version 2 of the question (“lower than,”
“same as,” and “higher than” that of a schoolteacher) can be ranked; there is a logical order
to them (“lower than” is less than “same as” is less than “higher than”). Therefore, the
data produced by the way of measuring status demonstrated in Version 2 of the question
enables you to say something about how many respondents evaluate the status of a specific
occupation as lower than or the same as or higher than that of a schoolteacher.
However, you cannot say anything about how much lower or higher the status of a
specific occupation is compared to that of a schoolteacher. Neither are you able to calculate
average opinions about the status of each occupation compared to that of a schoolteacher.
If this is what you need to say something about in order to answer your research question(s),
then you will need interval/ratio data for the statistical analyses you would need to do.
These examples illustrate the point we made previously, namely that not all types of
data will allow or enable all statistical procedures to be performed on those data. Therefore,
when designing your quantitative research, your design needs to enable you to obtain data
of a type that meets the requirements for the procedures for statistically analyzing the
research data you will undertake in order to enable you to answer your research questions.
TIP
BE AWARE YOU MAY NEED TO MANEUVER BETWEEN WHAT IS
IDEAL AND WHAT IS FEASIBLE
Collecting data meeting the data related requirements of procedures for statistically
analyzing that research data is not always a clear-cut issue. This is because collecting data of the type able to enable the type of statistical analyses needed to answer
your research questions often involves navigating trade-offs between what is ideal
data wise, and what is feasible.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
205
Why Making Sure You Collect Data of the “Right” Type Is Not Enough
to Ensure That Your Research Design Is Statistically Reasonable
It is possible to design your research to obtain the right type of data. However, if that right
type of data is obtained from the wrong people, for example, the wrong study population
in light of your variables, or from an inadequate sample from the right study population,
then it will not be the right data for your study.
To design your study in a credible way, all parts of that study design must be statistically reasonable. The design is only as credible as its weakest link. No part of designing
your research can be seen in isolation from any other part. This is because the credibility of
your quantitative research design, and findings, lies as much in the way that the numerical
data has been collected and the form that that data takes as it does in the statistical procedures
used to analyze that data.
TIP
TO USE STATISTICS WELL, YOU DON’T NEED TO BE A STATISTICIAN
IF YOU SEEK HELP WHEN YOU NEED IT
Remember: Statistics are a means in the design and conduct of research using some
form of quantitative approach, not an end in themselves. Therefore, you do not need to
know every detail of every concept involved in every type of procedure for statistically
analyzing the research data that you are planning to use. What you do need is sufficient knowledge that will enable you to avoid making major, irreparable mistakes in the
design of your research—mistakes that would ultimately cause your research design
to fail to achieve its purpose.
If you don’t have that knowledge, know that you will need to obtain that knowledge
and how you might do so. Discussions in textbooks about statistics, as well as in textbooks about statistics in relation to doing quantitative research, tend to be technical.
Therefore, if you
1. are not very familiar with statistics, or
2. feel you need to check that your thinking about statistical data analysis and its outcomes is on the right track,
we suggest that you seek expert help with the statistical procedures involved in selecting the participants of your research, as well as with the procedure(s) for statistically
analyzing the research data that will form part of your developing design.
A statistician can assist you sorting out what is essential for you to know, and
thereby help you avoid feeling like drowning in technical details that in themselves are
not really helping you develop a well-thought-through research design.
To reiterate, a strong word of advice from us is to seek timely help from a
statistician.
CONCLUSIONS
In this chapter, we have focused on aspects of a quantitative research design that will
underpin statistically reasonable answers to the research question(s) your research is
designed to address.
We have seen that designing quantitative research that is statistically reasonable includes
establishing clear and explicit links in the research design between your research
206 Research Design
questions, your statistical analyses, the data you collect, and the sample or population
that you collect your data from. Establishing these links is the result of a series of decisions about each of these aspects. We capture this series of decisions in Figure 8.3 below:
FIGURE 8.3 ■ Establishing Links Between Aspects of a Quantitative
Research Design
Data: The data of the
study is the raw
material for the
statistical analyses to
be done on those data
Sample: A subgroup of
the study population. The
people in the sample
are the ones that provide
the data of the study
Research questions: The
purpose of designing the
research in the first place
is to arrive at credible answers
to the research questions
Statistical analysis: The
outcomes of statistically
analyzing the data of the study
will provide the base for the
answers to the research questions
the study is designed to address
Study population: Doing quantitative
research, a general aim is to learn something
about large groups of people. Therefore,
the design must enable the credible answers
to the research questions being generalizable
to the study population—also in cases when
data are collected from a (relatively small)
subset of that population
After reading the chapter, if you understand
1.
how the aspects included in Figure 8.3 are linked in a quantitative research design,
2.
the importance of establishing these links clearly and explicitly,
3.
what you need to think about to be able to include such clear and explicit links in
your developing research design,
4.
that if you do not establish these links in your research design, you may end up in a
situation where you get some of your research design completely right but still end
up with a design that is completely wrong and that will not provide credible answers
to your research questions,
then we have achieved our goal when writing this chapter.
In addition, because you do understand these four points, now you are ready to move to
the next phase of your thinking about designing your quantitative research. This phase
involves thinking about how you will actually put your research design into practice and
do the research. To “do” your research you will first need to design a measurement instrument. This is an instrument that enables you to measure the variables of your research.
Those measures provide the data for the statistical procedures that can provide a credible
basis for answering your research questions.
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
207
Once you have your measurement instrument, you will need to put that measurement
instrument into practice and use it to collect the numerical data you need. This will
involve thinking about how you will approach and interact with the people included in
your proposed sample when actually collecting data from them using that measurement
instrument.
Therefore, in Chapter 9, we put the spotlight on the more operational aspects of putting
quantitative research design into practice. We do this by taking a closer look at what you
will need to think about when designing a measurement instrument, and then using it
to measure your variables of interest so that you can arrive at credible answers for your
research questions.
SUMMARY OF KEY POINTS
• For a quantitative research design to enable credible research findings, that research
design needs to be statistically reasonable.
• A statistically reasonable quantitative research design is one in which there are clear
and explicit links between the numerical data you propose to collect (who from, type,
and amount), the type of statistical analyses that you propose to do, and the research
questions that you are asking.
• In addition, a statistically reasonable quantitative research design is one in which
there are clear and explicit links between the group of people to which you want the
answers to your research questions to apply and the (often smaller) group of people
from which the numerical data will be collected.
• These links are established by adhering to probability-based principles and rules of
statistics that enable you to make claims about the generalizability of your findings
from the sample to the wider population from which that sample was drawn.
• A representative sample is a scientific ideal obtainable by true probability sampling,
but true probability sampling may not be feasible. In practice, sampling procedures
often involve a degree of non-probability sampling (such as convenience sampling).
• Only when knowing which alternatives there are to choose from, and their strengths
and weaknesses in terms of enabling or impeding achieving the goal of the research
being designed, are you in a position to make informed and thought through choices
about which procedure(s) for statistically analyzing the research data and which
sampling strategies to incorporate in that design.
• The credibility of a quantitative research design is only as strong as how clear and
explicit the links are in the research design between your research questions, your
statistical analyses, the data you collect, and the sample or population that you collect
your data from.
• Therefore, the mechanics of using statistical techniques or procedures as part
of a research design using a quantitative approach are not the same as designing
statistically reasonable research.
208 Research Design
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
estimate (of the numerical value of a
population characteristic)
external validity
generalizing in quantitative research
hypothesis
hypothesis testing
power of a hypothesis test
random selection process
representative sample
sample/study sample
sampling strategy
statistical validity
statistically reasonable
study population
variable
SUPPLEMENTAL ACTIVITIES
1.
The aim of this activity is to explore how it is possible to get some of your research
design completely right but still end up with a design that is completely wrong.
To begin your exploration of this question, we would like you to write down
what you will need to think about when building a house.
Your list probably includes points related to the following areas:
• Some sort of blueprint or plan—the aim of doing the building, knowing what
you are building, and what you want to achieve at the end of the building
process.
• Materials—What materials will you need to be able to build this? Type and
amount. This will vary depending on the planned house—how big it is planned
to be, what you want it to look like, the funds and resources you have available
to you. Where will you get these materials from to make sure they are up to the
standard required?
• The means of putting those materials together—knowing the building
procedures that need to be followed when using those materials that will
enable the house to be built correctly.
If you fail in the execution of, or make poor or inappropriate decisions about, any
of these areas, then it is likely that you will not achieve the aim of building the house.
For example, you might have a wonderful blueprint, but if you choose the
wrong materials for building what is in that blueprint, then you will not achieve
your aims. Similarly, you might have a wonderful blueprint, choose what can be
suitable materials for that blueprint, but not know what the building procedures are
that should be followed to ensure that the house is built properly. Therefore, while
you have some things completely right (e.g., blueprint and materials), others may be
completely wrong (the way you put them together). Now try this:
When reading the above points, make the following substitutions:
Some sort of blueprint or plan → Research question
Materials → Types of data and who you obtain it from
The means of putting those material together → Procedure for
statistically analyzing the research data
Chapter 8
•
Foundational Design Issues When Using Quantitative Methods
209
Doing so should give you insights into what we mean by saying that when
designing quantitative research, it is possible to get some (even most) of it
completely right (e.g., the research question is well developed and the planned
analytical procedures are statistically reasonable) but still end up with a design that
is completely wrong (e.g., the numerical data collected is of the wrong type and
therefore makes the rest of the research not credible).
2.
Is your research question well enough developed?
Consider your research question and answer the following questions about that
research question:
a.
b.
c.
Is who or what will need to comprise the population for the study clear
from your research question? If so, who or what is it?
Does the research question make clear what the main variables of interest
in the study are, that is, which aspects, characteristics, or features of the
study population you want your claims about the study population to be
about? If so, what are they?
Does the research question make clear what type of procedure for
statistically analyzing the research data will enable answering that
question? If yes, what is it and why? What type of data will you need to be
able to perform that analytical procedure? Why?
If your answer is no to any of the above questions, think about why you answered
no. This will reveal ways in which you will need to develop your research questions
further. Then return to your original research question and, using this thinking,
develop that question until all of the above questions can be answered with yes.
This example highlights the type of reflexive and iterative thinking that you will
need to do when designing your quantitative research study.
FURTHER READINGS
Gorard, S. (2003). Quantitative methods in social science: The role of numbers made easy.
Continuum.
Stockemer, D. (2019). Quantitative methods for the social sciences: A practical introduction with
examples in SPSS and Stata. Springer Nature.
NOTES
1. Participant here means a person from whom data is collected. You will also see participant used in qualitative research with a slightly different emphasis, namely as a
co-participant with the researcher in the research.
2. See Chapter 5 for an extended discussion of the nature and purpose of applying a quantitative approach in research.
3. “The size range of the algae spans seven orders of magnitude. Many algae consist
of only one cell, while the largest have millions of cells” (taken from https://www.
britannica.com/science/algae).
4. You will see the theory, methods, and practice of generalizing from a sample to a population being called statistical inferences in textbooks of statistics and quantitative
research methods.
210 Research Design
5. This is a consequence of mathematical principles derived from probability theory. For
example, suppose you flip a fair (i.e., getting heads and getting tails are equally likely
outcomes from flipping the coin) coin 10 times (i.e., 10 random selections). Probability
theory tells you to expect five head and five tails, but probably you would not get that
exact result. You could by chance end up with eight tails and two heads, for example.
However, by increasing the number of tosses (i.e., increasing the number of random
selections) to 1,000, you are more likely to get something close to 500 heads and 500
tails than getting 800 tails and 200 heads.
6. From our discussion in Chapter 3, you will remember that the variables related to your
research questions are “something of interest that varies in relation to either the people
or the places that you are studying.”
7. For example, of you want to find the average height of the people in your study population, and all of them have the same height, a sample of one is sufficient to find their
average height. However, if the variation in height is large among the people in the study
population, you will need a larger sample to make sure that the first few you select are
not extreme cases, and to make sure that you cover the variety in heights.
8. Quota sampling means that the population is divided into subgroups, where who
belongs to each subgroup is defined, for example, by a combination of sociodemographic factors. Then such a subgroup of the study population is chosen on a
non-random basis to be the subset of the study population from which sampling will
take place.
9. In Chapter 9, we will define what we mean by a variable in more specific terms.
10. Instructor-centred teaching is characterized by the teacher taking the role of being a
master of the subject matter. Learners, on the other hand, take the role of passive and
copious recipients of knowledge from the teacher. Practicing learner-centred teaching
methods, the teacher takes the role of a co-learner alongside the students and allows
for discussion about, and inquiry of, the content of the subject matter.
11. When reading about experimental designs, you will also come across the terms independent variable, dependent variable, confounding variable, extraneous variable, control,
and many more. These are all terms you need to understand the meaning of to be able to
design experimental research.
12. Corroborate: to support with evidence (from https://www.merriam-webster.com/
dictionary/corroborate).
13. This means that the exact way of testing a specific hypothesis depends on the nature of
that hypothesis (e.g., testing a hypothesis on relationships between specific characteristics or traits involves different statistical analyses than does testing a hypothesis on
differences in characteristics or traits across subgroups of a population) and the nature
of the data collected.
14. An integer is a number which is not a fraction.
15. By a search string we mean any combination of words, numbers, and symbols entered
into a search field.
9
COLLECTING DATA USING
QUANTITATIVE METHODS
PURPOSES AND GOALS OF THE CHAPTER
In Chapter 8, we highlighted that arriving at credible answers to research questions
using some form of quantitative approach depends on a statistically reasonable research
design. A statistically reasonable quantitative research design is one in which there are
clear and explicit links between (1) the numerical data you propose to collect, the type
of statistical analyses that you propose to do, and the research questions that you are
asking; and (2) the group of people to which you want the answers to your research questions to apply and the (often smaller) group of people from which the numerical data will
be collected.
Therefore, once you have decided on your statistically reasonable research design, you
are going to need to think about how you will actually obtain that numerical data from
those people. What method will you use to collect that data? The issues addressed in
quantitative research are often about abstract and complex concepts. Those concepts are
what make up the variables of the study.
Consequently, making sure that the data you collect can underpin credible answers to
your research questions includes thinking about how to
1. measure the variables of the study to make sure that the numerical data arising
from those measurements represents relevant information about the variables
that your research question(s) are about;
2. design, or select an existing, measurement instrument that can enable you to
obtain that numerical data in a reliable and valid way;
3. put that measurement instrument into practice in a way that optimizes both the
response rate of your study and the credibility of any findings based on those
responses.
Exploring these three points is what this chapter is about. However, a word of caution
before we begin. Chapter 8 provides the backdrop for many of the issues we take a closer
look at in this chapter. If you have not read Chapter 8, we strongly encourage you to do so
before reading this chapter.
The goals of the chapter are to
• Establish what we mean by valid research findings, as well as expand the
understanding of what underpins valid research findings beyond the issues of
statistical validity and external validity that we discussed in Chapter 8.
211
212 Research Design
• Make clear that arriving at credible research findings when using some form of
quantitative approach requires that the dataset is made up of credible information
about the variables identified as relevant for addressing the research question(s).
• Emphasize that obtaining data in the form of credible pieces of numerical
information about complex, and most often abstract, variables requires making
those variables measurable.
• Remind you that the data produced by measuring variables must be of the
type needed to be able to perform the procedures for statistically analyzing the
research data that you plan to use.
• Identify issues to think about when the measurement instrument is put into
practice when collecting data from the participants1 of the study.
• Make clear that assessing the credibility of the findings of a quantitative research
study involves assessing the effects of the choices you make about all aspects of
the research design, as well as assessing the degree to which those decisions are
consistent and in keeping with each other.
MEASURING VARIABLES TO ENABLE
VALID RESEARCH FINDINGS
When reading about quantitative research methods, you will come across the concept of validity. Validity is a complex concept, because the term validity focuses on different things depending on what we are talking about when using it. In short, we can say that research (or a research
finding) is valid when it is credible, well-founded, reasonable, justifiable, and defensible.
In Chapter 8, a large part of the discussion focused on considerations related to the
following:
a. whether the statistical analysis procedures that are part of a research design can
provide the kind of information needed to answer your research question(s);
b. whether the type of data you propose to collect will enable those statistical
analyses to be performed;
c. if your research design includes the use of a sample, whether what you learn
about that sample by statistically analyzing the data collected from the people
comprising it is generalizable beyond that sample (preferably to the study
population as a whole).
Points (a) and (b) are about statistical validity, and point (c) is about external validity.
Both statistical and external validity are facets of the overall concept of validity, but in
themselves they are not sufficient to ensure valid, that is, credible, well-founded, reasonable, justifiable, and defensible research findings.
What remains to think about related to the concept of validity as it applies to quantitative research are the parts of validity related to the methods used to obtain the data.
For example, if you are using some sort of survey as your method of choice in your study
design, then how will your survey related data actually be collected? We discuss this in
detail in later parts of the chapter.
Chapter 9
•
Collecting Data Using Quantitative Methods
213
TIP
ESTABLISHING THE MEANING OF SOME OF THE WORDS USED IN
THIS CHAPTER
Recall from Chapter 8 that your research design must include the design of a measurement instrument. That measurement instrument must be specifically designed to produce
numerical data that provides relevant information about measures of all the variables
included in the study. Moreover, the numerical data produced by that measurement
instrument must be in a form able to meet the requirement(s) of the statistical analyses
that you have identified you need to do in order to answer the research question(s).
Individual measurement items, each designed to measure (an aspect of) one of the
variables included in the study, collectively make up the measurement instrument
included in your research design.
The forms that measurement items can take include
•
questions (the complete set of questions, i.e., the measurement instrument, is then
called a questionnaire);
•
statements for which the respondents are asked to provide their response, usually
by selecting at least one option from a set of predetermined response options;
•
observation items (the complete set of observations is then a structured observation
schedule).
Other types of measuring instruments include web scrapers2 and technical instruments normally used in science labs (such as movement sensors).
Are You Measuring What You Think You Are?
One common method used to collect data from a group of respondents is a quantitative
survey. A survey is a research method used to collect data from a group of respondents by
way of asking people a set of predefined questions or asking them to respond to a set of
predefined statements.
If you are doing a survey, one of the things you will need to consider is how the measurement items making up your measurement instrument in that survey are phrased and
presented to the respondent so that you can be sure that the items measure what they are
intended to. If an item is not clear, then what also will not be clear are any measurements
gained from it. Sure, you will have a measurement but maybe not of what you think. We
illustrate this point by looking at an example inspired by Blair et al. (2014):
Assume one of the items in a measurement instrument is the following:
Select the option that best represents your opinion about this statement:
“The criminal justice system is performing well.”
1
2
3
4
5
6
Strongly
disagree
Moderately
disagree
Mildly
disagree
Mildly
agree
Moderately
agree
Strongly
agree
One of the problems here is that respondents may have different understandings of what is meant by “the criminal justice system.” For example, a respondent who
214 Research Design
understands the criminal justice system to include the police, the courts, and the prisons will provide an answer reflecting their overall impression of the performance of that
system—the police, the courts, and the prisons. On the other hand, a respondent who
understands the criminal justice system as being synonymous with the courts will provide
an answer based on, and reflecting, their opinion of the performance of the courts only.
Moreover, respondents may interpret “performing well” in very different ways. For
example, for one respondent, a criminal justice system that performs well could mean one
that can investigate and adjudicate criminal offenses in a timely and effective manner. For
a different respondent, a criminal justice system that performs well could be one that is
free of improper government influence.
Therefore, it is impossible to know what exactly is being measured by this item if what
the criminal justice system is, is not clear. You will have no way of knowing what understanding of this term participants of your study had when they responded to this item. Therefore,
it will not be possible to draw valid conclusions from the data collected from this item.
What this example highlights, is that the methods used to actually collect the numerical data for your study, and what is actually measured when doing so, enable valid research
findings if those measurements actually measure what they claim to. In the next section,
we take a closer look at what this means in practice when we are designing our research. Or
put another way, what will you need to think about to make sure that the measurements
you make actually do measure what they claim to?
MAKING SURE THE MEASUREMENTS YOU
MAKE MEASURE WHAT THEY CLAIM TO
You will recall from Chapter 8 that for each of the variables included in your study, you
need to define all possible values that the variable might take. For example, the variable “eye
color” can be assigned the values of “brown,” “hazel,” “blue,” “green,” “gray,” and “amber.“3
Whenever a value of a variable is assigned to a participant of the study, the variable is being
measured. For example, when a participant responding to the question “What is your eye
color? Please select one of the options: 1. brown, 2. hazel, 3. blue, 4. green, 5. gray, 6. amber,”
selects 1. brown, then 1 is the measure of the variable “eye color” for that respondent.
When designing your research, you will need to develop a way of measuring each of
the variables of interest in your study. For some variables (such as “eye color” or “age”),
you will only need one measurement item—what is your eye color or what is your age—to
obtain that measure. For other variables (such as “motivation” and “achievement”), you
will need a collection of measurement items to be able to measure those variables in a credible way. This is because the concepts of motivation and achievement are more complex
and made up of a number of different aspects or factors related to them.
This means that before you can decide what type of measurement instrument to use
in your study, and exactly what the measurement items making up that instrument will
look like, you will need to have asked yourself, and reached a decision about, the following
questions:
• What is my understanding of the variable in question?
• How can I define my understanding of the variable in such a way that it can be
measured?
Chapter 9
•
Collecting Data Using Quantitative Methods
215
• Given my understanding of the variable, which values can that variable take? Are
these values the only possible set of values?
• How can I make sure that I measure what I want to measure?
We illustrate the thinking you will need to do related to the above questions by using
two examples of putting this thinking into practice. Both examples illustrate why it is
important to be clear (to yourself and to others) about how you understand and operationalize your variables of interest.
Example 1
Suppose your variable of interest is “time spent on social media platform X” and you
choose to use a web scraper as the instrument for data collection. You will then need to
decide which data the web scraper needs to “fetch” in order to measure your variable of
interest—time spent on social media platform X. That will require you to be clear about
what you mean by “time spent on social media platform X.” Do you understand “time
spent on social media platform X” as equal to “time dedicated to being active (i.e., liking,
sharing, posting) on social media platform X”? Or is simply being logged on to platform X
enough to qualify as “spending time”?
This is an important decision to make, because if your understanding of what
“spending time” means is not clear, how can you go about measuring that variable? Or
put another way, how can you design a web scraper that fetches data that are relevant for
answering your research questions? If you are not clear about what your understanding
of the variable “time spent on social media platform X” is, then you are not able to decide
what data the web scraper needs to fetch. In other words, you are not able to design a web
scraper that will produce relevant data for your study.
For example, a web scraper designed to fetch data about when and where people log on
to social media platform X, and for how long they are logged on, does not provide relevant
data about the variable “time spent on social media platform X” if your understanding of
that variable is that spending time means engaging with the content on that social media
platform (by liking, sharing, posting, reading others’ posts and so on).
Consequently, if you go ahead and put a web scraper into practice without being clear
about what data that scraper is supposed to fetch, then you cannot really know what the
data produced by that scraper represents measures of. In other words, you are not able to
assess whether the data produced by the scraper is relevant in terms of enabling credible
answers to your research question(s).
Example 2
Suppose your variable of interest is “learning-friendly classroom environments.” You
are especially interested in the aspect of that variable that is about peer abuse at schools.
Even more specifically, you are interested in how teachers follow up on peer abuse in
classrooms. Now suppose you decide to use standardized observations to collect your
data.
If you are not clear about what your understandings of the variables “peer abuse” and
“following up on peer abuse” are, then you will not be able to decide what to look for while
observing. In other words, you are not able to decide what to include in your observation
schedule and what to leave out.
216 Research Design
For example, assume your understanding of the variable “peer abuse” is clear, and you
therefore are able to decide what type of physical actions, spoken comments, or behaviors
toward peers to look for and register in your observation log while making your observations. If, when making your observations, there is an episode of peer abuse in the classroom, and that episode is overlooked by the teacher at the specific time it occurs, does that
mean the episode is not followed up on by the teacher?
To answer this question, you need to make clear your understanding of “followed up
on by the teacher.” If “follow up on” means taking action related to the peer abuse when
it takes place, then you will register overlooking an episode of peer abuse at the time it
actually occurred as not following up on it. However, it may be the case that the teacher’s
experience is that acting upon peer abuse in the heat of the moment just makes the situation worse in the long run. Therefore, the teacher takes note of the episode when it takes
place but does not take action related to it until later in the day. Does the action taken by
the teacher at this later point in time qualify as following up on the peer abuse that took
place in the classroom earlier in the day?
If you do not know exactly what qualifies as following up on, and what does not, then
you will not be able to define in your observation schedule what it is that you are supposed to
look for related to this variable of interest. If you go ahead and do your observations anyway,
you will not be able to precisely define what it is that you have measured related to following
up, because you are not clear what following up means when making those observations.
What Can We Conclude From These Examples?
What both of these examples demonstrate is the importance of being clear about how you understand the variables of interest in the study. They highlight that if you are not clear about what
your variable of interest actually is, then it is impossible for you to develop a way to measure
your variables of interest in a way that will make sure that what is being measured is what
you actually need to measure about them in order to answer your research question(s). If
you cannot be sure of what you actually have measured, then the data produced by those
measurements may not be relevant to your research questions at all. Therefore, the data
will not enable credible answers to those questions.
On the other hand, when you have made clear how you understand each of the variables included in your study, you will know exactly what it is about those variables that is
of interest to you. Therefore, you will be able to develop a way to measure those variables.
This is because when you know what it is about each variable that you are interested in,
you are able to decide what to measure about the variable in order for the measurements to
produce relevant pieces of numerical information.
In the next section, we take a closer look at the thinking you will need to do about your
variables of interest in order to make them measurable.
TAKING A CLOSER LOOK AT WHAT ENABLES
VARIABLES TO BE MEASURED
How do you turn a variable such as “motivation” or “achievement” into something that
you can measure that is related to your research question(s)? The first thing you will need
to think about, and then do, is identify what you see as relevant about that variable in
terms of what you will need to know to be able to answer your research questions. When
you decide upon a specific way of understanding a variable of interest, you are developing
what is called the construct for that variable that you will use in your study.
Chapter 9
•
Collecting Data Using Quantitative Methods
By a construct, we mean a specific way of elaborating
on, or understanding, an abstract concept (Black, 1999).
For example, we all have an intuitive feel of what friendship is, but to investigate the concept friendship, and treat
it as a variable in a specific study, we need to state explicitly
what we mean by friendship. This involves us identifying
the aspects, factors, or indicators we understand to be key
elements of friendship.
What you identify variables to be about, and what
it is about them that is relevant in your specific research
study, will be informed by the disciplinary and theoretical traditions in which the research you are designing is
embedded. Consequently, deciding on the constructs
for the variables of your study anchors your study to the
total body of knowledge related to the area of research to
which the study you are about to design will add even more
knowledge.
Therefore, assessing the credibility of how a variable
is understood, includes asking yourself questions such as
these:
• Will experts in my area of interest agree with my
217
FIGURE 9.1 ■ The
Process of Developing
Operational Definitions
From Abstract
Concepts
Variable
Often abstract, and
specific meaning is not
defined
Construct
The variable is refined,
and you develop a
specific understanding
of it
understanding of the variable?
• Are there other possible ways of understanding
this variable?
• Is my understanding of the variable congruent
with the dominant theoretical and research
traditions within my discipline?
When you identify quantifiable factors, or indicators,
of a construct and specify which quantifiable factors or
indicators to include in the measurement of the variable,
you are developing what is called an operational definition
of the variable (see Figure 9.1). It is the operational definition of a variable, informed by the construct of that variable, that enables collecting pieces of information about
that variable that are considered relevant to enable learning something about that variable.
Operational definition
Observable and
quantifiable factors used
to measure/describe/
identify
the construct
Source: This illustration is developed by us, but is inspired by
Black (1999).
TIP
NOT ALL VARIABLES LEND THEMSELVES TO MEASUREMENT
For some variables, identifying quantifiable factors that would make up an acceptable
and appropriate way of measuring that variable would be a huge, difficult, and possibly
endless endeavor. This is because “not all concepts lend themselves to measurement
218 Research Design
and consequently will not be good variables to try to quantify—for example effective
teaching, quality of life, and good research” (Black, 1999, p. 38).
Therefore, if you do find yourself identifying concepts such as “good research”
as the variables of interest for your study, you will need to take a step back to think
through your research questions and consider (1) whether your research questions
are well-enough developed, that is, what exactly is it you want to know about related
to “good research”; or (2) whether the variables you have identified are the ones most
relevant to answer those questions.
HOW TO MAKE ABSTRACT VARIABLES MEASURABLE
Understanding the variable → construct → operational definition process outlined in
Figure 9.1 at a conceptual level is one thing; however, it is quite another to put this process
into practice. To help you think about what this process entails in practice, in this section
we provide an example of developing an operational definition of a specific variable. To do
so we will use this exemplar research question:
How is friendship related to well-being among children with attention-deficit
hyperactivity disorder (ADHD)?
In this research question, the variables of interest are “friendship” and “well-being.”
To make these variables measurable, you will need to follow these steps:
Step 1: Clarify Your Understandings (i.e., Develop the Constructs) of the
Variables
To measure the variable “friendship,” you must start by clarifying your understanding of
friendship in the context of your research. For example, one understanding of friendship
draws on Aristotle’s idea of three kinds of friendship, namely utility (established because
the one you befriend is useful to you), pleasure (established because of the pleasure you
gain from the one you befriend), and the “perfect” friendship (established because of the
other person being who they are) (Pangle, 2003).
However, this is not the only possible understanding of the variable “friendship.” There
are many other ways that friendship has been conceptualized and therefore understood.
For example, friendship can be understood as “a close relationship between two children
that is mutual and reciprocal” (Mikami, 2010, p. 181). Within this understanding of what
friendship between children is, the variable “friendship” is not a uniform entity. Rather,
“friendships vary in quality, in stability, and in the adjustment of the friend” (Mikami,
2010, p. 182; see also Hartup, 1995, 1996). Therefore, this understanding of the variable
“friendship” identifies quality, stability, and adjustment as key factors, aspects, or indicators of friendship.
Consequently, what you will need to think about, and decide upon, is which understanding of friendship you will adopt in your study and why. When refining and developing a specific understanding of the abstract variable “friendship” (or an aspect of that
variable, such as the aspect “adjustment”), you are developing a construct of that variable
(or an aspect of that variable).
You will undertake the same process with the variable “well-being.” For example, your
interest in the area of the well-being of children with attention-deficit hyperactivity disorder (ADHD), and your reading of the literature related to that area, may suggest that
Chapter 9
•
Collecting Data Using Quantitative Methods
219
within the population of children with ADHD, peer rejection negatively impacts their
well-being. For example, you find a study that reported this finding: “Peer rejection, if left
unchecked, leads to a variety of problems . . . that ultimately place the child on a perhaps
irreversible negative trajectory” (Hoza, 2007, p. 660). Moreover, other studies suggest that
“stable, high-quality friendships [have the potential] to buffer” (Mikami, 2010, p. 181)
such negative outcomes. Given this, your understanding of “well-being” might focus on
effects of peer rejection.
Consequently, the constructs related to the variables “friendship” and “well-being” for
your study could be (1) that quality, stability, and adjustment are understood as the key
elements of “friendship”; and (2) that “well-being” is understood in the context of peer
rejection.
Step 2: Identify Quantifiable Factors That Will Enable Measuring the
Constructs
To enable measuring the concepts or variables of interest (in this case “friendship” and
“well-being”), you will need to develop operational definitions from the constructs of those
variables. In other words, you will need to identify quantifiable factors, or indicators, of
the variables “friendship” and “well-being” and specify which quantifiable factors or indicators to include in your study.
The construct of the variable “friendship” discussed previously in Step 1 above identifies quality, stability, and adjustment as the key elements of that variable. Therefore, to
develop an operational definition of this variable, you will need to specify the quantifiable
factors or indicators for each of these three key elements.
Once again, you will need to draw on the work of others when doing so. For example,
from the studies by Mikami (2010), and Gülay and Önder (2013), you will be able to identify the following factors or indicators of the element quality of friendship:
• validation (acknowledging another person’s emotions, values, thoughts, and so
on),
• care (showing kindness and concern for others),
• trust (believing that another person is honest and will not do anything
deliberately to harm you or put you at risk),
• conflict (strong disagreement with someone, or argue with someone, about
something important),
• antagonism (hostility or aggression between two people), and
• competition (when two or more people are trying to get something which not
everyone can get).
Therefore, the six quantifiable factors above will make up the part of the operational
definition of friendship covering the key element quality of friendship.
Whether the above list of six quantifiable factors of the key element quality of the
variable “friendship” is suitable in terms of enabling valid research findings to emerge
from the data produced by including these factors in the measurement of the variable
“friendship,” depends. What it depends on is how well the list covers all relevant factors of
the element quality.
220 Research Design
Assessing whether the factors on the list cover the content of the key element quality
is done by comparing the list to the disciplinary and theoretical traditions in which your
study is based. If peers and experts within the disciplinary and theoretical tradition in
which your study is embedded accept these quantifiable factors as necessary and sufficient
for measuring the key element quality of the variable “friendship,” then this part of the
operational definition of the variable “friendship” is assessed as contributing to the validity of the research. You will see this understanding of validity being referred to as content
validity.
Making sure that the quantifiable factors included in an operational definition of
a variable cover all relevant elements of that variable is important because, if a factor is
missing in an operational definition, something will be missing in the data produced
by the measurement items developed from that incomplete operational definition.
In turn, this will lead to outcomes from statistical analyses performed on those data
which do not tell the whole story about the variable in question. Therefore, any claims
about the variable in question based on such outcomes are not well-founded (i.e., the
validity is threatened).
For example, suppose you only include validation, care, and trust as the quantifiable
factors used to measure quality of friendship among children with ADHD. Then if you
conclude that the quality of friendship has the potential to increase well-being among
those children, you implicitly ignore any potential negative effect on well-being from
conflicts, antagonism, and competition (the other factors or indicators of the element
quality of friendship) among children with ADHD. In other words, the claim you make
about how quality of friendship relates to well-being among children with ADHD is
based on data covering only half of the indicators of quality of a friendship that may
affect well-being.
Once you have identified the quantifiable factors related to quality, you will then follow the same process to identify quantifiable factors related to the other key elements,
namely stability and adjustment, that you have identified as being part of the construct of
the variable “friendship.”
In Figure 9.2 below, we provide a summary of quantifiable factors of all three key
elements of the variable “friendship” according to the construct of that variable discussed
previously in Step 1 above.
However, even when you have done all this, you still have more thinking to do. This is
because to be able to answer the research question How does friendship relate to well-being
among children with ADHD? you will need to develop an operational definition for the
variable “well-being” as well. You will recall that you already have decided on a construct
for the variable “well-being”—namely that the variable “well-being” is understood in the
context of peer rejection. Therefore, the operational definition of well-being would be
developed by identifying quantifiable factors enabling peer rejection to be measured.
Developing Measurement Items to Actually Measure the Variables
When you have developed operational definitions for each of the variables included in
your study, you are ready to move on to a more operational level. This is because simply
developing operational definitions enabling the variables of your study to be measured is
not enough. You still need to actually develop the items that will, when combined, make
up the measurement instrument of your study. Or you will need to find an existing measurement instrument that is in line with the operational definitions you have developed
Chapter 9
•
Collecting Data Using Quantitative Methods
221
FIGURE 9.2 ■ The Process of Developing Operational Definitions From the
Abstract Variable “Friendship”
Note:
The Example Is Developed From the Ideas of Mikami (2010), and Gülay and Önder (2013)
for the constructs of your variables of interest. It is that measurement instrument you will
put into practice when actually collecting the data.
In the next part of the chapter, we take a closer look at what you need to think about
when developing, and putting into practice, a measurement instrument that will enable
relevant data to be collected.
Activity
Deciding on Whether to Design Your Measurement Instrument
Yourself or Adopt One Developed by Others
In many cases, there are existing measurement instruments that can be used to
measure specific concepts relevant for your research questions. For example, if your
research question involves the variable “depression,” you will find many existing
instruments for measuring aspects of depression.4
Find a measurement instrument that has been developed by others and that includes
items designed to measure at least one of the variables relevant to the research you are
about to design. Answer the following questions about that instrument:
1. Do the understandings and assumptions about the concepts in question match
the thinking underpinning my research project?
2. Does the instrument focus on the same aspects of the concept in question as do I?
222 Research Design
3. Is it clear how the concept being measured by the instrument is operationally
defined?
4. Does the instrument measure all quantifiable factors of the concept, as
those factors are defined by the operational definition underpinning the
instrument?
If, based on the answers you gave to the above questions, you now consider adopting
that instrument, or specific items from it, you will need to be able to justify that such
an adoption is congruent with theoretical and disciplinary traditions in which the
research you are about to design is embedded.
If you are unable to find an instrument developed to measure at least one of the
variables relevant to the research you are about to design or you are unable to justify
adopting the instrument you have found, you know you will have to develop a measurement instrument yourself.
DEVELOPING A MEASUREMENT INSTRUMENT
The aim of any measurement instrument is to enable relevant and credible data to be collected that can be analyzed in a statistically reasonable way and provide you with the information you need to answer your research questions. This means that when developing the
measurement instrument for your study, you will need to think about
• how to develop items in that measurement instrument that measure what you
intend them to measure,
• whether or not responding to the measurement item(s) demands too much from
the respondents and therefore may not provide accurate data,
• whether the items in your measurement instrument enable consistent
measurements of the quantifiable aspects they are intended to measure,
• whether the form that a measurement item takes is in keeping with what that
item is supposed to measure.
In the discussion to follow, we take a closer look at what you will need to think about
related to each of these points.
Developing Measurement Items That Measure What You Intend Them
to Measure
As we have seen, when developing a measurement instrument to use in a specific research
study, you will need to be clear about which quantifiable factors you need to include measurements of in that measurement instrument. However, even when you have done this, it
is not always clear how to phrase an item that actually measures those specific quantifiable
factors. You will need to word each item in your measurement instrument carefully so that
you can be sure that each item measures what it is supposed to measure.
Chapter 9
•
Collecting Data Using Quantitative Methods
223
How do you do this? To answer this, we will return to the example we discussed earlier5 —about making the abstract variable “friendship” measurable.
In this example, quality, stability and adjustment were understood as the key elements
of the variable “friendship,” and one of the quantifiable factors making up the key element
quality was validation (acknowledging another person’s emotions, values, thoughts, and
so on). So how do you phrase an item to measure validation?
One way that you might do this is to design the following item:
Please rate the degree to which you agree with the following statement:
“My friend does not mind that I sometimes get angry”
1
I do not
agree at
all
2
3
4
I
moderately
agree
5
6
7
I fully
agree
However, what you will need to think about is if someone responds that they fully
agree that their friend does not mind that they sometimes get angry (i.e., 7 on the scale),
does that actually indicate that when giving this rating the person was thinking about
validation, that is acknowledging their friend’s emotions, values, and thoughts? Could
it be that what this item measures is not about the factor validation at all, but something
else? For example, the factor care (showing kindness and concern for others)? You will have
no way of knowing this. Or put another way, you will have no way of knowing if you have
measured what you set out to—the factor validation.
The result of not knowing if you have measured what you set out to measure is that you
have no way of assessing the construct validity of the measurement instrument, that is, the
degree to which a measurement instrument measures what it is supposed or expected to
measure. Construct validity is central to establishing the overall validity of the research
you are about to design.
If items in your measurement instrument do not measure what they are supposed
to measure about the variables of the study, no statistical analysis procedure using data
obtained from those flawed measures will enable credible research findings about the variables of interest to be achieved. This applies even when the data produced by putting the
measurement instrument into practice is of the type that allows those statistical analysis
procedures to be performed.
To make sure that the items intended to measure a specific variable (or aspect of that
variable) actually measure that (aspect of the) variable, you will also have to think carefully
about the response options you provide for answering those questions. These response
options need to be in keeping with how you have operationally defined that variable. For
example, to be able to measure the factor validation, it must be possible to choose response
options that are commonly agreed upon by fellow experts as being relevant to the factor
validation in relation to quality, where quality is understood as one of three key elements
of the variable “friendship.”
The response options you offer for a specific item affects whether peers and experts
within the disciplinary and theoretical tradition in which your study is embedded see the
224 Research Design
item as a good (or bad) measurement item for measuring the variable (or aspect of a variable) that the item is supposed to measure.
Do Your Measurement Items Demand Too Much From the
Respondents?
Tourangeau and Bradburn (2010) suggest that several cognitive processes are initiated
when a person fully engages in responding to an item in a measurement instrument such
as a questionnaire. First, the item is interpreted, and then the respondent deduces its
intent. Once respondents have made up their minds about what the item is about, they
search their memory for relevant information and then merge that information into a
single response. These cognitive processes occur in loops and evolve as each respondent
engages in a type of mental conversation with the researcher, via the measurement items.
At the opposite end from full cognitive engagement, you find respondents just ticking
a box, or scribbling down an answer, with little or no motivation other than just finishing
the measurement instrument (usually as quickly as possible).
What the items are about, as well as how they are phrased, may affect whether a participant is willing, or even able, to undertake the cognitive burden of providing an accurate, honest, or truthful response to a question or item in a measurement instrument.
Consequently, what a question is about, and how it is phrased, may affect the credibility of
the research findings based on the data produced from people responding to that question.
For example, when faced with a question that is hard to interpret, such as the question
How many university-level statistics courses have you taken? the respondent is likely to be
more inclined to choose the tick-a-box option and move on. This is because the respondent might not want to engage in interpreting what “taken a course” means. Does it mean
courses passed? Or does it include courses taken, but failed? Do courses with no credit
(such as prelecture summer courses) count? Or do they have time to actually work out how
many courses they have taken if, for example, courses were taken a long time ago or they
have taken many of them?
Moreover, a respondent may just tick boxes to finish the measurement instrument in
the shortest possible time because they are frustrated by not understanding what is being
asked, or simply frustrated by the number of items they have to respond to. In such cases,
you really cannot know whether those responses reflect the thoughts or actions or knowledge or attitudes of that respondent. Such non-correspondence between the information
you get in the form of data and what you will conclude about your variables based on that
information will threaten the validity of those conclusions.
TIP
PRETESTING THE MEASUREMENT INSTRUMENT TO ENABLE
COLLECTING (MORE) CREDIBLE DATA
To avoid the problematic situation of starting the main data collection, and then realizing that the measurement items in themselves or the measurement instrument as a
whole represent measurement problems, we strongly suggest pretesting.
If you have developed (parts of) your measurement instrument yourself, and that
instrument has never been used to collect data before, the individual items need to
be pretested for “clarity, ambiguity, and difficulty in responding to.” Moreover, the
Chapter 9
•
Collecting Data Using Quantitative Methods
225
complete set of measurement items (i.e., the measurement instrument) needs to be
tested for “length, and for time and difficulty to complete” (both quotes from Punch,
2003, p. 34). Even smaller scale pretesting on family and friends can provide valuable
feedback to make sure your measurement items work as intended. For example, pretesting the item
Select the option that best represents your opinion about the statement
“The criminal justice system is performing well”
1
2
3
4
5
6
Strongly
disagree
Moderately
disagree
Mildly
disagree
Mildly
agree
Moderately
agree
Strongly
agree
may include asking a group of people (1) what “the criminal justice system” means to
them, (2) what they see as characterizing a well-performing criminal justice system,
as well as (3) how they arrive at a response to the item. This will help you understand
the cognitive processes that the respondents undertake to arrive at a response to a
specific measurement item—and such an understanding will help you write items
enabling more credible data to be collected.
If you are now wondering whether pretesting is necessary, consider the following
hypothetical situation: Assume that you, after having received responses from 50 of the
500 people you have asked to participate in your research, realize that the item about
the criminal justice system above is not clear and therefore needs to be changed to
Select the option that best represents your opinion about the statement
“The courts are free of improper government influence”6
1
2
3
4
5
6
Strongly
disagree
Moderately
disagree
Mildly
disagree
Mildly
agree
Moderately
agree
Strongly
agree
The problem with changing an item when data collection has already started is that
you will have no way of knowing whether the data from the first 50 respondents related
to this item represents the same understanding of what the item is meant to measure
compared to data from respondents completing this item after you have changed it.
Then you must either
•
discard the data from the first 50 respondents (which is potentially a serious threat
to the credibility of your research, especially if you have limited resources and
therefore do not have the opportunity to plan for a sample so big that +/- 50 respondents does not matter much) or
•
keep the data from the first 50 respondents, and risk muddling up data that represent one measure of the variable in question with data representing a different
measure of that variable. This is potentially a serious threat to the credibility of your
research, for reasons we have emphasized throughout this chapter so far.
Consequently, while pretesting the measurement instrument will take some time,
in the end it will be time well spent.
Pretesting is one way to avoid, or at the very least minimize, bumps along the way
when putting your study design into practice. Other things can go wrong, too. The
key point is to seek help as soon as you become aware that not everything is going
as planned—for example, you are not getting the data you need (type or amount) to
answer your research question(s).
226 Research Design
Will the Measurement Items Enable Consistent Measurements of the
Quantifiable Aspects They Are Intended to Measure?
If an item is phrased in such a way that the interpretation of that item varies across the
group of respondents, you cannot really know whether you have measured what you
intended to by including that specific item in your measurement instrument. In other
words, that item does not serve as a consistent measure of the variable in question (or a
quantifiable factor of that variable). Consistency of measures allows comparing like with
like when statistically analyzing your data. The consistency of measures is related to the
validity of the measures and therefore to the validity of the research. In textbooks about
quantitative research or quantitative methods, you will see the degree of consistency across
multiple measures of the same construct referred to as the reliability of the measurement.
For example, you may expect that all students enrolled in a specific data science study
program at a specific university will respond consistently to the question How many
university-level statistics courses have you taken? This is because being enrolled in the
same study program means they all have the same course plan, and therefore they all have
taken the same statistics courses.
However, a student who has previously taken statistics courses elsewhere will probably respond with the total number of statistics courses taken in total—not just the ones
included in the course plan for the specific data science study program that the student is
presently enrolled in. Moreover, some students may include courses taken, but failed, in
their responses, while others include passed courses only. Therefore, the question does not
serve as a consistent measure.
To turn the question into a more consistent measure, you should consider rephrasing
it. For example, the question How many statistics courses have you taken and passed as
part of the study program you are currently enrolled in? could prove to be a more consistent measure of what you actually want to measure compared to the original version of the
question (How many university-level statistics courses have you taken?).
PUTTING IT INTO PRACTICE
DISTINGUISHING BETWEEN RELIABILITY OF MEASURES
AND VALIDITY OF MEASURES
Suppose Question X and Question Y are meant to measure the same variable (or aspect
of that variable). If Question X or Y or both do not serve as consistent measures of what
those questions intend to measure, a respondent may produce an answer to Question
X that is inconsistent with the answer given to Question Y.
For example, you may expect that students enrolled at a specific university, in a
specific data science study program that has a set number of statistics courses, will
respond consistently to these questions:
1. How many statistics courses have you taken as part of the study program you are
currently enrolled in?
2. What is your total course credit from statistics courses taken as part of the study
program you are currently enrolled in?
However, as pointed out earlier in this section, some students may include courses
taken, but failed, in their responses to Question 1, while others may include passed
courses only. Moreover, in Question 2, only courses passed will be included, as courses
Chapter 9
•
Collecting Data Using Quantitative Methods
227
not passed do not give any course credit. Therefore, their answer to Question 1 will not
necessarily be in line with their answer to Question 2. Consequently, the questions do
not serve as consistent measures of how many semester hours a student has spent (or
is supposed to have spent) on statistics.
However, even if a question is phrased in such a way that all respondents do interpret it in the same way (i.e., the measure is reliable), but for some reason they do not
interpret it in the way you intended, then the measurement still will not measure what
it is supposed to measure—what you wanted to know about. Rather, it will measure
what the respondents all interpreted the question to be about. Therefore, any measures obtained related to this reliable measurement item are not valid in relation to
you being able to make claims related to your research questions based on them.
Hence a measurement can be reliable but not necessarily valid.
On the other hand, if the measurement is not reliable, it cannot be valid. For example, if a question is phrased in such a way that different people interpret it in different
ways, you do not really know what that question measures and therefore the measurement is not valid.
From all this we can conclude the following:
• A measurement must be reliable to be valid.
• But just because it is reliable doesn’t necessarily make it valid.
Is the Form a Measurement Item Takes in Keeping With What That
Item Is Supposed to Measure?
For each item comprising the measurement instrument you are about to develop, you
must decide what form that item will take. An item may take the form of including a set
of predefined response options. These are known as closed-response items (or closed-ended
questions in cases when the item is a question) because they come with a set of predefined
response options that close off the range of responses that can be given to an item.
Examples of such closed-response items are the following:
1. Select the option that best represents your opinion about the statement
“The criminal justice system is performing well.”
1
2
3
4
5
6
Strongly
disagree
Moderately
disagree
Mildly
disagree
Mildly
agree
Moderately
agree
Strongly
agree
2. What is your eye color? Select one of the options: 1. brown, 2. hazel, 3. blue, 4.
green, 5. gray, 6. amber.
Other items may take the form of an open-response item (or open-ended question in
cases when the item is a question). Open-response items are phrased in such a way that the
respondent is free to provide whichever response that respondent sees as appropriate. An
example of an open-response item is, What is the most important issue for you that the
COVID-19 pandemic has raised? Answers to this open-ended question will be in the form
of text.
This text will need to be analyzed, and the outcomes emerging from that analysis
must be converted to and recorded as numerical data to enable statistical analysis of them.
228 Research Design
To do this will usually involve some form of deductive coding, where the researcher looks
specifically for examples of predefined codes (typically listed in some form of codebook)
in the textual answer provided by the respondent. The thinking behind this way of coding is much different from inductive coding strategies that are most often associated with
qualitative research.7
The open-ended question What is the most important issue for you that the
COVID-19 pandemic has raised? could have been posed as a close-ended question by providing a set response option based on issues related to the COVID-19 pandemic you have
identified from your reading of literature related to this question. For example,
What is the most important issue for you that the COVID-19 pandemic has
raised? Select one of the following options:
1. Ensuring vaccine supply
2. Unemployment
3. Loss of freedom
4. The need for strong leadership by governments
5. I don’t know
Then you can only get information about those five possible responses. Therefore,
what you will need to think about when deciding whether to use forms of open or closed
items in your measurement instrument—or a combination of both—is that the choice
you make about the form that each item takes will affect what type of information you are
able to get by asking that question in that way.
This means that thinking about the choice between open-response and closed-response
items is not just a matter of preference. Rather, the thinking concerns whether the measurement instrument is “any good as a measure of the characteristic it is interpreted to assess”
(Messick, 1980, p. 1012). In other words, choosing which form an item will take affects
the degree to which that item measures what it is supposed to measure. Consequently, the
form that an item takes affects the validity of the measure (and therefore the validity of the
research and the research findings).
TIP
A REMINDER: HOW THE RESPONSE OPTIONS YOU OFFER RELATE
TO YOUR RESEARCH BEING STATISTICALLY REASONABLE
What form the response options to a question take affects whether the research is
statistically reasonable (see Chapter 8). This is because the responses need to be able
to produce data of a type that enables performing statistical analyses that will produce
the kind of knowledge needed to answer the research question.
For example, if you need to know something about the respondents’ age, you could
choose to offer the options (youth, adult, senior) for an item meant to measure age.
Then that item will produce ordinal data (because youths are younger than adults, and
adults are younger than seniors). This data will not enable you to calculate average
ages of subgroups of people, for example, subgroups defined by the geographical area
Chapter 9
•
Collecting Data Using Quantitative Methods
229
in which people live. Therefore, if you need to know the average ages across subgroups
to be able to answer your research question(s), you will not have the type of data that
you need using these response options.
Takeaways From This Section
After reading this section, you should understand how and why the validity of your
research is affected by how your measurement instrument is designed. This includes how
each item making up that measurement instrument itself is designed. How each measurement item is designed includes the wording, content, and form of each item.
To ensure the validity of your research when designing the measurement instrument
that will be used in that research, and the items that comprise it, you will need to ensure
that
• how you develop items in your measurement instrument results in them being
able to measure what you intend them to measure,
• responding to an item does not demand too much from the respondent (if it does,
that item may not provide accurate data),
• the items in your measurement instrument enable consistent measurements of
the quantifiable aspects they are intended to measure,
• the form that an item takes is in keeping with what that item is supposed to
measure.
However, even if you have addressed all of these issues, you are only halfway there in
terms of the thinking that you will need to do related to putting your measurement instrument into practice. This is because it is one thing to have developed a credible and valid
measurement instrument such as a questionnaire, but it is quite another to actually put it
into practice. In the next section, we take a look at what you will need to think about when
deciding how you put that measurement instrument into practice—that is, actually use it
to collect your data.
PUTTING A MEASUREMENT INSTRUMENT INTO PRACTICE
On its own, a measurement instrument will not produce any data. It is only when that
instrument is put into practice as part of a data collection procedure that the actual data
are produced.
To explore what you will need to think about when putting your measurement instrument into practice, we use the example of developing and implementing a measurement
instrument in some form of survey research. We chose this example because surveying is a
frequently used method in quantitative research. In fact, survey research “is one of the pillars in social science research in the twenty-first century” (Stockemer, 2019, p. 26).
The measurement instrument in a quantitative survey is usually, but not always, some
sort of questionnaire made up of individual measurement items. It is when that questionnaire is distributed to people requesting their responses, and when those people actually
230 Research Design
provide their answers to the items comprising that questionnaire, that the numerical data
of that research study is produced.
Therefore, it is important to think about how to optimally put into practice the collection of that numerical data. For example, how will you approach people to ask them to
participate in your research? If they agree, in what form will they receive that questionnaire: Will it be in paper form, electronic form, over the telephone, or face-to-face? When
they are filling in the questionnaire, will your items be understood by your participants,
or is the language you use in your items not clear enough? Have you considered how
long it might take your participants to complete that questionnaire? And throughout all
of this, have you thought about the ethical issues related to what is asked and how it is
asked?
Remember, no matter how well designed your questionnaire is, if you do not think
through these questions, you are unlikely to gain the credible data you need. In the rest of
this section, we take a closer look at what you will need to think about in order to address
these sorts of questions. We begin by looking at what you need to think about to optimize
the response rate of your survey.
Response Rates and How They Relate to Sampling and Validity
If enough people don’t respond to a request for them to answer a questionnaire, then
that questionnaire will not give you the numerical data you need. The proportion of
sampled individuals that are willing to provide information by responding to the measurement instrument that is part of a specific survey-based research study is referred
to as the response rate of that survey. Response rates matter because very low response
rates are more likely to cause your final study sample, that is, the group of people who
actually provide the data for the study (as opposed to the sample you originally planned
for) to be incongruent with the sampling strategy you initially designed, and that was
in line with your plan for arriving at statistically reasonable answers to your research
question(s).8
For example, a probability sample will secure approximate representativeness of the
study population, but only to a point. If the response rate is low, the final study sample
may be too small to capture the variability in the variables you intend to measure. In such
cases, even probability sampling will not enable research findings about the sample to
underpin statistically reasonable research findings about the study population. In other
words, if not enough people respond to ensure approximate representativeness of the study
population, then no matter how good the design is, technically speaking, the research will
not achieve the purpose it was designed for.
Therefore, the easiest way to avoid problems related to nonresponse is to make sure
that response rates are high across all subgroups of the study population relevant to the
research in question. This means that you will need a very well-thought-through sampling
strategy, able to optimize response rates, to form part of your research design.
Ways of improving response rates include
• providing advance letters or some other form of information to the people in your
sample announcing the survey and outlining what participating in it involves, or
• follow-up efforts such as reminders or multiple attempts to obtain responses to
motivate a potential respondent to actually respond.
Chapter 9
•
Collecting Data Using Quantitative Methods
231
Improving response rates might also include using incentives for participation.
However, it is important to recognize that there are ethical considerations you will need to
think through related to using incentives to motivate people to respond to a measurement
instrument. Such considerations are discussed on the box below.
PUTTING IT INTO PRACTICE
ETHICAL CONSIDERATIONS WHEN USING INCENTIVES FOR
PARTICIPATION
Sometimes, a token of gratitude in the form of some sort of payment or reward is
offered to potential respondents in return for participating in a study. However, this
can raise ethical issues that will need to be thought through.
Empirical evidence shows that prepaid monetary or nonmonetary incentives to
participate, such as cash, pens, and lottery tickets, do increase response rates (Singer
& Ye, 2013). However, while monetary and nonmonetary incentives can help with
recruitment of respondents for a survey and also act to motivate some respondents to
participate in the survey who otherwise might not (Singer & Bossarte, 2006), increasing recruitment and response rates does not outweigh the importance of thinking
about the ethical issues involved in using incentives.
For example, surveys on sensitive topics such as family or self-directed violence,
sexual abuse, underage substance use, and weapon possession, may cause psychological trauma or otherwise place respondents at risk or cause burden in their lives
(Schirmer, 2009; Singer & Bossarte, 2006). In such cases, “[i]ncentives are improper
when they are used to induce participation in the presence of avoidable or unreasonable risks. What is unethical in such a situation is not the use of incentives, but the
failure to protect against risk” (Singer & Bossarte, 2006, p. 416).
This is an example of why, in Chapter 2 of this book, we emphasized that thinking
about ethics should continue throughout the entire research process as opposed to
ethics being thought about as a static list of standardized steps to be followed that can
be made into a checklist and then ticked off as complete one by one.
To sum up, we have established that response rates matter in terms of you being able
to make statistically reasonable claims about the study population based on statistically
analyzing the data you collect. Therefore, when designing your research, you will need to
think through matters such as the way you will approach potential respondents and what
you will expect from them in terms of, for example how much time it takes to complete the
measurement instrument or the cognitive or emotional burden such responding puts on
the respondent.
Ways of Collecting Data When Doing Survey Research, and Why They
Matter
Another key consideration when putting the measurement instrument into practice, is
how you will actually collect the data. For example, data collection in a survey-based
research design can take various forms. The measurement instrument could be presented
to a participant verbally face-to-face, or by using the telephone. Or, a written measurement instrument could be distributed by mail or digitally on email. For example, participants of the study could receive an email containing a link to an online questionnaire.
232 Research Design
Thinking about, and choosing, a specific form that the data collection will take involves
assessing the feasibility of each option. For example, do you have the resources in terms of
time and money to do face-to-face interviewing? What will it take to get hold of the necessary information, such as email or postal addresses, to contact the people in your sample?
However, it is not just feasibility considerations that you will need to think about
when deciding how to collect your survey data. You will also need to consider and think
through what type of items you have included in your measurement instrument, whom
they are presented to—when, where, and how—and whether the way you have decided to
administer the measurement instrument is appropriate in terms of you obtaining credible
and reliable data.
We will illustrate why considering these issues is important in terms of you being able
to arrive at credible answers to your research questions by exploring three scenarios.
Scenario 1
Assume your study is about students’ attitudes toward cash-in-hand payments (i.e., payments for which no tax is paid) in cases of casual employment. You have developed a measurement instrument comprising items measuring aspects of the variable “attitude toward
cash-in-hand payment,” and the items are phrased using expressions such as “unofficial
work,” “under-the-table payment,” “untaxed payment,” and so on. Moreover, assume your
data collection procedure includes distributing paper copies of your measurement instrument to a group of financial law students during a lecture on tax revenue. You ask them to
return the completed measurement instrument before leaving the auditorium.
The problem here is that financial law students might be more prone to underreport positive attitudes toward cash-in-hand payment for work during a lecture on tax
revenue than if they had responded to the measurement instrument in an off-campus
context. This is because even though the measurement instrument is self-administered,
and anonymous, the students are likely to want to “look good against the background
of social norms and common values” (Stockemer, 2019, p. 41)—in this case the norms
and common values associated with being a financial law student. As such, the students
might think it is socially unacceptable to report truthfully about having a positive attitude
toward breaking laws related to work and paying taxes.
You will see this phenomenon, the tendency to depict oneself as conforming to social
norms, described as social desirability discussed in many textbooks. Social desirability is
one of the most common issues related to how the items comprising a measurement instrument can, when combined with a way of collecting data that does not take the nature of
the item into account, affect the credibility of research (Nederhof, 1985; Stockemer, 2019).
This is because, in this case, people do not answer accurately and truthfully the
questions comprising your measurement instrument. Instead, they answer what they
think they should answer, or what it is acceptable to answer in relation to that question.
Consequently, whatever you claim to know about your study population is based on estimates of population characteristics computed from numerical data about what people
thought they should answer in relation to the variables of the study.
These answers may, and probably will, not correspond well to what the actual situation is
that you claim to have learned something about. For example, you may find that the financial
law students in your sample have high levels of negative attitudes toward all factors associated
with cash-in-hand payments in cases of casual employment, yet in reality a number of them
are actually taking such payments to support their studies, and believe this is justified.
Chapter 9
•
Collecting Data Using Quantitative Methods
233
Scenario 2
Suppose you have decided that living in a residential aged care facility is an inclusion criterion for the study population of your study. You design a strategy for selecting who will be
the participants of your study, and make sure that the sample represents the study population well.9 Moreover, because an online survey is both cost and time efficient, you choose
to distribute an online self-administered questionnaire by way of sending a link by email.
The older people receiving the link will then access the questionnaire by clicking that link.
When a respondent has completed the questionnaire, their response will then be sent back
to you in digital form. In this way, the data is automatically ready to be fed into software
designed to perform statistical analyses. All good so far.
However, many elderly people living in a residential aged care facility may not have an
email address. And even if they do have an email address to which you can send the link they
are supposed to use, to access the questionnaire they would need access to a computer, as well
as the knowledge and health required to operate that computer. While some of the elderly
people in the residential aged care facility may be able to do this, many may not. In effect,
those who cannot do this will not be able to participate in the research even if they wanted to.
Consequently, only a fraction of the people in your study sample will have the opportunity to reply to the questions comprising the questionnaire. In other words, the response
rate (i.e., the proportion of sampled individuals that actually provide information by
responding to the questionnaire) is low. As we pointed out at the outset of this section
about putting your measurement instrument into practice, low response rates may pose a
serious threat to the validity of your research.
Scenario 3
Suppose you decide to collect data by way of asking people questions face-to-face, or by
telephone. In each case, there will be some form of interaction between you and the respondent. Consequently, the respondent has the opportunity, for example, to ask you to clarify
aspects of the questions being asked if needed. Such interaction will help avoid questions
being interpreted differently across the group of respondents. This will strengthen the
validity of your research.
On the other hand, the personal interaction between you and the respondent may
make the respondent feel uneasy about providing answers to certain types of questions,
for example, if your questionnaire includes questions that are perceived as judgmental
or questioning the respondent’s competence, questions about sexuality, questions about
abuse, or questions about political preferences. These are typical examples of questions
that may make participants prone to opt out of providing an answer when face-to-face or
telephone interviews are used simply because of the lack of possibility for the respondent
to remain completely anonymous. The more respondents who choose to opt out of providing answers to your questions, the lower the response rate. As we have already discussed,
low response rates represent a possible weakness in the validity of your research.
Therefore, when planning to ask people questions that may make people feel uneasy
in some way, distributing the questionnaire by mail or email may be a better option than
choosing the face-to-face (or telephone) option. This is because you will then be absent at
the time the respondents provide their responses to the questions making up that questionnaire, thereby maintaining their anonymity when responding.
However, the trade-off will be that when you are absent, “assistance cannot be provided, and there is nobody who can clarify unclear questions or words” (Stockemer, 2019,
234 Research Design
p. 65) should the respondent need it. You will need to make choices about this trade-off
in terms of what will matter more in terms of the credibility of the data that you collect.
TIP
PRETESTING THE ENTIRE DATA COLLECTION PROCEDURE
As highlighted previously in this chapter, pretesting the individual items that make
up the measurement instrument is part of ensuring the credibility of the research.
Pretesting also provides an opportunity to test the way you plan to put that measurement instrument into practice (i.e., test your plan for approaching people, how you plan
to actually collect the data, test any effect the use of incentives has on who decides to
participate in your study, and so on).
Of course, such pretesting requires that you do make up a test sample comprising
people that could just as well have been included in the “real” sample of your study,
approach them, and collect data from them, in the same way that you are planning
to approach and collect data from the people in the “real” sample. You do this to test
whether your plan for collecting data works as intended in your research design.
If you are a student designing a smaller scale study with a very limited time frame,
the extent of pretesting that you are able to achieve may be limited. This is a feasibility
consideration. However, any pretesting of the data collection process that you are able
to manage “is likely to help increase response rates” (Punch, 2003, p. 34), and therefore may potentially strengthen the credibility of your research.
Take-Home Messages From These Scenarios
The above scenarios illustrate that choosing how to go about putting some sort of measurement instrument into practice, and actually collect the data for your study, is not
as straightforward as it might seem. You will need to think through the effects of each
decision you make related to how you will put that instrument into practice and justify
them when you are designing your research. This is important, as decisions about how you
will go about to collect the data of your study may affect the credibility or validity of the
answers to your research questions.
TIP
TIPS FOR WHAT YOU NEED TO THINK ABOUT IF YOU ARE
CONSIDERING USING DATA COLLECTED BY OTHERS
Sometimes research is designed involving the reuse of existing data already collected
by others. While reusing data collected by others might be a tempting option in that
the data is already there waiting for you, reusing data comes with its own challenges.
Therefore, there is still much to think about if you choose to do this.
This is because when reusing data, it is important to remember that the researcher
who collected the data in the first place made a series of choices—the sum of which
led to those specific data being collected. If you use that data set in your own research
study, you will not only be using the numerical data comprising the dataset. Rather, you
will also be using the understandings of the variables which the data set is about, and
the way that those understandings were reflected in a measurement instrument and
the way that this measurement instrument was put into practice.
For example, when the measurement instrument used to collect a set of data takes
the form of a questionnaire, the researcher designing that questionnaire has actively
Chapter 9
•
Collecting Data Using Quantitative Methods
235
chosen which questions to ask, which not to ask, and how the questions are phrased.
Equally, an observation schedule is the outcome of choices made by the researcher
about what to look for, and record, when observing. When using measurement instruments such as web scrapers or technical devices, the researcher has chosen which
data the scraper will extract or what type of information the technical measurement
instruments will record.
Consequently, a dataset is not a neutral, value-free “thing” that you can simply
“pick up” and insert into your study. Should you decide to reuse data collected by somebody else, you adopt their operational definitions of the variables in question, as well as
the measurement instrument they developed based on those operational definitions.
Therefore, any weaknesses in the operational definitions developed by those who
collected the data will also be weaknesses in your research. So will any discrepancies
between the issues the research is designed to address, the measurement instrument used to measure the variables of interest to those issues, and the way that the
researcher went about to collect the data using that measurement instrument.
What this means is that if you are intending to use an existing data set in your
research design, you will need to be able to describe the operational definitions that
underpin the items making up the measurement instrument that was used to produce
the data. Moreover, it must be possible for you to acquire knowledge about how that
measurement instrument was put into practice. Only then will you be able to assess
the credibility of using that data in your study.
CONCLUSIONS
This conclusions section is in two parts. The first part is focused on conclusions specific
to the discussion in this chapter. The second part is wider in focus and is designed to put
together, and round off, the discussion that we have had about quantitative design related
considerations across both this chapter and the one before it (Chapter 8). This second
part is important as we have covered a lot of interconnected ground in these two chapters
about quantitative research design (Chapters 8 and 9) and therefore neither chapter can
be read in isolation. Consequently, the second part of the conclusion is about connecting
the dots between each of the chapters.
Part 1: Conclusions Related to This Chapter
This chapter has highlighted that putting into practice the collecting of the numerical
data that you will then base your statistical analyses on involves a series of interrelated decisions. To arrive at credible research findings by way of statistically analyzing
numerical data, that numerical data needs to represent relevant and credible information able to be used to answer the research questions you are interested in. This
involves identifying variables of interest related to those research questions and making those variables measurable. It is the measurements of these variables that provides
the numerical data you will apply statistically based analyses to, the interpretation of
which will provide you with the findings of your study.
Once you have decided what you will need to measure in order to answer your
research questions, you will then need to develop some sort of instrument able to take
those measurements. In quantitative survey research this instrument (usually a questionnaire) is made up of a set of measurement items related to aspects of the variables of interest. The way each item in the measurement instrument is worded is an
236 Research Design
important part of the validity of your research, as is the way that the measurement
instrument is administered.
Therefore, a key message in this chapter has been that making variables measurable,
developing items that will measure them, and then finding a way to put the measurement
into practice to actually collect the data requires a lot of careful thinking about a series
of interconnected aspects of both the design, and conduct, of the research. It is central to
the validity of the research. This is because if, for example, the items don’t measure what
they are supposed to, or the response rate is too low, then this will affect the validity of the
research design and the findings that arise from it.
Therefore, it is important to remember that simply obtaining numerical data about a
study population, or a sample drawn from a study population, does not ensure credible
research findings. What does make such findings credible is when the numerical data
represent relevant and credible information about whatever it is the research is being
designed and conducted to address.
Part 2: Putting It All Together—What Have We Learned From Chapters 8 and 9?
We have covered a lot of ground in the two chapters about quantitative research design
(Chapters 8 and 9). Much of this ground has been related to issues around ensuring that
your research design is statistically reasonable. Our discussion has highlighted that using
statistical procedures to analyze your data does not in themselves make your research
findings statistically reasonable. Used well, procedure(s) for statistically analyzing the
research data can increase the credibility of your research. However, used poorly, or in
an unthinking manner, they can reduce or even undermine that credibility. Therefore, in
Chapter 8 we emphasized
• what you need to think about to be able to decide what procedure(s) for statistically
analyzing the research data will enable credible answers to your research questions to
be deduced from the outcomes of that analysis, and
• what you need to think about to be able to make statistically reasonable claims about
a large group of people if your plan is to collect data from a relatively small subset of
that group—that is, a sample of that large group of people.
In addition, we have discussed the following design related matters focusing on enabling
the collection of relevant and credible data in this chapter, Chapter 9:
• what you need to think about to measure the variables of the study to make sure that
the data represents relevant information about the variables addressed by the research
question(s);
• what you need to think about to design a measurement instrument that, when put
into practice, enables collecting relevant and reliable data;
• what you need to think about when putting the measurement instrument into
practice and collecting data to make sure that the data produced are credible. This
includes (a) the type of measurement instrument you will use and why, (b) what
variables that instrument will measure and why, and (c) the way you will approach
the people who will provide the measurements and why.
Chapter 9
•
Collecting Data Using Quantitative Methods
237
What we hope to have made clear across these two chapters is that all of the above points
need to be taken into account in order to arrive at credible research findings when using
some form of quantitative approach. In a nutshell, when developing your research design,
you will need to
• identify and develop precise research questions,
• select statistically reasonable analysis procedures that will enable you to obtain the
information that you will need to know to answer those questions,
• design a sampling strategy and a measurement instrument when planning for how
you will obtain the numerical data you need for those analysis procedures,
• and then put that measurement instrument into practice and actually collect the data.
However, you are likely to find that what happens when you put your research design into
practice does not always go exactly according to your plan. For example, you may not be
able to practice probability sampling as you planned to do, because you find that you do
not have access to every member of the study population (and therefore, some members
of the study population are in fact excluded from participating in your study). Or you
may not be able to obtain a sample large enough to obtain enough data to fully meet the
requirements of the procedure(s) for statistically analyzing those data that you need to use
to be able to answer your research question(s).
Part of responsible research and research design is “showing your warts” (Gernsbacher,
2018, p. 403) about what you actually did, and acknowledge ways in which the compromises you have made affect how the findings of the research can be or might be or are
interpreted. When “showing your warts” (Gernsbacher, 2018, p. 403) about what you
actually did, you acknowledge any detours taken from the path planned for when putting the research design into action, as well as enable yourself and others to assess whether
those detours matter in terms of what, and about whom, you will be able to say something
(related to your research questions) at the end of the research.
In the next chapter, we continue our exploration of putting research methods into practice when designing research by taking a look at issues related to designing and conducting research using more than one method.
SUMMARY OF KEY POINTS
• When employing quantitative approaches in your research design, you will identify
your research questions, select statistically reasonable analysis procedures based
on what you will need to know about to answer those questions, design a sampling
strategy and a measurement instrument for obtaining the numerical data you need for
those analysis procedures, and then put that measurement instrument into practice
and actually collect the data.
• The degree of consistency across these aspects of the overall research design affects
whether the data obtained are likely to carry credible information about the variables
in question.
238 Research Design
• Therefore, the form that a data collection procedure takes must be congruent with the
plan for statistically analyzing the data produced by them, as well as congruent with
the group of people from whom you want to collect data.
• An important consideration when designing your research is to make sure that how
you collect your data enables you to obtain relevant and credible pieces of numerical
information able to underpin credible answers to your research questions.
• Research questions are often about abstract and complex concepts. Those concepts
are what make up the variables of interest in your study.
• Obtaining numerical data representing credible information about the issues of
interest requires making the often complex and abstract variables identified as
relevant for addressing the research question(s) measurable.
• When you decide upon a specific way of understanding a variable of interest, you are
developing what is called the construct for that variable that you will use in your study.
• You will use some form of measurement tool or instrument to obtain numerical data
about those variables in a reliable and valid way.
• The individual measurement items you develop related to the variables in your study
collectively make up the measurement instrument to be used in your study.
• If your measurement instrument does not measure what it is supposed to measure
about the variables of the study, no statistical analysis procedure will, in itself, enable
valid research findings about the variables of interest to be achieved.
• A fundamental issue you will have to think through is whether you will design your
measurement instrument from scratch or adopt one developed by others and the
implications of doing so.
• When using your chosen measurement instrument, you will need to ensure optimal
context specific ways of collecting data from your population of interest in order
to maximise the response rate for your study, and therefore the credibility of any
findings based on those responses.
• For each item comprising the measurement instrument, you must decide what form
that item will take. For example, closed- or open-response items.
• The validity of your research is affected by how your measurement instrument is
designed. This includes how each item making up that measurement instrument
itself is designed. How each measurement item is designed includes the wording,
content, and form of each item.
• Validity is a complex concept, because the term validity focuses on different things
depending on what we are talking about when using it.
• Types of validity include statistical validity, external validity, construct validity, and
content validity.
• A research design, or a research finding, is valid when it is credible, well founded,
reasonable, justifiable, and defensible.
Chapter 9
•
Collecting Data Using Quantitative Methods
239
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
construct (of a variable)
construct validity
content validity
measurement instrument
measurement item
operational definition of a variable
quantitative survey
reliability
response rate
social desirability
validity
SUPPLEMENTAL ACTIVITIES
1.
In light of the discussion in this chapter, we would like you to return to these questions
we asked you to think about, and write answers to, after you had read Chapter 5:
•
•
•
•
Why have I chosen a qualitative or quantitative approach for this study?
What type of qualitative or quantitative approach am I using and why?
What method(s) will I use to collect the qualitative or quantitative data and why?
Once I have decided that, what else will I need to decide in order to be able to
put those methods into practice?
The discussion in this chapter has identified additional things you will need to
think about when answering these questions when using a quantitative approach in
your research design.
If your research design uses a quantitative approach, fold back on your thinking
about these questions or go back to what you wrote. Now extend and develop that
thinking or writing in light of the discussions in this chapter about how to
•
•
•
•
select participants for your research,
obtain data from them,
operationalize your variables so that you can measure those variables of interest,
choose which statistical analysis procedures to use in your research.
At the end of this reflexive process, you will have developed a large part of the
methodology and methods sections of your research proposal in terms of what you
will do, how, and why.
2.
Imagine you have designed your research using some sort of quantitative approach.
You take this design to your supervisor or your fellow students to get their feedback
about what you have done. During the discussion they ask you a number of
questions that require you to justify the decisions that you have made. How would
you answer questions related to the appropriateness of
a.
b.
the operational definitions of the variables of the study you have decided to
use as part of your research design?
the data collection procedure you are planning to use? What features of
that data collection procedure would you identify as contributing to the
rigor of your design?
240 Research Design
c.
d.
e.
who you will include in your study—who they are and how many they are?
how you will actually collect the numerical data from people in your study?
How will this ensure you have optimized the chance of having a high
response rate?
the measurement instrument you have designed or decided to adopt?
FURTHER READINGS
Gorard, S. (2003). Quantitative methods in social science: The role of numbers made easy.
Continuum.
Nardi, P. M. (2018). Doing survey research: A guide to quantitative methods (4th ed.). Routledge.
Stockemer, D. (2019). Quantitative methods for the social sciences: A practical introduction with
examples in SPSS and Stata. Springer Nature.
NOTES
1. Participant here means a person from whom data is collected. You will also see participant used in qualitative research with a slightly different emphasis, namely as a
co-participant with the researcher in the research.
2. “A web scraper is a specialized tool designed to accurately and quickly extract data
from a web page” (scrapinghub, 2020). See also the discussion in Chapter 2 about the
OKCupid case.
3. See https://en.wikipedia.org/wiki/Eye_color
4. One example is the Beck Depression Inventory-II (Beck et al., 1996) used to screen for
depression, and to measure severity and behavioral manifestations of depression; a
second example is the Hamilton Depression Rating Scale (Hamilton, 1960) for individuals already diagnosed as suffering from depression, used to measure symptoms
of depression in individuals before, during, and after treatment. They are two among
many that are seen as relevant to the treatment of depression (American Psychological
Association, 2019).
5. See the section above, How to make abstract variables measurable.
6. We have discussed this example in more detail in the opening section of this chapter.
7. See Chapter 7.
8. See Chapter 8.
9. See the discussion of representative samples in Chapter 8.
10
DESIGNING RESEARCH
USING MIXED METHODS
PURPOSES AND GOALS OF THE CHAPTER
This chapter focuses on the type of reflexive thinking that underpins the development of a
mixed methods1 research design.
A key question that underpins the discussion is Why, and how, might using mixed
methods thinking when designing research enable you to better answer your research
problem or question(s) and contribute new knowledge to your area of interest? Or put
another way, how can we design mixed methods research in such a way that Greene’s
inspirational vision for mixing methods might be realized? This is a vision where
[t]he core meaning of mixing methods in social inquiry is to invite multiple mental models2 into the same inquiry space for purposes of respectful conversation,
dialogue, and learning from the other, toward a collective generation of better
understandings of the phenomena being studied (Greene, 2007, p. 13, footnote
not in original).
Therefore, as you are reading the chapter, we encourage you to think about the
following:
• What does “mixed methods” actually mean?
• Why use mixed methods in the first place?
• What do I need to think about in order to decide what I will mix, how, and why
when I design and then conduct my research?
• How can I navigate the diversity of thought that makes up the heterogenous
mixed methods field?
Asking these questions of yourself, and of your design, is an important part of developing a credible mixed methods design that can address the research problem that it is
designed to. The discussion in the chapter provides a guide, as well as tips, for how you
might reflexively think your way through questions such as these.
Consequently, in the discussion to follow, the focus is not on providing a quick and
simplified overview of “how to do” mixed methods. There has been a lot of writing in that
genre already. Rather, in this chapter the focus is on what you will need to think about
when choosing, and then using, a mixed methods research approach in your research
design.
241
242 Research Design
The goals of this chapter are to
• Explore different views about what mixed methods research is.
• Demonstrate how the view held by a researcher about what mixed methods
research is impacts on the way that they design their mixed methods study.
• Highlight some of the reasons that you might choose to use a mixed methods
approach when designing your research.
• Establish what is meant by priority, timing, and mixing when designing mixed
methods research.
• Illustrate types of basic mixed methods designs that reflect different decisions
related to the priority and timing of the components that make up that
design.
• Introduce accepted notations for describing different types of mixed methods
designs.
• Emphasize that mixing is an analytical concept that permeates the entire process
of designing mixed methods research. It is not simply a procedure or point in that
design.
• Highlight that matters related to mixing are first and foremost matters of
thinking about what will be mixed and why, after which considerations and
decisions about how that mixing might occur can be made.
• Demonstrate that there are a number of possible levels at, and ways in which,
mixing can be thought about or occur.
• Emphasize that what levels mixing occurs at, and what associated strategies
are used for such mixing at those levels, in any specific mixed methods study,
depends on the purpose for the mixed methods study in the first place.
• Provide concrete strategies for navigating the complex and contested field of
mixed methods research.
WHAT IS A MIXED METHODS RESEARCH APPROACH?
All understandings, and therefore definitions, of mixed methods research approaches
have in common that “something” related to two (or more) different research methods
or approaches is being “mixed” in some way at a “point or some point(s)” in a research
study to add “something more” to the study than using only one method would. However,
after this there are various understandings, and different emphases, about what the “something” being mixed is, what aspect of that “something” is “being mixed,” at what point in
a study that mixing occurs, as well as what the “something more” actually is that you will
gain by that mixing.
For example, you will find definitions of mixed methods that emphasize mixed
methods as a single study combining qualitative and quantitative research approaches.
However, you will also find definitions of mixed methods research that put the emphasis
Chapter 10
•
Designing Research Using Mixed Methods
243
on mixed methods as a single study where one of the methods is incomplete and cannot
stand alone. In this view, the single study does not necessarily have to comprise qualitative
and quantitative approaches in it.
In the next section, we take a closer look at these different emphases in the way mixed
methods is defined. To do so, we use brief snapshots of three mixed methods scholars’
work. These scholars differ in what they emphasize when thinking about what mixed
methods is, and is for.
Mixed Methods Research as Combining Qualitative and Quantitative
Research Approaches
The most common understanding of mixed methods is that it is a research approach
that uses and “mixes” in some way aspects of both qualitative and quantitative thinking and/or methods and/or data in the same study. An often-cited definition of
mixed methods research developed by Johnson et al. (2007) reflects this understanding: “Mixed methods research is the type of research in which a researcher or team
of researchers combines elements of qualitative and quantitative research approaches
(e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference
techniques) for the broad purposes of breadth and depth of understanding and corroboration” (p. 123).3
However, there are differences among mixed methods researchers in terms of what
they emphasize when designing research to combine in some way elements of qualitative
and quantitative approaches.
Snapshot 1: Mixed Methods as a Method That Combines Qualitative and
Quantitative Approaches
John Creswell, an influential scholar in mixed methods research, makes it clear that when
thinking about designing mixed methods research, his emphasis and understanding
of mixed methods is as a method (Creswell, 2015). Understanding mixed methods as a
method puts emphasis on developing detailed sets of procedures for the collection and
analysis of different types of data—quantitative and qualitative. For example, how that
data is collected, what form it must take, and the procedures that should be applied to that
data once it has been collected.
Such elaboration and development of procedures has resulted in typologies or categories of types of mixed methods designs. It is then possible for researchers to choose
one of these types of core mixed methods research designs and adapt it to meet the
purpose and address the questions underpinning the study (Creswell & Plano Clark,
2018).
This emphasis on methods, and procedures related to the data collected by those
methods, is reflected in the core characteristics of mixed methods that he and Plano Clark
(2018) identify, and state, “[A]dequately describe mixed methods research” (p. 5). These
characteristics are that mixed methods research
• collects and analyzes both qualitative and quantitative data rigorously in response
to research questions and hypotheses,
• integrates (or mixes or combines) the two forms of data and their results,
244 Research Design
• organizes these procedures into specific research designs that provide the logic
and procedures for conducting the study, and
• frames these procedures within theory and philosophy. (Creswell & Plano Clark,
2018, p. 5)
Snapshot 2: Mixed Methods as a Way of Thinking That Combines Aspects of
Qualitative and Quantitative Thought
Jennifer Greene, another prominent mixed methods scholar, agrees that mixed methods studies comprise qualitative and quantitative approaches. However, she does not put
the emphasis on methods and procedures in her understanding of what mixed methods
research is. Instead, she understands, and therefore places emphasis on, mixed methods as
a way of thinking.
Reflecting the complexity, and at times messiness, of social contexts, she emphasizes the importance in social inquiry of “engaging with difference” (Greene, 2002, p.
23). Such engagement applies to both the differences in what is being studied (i.e., the
complexity or messiness of a social setting or phenomena of interest) and the differences
in the ways that it can be studied (i.e., different research methods, methodologies, or
approaches).
This way of thinking about what mixed methods is, and is for, enables “possibilities of including not just more than one method in an inquiry study but also the
possibilities of including more than one methodology, more than one philosophical paradigm, as well as more than one discipline and substantive theory” (Greene, 2015, p.
607). It is a thinking driven by a desire to better capture the multidimensional reality
that we study.
Consequently, she views designing mixed methods research as a reflexive,
thinking-driven process. This is a process that enables multiple ways of thinking to enter
into the same inquiry space for purposes of respectful conversation, dialogue, and learning. It involves creative (not set or prescribed) ways of thinking. Thus, mixed methods
research is much more than a type of method or procedure. Rather, it is a whole of study
approach and a way of thinking that impacts all parts of the research design including
what will be studied, how, and why (Greene, 2015).
A Different Definition and View of What Mixed Methods Research Is
Some mixed methods scholars do not agree that a study must be comprised of at least one
qualitative and one quantitative component in order to qualify as mixed methods research.
Rather, they put the emphasis on mixed methods research and its design as a single study.
Therefore, they do not necessarily agree that the single study must have both a qualitative
and quantitative component in it in order to qualify as a mixed methods design.
Snapshot 3: Mixed Methods As a Single Study, Where One of the Methods Is
Incomplete and Cannot Stand Alone
Foundational scholar in mixed methods research, Janice Morse, views mixed methods
research as a single study—hence her use of the term mixed-method rather than mixed
methods. This mixed-method single study always has a core component and one or more
Chapter 10
•
Designing Research Using Mixed Methods
245
supplemental component(s). The core component is the primary method in the study
which “must be conducted to a standard of rigor, such that, if all else were to fail, it could
be published alone” (Morse & Niehaus, 2009, p. 23). In other words, the core component
is a complete method in its own right.
In contrast, the supplemental component(s) in a mixed-method study only makes
sense in terms of supplementing the core component in some way. This is because the
supplemental component(s) is in some way incomplete and cannot stand alone as a complete study in its own right. Using one or more supplemental components is what can
enable you to address a research problem or question in a fuller or more complete way.
This is because the results of the core component are enhanced in some way by the
results of the supplementary strategy(ies).
In this view, thinking about mixed methods is not restricted to only combinations
of qualitative and quantitative components (see, for example, Morse & Niehaus, 2009;
Morse, 2017; Morse & Cheek, 2014, 2015; Cheek & Morse, 2022). A mixed methods
study can comprise two or more qualitative methods or strategies or two or more quantitative methods or strategies.
TIP
BE AWARE: DIFFERENT WAYS OF UNDERSTANDING WHAT MIXED
METHODS RESEARCH IS IMPACTS UNDERSTANDINGS OF WHAT
MULTIPLE METHODS RESEARCH DESIGN IS
There is often confusion about the difference between mixed methods and multiple
methods (also known as multimethod research). Different understandings of mixed
methods research can lead to different understandings of what multiple methods
research is. For example,
•
If one of the defining features of mixed methods (and therefore mixed methods
design) is the use of some form of qualitative and quantitative methods, then a study
only using more than one quantitative method, or a study only using more than one
qualitative method, cannot by definition be a mixed methods study. Studies that use
two qualitative methods or two quantitative methods would be considered multiple
methods (Creswell, 2015). 4
•
If, as in Morse’s thinking, the defining feature of mixed-methods research is the
combining of one complete method and one incomplete method, then the use of
two qualitative methods, or two quantitative methods in the same study, can meet
her definition of mixed-method research as long as one of those methods is incomplete. However, designs comprising one complete quantitative and one complete
qualitative method would not meet her definition of mixed-method research but be
considered multiple methods (Morse, 2017).5
This highlights that the way a researcher thinks about what mixed methods
research is affects not only the way that they think about aspects of a mixed methods research design when developing it. It also affects the way that they think about
research designs using multiple methods.
It is also another example of why there is a lot of thinking to do when designing
research in this area as the terms are not stable and even the same term can mean
quite different things depending on the point of view of the researcher.
246 Research Design
What Can We Learn From These Snapshots About What Mixed
Methods Is?
These snapshots highlight (and remember these are only three of many possible snapshots
of scholars’ work that we could have used) that different mixed methods scholars, while
agreeing on some things, put different emphases on aspects of mixed methods research.
Consequently, there is not full agreement among scholars about what mixed methods is.
Therefore, when you are reading reports of mixed methods research studies, or a textbook about mixed methods research, you will need to think about the implicit understanding of mixed methods research that report or that textbook is premised on. This will
enable you to locate that textbook or report in a specific way of thinking about, and understanding of, mixed methods research. It will also help you navigate what at times can seem
to be a bewildering, and frankly overwhelming, array of terminology built up in this area.
There has been a lot of terminology related to the field of mixed methods research
developed in a relatively short time by researchers working in many different disciplines.
This has sometimes resulted in new terms for the same idea being developed in parallel—mostly unwittingly but not always.6 In such cases, we have ended up with more than
one term for the same idea or a (very) slightly different idea. And a situation in which “terminology, structure and even the principles—the rules—for what is good mixed-method
research, have become complicated, contradictory and contested” (Morse et al., 2018, p.
564; Johnson et al., 2007).
This situation has been compounded by the fact that some scholars may have changed
or modified the terms they use as their ideas, or the ideas in the field of mixed methods
research itself, developed, changed, or agreed upon terminology emerged.7 Of course, in
itself this is not necessarily a bad thing as it reflects the emergence of a field in development. However, what is not helpful is if the researcher does not provide insights into what
changes, and why, they have made over time to their mixed methods research thinking
and therefore terminology that they use. Much confusion can be created for the reader
new (or even not so new) to the field when no reason for, or even acknowledgment of, those
changes is given.
The take home message from all this is that when designing research using mixed
methods research, you will need to think through, make a decision about, and then
declare the understanding of mixed methods research that you will use to provide the
methodological and theoretical framework for that design. The development of the various areas that together make up that design, as well as the terms used to describe those
areas, will need to be consistent with that declared understanding. Otherwise, you may
find that your study becomes a muddle very quickly both at a conceptual level but also at
the level of the terminology you are using.
TIP
SOMETHING ELSE TO THINK ABOUT: IS MIXED METHODS REALLY A
NEW FIELD?
The fact that mixed methods research is a contested field still in development does
not mean that it is a new field (see Maxwell et al., 2015; Fetters, 2016). As Pelto notes
“[S]ome anthropologists and sociologists (and others) have used mixed methods in
Chapter 10
•
Designing Research Using Mixed Methods
247
field work for the past 80 years, and there are studies from early in the twentieth
century that clearly fall within the definition of ‘mixed methods’” (Pelto, 2015, p. 734).
Given this, he expresses surprise at some writers suggesting that mixed methods
research per se is a relatively recent development.8
However, what certainly is a relatively recent development is the exponential
growth of interest in the area of mixed methods research. This has been accompanied by an ever-increasing array of notations, terminologies, procedures, and types
of research designs related to mixed methods. This is reflected in the rapid proliferation of journals, books, textbooks, book chapters, journal articles, methods manuals,
workshops, conferences, and conference papers related to mixed methods.
This has led to claims that “mixed methods research has evolved to the point where
it is a separate methodological orientation with its own worldview, vocabulary, and
techniques” (Tashakkori & Teddlie, 2003, p. x).
WHY USE A MIXED METHODS RESEARCH DESIGN?
Mixed methods approaches are designed to provide in some way a better understanding
of what you are studying than only using a single method can. This is because “any given
approach to social inquiry is inevitably partial” (Greene, 2008, p. 20). Therefore, using
more than one research approach and way of thinking in a research design can
• provide more comprehensive and multidimensional understandings of complex
social settings and problems.
• allow for the use of different approaches to research, and the data those
approaches produce, to be mixed in some way to enable those comprehensive and
multidimensional understandings to emerge.
Given this, it follows that if you choose to use a mixed methods approach in your
study, then you have decided that mixing different types of information, perspectives,
or data about the particular issue or problem that is the focus of your study will in some
way assist you to address that issue or problem. This means that very early in the research
design process you will need to think through, and say something explicitly about, why
mixing methods in your study design will give you “more” (e.g., a better, fuller, or deeper
answer to your research questions) than using a single method.
Morse, Cheek, and Clark identify some of the reasons for using a mixed-method9
research design. These include the following:
1. Sequentially increasing the understanding of a phenomenon
2. Obtaining different perspectives on the same phenomenon
3. The phenomenon demands different approaches
4. Increasing complexity
5. Increasing scope; determining boundaries (Morse et al., 2018, p. 565)
Points 1 through 3 are usually planned at the outset of the project whereas Points 4 and
5 reflect that the study design may evolve as the research unfolds. For example, the results
of the study indicate that the problem you have set out to study may be more complex than
248 Research Design
you first thought, or the boundaries for what is to be studied are “more expansive than first
understood” (Morse et al., 2018, p. 567).
All five reasons are related in some way to providing a “better” answer to a specific
problem by incorporating different perspectives and methods to produce a more complete
picture or understanding of what is going on. For example, why people answered a survey
in the way that they did. In this case you might, for example, conduct a quantitative survey about the area of interest of your research and then interview people about particular
aspects of those survey findings in order to help explain them.
TIP
FOUNDATIONAL AND IMPORTANT WORK ON THE IDEA OF
PURPOSE IN RELATION TO MIXED METHODS RESEARCH
In 1989, Greene and her fellow researchers Caracelli and Graham developed one of the
first typologies of purposes of mixed methods. This typology classified types of mixed
methods studies by purpose. It identified five categories of purpose—triangulation,
complementarity, development, initiation, and expansion:
TRIANGULATION seeks convergence, corroboration, correspondence of results
from different methods.
COMPLEMENTARITY seeks elaboration, enhancement, illustration, clarification of
the results from one method with the results from the other method.
DEVELOPMENT seeks to use the results from one method to help develop or inform
the other method, where development is broadly construed to include sampling
and implementation, as well as measurement decisions.
INITIATION seeks the discovery of paradox and contradiction, new perspectives of
frameworks, the recasting of questions or results from one method with questions or results from the other method.
EXPANSION seeks to extend the breadth and range of inquiry by using different
methods for different inquiry components. (Greene et al., 1989, p. 259)
Since then, others have picked up this work and developed it (e.g., Bryman, 2006).
Therefore, you will see some variation in typologies of purpose for mixed methods
research when reading in this area.
The Importance of Thinking About Why You Might Use a Mixed
Methods Approach
What is absolutely crucial to think about from the outset of the design process is why
using mixed methods in some way will strengthen your research. Why will it provide
enhanced understandings of the problem or issue that your research is focused on and
being designed to address? Decisions about how methods might be mixed and why, must
be thought through and justified in relation to a specific piece of research. Just having more
or different types of data or methods does not necessarily, and in and of itself, give a better,
fuller, or deeper answer to your research problem or questions.
Therefore, whether having more, or different types of, data or methods will give a
better, fuller, or deeper answer to your research question depends. It depends on what the
purpose is of mixing one or more methods in some way, and how that purpose assists in
Chapter 10
•
Designing Research Using Mixed Methods
249
achieving the overall goal (purpose) of the research. An important part of mixed methods
research design is being able to justify how the design that you come up with (including
your choice of which methods will be mixed, when, and how) in that research will enable
you to achieve the overall goal of the research.
We develop this point in the next section of the chapter where we take a closer look at
the decisions you will need to make about the priority and timing of the components that
make up your mixed methods study.
PRIORITY AND TIMING OF THE COMPONENTS
IN A MIXED METHODS STUDY
There is broad agreement among mixed methods scholars that when designing mixed
methods research, consideration needs to be given to three key, and interconnected,
dimensions related to the components that make up the mixed methods study. You
will often see these dimensions, and the thinking about them that produced the mixed
methods design, referred to as considerations about the “priority, timing, and mixing”
(Creamer, 2018, p. 61) of the components.10
Thinking About Matters Related to Priority or Weighting of
Components
A notation system developed in 1991 by Morse, and subsequently developed by others, is commonly used when describing or writing about mixed methods and mixed methods research
designs. It uses the notations QUAL or qual to refer to qualitative components in that design
that draw on qualitatively derived logics of inquiry. The notations QUAN or quan are used to
refer to quantitative components that draw on quantitatively derived logics of inquiry.
The capitalized notation QUAL means that the overall study is a qualitatively driven
one in terms of its overriding methodological or philosophical emphasis. Likewise, the
notation QUAN indicates that the overall study is a quantitatively driven one in terms of
its overriding methodological or philosophical emphasis. Therefore, QUAN is used when
“the project is deductive, driven by an a priori theoretical framework” and QUAL when
the project is “driven by an inductive process and the theory developed qualitatively”
(both quotes from Morse, 1991, p. 121).
In other words, capitalization reflects the priority or weighting of the components
in the study in relation to the overall logic of inquiry that is shaping the mixed methods
study. Morse’s use of the noncapitalized and italicized qual or quan notation indicates that
that component is a supplementary one. Lesser priority or weighting is given to this component in the context of the overall design. When conducting the supplementary component, you adhere to the logic of inquiry underpinning that component. As Morse explains,
• “With QUAL-quan designs, you are thinking inductively overall and when doing
the core component, but you are thinking deductively when conducting the
supplemental component”
• “With QUAN-qual designs, you are thinking overall deductively to answer
your research aim and in the core component, but when you are working in
the supplemental component, you are thinking inductively” (both quotes from
Morse, 2017, p. 5)
250 Research Design
Creswell and Plano Clark (2011) identify three options when designing mixed methods research related to the priority given to the various components:
• “The two methods may have an equal priority so that both play an equally
important role in addressing the research problem” (Creswell & Plano Clark,
2011, p. 65)
Our Comment: In this case, the notation used would be QUAL and QUAN
to reflect different logics of inquiry, both of which are equally weighted in the
overall study.
• “The study may utilize a quantitative priority where a greater emphasis is placed
on the quantitative methods and the qualitative methods are used in a secondary
role” (Creswell & Plano Clark, 2011, p. 65).
Our Comment: In this case, QUAN would be capitalized and qual not, reflecting
the greater relative weighting of, and priority given to, quantitatively and
deductively derived logics of inquiry in the overall study design.
• “The study may utilize a qualitative priority where a greater emphasis is placed
on the qualitative methods and the quantitative methods are used in a secondary
role” (Creswell & Plano Clark, 2011, p. 65).
Our Comment: In this case, QUAL would be capitalized and quan not,
reflecting the greater relative weighting of, and priority given to qualitatively and
inductively derived logics of inquiry in the overall study design.
However, if we adopt Morse’s (2017) understanding of mixed methods as mixed-method
comprised of one component that is complete and one or more supplementary components
that are incomplete,11 then the first weighting option, QUAL and QUAN, identified by
Creswell and Plano Clark is not possible. This is because the supplemental component, and
therefore the strategies it uses, is by definition incomplete. It provides additional data that
can supplement in some way the findings of the core component and thereby extend the findings of the overall study. Therefore, it cannot be of equal weight or afforded the same priority as the core component. This means that the weighting option QUAN QUAL in the one
mixed-method study is not possible. QUAN QUAL would be a multiple methods study where
both components are full studies and the results of each of the studies can be compared to
build a picture about different aspects of a research problem area of interest.
What these different views about the possibility of having QUAL and QUAN weighting options in the same study highlights is a point that we have made previously, and that
we will continue to emphasize in this chapter. This is the point that what understanding
you have of what mixed methods is affects all parts of your thinking when designing your
mixed methods research. It is important that your design is congruent with those understandings. This means that when you are designing a mixed methods study, you will need
to make explicit what this understanding is and then make sure that the entire design is in
keeping with that understanding.
Thinking Through, and Deciding About, Matters Related to the Timing
of the Components
A second key consideration when thinking through the design of your mixed methods
study relates to when the various components that make up the design will be undertaken
in the study. This is a consideration related to the relative timing or pacing (Morse &
Chapter 10
•
Designing Research Using Mixed Methods
251
Niehaus, 2009) of the components. For example, you will need to think about whether the
components will be conducted at the same time or will one method/type of collection of
data follow the other, and why.
In the mixed methods research literature, a + sign is commonly used to indicate that the
components will be conducted at the same time (referred to as concurrently or simultaneously)
and an arrow → used to indicate that they will be conducted one after the other (referred to as
sequentially). Mixed methods designs are often drawn, named, and/or described using these
notations about pacing/timing as well as those related to weighting/priority.
Often, three types of “basic” or “core designs” for mixed methods research are identified. These are the following:
• The convergent design – “involves the separate collection and analysis of
quantitative and qualitative data. The intent is to merge the results of the
quantitative and qualitative data analyses” (Creswell, 2015, p. 36)
• The explanatory sequential design where the intent is to “begin with a quantitative
strand and then conduct a second qualitative strand to explain the quantitative
results” (Creswell, 2015, p. 38)
• The exploratory sequential design where the intent is to “first explore a problem
through qualitative data collection and analysis, develop an instrument or
intervention, and follow with a third quantitative phase” (Creswell, 2015, p. 39)
Figure 10.1 is a diagrammatic representation of these three types of core designs developed by Creswell and Plano Clark (2018) and reproduced with permission here.
FIGURE 10.1 ■
General Diagrams of the Three Core Designs
Note:
Taken from Creswell and Plano Clark (2018, p. 66). Reproduced with permission
On the other hand, Morse (2017; Morse & Niehaus, 2009) identify eight possible
types of basic mixed methods designs. These basic designs can be divided into two groups:
1. basic types of mixed methods research designs using qualitative and quantitative
components, and
252 Research Design
2. basic types of mixed methods research designs using two different qualitative
components or two different quantitative components.
Each of these groups is comprised of four possible basic designs. Morse explains:
Considering simultaneous (+) and sequential (→) pacing, and combinations of the
qual and quan supplemental projects, we have four basic mixed-method designs
that use both qualitative and quantitative methods:
QUAL + quan (Qualitatively-driven with a simultaneous quantitative supplement)
QUAL → quan (Qualitatively-driven with a sequential quantitative supplement)
QUAN + qual (Quantitatively-driven with a simultaneous qualitative supplement)
QUAN → qual (Quantitatively-driven with a sequential qualitative supplement)
(Morse, 2017, pp. 6-7)
She continues:
Further if we add two different methods that use two qualitative or two quantitative components, there are four more designs:
QUAL + qual (Qualitatively-driven with a simultaneous qualitative supplement)
QUAL → qual (Qualitatively-driven with a sequential qualitative supplement)
QUAN + quan (Quantitatively-driven with a simultaneous quantitative
supplement)
QUAN → quan (Quantitatively-driven with a sequential quantitative supplement)
(Morse, 2017, p. 7)
This way of grouping types of mixed methods research designs builds on, and is congruent with, Morse’s thinking about, and understanding of, mixed methods research—
a single mixed-method study comprising a core and incomplete supplementary
component(s).
The second group of four basic designs identified by Morse would not be considered
mixed methods by other mixed methods scholars if their understanding of mixed methods requires the use of both qualitative and quantitative components or strands in the
same study.
In line with Morse’s view of what mixed methods research is, the design QUAN +
QUAL does not appear in this list12 —however, other researchers would add this if in their
thinking about, and understanding of, mixed methods research, it is possible to give equal
priority to the components in the design.
This highlights that any typology or classification of mixed methods research
designs, not just the one by Morse, is built on assumptions about what mixed methods is, and what mixed methods are for. Any typology or classification of basic mixed
methods designs cannot be understood apart from the thinking and understanding
of mixed methods research that gave rise to it. If you copy and paste and/or “select” a
design from some sort of typology of mixed methods designs, you also copy, paste, and
select those understandings—therefore, you need to make sure that you know what
they are. How you design your research should then be in keeping with this view of
what mixed methods is.
Chapter 10
•
Designing Research Using Mixed Methods
253
TIP
NAVIGATING CHANGING USE OF TERMINOLOGY, AS WELL AS
VARIATIONS IN TERMINOLOGY USED FOR THE SAME IDEA
One of the problems that you no doubt will have when reading about basic or core
mixed methods designs is the changing use of terminology in the area as scholars’
thinking has developed rapidly over relatively short periods of time. For example, the
following table from Creswell and Plano Clark (2018) shows the changes and adjustments in their thinking about types of core mixed methods designs over time.
FIGURE 10.2 ■
Our Changing Typologies
Our 2003 Typology
Our 2007 Typology
[Creswell, Plano Clark, [Creswell & Plano
Gutmann, & Hanson,
Clark, 2007]
2003]
Our 2011 Typology
[Creswell & Plano
Clark, 2011]
Our Present Typology
of Core Designs
Sequential explanatory
Explanatory
design
Explanatory
sequential design
Explanatory
sequential design
Sequential exploratory
Exploratory
design
Exploratory
sequential design
Exploratory
sequential design
Sequential
transformative
Transformative
design
Concurrent
triangulation
Triangulation
design
Convergent
parallel design
Concurrent nested
Embedded design
Embedded design
Concurrent
transformative
Convergent design
Transformative
design
Multiphase design
Note:
Taken from Creswell and Plano Clark (2018, p. 59). Reproduced with permission.
Rapid and constant change in terminology and typologies, such as in Figure 10.2,
can make the field of mixed methods research very confusing at times. It is important
to make sure that you are aware of possible changes and adjustments such as these in
order not to be overwhelmed by the seemingly endless amount, and layers, of terminology that many scholars believe has made the field of mixed methods overly complicated.
It is also important that you are alert to the variations in terminology used for the
same idea (e.g., timing and pacing or priority and weighting) when reading a textbook
or a journal article about mixed methods research.
AN EXAMPLE OF HOW TO CONNECT PURPOSE,
PRIORITY, AND TIMING AND WHY THIS MATTERS
An important part of designing mixed methods is to establish why using mixed methods
in some way will strengthen your research. How will it provide enhanced understandings of the problem or issue that your research is focused on and being designed to
254 Research Design
address? Decisions about how methods might be mixed or how components might be
prioritized and paced can only be made if first the purpose for that mixing is clearly
understood. Deciding where, when, and how they will be brought together or interfaced
(Morse, 2017) in a mixed methods design depends on what the problem you are addressing is. As Greene (2008) points out, “[A] given mix of two methods, say structured
surveys and in-depth interviews, can accomplish several different mixed methods purposes, and thus be characterized by several different mixed methods design dimensions”
(Greene, 2008, p. 17).
For example, consider the following scenario:
• A researcher has been asked to design and conduct a research study for a local
government authority that is concerned about low COVID-19 vaccination rates
in that local government area. The local authority needs information able to be
used for developing strategies to optimize COVID-19 vaccine uptake in the area
that it is responsible for.
• The study purpose has been identified as to
1. identify barriers and enablers related to why people in the area decide or don’t
decide to be vaccinated and then
2. use this information to inform the development of strategies for optimizing
COVID-19 vaccine uptake in the geographical area.
• Mixed methods has been identified as a suitable research approach to be used.
This is because a mixed methods study design is well suited to this study as it
can provide a more complete picture of what is going on in this context related
to why people decide to vaccinate or not vaccinate. It can do this by combining
information from complementary kinds of data or sources.
However, the researcher still has a lot more thinking to do before they are in a position
to actually begin this mixed methods study. This is because there are different options
for how this study might be designed in relation to the priority and timing of the components (structured surveys and in-depth interviews) that will make up that study. Which
option the researcher chooses will depend on what they want to know more about related
to the purpose of the study. For example, consider the following options for designing
this study.
Option One: A QUAL → quan Design
The researcher decides that they need to know more about what the barriers and enablers
are in this specific context. To do this, they decide that they will use qualitative in-depth
interviews, the purpose of which is to find out how and why people in this geographical
area think, and then make decisions, about whether to be vaccinated or not. What are
participants’ perceived barriers or enablers related to deciding to be vaccinated? The analysis of the interviews will reveal a range of perceptions, understandings, and experiences
related to those barriers and enablers.
This range of perceptions, understandings, and experiences about barriers and
enablers can then be used to inform the development of items for a structured questionnaire forming part of a quantitative survey study. This will enable the researcher
to identify, among other things, how common these perceptions, understandings, and
Chapter 10
•
Designing Research Using Mixed Methods
255
experiences are in the local population of interest as well as the degree of importance particular segments of the population of this local geographic area attribute to each of these
characteristics.
Timing and Priority. The design of this study is QUAL → quan. The core component providing the overall inductive theoretical drive of the project is the in-depth
interviews (QUAL), the emergent findings of which are then supplemented by the quantitative data provided about the findings of the interview study using a quantitative
survey (quan).
Type of Basic Design. As such it is an example of a qualitatively driven (QUAL) mixed
methods project comprising a core QUAL component and a supplementary quan component designed to provide more information about the findings of that QUAL component.
It is a sequential mixed methods design: QUAL → quan.
Purpose:
• Producing a more complete picture of what is going on the context of interest
(perceived barriers or enablers related to deciding to be vaccinated in the local
government area) using inductive analysis of qualitative in-depth interviews to
identify what these barriers and enablers are perceived to be, and why
• Using supplementary data from the survey to identify the frequency of and
degree of importance particular segments of the population of this geographic
area attribute to each of these characteristics
• Enabling the development of targeted, derived from the ground up rather than
imposed, strategies to address the issues around vaccine uptake.
Option Two: A QUAN → qual Design
The researcher decides that they need to know more about what the barriers and enablers
are in this specific context. They decide that to find this out they will develop a closed
questionnaire for a quantitative survey that they will conduct in the geographical area.
They also decide that they will use variables and factors that have been identified in other
studies related to uptake or non-uptake of vaccinations to inform the items they will use
for the questionnaire. The analysis of the survey will provide a picture of the frequency
and distribution of those variables or factors throughout the geographic population of
interest.
To gain a better understanding of the reasons for this frequency and distribution, the
researcher conducts a series of in-depth interviews designed to probe and add qualitative dimensions to the findings of the survey. A purposive sample of participants for the
in-depth interviews is developed from information-rich people living in the geographical
context of interest. The interviews probe areas such as these: What is it about a specific
variable identified in the survey as affecting the rate of vaccination, for example, the perceived trustworthiness of sources of information about a vaccine, that affects the rate of
uptake of a vaccination? Why is this? How might issues identified as being related to this
variable be addressed?
Timing and Priority: The design of this study is QUAN → qual. The core component
providing the overall deductive theoretical drive of the project is the quantitative survey
(QUAN), the findings of which are then supplemented by the supplementary qualitative
interview data (qual) about the findings of that survey.
256 Research Design
As such, it is an example of a quantitatively driven (QUAN) mixed methods project
comprising a core QUAN component and a supplementary qual component designed to
provide more information about the findings of that QUAN component. It is a sequential
mixed methods design: QUAN → qual.
Purpose:
• producing a more complete picture of the information gained from the survey
alone by combining information gained from that survey with complementary
kinds of data or sources (the in-depth interviews)
• enabling development, further analysis, and refinement of the survey findings
using the contrasting kind of data gained from the in-depth interviews about
those survey results. This includes context specific rich and in-depth data about
the views and perceptions of people in the area of interest about the variables of
interest.
• enabling development, further analysis, and refinement of the survey itself for
future research purposes by gaining insights about how the items in the survey
questionnaire were understood or identifying other possible variables or items of
interest for that survey.
What Can We Learn From This Example?
This example reveals two different options for a mixed methods research design where
• both address the same issue (barriers and enablers related to why people in an area
decide or don’t decide to be vaccinated and then use this information to inform
the development of strategies for optimizing COVID-19 vaccine uptake in the
geographical area),
• both use the same methods (structured surveys and in-depth interviews),
• but each assigns different priority and timing to the structured surveys and the
in-depth interviews reflecting differences in purpose for the mixed methods
research being undertaken in each of the options. In option one, the structured
survey provides supplementary data for the interviews, whereas in option two the
interviews provide supplementary data for the structured survey.
The key point in all this is that priority and timing are not just simply decisions about
which type of data will be collected and when in a mixed methods study. Priority and timing related decisions are also based on thinking about why that data will be collected in the
way it is. How will collecting the different types of data in the way that you do help you
answer your research questions? What do you want those sets of data to be able to do or
add that just having one set of data cannot?
Considerations related to timing or pacing also require you to think about when and
where in the research design process the data and results related to the various components
of the study will be brought together, mixed, or integrated in some way. What will be
brought together and how? When and where will this happen in the research process? And
very important, how will this enable the production of “findings that are greater than the
sum of parts” (Woolley, 2009, p. 7)?
Chapter 10
•
Designing Research Using Mixed Methods
257
In the next section, we develop this point by taking a closer look at the idea of mixing,
which, along with priority and timing, is a central consideration when designing mixed
methods research.
Activity
Thinking About How to Report “Findings That Are Greater
Than the Sum of Parts”
If the aim of using a mixed methods approach is to enable the production of “findings that are greater than the sum of parts” (Woolley, 2009, p. 7), then is it possible
to report the components of a mixed methods study separately? This is an issue about
which you will come across very different viewpoints. Different journals may even
have different views and policies related to this.
Explore this issue further by considering the following questions. If possible, debate
them with other mixed methods researchers or students:
• Is “a helpful way to think about writing empirical articles for mixed meth-
ods research is to think about generating three written products from a single
study: a quantitative article, a qualitative article, and an overall mixed methods article” (Creswell, 2015, p. 92)? Why or why not?
• Is it sufficient to simply state that a written product is a report of one of the
components of a larger single mixed methods study? Why or why not?
• Why do you think that the journal Qualitative Health Research does not publish stand-alone reports of “the supplemental component from a quantitatively
driven study—QUAN-qual?” (Cheek, 2021, p. 1017; you might find it useful
and interesting to read the entire editorial from which this quote was taken).
MIXING—A CENTRAL CONSIDERATION
IN MIXED METHODS RESEARCH
Mixing is a reflexive process in which conceptual ideas, derived from different ways of
thinking, for example, data or methods, are woven and re-woven “into a meaningful pattern and a practically viable blueprint for generating better understanding of the social
phenomena being investigated” (Greene, 2007, p 16). The idea of “mixing” is central
to mixed methods thinking and a “part of the specialized language of mixed methods
researchers” (Creamer, 2018, p. 5).
Considerations, such as what will be mixed, when, and how in your study, are key
decisions when designing mixed methods research. Such mixing can occur at one or more
levels—“methods, methodologies, and paradigms—as well as substantive mixing across
disciplines and theories” (Greene, 2015, p. 607). This means that when you are designing
your mixed methods study, you will need to think about, and make explicit, at which of
these levels, and how, such mixing will occur.
For example, if your mixed methods study design comprises qualitative and quantitative components, you will need to consider what it is related to those qualitative and
quantitative components in the research design that will be mixed. Will it be data, methods, ways of thinking, logics of inquiry, all or some of the above? How, and why, will this
258 Research Design
choice of what is to be mixed enable you to better answer the question(s) driving your
research in the first place?
The same types of questions apply if you take the view that mixed-method research
involves the mixing of data obtained from one component that is complete in that study,
with data from another component that is incomplete in that study. How will the findings
from the supplemental component(s) be integrated with the findings of the core component? When and how in your design will the point(s) of interface between the two types of
data be? How will the integration of findings from the supplemental component(s) with
the findings of the core component enhance the result of the core component in some way,
and in turn provide findings that are greater than the sum of the parts?
“Mixing” as a Concept Not Just a Procedure
When designing mixed methods research, it is important to focus your thinking on mixing and what it involves throughout the entire process of designing your research. Mixing
is not a self-contained step or discrete procedure in a linear step-by-step mixed methods research design. Rather, the intention to “mix” (Creamer, 2018), or “mixing” of some
kind, is the thread that runs throughout your research design giving it coherence as a
single study. Put another way, mixing is a conceptual process, not simply a procedural one.
The point of interface in your research design is where the components making up
your research design are brought together in some way, usually (but not always) occurring
“once the analysis of each component is completed” (Morse, 2017, p. 9). However, simply
identifying the point(s) of interface of components in a mixed methods research design is
not what mixing is or what thinking about mixing is limited to. There is more thinking to
do about mixing than determining a single or predetermined point in your research design
where “mixing” happens.
For example, there is a lot of mixing or intention to mix related thinking that has
had to occur to get to that point of interface. This includes the thinking underpinning your choice of the components that will make up your design, as well as priority
and timing considerations related to them. Further, there is still a lot more mixing
related thinking to do after identifying that point of interface. For example, how will
the results of those interfacing components be synthesized, presented, and reported in
an integrated way.
This makes the question “Around what does the mixing happen?” (Greene, 2008, p.
17) an important one to consider from the outset, and throughout the process, of designing mixed methods research. There are different levels of focus at, and around, which mixing can occur. In the next section, we take a closer look at these levels.
Incorporating Different Levels of Focus Into Your Thinking About
Mixing
When designing your mixed methods study, mixing, as we have emphasized in the previous discussion, should not be thought of as, or occurring at, a static or fixed point. There
are various levels at, and ways in which, this mixing might occur. These multiple levels
include “different types of methods and data . . . different logic of inquiry, different ways
of knowing . . . different perspectives on understanding important social phenomena”
(Greene, 2015, p. 608). However, not all mixed methods studies will mix components of
the research within or across all these levels.
Chapter 10
•
Designing Research Using Mixed Methods
259
Which levels mixing occurs at in a specific study will depend on what the study is
being designed to do— the research problem and questions that the mixed methods study
is designed to address. As Greene notes,
This possibility of mixing at multiple levels is integral to the character of mixed
methods approaches to social inquiry, even though not all mixed methods studies
do or even should mix at multiple levels. That is, in some contexts a mix of methods alone is most justifiable, while in others a limited methods mix may represent
a missed opportunity for deeper insights (Greene, 2015, pp. 607–608).
What all this means is that there is not a recipe or set procedure to follow about what
should be mixed and at what level. Instead, in the context of your specific mixed methods
study, you will need to think through what these levels might be, and then make decisions
about at which of these level(s) mixing will occur, when designing that study. Explicitly
declaring what your decisions have been, and why you made them, is an important part
of ensuring the quality of a mixed methods study’s design, findings, and the inferences
derived from them (Greene, 2015).
Consequently, when designing and conducting mixed methods research, you will
need to think about how mixing will occur at a number of levels such as
• the overall project design (e.g., methods or components used),
• the data collected (e.g., textual and numeric data),
• the analyses of that data (e.g., the findings of each component). (Yin, 2006)
It also means that you will need to think about how within each one of these specific
areas of the project (design, data, and analyses) mixing will occur. Remember, you may
not necessarily need to employ mixing in all of these areas, depending on the purpose of
your study. Which specific mixing strategies you employ in your design, and in which
areas of that design, will depend on the purpose of your research, and in turn, the purpose
of collecting different types of data or employing different integration strategies related to
that data.
Therefore, matters related to mixing are first and foremost matters of thinking about
what will be mixed and why, after which considerations and decisions about how that
mixing might occur can be made.
Keeping your focus on mixing when designing and conducting mixed methods
research will enable the integrity of the single mixed methods study to be maintained and
not, as Yin (2006) put it, “decompose into two or more parallel studies” (p. 41).
What About Paradigmatic-Related Considerations When Mixing
Methods?
Some years ago, one of us was asked the following question by leading mixed methods
researcher Sharlene Hesse-Biber: “What do you think about researchers who say you can’t
mix quantitative and qualitative research methods because they arise from two different
paradigms and that’s not acceptable on philosophical grounds?” (Hesse-Biber, 2010b, p.
31, italics removed from original). This is a question that you will need to think about
when considering using mixed methods research. It is an important question about
inquiry paradigms13 and warrants some thinking time.
260 Research Design
Qualitative and quantitative approaches to research differ with respect to the methodological and onto-epistemological understandings embedded within them. Methods
derived from qualitative and quantitative approaches and methodological thinking cannot be viewed apart from the thinking that gave rise to those approaches in the first place.
Therefore, if we are going to mix qualitative and quantitative approaches in a single study,
we are also going to have to think through how to navigate the mixing of the different
assumptions and understandings implicit in qualitative and quantitative methodological
thinking.
When reading in the area you will find different views among scholars about the
answer to this question posed by Hesse-Biber. They include
Reply 1: Pragmatism as a possible alternate paradigm for mixed methods research.
Some mixed methods scholars suggest pragmatism is an alternate paradigm that
can enable us to move past the paradigm debates of the past. Pragmatism is a perspective that
sets aside debates about philosophy in favor of what works in a particular setting
or for a particular set of research questions. . . . Pragmatists approach research
with the purpose of producing something that will be both practical and useful
(Creamer, 2018, pp. 45–46, italics removed from original).
However, a word of caution is needed at this point. Phrases such as “what works”
should not be read as meaning “anything goes” when designing mixed methods
research employing pragmatism as a paradigmatic or organizing construct for that
study. As Greene notes, “[F]or practitioners to be meaningfully guided by an alternative paradigm [such as pragmatism], they must not only understand it but also understand just how it is intended to influence their methodological decisions” (Greene,
2008, p. 13). Therefore, she urges us to ask questions of our research design as we
think it through such as “How do the assumptions and stances of pragmatism influence inquiry decisions? . . . What does knowledge that integrates knowing and acting
look like and how is it validated?” (p. 13).
Reply 2: Take an a-paradigmatic stance.
Some scholars adopt what is sometimes referred to as an a-paradigmatic stance in relation to mixed methods research and what can be mixed. These “scholars see the epistemology–methods link as distracting or unnecessary and simply ignore it, continuing to work
as they always have worked, using whatever methods seem appropriate for the question
at hand” (Teddlie & Tashakkori, 2003, p. 18). In such a stance, while paradigms are useful for providing insights into, and contexts for, the thinking behind the methods and
methodologies that make up mixed methods studies, they do not determine the choice of
what mix of methods can be used in a particular study. What does is the question(s) being
asked.
Critics of the idea of an a-paradigmatic stance suggest that such a stance “sidesteps the
paradigm issue by ignoring it” (Hall, 2013, p. 73). They argue that all researchers make
decisions, such as what is data and how it can be interpreted, based on their paradigmatic
stance—even if they don’t acknowledge it. Therefore, while the
paradigmatic stance adopted by a researcher may not be made explicit . . . [t]his
does not mean that they don’t have one. . . . This means that the a-paradigmatic
Chapter 10
•
Designing Research Using Mixed Methods
261
stance . . . can’t be sustained as a viable approach to justify mixed methods research
since no research is paradigm free. (Hall, 2013, p. 73)
Reply 3: An incommensurable stance
Other scholars argue that it is not possible to mix paradigms. This view draws on the
foundational work by Lincoln and Guba (1985) and their idea of the “incommensurability” (Guba & Lincoln, 1994) or incompatibility of some paradigms such as post positivism
and constructivism where “the basic incompatibility of the two paradigms means that no
single research study can credibly embrace both” (Yin, 2015, p. 653). For example, the
contradictions between paradigms related to assumptions about causality and generalizability (Yin, 2015) or the nature of truth (Small, 2011, p. 77). Therefore, it is not possible
to mix paradigms in the same study. These scholars question whether a single mixed methods study design using methods derived from incompatible paradigms is actually possible.
Critics of this thinking term it “purist” thinking and argue that only very few methods
are linked to a single paradigm. They point out that qualitative approaches, the methods
they use, and the data they produce can draw on different paradigms. So can quantitative
approaches, the methods they use, and the data they produce (Creamer, 2018).
TIP
IF YOU WANT TO READ MORE IN THIS AREA
Greene developed a very useful table to reflect what various scholars “think about
the character, value, and role of traditional paradigmatic assumptions in mixed methods inquiry” (Greene, 2008, p. 11). She identified six positions that she called “Mixed
Methods ‘Paradigm Stance’” (Greene, 2008, p. 11). This provides a useful overview of,
and entry point for, the different things you will need to think about to get a sense of the
lie of the land in this area.
Summing Up Our Discussion of Mixing
What our discussion of mixing highlights is that like most other areas of mixed methods
research, the idea of “mixing” is an area about which there are different points of view and
emphases. However, there is at least one point that all mixed methods researchers agree
on. This is that simply having more than one method, component, or strand present in
our research, or collecting more than one type of data when “doing” our research, does not
necessarily mean that we have some sort of mixed methods study. This is the case no matter how well developed the procedures related to those methods, and the data collected,
are. In fact, Teddlie and Tashakkori (2011) consider “designs in which two types of data
are collected, but there is little or no integration of findings and inferences from the study”
(p. 294) as “quasi-mixed designs” (p. 293) rather than mixed methods research designs.
Rather, it is what those methods or that data are being used for, and why integrating them in the way that you do in your design enables you to better address the issue or
problem that the research is centered on, that makes your study a mixed methods one.
Therefore, mixing is “the key mechanism for the value-added of mixed methods research
and its potential to produce creative and coherent explanations and conclusions . . . which
increases the comprehensiveness, cohesiveness, robustness, and theoretical power of inferences
and conclusions” (Creamer, 2018, p. 82).
262 Research Design
STRATEGIES FOR NAVIGATING THE COMPLEX AND
CONTESTED FIELD OF MIXED METHODS RESEARCH
Given the range of views about most aspects of mixed methods research, designing mixed
methods research can seem a daunting task. In this section, we offer some strategies for
how you can navigate the large, complex, contested, and frankly at times, muddy terrain
of mixed methods research.
Strategy One: Use Diagramming (Not Just Diagrams) as a Way of
Keeping Track of the Decisions You Make About Your Emerging Mixed
Methods Design.
A useful way of keeping track of priority and timing in mixed methods designs is the use of
diagramming. Diagramming (Morse, 2017), as its name suggests, is the process of developing a diagram about, or representing in diagrammatic form, what you have thought
through, and then made decisions about, when designing your mixed methods research
study. The diagram produced is a static representation of this process of thinking and
therefore cannot be fully understood removed or decontextualized from the process of
diagramming giving rise to it.
Diagramming the process of designing your mixed methods research forces you to
think through the reasons for your decisions during that process and justify them—both
to yourself, and to others. It also helps you to track how your research actually is unfolding. Few, if any, research projects follow a set plan or procedure without at least some
modifications and changes—all of which need to be tracked, explicitly acknowledged,
and justified.
Diagramming may outline in overview the entire mixed methods research design. For
example, at the outset of the research diagramming can be used to
illustrate and communicate . . . how each component will be conducted—how
they will be paced and sampled, what type of data will be collected and analyzed,
and what findings are expected. It will show the location of the point of interface.
It may even show what the final results will look like, and may extend to dissemination. (Morse, 2017, p. 10)
However, remember this can change as the design unfolds. Therefore, diagramming
may also be used to focus on what happened when conducting the mixed methods project and indicate where modifications were made to the original design. In this way, diagramming can help both the researcher, and the reader of reports of the research, track
or follow what was happening throughout the mixed methods research process in terms
of design related decisions or modifications made—including when and why they were
made. Diagramming is thus a type of audit trail of the thinking and decision-making
throughout the mixed methods research process.
The key point here is that diagramming is an active, dynamic, and reflexive process
that arises from thinking about the active, dynamic, and reflexive process of designing
mixed methods research. Diagramming is part of the craft of designing mixed methods
research enabling you to track and make sense of the decisions you make about the shape
that your mixed methods design takes. It also enables others to follow the decisions that
you have made.
Chapter 10
•
Designing Research Using Mixed Methods
263
Strategy Two: Keep Folding Back Reflexively on Your Own Thinking,
and the Decisions You Are Making or Have Made, When Designing and
Reporting Your Mixed Methods Research.
At the heart of any process of navigating the mixed methods research-related terrain is
reflexive thinking. Any mixing of methods, purpose, data, or theories will only be as good
as the reflexive thinking work that you have done about that mixing. Therefore, it is somewhat surprising that there has not been a lot of writing about the reflexive thinking that
researchers actually did about how, and why, “mixing” of some kind added to or assisted
them in answering the research questions that their research was designed to address.
What led them to make the decisions that they did about their mixed methods design?
What did they base those decisions on? What struggles did they have along the way?
Published research reports and presentations about mixed methods research studies
tend to gloss over these struggles and how they were resolved. Instead, the mixed methods
study is presented as unfolding step by step exactly according to some preset plan or typology. Yet as all researchers know, no matter what approach to research is being employed,
a research design on paper, and actually putting that research plan into practice, are two
very different things.
This is because when making decisions about your research, both when designing and
then doing that research, you are putting “if then” thinking (Morse & Niehaus, 2009)
into action. For example,
If I ask this question, I will need to sample here from these folk, data will look like
this-and-that, I will analyze it this way, and at the end of the day I will know
thus-and-so . . . but I still would not have information about this or that. (Morse &
Niehaus, 2009, p. 78)
And if I don’t have information about this or that, will that matter? What will I gain
or lose by having or not having it? If I do add more components than originally planned in
some way to the research, how will the methods be integrated in some way and where in
my study design will that integration take place? Asking these types of questions requires
you to constantly fold back on your thinking and decisions in the iterative development of
your mixed methods research design.
PUTTING IT INTO PRACTICE
AN EXAMPLE OF PUTTING THIS STRATEGY INTO ACTION
One of us was involved in a study “designed to investigate the experiences of, and
effects on, a cohort of young adults who participated in a mindfulness training (MT)
curriculum at an elementary (primary) school in the mountain west region of the
United States in the 1990s” (Cheek et al., 2015, p. 752). The study design is summarized in Figure 10.3. It comprises two projects, one of which is a mixed methods
design.
While the figure looks neat and orderly, in fact the development of the design
required a great deal of thinking and reflexivity. At the end of the study, the team wrote
a paper specifically about putting such “if/then” thinking into action throughout the
264 Research Design
FIGURE 10.3 ■
Study Design
Overall programmatic research question: To explore the effects of
mindfulness training experienced by elementary school students
15 – 20 years ago
Note:
From Cheek, et al. (2015, p. 753). Reproduced With Permission.
process of developing this mixed methods research and how this affected the study
design. The article focused “on what we did, why we did it, what happened when we
did it, and what we have learned from all this in a specific research study” (Cheek et
al., 2015, p. 752). For example, there is a detailed description and then analysis of how
the team’s thinking about the study design changed as they thought through what they
really wanted to know about and why.
The team proposed that this sort of reflexive thinking constitutes a “‘second’ set of
findings” (Cheek et al, 2015, p. 754) in the study. The first set of findings relate to the
“more ‘traditional’ substantive results/findings/discussions . . . [offering] insights into
the experiences of the children in the classroom of participating in the MT curriculum,
as well as the effects of such participation at the time” (p. 754). The second set of
findings relate to “methodological and research design aspects of the study. These
findings provide new insights into mixed-method and multiple methods research
approaches derived from actually using these approaches in a project” (p. 754).
Therefore, when writing the article, and reporting this second set of findings, the
aim was to surface and highlight the dynamic and reflexive thinking which underpinned
the design of the study—thinking that “is not made explicit, and therefore remains hidden, in most research reports: an absence that this article seeks to redress by focusing on this ‘second’ set of findings” (Cheek et al., 2015, p. 754).
Exposing and analyzing this second set of findings can counter simplistic and
unthought-through assumptions about mixed methods research such as the argument that simply having more methods results in a better study or better results.
Distilling all this down to one key message, it is important to show how and why
designing and conducting your mixed methods study in the way that you did resulted in
richer data and stronger inferences.
Chapter 10
•
Designing Research Using Mixed Methods
265
Strategy Three: Don’t Go It Alone. Get Help and Find Support Along
the Way.
Teaming up with others could prove to be a good strategy to make sure you move forward in your thinking when navigating the complex field of mixed methods research.
Greenwood and Terry (2012) provide a unique reflexive account of how a group of
doctoral researchers from the field of nursing navigated and addressed the challenge of
“[d]esigning and implementing a mixed methods project” (Greenwood & Terry, 2012, p.
98). A reading group comprised of four students and two supervisors formed as a means of
“grappling with the methodological and design issues of mixed methods research” (p. 98).
Their account identifies “[f]ive principal challenges for novice researchers” (p. 100)
namely,
• Adopting and defending a mixed methods design
• Understanding the meaning of paradigm and its implication for the research project
• Identifying stages of mixing or integration in the individual projects
• Examining issues that threatened rigor
• Exploring and developing skills in “writing up” a mixed methods project (points
are taken from Table 2 of Greenwood & Terry, 2012, p. 100)
For each of these challenges they provide a description of the challenge and a possible way
forward produced by the collective reflexive thinking work done by that reading group.
For example, a way forward identified by the group in relation to the first challenge—
“adopting and defending a mixed methods design”—was as follows:
Perhaps this is not such an issue because mixed methods emerged to allow
research questions to be addressed with greater flexibility, and a predetermined
design strategy could inhibit future creative efforts that might fall outside of these
perspectives. . . . The group struggled with the notion that a research question should
“fit” into an established typology. Instead the group supports innovative and flexible
designs driven by the research question thereby removing self-imposed constraints
and thus allowing the research design to become an “enabler” rather than a “constrainer” for addressing real world problems. (Greenwood & Terry, 2012, p. 101)
In addition, based on their experience of being part of this reading group, the authors
provide an excellent reflexive account of considerations for forming a reading group, how
they managed that group including challenges that arose, and what made it work.
Both of these interconnected reflexive accounts—one focusing on the issues faced by
novice researchers entering the mixed methods terrain landscape, and one focusing on
how a reading group was used to navigate this terrain, provide “sign posts” (Greenwood &
Terry, 2012, p. 98) to assist others on their mixed methods journeys.
CONCLUSIONS
The discussion in this chapter has demonstrated that using mixed methods well in a
research study requires a lot of thinking. Thinking not just about how to mix methods in
some way, but also reflexive thinking about why to mix those methods in the first place,
266 Research Design
and why in this way and not that way. To think in this reflexive way, you will need to be
well versed in the approaches and methods that are being mixed, as well as well versed
in the body of knowledge about the idea of mixing methods itself. Consequently, when
thinking about how to structure, as well as where in the book to place, this discussion of
designing mixed methods research, we faced the same problem that Morse (2017) identified when she was teaching students about mixed methods.
My problem, when teaching mixed-method design, is that readers and students
are expected to know a lot about research at the start—to know and understand
many concepts, procedures, rules, and research principles-in-action, all at once
(Morse, 2017, p. xi).
This is why this chapter appears so late in this book—the next to last chapter. We needed
to have had other discussions first. The discussions in Chapters 4, 5, 6, 7, 8, and 9 needed
to precede this chapter.
What we hope you will have as a result of reading this chapter in conjunction with the other
chapters in this book, is a better understanding of the various layers of thinking needed
in order to make informed decisions about the type of “mixing” you will employ in your
research design and why. This is a type of thinking that will enable you to justify those
decisions and thereby add to the rigor and trustworthiness of your research. It is also a
type of thinking that can enable the potential offered by mixing methods to be met.
Throughout the chapter, we have aimed to open up rather than shut down thinking about
designing research using mixed methods. We did not want the chapter to default to what
some (actually many) discussions of mixed methods have become: versions of “the comparatively drab days in the middle of the previous century, when becoming a social inquirer
was a matter of learning the ‘the proper methods properly applied’ (Smith, 1983)” (Greene,
2015, p. 606). Mixed methods is a lively and alive field, not a drab and prosaic one!
The type of reflexive thinking the chapter advocates is part of responsible research design
and being a responsible and reflexive methodologist (Kuntz, 2015). It requires us to think
about why, and then (and only then) how we will use mixed methods so that our design
is more than a set of techniques. For as Greene reminds us, “Responsible social inquiry
today requires thoughtful consideration of methodology’s multiple layers, well beyond
the technical layer” (Greene, 2015, p. 606). Thinking how to do this and then designing
research around those considerations is what designing research mixing more than one
method in some way is all about.
SUMMARY OF KEY POINTS
• There are various understandings, and different emphases, about what a mixed
methods research approach is.
• One understanding is that a mixed methods research approach is a method that
combines qualitative and quantitative approaches in one study.
• Another understanding is that employing a mixed methods research approach means
employing a way of thinking that combines aspects of qualitative and quantitative
thought.
Chapter 10
•
Designing Research Using Mixed Methods
267
• A third understanding is that a mixed methods research approach is an
approach where (1) more than one method is used in a single study, (2) where
one method is complete but the other method(s) is incomplete in some way, and
(3) those methods do not have to comprise both a qualitative and a quantitative
component.
• If you choose to use a mixed methods approach in your study, then you will need
to think through, and say something explicitly about, why mixing methods in your
study design will give you “more” (e.g., a better, fuller, or deeper answer to your
research questions) than using a single method.
• Just having two methods present in a study, or collecting two types of data, does not
necessarily make your study mixed or your study design a mixed methods one.
• When designing mixed methods research you need to think about (1) the priority
given to the components in the study in relation to the overall logic of inquiry that is
shaping the mixed methods study; (2) when the various components that make up the
design will be undertaken in the study—timing; and (3) what will be mixed, when,
and how in your study—mixing.
• Different basic research designs for mixed methods research reflect differences in the
timing and priority of the components making up that research.
• Mixing is an analytical concept that permeates the entire process of designing mixed
methods research: it is not simply a procedure or point in that design.
• There are a number of possible levels at which mixing can be thought about or occur,
as well as strategies associated with that mixing at those levels. This can range from
incorporating findings from one component that builds onto a second component
to integrating or weaving two components such that one informs the other. The
bringing together of qualitative and quantitative components can also take place
across the research process through mixing of methodologies, data collection
methods, and mixing of analytical and interpretive techniques.
• What levels mixing occurs at, and what associated strategies are used for such mixing
at those levels, in any specific mixed methods study depends on the purpose for the
mixed methods study in the first place.
• Aids that can assist you navigate this complex research approach include
diagramming, folding back reflexively on your own thinking, and finding support
along the way when designing your research.
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
capitalization
core component
diagramming
mixed methods research approach
mixing
notation systems
point of interface
priority or weighting of components
qualitative component
quantitative component
quasi-mixed designs
supplemental component
timing or pacing
268 Research Design
SUPPLEMENTAL ACTIVITIES
1.
Putting diagramming into practice.
Find a research article reporting what is described as a mixed methods study.
After reading it, try diagramming the study to provide an overview of the design of
the mixed methods project.
OR
If you are in the process of either designing, or conducting, a mixed methods
study, try diagramming the study design you have come up with or are developing.
When doing so,
• at the top of the diagram indicate whether it is a qualitatively or quantitatively
driven design.
• Next identify the components that make up the research design.
• Indicate the priority and timing of the components.
• Use the notations QUAL/ QUAN; qual/quan; +/→ when doing so
(remember there may be more than one supplementary component).
• Identify the type of data produced by each component and what methods or
strategies were used to do so.
• Indicate where the point(s) of interface of the results of each component in
the study are in the research design.
If the article you choose already has a diagram, then work backwards to match
what is represented in the diagram with what is reported in the text of the article.
If you are having trouble finding an article, try using
Leung, D. Y., Kumlien, C., Bish, M., Carlson, E., Chan, P. S., & Chan,
E. A. (2021). Using internationalization-at-home activities to enhance the cultural
awareness of health and social science research students:
A mixed-method study. Nurse Education Today, 100, Article 104851. https://doi.
org/10.1016/j.nedt.2021.104851
2.
If you are in the process of designing research, and are considering using a mixed
methods approach in that design, write down as precisely as you can
a.
b.
c.
Why using more than one method will enable you obtaining better, fuller,
or deeper answers to your research questions than using a single method.
What understanding you have of what mixed methods is.
How that understanding affects different aspects of your research design
If you are not in the process of designing research, think of a topic that
you might consider using a mixed methods research approach for investigating.
Thinking about that topic, answer (a)–(c) above.
Addressing point (a) will provide the rationale for using a mixed methods
approach in your research design in the first place. Addressing point (b) will enable
both you and the readers of your work to locate your study in a specific way of
thinking about, and understanding of, mixed methods research. Making clear
what your understanding of mixed methods research is is imperative in order to
Chapter 10
•
Designing Research Using Mixed Methods
269
be able to design research that is congruent with that understanding of mixed
methods research and for making sure that the entire design is in keeping with that
understanding. Therefore, being able to address (b) is a necessary prior condition to
addressing point (c).
When you have clear statements about (a)–(c) above, continue your writing by
answering the following questions:
d.
e.
f.
g.
h.
What are the methods you will use in your design? Why those and not
others?
What priority and weight will each method be given, and why?
What is the timing of using one method in relation to using the other
method?
How would choosing a design with different priority and timing affect the
answers to your research questions enabled by the research design? Why is
this alternative design not the one you have decided to employ?
What will be mixed, when, and how in your study?
Answering (d)–(h) will help you develop a research design for a mixed methods study
that will not “decompose into two or more parallel studies” (Yin, 2006, p. 41).
FURTHER READINGS
Creamer, E. G. (2018) An introduction to fully integrated mixed methods research. SAGE.
Creswell, J. W., & Plano Clark, V. L. (2018) Designing and conducting mixed methods research
(3rd ed.). SAGE.
Greene, J. C. (2007) Mixed methods in social inquiry. John Wiley.
Morse, J. M., & Niehaus, L. (2009) Mixed method design: Principles and procedures. Left Coast
Press.
NOTES
1. We will use the terms mixed methods and mixed methods research throughout this chapter as overarching generic terms. When discussing specific understandings of mixed
methods research that use a specific form of referring to mixed methods in line with
those understandings, we will use that form of the term, for example, when the term
mixed-method is used to reflect a specific understanding of mixed methods—see
Snapshot 3 in this chapter.
2. Greene defines mental model this way: “[A] mental model is a set of assumptions,
understandings, predispositions, and values and beliefs with which a social inquirer
approaches his or her work. A mental model includes inquirer stances, values, beliefs,
disciplinary understandings, past experiences, and practical wisdom” (Greene, 2007, p.
53). In other words, a mental model is an explanation of someone’s thinking about how
something works in the real world, and can help set an approach to solving problems:
Mental models are personal, internal representations of external reality that
people use to interact with the world around them. They are constructed by
individuals based on their unique life experiences, perceptions, and understandings of the world. Mental models are used to reason and make decisions
and can be the basis of individual behaviors. They provide the mechanism
through which new information is filtered and stored (Jones et al., 2011, p. 1).
270 Research Design
3. Creswell (2011) describes this as a “composite definition for mixed methods based on 19
definitions provided by 21 highly published mixed methods researchers” (p. 271).
4. Creswell explains: “Mixed methods further is not simply the collection of multiple forms
of qualitative data (e.g., interviews and observations), nor the collection of multiple
types of quantitative data (e.g., survey data, experimental data). It involves the collection, analysis and integration of both quantitative and qualitative data. . . . When multiple
forms of qualitative data (or multiple forms of quantitative data) are collected, the term
is multimethod research, not mixed methods research” (Creswell, 2015, p. 3).
5. Morse explains: “Some researchers who used mixed methods (note: pluralized with no
hyphen) are actually referring to two research methods conducted simultaneously or
sequentially—and although they use the same name (‘mixed methods’), they are actually
conducting multiple methods. . . . Their way of doing mixed methods is considerably easier than doing a mixed-method designed study, because each method (or component) is
complete (capable of being published alone)” (Morse, 2017, p. 4).
6. Hesse-Biber (2010b, p. 210) describes this as a “gold-rush mentality.”
7. For example, in 1989, Greene and her colleagues refer to mixed-method evaluation
designs (Greene et al., 1989) but later work by Greene uses the form mixed methods.
8. For example, Creswell´s description in 2015 of mixed methods research “(a)s a field of
methodology about 25 years old” (Creswell, 2015, p. 1).
9. While Morse, Cheek, and Clark (2018) identified these as reasons for introducing a qualitative component, either as a core or a supplemental component, into a mixed-method
study, the same type of reasons apply to the introduction of quantitative component as a
core or a supplemental component into a mixed-method study. Note also the terminology that they use (mixed-method), which gives you insight into what understanding of
mixed methods they are employing.
10. However, as with the idea of mixed methods, you will soon find that different scholars use different terms when describing these dimensions or have different ideas or
emphases related to them. While some of the alternate forms of words used on the surface may appear to be just different names for the same idea, this may not always be
the case. Often these terms reflect different understandings of, or emphases about,
aspects of mixed methods research.
11. See Snapshot 3 in the section “What is a mixed methods research approach” earlier in
this chapter.
12. See previous discussion of this in Snapshot 3 of this chapter. Different understandings of
mixed methods also result in different understandings of multiple methods.
13. You will remember from Chapter 4 that all researchers approach “the world with a set
of ideas, a framework (theory, ontology) that specifies a set of questions (epistemology) that he or she then examines in specific ways (methodology, analysis)” (Denzin &
Lincoln, 2005a, p. 21). This set of ideas, or inquiry paradigm, define “for inquirers what it
is they are about, and what falls within and outside the limits of legitimate inquiry” (Guba
& Lincoln, 1994, p. 108).
11
WHY KNOWING AND
DECLARING YOUR
RESEARCH DESIGN
HAND MATTERS
PURPOSE AND GOALS OF THE CHAPTER
The purpose of this chapter is to round off, and consolidate, the key points we have made
about the process of designing research throughout the various chapters of this book. To
do this, we use the idea of declaring the cards in your research hand as the vehicle for the
discussion in the chapter.
When designing our research, we hold many possible types of research design–related
“cards” in our hand. These are cards such as preferences we might have for, or assumptions
we might make about, particular types of methods, methodologies, or research questions.
These are assumptions such as what a “good” or “better” way of designing and conducting research is. Often, they are made on the basis of the disciplinary and methodological
traditions we have come into contact with, or been trained in.
Recognizing these preferences and assumptions, where they come from, and how they
affect the choices and decisions we make when designing our research is important. Such
preferences and assumptions affect the way we think about all aspects of our research
design and influence the design related decisions we make when developing it. Knowing
what these preferences and assumptions are, and what effect that they have had on our
thinking, is central to being able to justify what we did, and why, when designing our
research.
Therefore, the choices and decisions that you make, and which collectively make up
your research design, need to be declared. They also need to be justified in terms of establishing the credibility of that design. To do this requires a great deal of careful reflexive
thinking work on your part.
However, what these decisions were, and what the basis for them was, is often missing
from reports of research. It is hard to find examples of the form such reporting might take.
Therefore, the chapter has a practical edge to it. It explores how you might actually write
about, declare, and justify the decisions you made that gave rise to your research design.
Or put another way, capture the thinking that gave rise to it.
At the end of the chapter, because it is the final chapter in the book, we provide two
sets of related conclusions which differ in their level of focus. The first set rounds off our
discussion in this chapter about the idea of declaring your research hand and how this
is linked to establishing the credibility of your research design. The second set takes the
form of overall key points that have emerged about designing research throughout the
entire book. Read in conjunction with the key point summaries at the end of each of the
individual chapters in the book, this list of overarching key points provides you with a
good overview of what we have covered in the book.
271
272 Research Design
The red thread holding the chapter (and the book) together is the type of thinking you
will need to do throughout the entire process of designing your research, as well as when
putting that design into practice and when reporting the findings that the design enabled.
The goals of this chapter are to
• Highlight that the choices and decisions that collectively make up your research
design need to be declared.
• Establish that like actual playing cards, in themselves these research design–
related cards are neither necessarily good or bad, or more or less suitable for a
research design. Rather, it is what they are used for, and how they fit with each
of the other cards in your research design hand, that makes them more or less
suitable for a particular research design.
• Demonstrate that it is the combination of, and connections between, these
metaphorical research cards that produces your research design, for example,
connections between the purpose of the research, the combination of research
related cards you hold in your hand, and how that combination will enable that
purpose to be achieved.
• Point out that we have collected the cards in our hand from our experiences when
interacting with the disciplinary and methodological traditions we have come
into contact with or been trained in.
• Emphasize that when designing research, we will need to expose and revisit the
beliefs and assumptions we hold both about research, and designing research,
that we bring with us to the process of designing our research.
• Establish that designing credible research, no matter what research approach is
taken, involves knowing and declaring what the effect of such choices are on both
the overall design of the research and the results of that research.
• Illustrate that to declare and justify your research hand will require reflexive
consideration and acknowledgment of the choices you have made and the effect
that those choices have had on your research design. In other words, you will
need to declare what these decisions are and justify them.
• Provide practical tips for how you might think through, identify, and declare
your own research hand when designing, and writing about, your own research
study.
• Revisit and highlight that a central theme in the book has been to emphasize that
designing research involves thinking through a dynamic and complex web of
interconnected decisions. In other words, designing research is a process driven
by a way of reflexive thinking that requires us to constantly revisit and assess our
decisions when developing the various aspects of our research design.
• Illustrate that this is a type of thinking that requires us to move forward and
backward through our developing research design to continuously revisit, review,
revise if necessary, and then build on the decisions we have made about that
design.
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
273
• Conclude, while at the same time keeping open, our extended discussion about
designing research by providing a summary of overarching key points that have
emerged throughout the book.
KNOWING AND DECLARING WHAT YOUR
RESEARCH DESIGN–RELATED HAND IS
Think of yourself pulling up a chair at a card table to join a group of people in a game of
cards. During that game, you will be dealt a number of cards. At various times, you will
think about these cards as either “good” or “bad” cards. Now think about what makes you
able to say that. It is not the cards themselves—after all, a card that has the number 2 on it
is a card with a 2 on it, no more or no less. Rather, it is what that card with the number 2 on
it can be used to do at a particular point in a game that makes it either a good or bad card.
What it can be used for depends on the rules of the card game that is being played. In some
games the card that is a 2 might be a key card, in others not at all.1
However, what also makes the card with the number 2 on it a good card, or a not-sogood one, is what the other cards are that you have in your hand. How does the 2 fit with
the rest of these cards that you have in your hand? For example, if the rules of the card game
require you to have a numerical run of cards, is the 2 the missing card you need in terms of
being able to put your hand down on the table, go out, and win the hand?
Or is it a card that stops you from being able to go out because it doesn’t give you the combination of cards you need to do so?
274 Research Design
In other games having a winning hand might involve doing something else, for example having four cards all of which are spades. Then the 2 of spades is a very useful card
because it is a spade, not because it is a 2.
Therefore, it is the purpose of the card game that you are playing that determines
which cards, or combinations of cards, you will need to collect in order to have a winning hand. It is how, in light of that purpose, a card can be used individually or in
combination with the other cards in your hand, that makes that card “good” or “bad.”
In one game, a particular card can be a “good” card, in another game the same card can
be a “bad” one.
The point we are making here is that no card is innately better than another is.
Rather, it is how they are played, both individually and in combination, related to the
purpose of the game in the specific hand being played, that makes an individual card a
better card than another one, or another card better than it. Therefore, it is that hand- or
game-specific purpose that will lead you to either keep or discard cards.
Pulling Up a Chair at the Research Design Table
Now, holding in your head this image of players sitting around a card table with hands
full of cards considering how they can, or might, use those cards, think of a research
design table. This is a metaphorical table where combinations of scholars, researchers,
or those with an interest in research can pull up a chair and enter into dialogue2 about
the process of designing research. Those sitting at this table are encouraged to reveal
and share their ideas about aspects of research design. There is room for everyone at
this table.
When you pull up a chair at this metaphorical table of research design, there are many
possible types of research design–related “cards” that you may already hold in your hand.
These are cards such as preferences you might have for particular types of methods, methodologies, research questions, as well as the assumptions about knowledge, research, and
science that these preferences are connected to, and derived from. You have collected these
cards from your experiences when interacting with the disciplinary and methodological
traditions you have come into contact with or been trained in. The cards you hold are
related to your view about
• paradigms and onto-epistemological assumptions and their effect on the way
research, and consequently research design, are thought about and enacted
(Chapter 4);
• ethical considerations that permeate the entire research design process (Chapter 2);
• what constitutes a research problem and what form that problem and associated
questions might (even must) take (Chapter 3); and
• what methodologies and methods can be used to address that problem
(Chapters 5−10).
Like actual playing cards, in themselves these metaphorical research design–related
cards are neither good or bad. For example, as we saw in Chapters 5 through 10, particular
types of methods such as qualitative interviews, quantitative surveys, and forms of mixed
methods research, are neither more or less suitable for a research design in themselves.
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
275
Rather, it is what they are used for, and how they fit with each of the other cards in your
research design hand (e.g., your research questions, your ethical considerations when putting a method into practice) that makes them more or less suitable for a particular research
design.
Therefore, how each card relates to each of the other cards in your hand in light of
the purposes of the research will determine which of these metaphorical research design–
related cards you will keep, and which you will discard. It is the combination of, and connections between, these metaphorical research cards that makes up your research design.
Just as in a card game no one card in your hand can be viewed in isolation from either the
purpose of the game, or the other cards that you hold, when designing research no one
aspect of that process or choices you make about that aspect can be viewed in isolation
from the purpose of the research and the overall research design. The following box develops this point further.
PUTTING IT INTO PRACTICE
IT IS THE COMBINATION THAT MATTERS
In this box, we use an insightful and wise observation from Ian Greener (2011) to
demonstrate the key point we have been making—namely that it is the combination
of metaphorical research design–related cards viewed in light of the purpose of the
research that matters when designing research.
The discussion here takes the form of us having a dialogue with Greener’s
observation. The italics are our comments on the points he makes.
[D]ifferent methods have very different underlying philosophies, and it is
important to have an understanding of these differences, not because it will
allow you to show how clever you are, but because it can help you to design
social research so that it is more coherent and so has a better chance of success. (Greener, 2011, p. 21)
Or for the purposes of the discussion here, it is important to understand what cards go
with each other (i.e., methods with methodologies, and methodologies with ways of viewing knowledge) to form a hand that is a “good” one in terms of being made up of cards that
together will enable you to achieve the purpose that the research is being designed for.
Knowing all the philosophy in the world can’t guarantee that you do a good project. (Greener, 2011, p. 21)
Nor does knowing all about specific methods or being a brilliant exponent of techniques
and procedures related to those methods guarantee that you do a good project. It might, but
it depends. Just as having an ace, for example, in your hand in a card game doesn’t guarantee that you have a good hand of cards as it depends on what that ace can be used for which
in turn depends on the purpose of the game and the other cards that you hold, so having
a particular method in your research design hand doesn’t guarantee that your research
design will be a good one. It depends on what the purpose of that research design is and
the other aspects that make up that design whether or not that method is a good choice.
But it can prevent you from making mistakes, such as claiming that you are
going to prove or disprove a theory by using a method which simply doesn’t offer
this as a possibility. (Greener, 2011, p. 21)
This is why it is important to know what the purpose of the research you are designing
is, and then make sure that you collect the research related cards you will need to enable
276 Research Design
the possibility for that purpose to be met. Prematurely shutting down your thinking about
methods or the purpose of research, and at times not even realizing or declaring that you
have done so, may mean that you simply do not have the means available to address the
problem that your research is being designed for. This is the equivalent of if your card game
requires jokers and you remove them from the pack (as you might do in some other card
games with different rules), then you simply do not have the possibility of achieving the
purpose of the game if it requires the jokers to remain in the pack.
As it would be odd to try and generate an in-depth understanding of a social
setting using an experimental design (odd, but not impossible), it would be odd
to try and test a theory using qualitative data because words, videos or other
recordings, the primary forms of qualitative data, can’t really be used to prove
or disprove anything. (Greener, 2011, p. 21)
It is (1) the purpose of the research and (2) the combination of metaphorical research
related design cards that you are holding in your hand, and (3) the congruency and consistency between them that matters. Does that combination enable you to achieve the aim
of the research? If it does not, then you may need to discard some of those cards (e.g.,
methods that will not enable you to address your research problem) and replace them with
others that will give you the combination of cards that does enable you to achieve the aim
of the research.
Putting All of Your Cards on the Table and Declaring Your Hand: An
Important Part of Research Design
In any card game, at some point you will have to put the cards you have been collecting
on the table and declare your hand. This is so that the others at the table are able to evaluate the worth of that hand in terms of the understandings and purpose of the card game
being played. In the same way, when designing research, after you have thought through
and made your decisions about the various aspects of your research design, you will need
to show and declare that hand. This involves declaring the decisions that you have made
when designing and thinking through that research. It is also about being able to justify
those decisions.
To declare and justify your research hand will require reflexive consideration and
acknowledgment of the choices you have made, and the effect that those choices have had
on your research design. At times, this might take your thinking into areas that, even
though they may have profoundly affected the shape that a research design has taken,
often remain undeclared and therefore invisible.
For example, putting all of your cards on the table when declaring your hand may
include revealing cards related to the politics of research such as the effect that trying to
obtain funding for the research had on the research design.3 Was, for example, your study
design influenced by considerations such as the preferred method of a particular funder of
research or dissertation committee?
DECLARING YOUR HAND: MISSING IN ACTION IN
MUCH OF THE REPORTING OF RESEARCH
The previous discussion has emphasized the importance of recognizing, and then declaring, your hand in relation to both the decisions that you make when designing your
research and the reasons for those decisions. Yet, this is something that is not often done
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
277
in reports of research. What one of us noticed several decades ago remains the norm. This
is that “[r]arely, if ever, are research proposals themselves published so that the conceptual
development of the research project can be explored” (Cheek, 2000b, p. 67). In other
words, the thinking that shaped the design and its development of the research being
reported is rarely, if ever, declared or published.
Consequently, in reports of research we are usually only presented with the end point
of the research design process—what the research design became or finished up as. How
that design became what it did remains invisible or undeclared. This can be for a number
of reasons ranging from word count restrictions by journals limiting what can be reported
in an article, through to researchers assuming shared understandings of what a research
hand is (or can be).
However, no matter what the reason, the effect is the same. Part of the researcher’s
hand is missing and not declared. This is because we join the conversation about that
research and its design partway through. We begin reading about the research when the
design is set and the research has been conducted. The part of the conversation that we
are missing is what happened before that. What was the thinking that led to the research
being designed and conducted in the way that it was?
For example, many published papers reporting forms of “quantitative” research present the results of that research by referring to findings in the form of lists of outcomes from
statistical analyses. Most often such reports omit to discuss how, where, and why the data
were collected, and on what basis those decisions were made. This is despite the fact, as
we discussed in Chapters 8 and 9, that these decisions affect what data were collected and
consequently what analyses were able to be done (or not done). This includes the choices
made about how concepts or constructs to be measured were defined, what was deemed to
be important about those concepts to measure and why, as well as what tradeoffs needed
to be made when designing the research. Tradeoffs could include needing to use a sample
of participants because of feasibility considerations, or using convenience sampling while
still aiming to say something about the entire study population when employing quantitative approaches to research (Chapters 8 and 9).
Similarly, when reading reports of qualitative research, we might read that six focus
groups were conducted and then move onto reading about the findings of the research
based on interpreting the information gained from those focus groups. What we don’t
often read is why focus groups were chosen to be part of the research design and what was
gained and lost as a result of making that choice, for example, explaining why choosing to
use focus groups because the interaction in the group was deemed of greater importance
in terms of achieving the purpose of the research than the depth of information able to be
obtained from each individual participant.
What Happened Along the Way?
Another part of the researcher’s hand often missing from reports of their research is what
happened along the way when the design was actually being developed, and then put into
action. What iterations did that design go through before the researcher landed on its final
form? Did what happened when putting the research into practice affect the final form
that the research design took? If so, how and why?
Things happen and change when doing your research. For example, unexpected
events such as recruitment difficulties may lead to further decisions related to the design of
the research having to be made. This happens in all research approaches—even the most
quantitative and seemingly objective ones. For as Becker noted,
278 Research Design
Sociologists of science . . . have shown us how natural scientists work in ways
never mentioned in their formal statements of method, hiding “shop floor practice”—what scientists really do—in the formal way they talk about what they do.
(Becker, 1998, p. 5)
This missing or hidden part of the research design conversation matters. Designing
credible research, no matter what research approach is taken, involves knowing and
declaring what the effect of such choices, and any changes related to them, are on both the
overall design of the research and the results of that research. It also involves knowing and
declaring that research design considerations do not end once you have developed your
initial design that will guide your study.
In the next section of the chapter, we put the spotlight on how you might think
through, identify, and declare your own research hand when designing, and writing
about, your own research study.
TIP
LEARNING FROM OTHERS
Although what happened along the way when the design was actually being put into
action is often missing when that research is reported, there are some good examples
of such reflexive reporting. We can learn much from these researchers and scholars’
description and reflexive analyses of how they thought through, and then declared,
their research design–related hand when reporting or writing their research.
Examples of this type of writing include the following:
1. The article we referred to in Chapter 10 by Cheek et al. (2015) that was written to
deliberately focus on this often absent and therefore hidden part of the conversation about designing research—the thinking that “is not made explicit, and therefore remains hidden, in most research reports” (p. 754). The team of researchers
considered that exposing and exploring this missing part of most research design–
related conversations constituted a “‘second’ set of findings” (p. 754) for their
study. These were findings about the dynamic and reflexive thinking that led to the
research being designed and conducted in that way that it was—findings that provide an otherwise missing part of the conceptual context (Maxwell, 1996) for both
the research design itself, as well as for the empirical findings that emerged from
that study.
See Cheek, J., Lipschitz, D. L., Abrams, E. M., Vago, D. R., & Nakamura, Y. (2015).
Dynamic Reflexivity in Action: An Armchair Walkthrough of a Qualitatively
Driven Mixed-Method and Multiple Methods Study of Mindfulness Training in
Schoolchildren. Qualitative Health Research, 25(6), 751–762.
2. Elaine Demps’s (2013) reflexive account of the rationale for why she chose to use
what she calls “the blurred genre of interpretive critical inquiry” (Lincoln & Guba,
2013, p. 90) in her doctoral study. This reflexive account is a very clear and helpful
example of declaring one’s hand with respect to the decisions made, and the reasons for them, when developing a research design. Her account provides a practical and thoughtful example of what a study might look like as a result of these
decisions and, more importantly, why that is so.
See Demps, E. L. (2013). Excerpts from Elaine Demps: Understanding the Faculty
Experience of Teaching Using Educational Technology. In Y. S. Lincoln & E. G. Guba
(Eds.), The Constructivist Credo (pp. 83–198). Left Coast Press.
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
279
3. The idea of “showing your warts” (Gernsbacher, 2018, p. 403) when designing your
quantitative research and acknowledging what you actually did. For example,
If experiments were conducted in an order different from the
reported order, state that. If participants participated in more
than one study, state that. If measures were recalculated, stimuli
were refashioned, procedures were reconfigured, variables were
dropped, items were modified—if anything transgressed the prespecified plan and approach—state that. (p. 405)
See Gernsbacher, M. A. (2018). Writing Empirical Articles: Transparency,
Reproducibility, Clarity, and Memorability. Advances in Methods and Practices in
Psychological Science, 1(3), 403–414.
HOW TO DECLARE YOUR RESEARCH HAND: CIRCLING BACK
TO TELL THE STORY OF THE DESIGNING OF YOUR RESEARCH
Declaring your research hand involves “telling the story of the project” (Schostak, 2002,
p. 228). Such telling is about what the research was about and why. It is about the choices
that arose along the way when designing that research and the decisions that were made
about them. It is about changes to the design that were made during the design process,
what they were and why they were made. Schostak likens this to “a return to the beginning, a circling back like a bird of prey” (Schostak, 2002, p. 228).4
Such circling back involves a great deal of reflexivity or folding back on what you
have done. This is because in order to be able to tell the story of the project, you need
to know what that story is. In order to know what the story is, you will need to circle
back on the series of interconnected decisions you have made which has produced the
shape of your research design, and which therefore make up the story of your research
design.
This means that a starting point for declaring your research design–related hand
when writing about and presenting the final form that your research design takes is
to think through your research design and the process that produced it. Thinking
through your process of producing that design enables you to identify the decisions
that you made about, and which underpin, that design. However, just identifying what
those decisions were, in itself, is not sufficient in terms of declaring your hand. Once
you have identified what the decisions were, you will also need to circle back on those
decisions and think through your reasons for making them. Why did you make them,
based on what?
To do this will require you to expose and revisit the beliefs and assumptions about
both research, and designing research, that you have brought with you to the research
design table. These are the beliefs and assumptions on which your decisions are based. Or,
put another way, these are the beliefs and assumptions that have produced the combination of research design–related cards that you now hold in your hand and are declaring.
This is why Koro-Ljungberg (2016) suggests that “[r]esearchers should ask themselves
. . . Why are they drawn to a particular set of beliefs? What are labels such as ‘paradigms,’ ‘reflexivity,’ or ‘triangulation’ expected to signify? What do particular
labels do? How do they operate? Who might gain from the use of these methods?”
(p. 14).
280 Research Design
To think through, and tell, your research design story in this way requires a form of
thinking out aloud to yourself when designing your research underpinned by forms of the
“if . . . then . . .” thinking (Morse, 2017, p. 48) and questioning that we have alluded to previously throughout various parts of the book. Identifying these choices and the effects that
they have had on your thinking about, and the subsequent form of, your research design is
part of declaring your hand. Such circling back on your thinking is part of what Guyotte
and Kuntz (2018) describe as asking students in their research classes “to give language
to how they come to know and become, thus opening the possibility for knowing/being
otherwise” (p. 257).
Telling the story of the process of designing your research in this way informs readers of your research about the assumptions, and decisions related to them, that constitute
your research hand. In so doing, it gives a reader the information that they will need, and
sometimes might otherwise not have, in order to make informed judgments about both
the way that the research was designed and enacted, as well as the veracity of the conclusions reached as a result (Schostak, 2002).
It also has the very useful effect of raising awareness and informed thinking among
researchers from different traditions about the various ways that research can be
designed, and what to look for when judging or making decisions about research from
those different traditions. This is important, for as Janesick (2008), commenting on
Egon Guba’s contribution to raising awareness of, and understanding about, qualitative
inquiry, notes, Guba “pointed out how many at the time did not understand qualitative
methods because so many qualitative researchers forgot to carefully document what
they did as researchers” (Janesick, 2008, p. 565). Such an observation, of course it can
be argued, applies equally well to mixed methods research and also forms of quantitative inquiry as well.
Reflexively thinking forward and backward through the process of designing research
enables you to consider questions such as these: Are the choices you have made about
which methods you will use, and the way that you will use them, consistent with the
choices you have made about methodology and the onto-epistemological assumptions
those methodologies make? How are all these choices connected to the research problem
that your research is being designed to address? How in turn does this constellation of
choices interconnect with the theoretical and empirical constructs that the design is being
built on? How does every decision that you make at every point of the process of designing
research and every set of connections arising from those decisions reflect an emphasis on
ethical and responsible thinking when designing your research, for example, connections
such as what type of data you collect, what about, who from, and how? This type of questioning, and the story that we tell as a result of it, is central to the credibility of our research
design and enables us to remain “faithful to how we practice research and scholarship”
(Preissle & deMarrais, 2011, p. 32).
Activity
Try This: What Happens When One of Your Cards Changes?
Drawing on the metaphor of the research design table and research design–related
cards, try to work out what research cards you bring to the table. These cards are the
ways that you think about, for example, what a research question is and the form it
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
281
takes, what a research method is (and is not), what form research data takes, and what
analyzing that data means.
Now try this. Discard the card that you are holding about what a research question is
and the form it takes. Mentally pick up another card that has a different view of what
a research question is and the form it takes. What does this do to all the other cards
in your hand?
How to Reflexively Circle Back to Tell the Story of Your Project When
Reporting on Your Research
You may be thinking to yourself at this point that while this all seems well and good, the issue
remains how you might actually go about telling this story. How can you do this? One possible way of doing this is offered by Schostak (2002). He provides a useful guide in the form
of a series of statements for you to think or write about that can help you both think through
the types of decisions indicated above that make up your research design, and the reasons
and effects of those decisions. Although his focus was on telling the story of the project when
writing up a thesis, these types of tips can be used equally well to tell the story of the process of
designing a research project. It provides a great example of reflexive thinking in action.
The statements are as follows:
1. This is what I intended as aims and objectives, these were my beliefs that
constructed my initial rationale.
2. These were how I conceptualized the structures, resources, mechanisms that
framed my thinking and that of others.
3. This is how I conceived the structure of problems and opportunities that I faced
at the outset.
4. And this was my initial methodology and chosen methods to obtain the data that
I wanted.
However, after an exploration of the literature, or experience in the field of some
combination of [sic] both I discovered:
5. More about my and others’ ethical, political and so on value positions.
6. I discovered more about the implications of my theoretical frameworks.
7. I explored alternative methodologies, philosophies and theories.
And these are discussed in Chapter ‘X’. Thus I needed to reframe any project in the
following ways:
8. This led me to refine my data collection etc. according to new or modified
rationales.
9. Which in turn led to innovative approaches to . . . , forms of representation of . . .
and theories or models of . . . and facts or information, or findings concerning . . .
And what I finally learnt from reflecting back on all these substantive, theoretical and
methodological investigations and experiences was . . .
282 Research Design
The story creates an air of continuous critical reflection, debate and ethical
soundness, and focuses on the agenda of concerns the writer wishes to emphasize.
(Schostak, 2002, pp. 228-229)
Activity
Circling Back to Some Activities From Chapter 1
In terms of thinking about how to declare your hand and tell the story of your project
when reporting your research, you will find it useful to circle back to the two activities
at the end of Chapter 1 to help you get a sense of how to reflexively think through and
tell the story of how your research design came to be the way it is.
In Activity 1, we asked you to obtain a report outlining the findings of a research
study and look at the level of detail about the way the research was designed, and
what was discussed, and what was not. We provided a series of points for you to think
through related to that report. Now we want you to think about how these points could
also guide you when writing about how your research design, and your research, came to
be the way it is.
Next revisit Activity 2 at the end of Chapter 1. Here we asked you, if you were in
the process of designing your research, to journal the decisions you make during
that process, and why you make them. We also asked you to continue this type
of journaling at various points in the book.5 If you have done this, collect all the
pieces of journaling that you have done and put them together to form the basis
for developing a reflexive account of your thinking during the process of designing
your research.
The type of thinking that these activities require of you is the type of thinking you
will need to do when declaring your hand.
CAPTURING ALL THIS IN A DIAGRAM OF SOME SORT
From the outset of this book-length conversation6 about research design, we have made
it clear that our focus is on the thinking that occurs when we are designing our research.
This is a thinking that affects the choices and decisions we make in the process of developing our research design.
Based on the many conversations we have had about this type of thinking throughout
the book, we thought that it could be useful at this point for readers if we developed a diagram that captured the iterative process of research design development that has been the
focus of this chapter and the entire book. However, as we began to develop our diagram,
we found ourselves facing the same problem that others who have tried to develop this sort
of diagram have had.
This problem is how to summarize and capture in a static diagram the iterative process
of designing research—a process that does not keep still and which requires us to constantly think backward and forward between the various areas of that design. And how
to stop any diagram produced from reverting to ones simply about research procedures
and not research design? As a result, we began to question if it is possible to capture all of
the areas, and aspects of those areas, related to designing research in a succinct and useful
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
283
diagrammatic summary of some sort that does not strip the representation of a research
design of the thinking that underpins it.
Developing, and Then Diagramming, an Overview of the Process of
Research Design
When we worked through the above questions, we realized that we needed to keep the
emphasis on the thinking that shapes the way our research is designed when developing
this diagram. After many attempts to do so, the result was Figure 11.1 below that we have
called Research Design: Putting All the Thinking Together to reflect this emphasis. In this
diagram, double-headed arrows are used across all the interconnected areas that make up a
research design in order to capture the type of reflexive thinking underpinning designing
research as an iterative process. Responsibility, ethics, and reflexivity frame and permeate
the entire research design process.
In this figure, we have tried to remove any impression of step-by-step linearity in the
design process. Thinking about what the research is being designed to do or be used for is
centered in this visual presentation and shown as constantly connecting and reconnecting
with all other areas—none of which stand alone. The multiple reflexive conversations
FIGURE 11.1 ■
Ethics
Research Design: Putting All the Thinking Together
Making explicit
methodological
assumptions about
what type of knowledge
will be produced
what type of methods
will enable the
production of that
type of knowledge
Making explicit theoretical
assumptions about
what theory is and is for
specific theoretical
concepts in use
Some sort of
problem/hunch
Substantive research
area
Research
question/hypothesis
about methodological matters
about specific methods for data
collection and data analysis
Using the literature
to join
conversations
Responsibility
about theoretical and conceptual
writing relevant to your study
about empirical data-based
studies related to substantive
focus of your study
to compare the findings of
your study to existing
empirical and theoretical work
Reflexivity
284 Research Design
with different types of literature that occur throughout the process of designing research
are represented, as are the connections between those literatures. We have used double
arrows in and out of the circle “Using the literature to join conversations” to reflect that
working with the literature related to any area of the research affects our thinking about
that area, and in turn, that thinking may lead to us consulting further or different literature as our thinking and ideas develop.
Imperfect as it may be, this visual presentation is a useful one. This is not because it is
a simple, easy to understand presentation. Quite the contrary! We think that this visual
presentation is useful because it captures, and thereby exposes, the complexity of designing
research.
The Importance of If . . . Then Thinking
The description of how we developed this diagram and why it takes the shape that it does
reinforces a central theme running through this book. This is that designing research
involves a lot of “If . . . then . . .” thinking (Morse, 2017, p. 48), thinking such as, If I do
this . . . then . . .; or Because I have done this . . . then . . .; or I did this but I know that
another way I could have done this was . . ., or consequently I will be able to say this as a
result of my research, but I will not be able to say this because of the choices I have made.
As we have discussed at numerous points in the book, reflexivity is a process in which
you will revisit, think through, challenge, and refine the decisions you make throughout the entire research design process. It is an important part of the dynamic process of
designing your research.
Turning your gaze on yourself, and the decisions that you make when developing your
research design, brings into focus what Maxwell (1996) termed the “conceptual context”
(p. 25) of your research: the “goals, experiences, knowledge, assumptions, and theory you
bring to the study and incorporate in the design” (p. 6). For example, how your background and assumptions about what scientific research is, and ways it can, should, or must
be done, affected the choices you made when designing your research. It can expose things
that you may have been taking for granted—for example, that this is the only way that this
could be studied, or that one method is necessarily better than another.
To think through, and then declare your hand in this way, relies on, and therefore
requires, the dynamic reflexivity7 that is central to the process of designing research. This
applies for all types of research design. Remember, “all research is based on the researcher’s
basic set of beliefs that guide action” (Janesick, 2008, p. 566).
TIP
USING OUR DISCUSSION TO HELP YOU WHEN YOU ARE
CONSIDERING DRAWING A DIAGRAM OF YOUR RESEARCH DESIGN
Our discussion of what we need to think about when trying to capture and summarize
our discussion of research design in a diagram provides a useful guide for some of the
things you will need to think about when providing some sort of summary or overview
of both your research design and the process of designing that research. Such an overview may take the form of either a diagram or a text, or both text and a diagram.
Asking such questions of yourself and your research design is central to iterative
and reflexive research design development.
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
285
CONCLUSIONS
When thinking about how to end this chapter, and in effect the book since this is its
final chapter, we struggled. This is because our discussion has covered so much related to
research design. How could we “round off” the discussion by providing an overview of
the key points and messages that have underpinned the discussion but at the same time
not give the impression that this wide-ranging discussion is able to be reduced to a series
of key points—a type of dot point thinking that we have critiqued in the discussion in
this book.8
In the end, we decided on two foci for this discussion. The first focus relates to rounding
off, and drawing conclusions specifically related to, the discussion in this chapter. This
has been a discussion about the importance of knowing and declaring the cards you hold
in your research design–related hand. The second focus aims to round off the discussion we have had throughout the book. We do this by indicating not only the key points
emerging from the discussion in this chapter, but also key points and messages about
designing research that we have distilled from our book-length discussion.
Focus 1: Rounding Off This Chapter
The discussion in this chapter has been about recognizing the many possible types of
research design–related “cards” we hold in our hand when designing research. These are
cards such as particular types of methods, methodologies, research questions, as well as
the assumptions about knowledge, research, and science that these preferences are connected to, and derived from. We collect (or discard) particular types of cards related to
research design, such as the way a concept is understood or a preference for a particular
way of doing research, from the disciplinary and methodological traditions we have come
into contact with or been trained in.
None of these cards, like actual playing cards, are innately better than any other card to
have in your hand. It is what they can be used for, and how they fit with each of the other
cards in your research design hand, that makes them more or less suitable to have in your
hand when designing a specific piece of research. When we design our research, we need
to know what these cards are, what the effects of having them in our hand when designing our research are, justify why we have chosen to keep these cards and discard others in
light of the purpose of the research, and thereby declare and justify our research hand.
Consequently, declaring your research hand is not just about what is in your research
hand but why it is in it. Designing research requires us to think reflexively throughout the
entire research design process. Adding an -ing to the word “design” emphasizes the active
and engaged process that designing research is, and from which a specific research design
emerges. Designing research involves thinking through a complex web of interconnected
decisions. In other words, designing research is a process driven by a way of thinking
that requires us to constantly revisit and assess our decisions when developing the various
aspects of our research design. This enables you to declare and defend not just what is in
that hand, but why it is in that hand.
Declaring and defending your research hand is a process that requires reflexivity on the
part of the researcher. Reflexivity is about adding an and why to the thinking we do and
286 Research Design
the decisions we make about our design. When thinking reflexively, we continually ask
ourselves questions about the decisions we have made about the emerging design in order
to modify or confirm those decisions. The goal of asking these questions is to improve
and refine the emerging research design.
Such iterative and reflexive thinking is what lies at the heart of the process of designing
research and is embedded in any research design produced as a result of that process. It is
an important part of ethical and responsible research, and research design. This is the case
no matter what research approach we are using. It is not an optional extra.
Rather, reflexive thinking is “a way to examine the complete research process and a vital
procedure for enhancing validity” (Lahman, 2018, p. 35) of all types of research—a point
that we have emphasized many times in the book. Understanding this means that you are
well on the way to becoming an informed, responsible, and thoughtful scholar, and therefore exponent, of designing research.
We do not pretend that such thinking will be a straightforward or easy process. It will be
challenging and messy at times and “more work than if you did things in a routine way
that didn’t make you think at all” (Becker, 1998, p. 7). However, in the end you will save
time and work as your research design will be well thought out, clear, and the decisions
that make it up and give it its shape will be able to be justified. In other words, you will
know and be able to declare and defend your research hand.
Focus 2: Ending With Some Key Points About This Chapter and Also the Book as a
Whole
As we have done in each chapter of this book, we will end this chapter with a list of key
points. However, because this is the last chapter of the book, these key points will serve
two purposes. The first is to give an overview of the key points that emerge from this
chapter. The second is to give an overview of key points about the idea of designing
research that have emerged from the book.
However, in doing so, we add the caveat that these key points and messages should not
be read as sufficient or complete in themselves. Each of these key points and messages
emerged from, and summarize, much thinking that sits behind them. It is that thinking
and its effect on the process of designing research that makes them the key points and
messages. This thinking has been the red thread that has run through all the chapters of
the book.
If you read and understand the thinking that gave rise to these key points, then you are well
on your way to knowing what you will need to think, consider, and make decisions about
when you are designing your research.
SUMMARY OF KEY POINTS
Key points arising from the discussion in this chapter, Chapter 11, are the following:
• An important part of the dynamic and reflexive process of designing research is
recognizing, thinking through, and declaring your research related hand when
designing and writing about your research.
Chapter 11
•
Why Knowing and Declaring Your Research Design Hand Matters
287
• This includes declaring the understandings, beliefs, and assumptions you have about
research and bring with you when you design your research.
• These understandings, beliefs, and assumptions provided a conceptual context
(Maxwell, 1996) for your research design, and the way in which you went about
designing it.
• Reflexive thinking enables you to recognize, think through, and declare your
research related hand.
• Declaring your hand requires you to be aware that designing research is complex and
not the same as selecting a research design.
• During the research design process, you will constantly examine and reexamine every
part of the overall design you are developing.
• This may require you to rethink prior decisions you have made or ideas you have
about that design—this is a necessary and key part of designing research,
Succinctly summing up key points that can be distilled from the discussions in the
10 chapters about designing research that have preceded this one, would produce a list of
points including the following:
• Designing research is a dynamic process involving reflexive thinking focused on, and
moving between, iterations of interrelated and interconnected theoretical, substantive,
disciplinary, methodological, methods-related, and ethical considerations.
• It is the decisions that we make about these considerations when designing our
research that collectively make up what we term a research design.
• Any research design is only as strong as the credibility of every one of those decisions
and the thinking behind that decision.
• Designing research involves developing a plan to guide (not constrain) a specific
research project or undertaking.
• The various parts of that research plan and how they relate to each other must make
up a consistent and congruent whole.
• Neither designing research, nor a research design, are synonymous with the selection
of methods to be used in the research.
• Methods, as well as diagrams, techniques, and procedures related to those methods,
are part of a specific research design. However, they do not make sense in the context
of that specific design and project unless they are viewed and understood in relation
to the other facets of that design, for example, the purpose of the research and the
type of knowledge required to address that purpose.
• Responsible and reflexive research design is about moving across, and between, the
thinking that connects and holds the areas that make up that design together, thereby
giving sense and coherence to that design.
288 Research Design
KEY RESEARCH-RELATED TERMS INTRODUCED IN THIS CHAPTER
declaring your hand
designing research
SUPPLEMENTAL ACTIVITIES
1.
Re-read the eight dot points about key points that can be distilled from the
discussions in the 10 chapters about designing research. You will find them at the
end of the conclusions section of this chapter.
Now working alone or with others, discuss the following:
a.
b.
c.
The thinking that underpins each of these key points, and from which they
have arisen
Why we consider them to be key points
Other key points that might be added and why
FURTHER READINGS
Becker, H. S. (1998). Tricks of the trade. How to think about your research while you’re doing it.
The University of Chicago Press.
Lumsden, K. (2019). Reflexivity: Theory, method, and practice. Routledge.
NOTES
1. For example, in Canasta it would be a wild card and therefore very useful; in other
games it might simply be a very low scoring card.
2. Hesse-Biber has used the idea of a “dialogue table” in her discussion of mixed methods
(see Hesse-Biber, 2010a, p. 417)
3. See Cheek’s (2022) discussion of the effect of funding on mixed methods research
design.
4. Schostak’s focus was on a student’s research thesis, but the ideas he puts forward
apply equally well to a student’s or researcher’s research design (which of course is a
central part of that thesis and the judgements made about its credibility).
5. See Activity 2 of Chapter 5, Activity 1 of Chapter 7, and Activity 1 of Chapter 9.
6. See for example the Preface and Chapter 1.
7. See section The Importance of Reflexive Thinking When Designing Research in Chapter
1 of this book for more about what reflexivity is and does.
8. See Chapter 5 where we critiqued the idea of dot point thinking.
GLOSSARY
analysis of qualitative data. Analysis of data involves cleaning, ordering, transforming and modelling the data collected to produce useful insights and information. Data collection and analysis continue iteratively until trustworthy and credible interpretations of that qualitative data
can be made.
anonymity. Measures used by researchers to protect the identity of the participants in their
study. In research design, anonymity often involves using pseudonyms when referring to participants or sites in the study.
audit trail. The thinking and decision-making throughout the process of designing your
research. Highlights design-related decisions and/or modifications made during this process—including when and why they were made. In this way, an audit trail builds the credibility
and trustworthiness of the design (Lincoln & Guba, 1985).
capitalization. Capitalization of notations related to components in a mixed methods study
reflects the priority given to the components in relation to the overall logic of inquiry that is
shaping the mixed methods study. The capitalized notation QUAL means that the overall study
is qualitatively driven whereas the capitalized notation QUAN indicates that the overall study
is quantitatively driven. The use of the noncapitalized qual or quan notation indicates that that
component is a supplementary one.
category. A grouping of segments of text (i.e., codes or parts of memos) that talk about/relates
to a particular aspect of a line of inquiry or that are instances or examples of the same idea.
This is a higher level of focus that captures what the codes in that category are instances of.
codebook. A list of predefined codes, sometimes with content descriptions and brief examples, that the researcher refers to when reading the transcripts of interviews or field notes of
observations. The researcher looks specifically for examples of those codes in the data.
coding. The “active process of identifying, labeling and systemizing data as belonging to or repre-
senting some type of phenomenon” (Tracy, 2020, p. 234). When coding, you will identify a segment
of data, such as a word or a series of words in an interview transcript, and give that segment of
data an initial label or code that captures what the segment of data is about or relates to.
computer-assisted qualitative data analysis software (CAQDAS). A generic name for software
designed to assist qualitative data analysis. While such software may help you “find, categorize, and retrieve data and texts more quickly than using a manual search” (Liamputtong,
2020, p. 269), it does not in itself analyze the data.
confidentiality. Keeping a research participant’s identity anonymous is part of confidentiality.
However, confidentiality is more than just not disclosing the name, identity, or identifying features of participants. It is also about the way that any data that a participant has provided, or is
related to that participant, is shared or not shared, and with whom.
construct (of a variable). A specific way of elaborating on, or understanding, an abstract concept
(Black, 1999). When you decide upon a specific way of understanding a variable of interest,
you are developing the construct for that variable that you will use in your quantitative study.
construct validity. The degree to which a measurement instrument measures what it is supposed or expected to measure in a quantitative research design.
constructivism. A paradigmatic stance in which the researcher does not seek to eliminate the
subjective thoughts, feelings, and opinions of those being researched to concentrate only
on specific objective “facts” and variables that must be controlled. Instead, the researcher
actively seeks to understand “the complex world of lived experience from the point of view of
those who live it” (Schwandt, 1994, p. 118).
289
290 Glossary
content validity. The degree to which a measurement instrument used in a quantitative
research design includes the measurement items necessary and sufficient to measure every
element of the construct in question.
core component. In a mixed-method study, the primary method in the study which “must be
conducted to a standard of rigor, such that, if all else were to fail, it could be published alone”
(Morse & Niehaus, 2009, p. 23). In other words, the core component can stand alone as a
research method for the study. See also supplemental component.
correlational approach. An approach in quantitative research design in which the researcher
investigates the relationships between variables without controlling or manipulating any
of them. Correlational approaches have the potential to address research questions about
“what goes with what” (Oppenheim, 1992, p. 21) in a population.
credibility of research. Whether or not research is considered credible (or even research at all)
will be based on the basic beliefs that make up the paradigmatic stance of the researcher.
Therefore, judgments about the credibility of research make no sense unless they are in relation to a particular worldview, or basic set of beliefs, about research. Therefore, to be credible the design of the research, and its conduct, must be in keeping with the paradigmatic
stance underpinning it. Criteria for making judgments about the credibility of research do not
automatically transfer from one paradigmatic stance to another.
data. Individual items of information that reduce or “chunk” reality into manageable units
(Bernard et al., 2017) able to be analyzed and interpreted to produce the findings or results
of our study. What is “chunked” (i.e., what is considered data and why) and how it is “chunked”
(i.e., what methods are used or not used to collect and analyze that data, and why) depends on
the way that the idea of data itself is viewed and thought about.
data condensation. In qualitative approaches refers to the “process of selecting, focusing,
simplifying, abstracting, and/or transforming the data that appear in the full corpus (body)
of written-up field notes, interview transcripts, documents, and other empirical materials”
(Miles et al., 2014, p. 12). Data condensation is about making choices about what parts of the
large body of data you have collected are relevant for addressing your research questions.
declaring your research hand. Declaring the decisions that you have made when designing and
thinking through your research is an important part of research design. It is also about being
able to justify those decisions. To declare and justify your research hand will require reflexive consideration and acknowledgment of the choices you have made, and the effect that those
choices have had on your research design.
deductive reasoning. In research based on deductive reasoning, empirical evidence is collected and used to test an existing theory or a theoretically based assertion. The researcher
begins by thinking about that theory, or aspects of that theory, and predicts what the empirical evidence—that is, the data collected—should show if that theory, or those aspects, are
supported.
descriptive quantitative approach. A descriptive quantitative approach seeks to collect and analyze numerical data to identify and numerically describe population characteristics. Such
descriptive quantitative approaches have the potential to address research questions about
what is going on in a study population in terms of how many, or how much, of something of
interest is occurring within the population, without drawing inferences about cause and
effect.
designing research. Designing research requires us to think reflexively throughout the entire
research design process. Adding an -ing to the word design emphasizes the active and engaged
process that designing research is, and from which a specific research design emerges.
diagramming. An active, dynamic and reflexive process of developing a diagram about, or
representing in diagrammatic form, what you have thought through, and then made decisions about, when designing your mixed methods research study (Morse, 2017). The diagram
produced is a static representation of this process of thinking and therefore cannot be fully
understood removed or decontextualized from the process giving rise to it.
Glossary 291
discourse analysis. A research approach that seeks to study written or spoken language in
relation to its social context. Discourse analysis is premised on the understanding of “language as a meaning constituting system which is both historically and socially situated”
(Cheek & Rudge, 1994, p. 61). Therefore, the focus of analysis is the meaning constituting systems in texts generated from, for example, interviews, news articles, or visual texts such as
pictures and films (Taylor, 2013).
empirical. Empirical research is an evidence-based approach that draws conclusions from
results derived from observations, experiments, interviews, or other forms of data derived
from real-life situations. An empirical approach relies on real-world data, measurements,
and results rather than theories and concepts.
epistemology. The study of knowledge. Epistemological considerations focus on questions
such as, What do we mean when we say that we know something? What enables us to claim
that we know that? How does knowledge differ from opinion or belief? Epistemological considerations when designing our research focus on the type of knowledge we will need to produce to be able to reach credible or warranted conclusions or inferences.
estimate. Used in quantitative research designs, an estimate for a population characteristic is the numerical value of the corresponding characteristic of a sample drawn from that
population.
ethics committees. Formally constituted committees, also known as Institutional Review
Boards, or IRBs, that regulate ethical matters relating to research design and conduct.
ethnography. A specialist qualitative research approach that seeks to understand the culture
of the social context being studied. It is the focus of an ethnographic research approach on the
theoretical idea of culture that makes the study ethnographic. Put another way “to be an ethnographic study, the lens of culture must be used to understand the phenomenon” (Merriam &
Tisdell, 2016, p. 31) that is being studied.
experimental/quasi-experimental approach. A quantitative research approach that manipulates variables under controlled conditions to identify cause-and-effect relationships.
Experimental approaches have the potential to address research questions about why something happens. Quasi-experimental approaches likewise involve the manipulation of an independent variable but without the randomization that is central to experimental approaches.
external validity. The extent to which the numerical based findings of a quantitative study using
a sample can be generalized to a population beyond that sample.
feasibility. Whether the research can be done within the time and resources that are
available.
focus group. A specific type of interview involving a group of people. “Focus groups are a
research method that collects qualitative data through group discussions. This definition
contains two components: first, the goal of generating data, and second, the reliance on interaction” (Morgan, 2019, p. 4). The reliance on the group interaction sets focus groups apart
from individual interviews.
generalizing in quantitative research. Arriving at conclusions about a study population based
on numerical data from a sample drawn from that population. Generalization is enabled by
establishing a relationship between what is going on in a sample and what is going on in the
population from which that sample is drawn.
grounded theory. A whole-of-study theoretical and methodological approach underpinning
your research design. Uses strategies of inquiry designed to enable you to develop a theory
able to address your research problem or provide answers for your research questions. The
theory is grounded in, and arises from, the analysis and interpretation of the data collected—
hence the name grounded theory.
hypothesis. A statement about what we predict empirical evidence to reveal in a specific situation in a specific context (and in the social sciences often for a specific group of people). It is
a prediction “about the nature of relationships between two or more variables expressed in
the form of a testable statement” (O’Leary, 2017, p. 377) based on “an informed reading of the
literature, a theory, or personal observations and experience” (Nardi, 2018, p. 48).
292 Glossary
hypothesis testing. Testing an hypothesis means applying a procedure for statistically analyzing the research data designed to detect any inconsistency between the data collected about
one or more variables and the hypothesis made about that (those) variable(s) if there is such
an inconsistency to detect.
hypothetico-deductive thinking. A type of thinking that begins with defining the general assumptions and understandings about how things work, and which variables are involved in a specific situation. From that understanding, a testable statement (a hypothesis) about how things
work is deduced. That hypothesis is then tested by analyzing numerically data collected specifically to enable assessing whether the hypothesis is supported by that numerically based
empirical evidence.
individual interviews. One of the, if not the most, common way that data is collected in qualitative research approaches. This form of interview involves a one-to-one dialogue between
an individual participant and the researcher. Therefore, you will see an individual interview
sometimes referred to as an in-depth individual interview.
inductive reasoning. In research based on inductive reasoning, empirical evidence is collected and used to build theoretical and empirical concepts and understandings. The
researcher begins by collecting data and seeks to develop concepts and understandings
about the area (rather than starting with a predetermined or a priori set of theoretical concepts to test).
informed consent. Informed consent is when participants agree or consent to participate in
the research. It means that the participant understands both that they are giving consent and
what they are consenting to. Such consent relies on full disclosure by the researcher of what
participating in the research involves.
inquiry paradigms. Guba and Lincoln (1994) developed the idea of inquiry paradigms to make
explicit the intersections between ontology, epistemology, methodology, and methods and
how they affect what is studied, how it is studied, and the role that the researcher plays in that
study. This includes “what falls within and outside the limits of legitimate inquiry” (Guba &
Lincoln, 1994, p. 108).
Institutional Review Boards (IRBs). See ethics committees.
interview guide. A prepared list of the questions or areas that you will use to guide the discussion in an interview or focus group, including follow-up questions, prompts, interview probes.
interview probes. Follow-up questions or statements used in qualitative research interviews
designed to enable you to pursue new and interesting leads as the interview progresses or to
clarify aspects of the answers given to the questions you have asked.
iterative process. An iterative process involves cycles of thinking where you begin with an idea,
think it through, and then revisit, refine, or change the initial idea in light of that thinking,
then think that change through and so on. This continues until you have landed on a research
design that you believe will enable you to complete your research in a credible, systematic,
and well-thought-through way. This thinking also continues as you put that design into action.
layers of consent. A strategy that can be used to address issues about consent related to the
repurposing of data. Research participants giving informed consent for the primary study are
also explicitly asked whether they consent to their data (1) only being used in that study, (2)
being reused in later studies, (3) being archived or stored to be accessed by other researchers and reused in other studies.
lines of inquiry. Used to guide qualitative interviews. These initial lines, or areas, of inquiry
emerge from your thinking about what it is important for you to know more about to get an
in-depth picture of what is going on in the situation that your research is focused on. Once you
have identified your lines of inquiry, you can then develop interview questions related to each
of them.
Glossary 293
measurement instrument. The complete set of measurement items designed to measure each
of the variables that are identified as having the potential to enable you to answer your specific research question(s) make up the measurement instrument included in your quantitative
research design.
measurement item. A question, statement, or observation designed to measure a specific vari-
able (or an aspect of a variable).
memos. The preliminary and developing analytical notes or hunches written by a qualitative
researcher during analysis of data. Memos can be used to describe what happened during
data collection, thoughts about what is going on and/or questions, or thoughts about what you
need to find out more about. Memos can also be used to highlight consistencies or inconsistencies in the data collected that require further elucidation or follow-up.
methodology. The thinking that gives rise to the choice and use of particular methods in your
research design. An example of a methodological consideration is whether or not a particular
method will enable you to obtain the type of data needed to generate the type of knowledge
needed to address your research question(s).
methods. The techniques used to obtain and analyze research data. Each research related
method has specific procedures associated with it that are designed to obtain a particular
type of knowledge or information—usually referred to as data. The methods chosen must be
consistent with the type of knowledge or data we want to obtain from our research (our methodological considerations), which in turn must be consistent with the nature of the research
problem.
mixed methods research approach. A research approach that integrates more than one method
to provide a better understanding of the phenomenon being studied. Some definitions of
mixed methods describe “a single study combining qualitative and quantitative research
approaches” (e.g., Creswell, 2015) while others limit the definition to studies where one of the
methods is incomplete (see supplemental component) and cannot stand alone (e.g., Morse &
Cheek, 2014).
mixing. In mixed methods design, the conceptual process of integrating ideas from different
ways of thinking “into a meaningful pattern and a practically viable blueprint for generating better understanding of the social phenomena being investigated” (Greene, 2007, p 16).
Mixing occurs at a number of levels, including overall project design (e.g., methods or components used), data collected (e.g., textual or numeric), and the analyses of that data (e.g., the
findings of each component) (Yin, 2006).
multiple methods/multimethod research. If mixed method research is defined as using both
qualitative and quantitative methods, then research using more than one qualitative method,
or more than one quantitative method—but not both—can be defined as multimethod.
However, if mixed method research is defined as combining one complete method and one
incomplete method, then a study using one complete qualitative method and one complete
quantitative method would be considered a multiple methods/multimethod study.
notation system. In mixed methods research, a notation system, developed in 1991 by Morse
and subsequently expanded by others, is commonly used to describe or write about mixed
methods and mixed methods research designs. For example, it uses the notations QUAL or
qual to refer to components in that design that draw on qualitatively derived logics of inquiry.
The notations QUAN or quan are used to refer to components that draw on quantitatively
derived logics of inquiry.
onto-epistemological assumptions. The stances on different ontological and epistemological
assumptions about the nature of reality, and how we know what we know about that reality
(Crotty, 1998) under a particular paradigmatic approach. Which onto-epistemological view we
adopt affects the methodological decisions we make when we design our research.
ontology. The study of “the nature of truth and reality and whether it is external or constructed” (Creamer, 2018, p. 43). It is about what exists and what can be considered real.
There are different views or ontological positions about the nature of reality, for example,
constructivism, positivism.
294 Glossary
operational definition. The identification of quantifiable factors, or indicators, of a construct
and specification of which quantifiable factors or indicators to include in the measurement
of the variable. The operational definition of a variable, informed by the construct of that
variable, enables collecting information that is relevant to learning something about that
variable.
paradigm. A type of world view or set of basic beliefs (Guba & Lincoln, 1994) that guides the
thinking of, and therefore the decisions made by, researchers related to what a researcher
might decide to study, how they will study it, and the way that they interpret and make conclusions about what they have studied.
paradigmatic stance. The set of basic beliefs held by a researcher about what research and sci-
ence are and how research can be designed and conducted in order to be scientific and therefore credible.
point of interface. The point of interface in your mixed methods research design is where the
components making up your research design are brought together and integrated in some
way, usually (but not always) occurring “once the analysis of each component is completed”
(Morse, 2017, p. 9).
positivism. A dominant inquiry paradigm in western scientific thought for hundreds of years,
positivism is based on a “conviction that scientific knowledge is both accurate and certain”
(Crotty, 1998, p.27). The task of the researcher is to study the reality that already exists “out
there” (Lincoln & Guba, 2013, p. 38). Consequently, positivism is premised on a realist ontological position.
post-positivism. A paradigmatic stance under which research seeks to distinguish between
beliefs and valid beliefs, rather than to produce absolute truths (Campbell & Russo, 1999).
Whether a belief is valid is based on judgments about the validity of the empirical evidence it
is based on and how that evidence was obtained and analyzed. Experimentation is retained as
the basic methodological strategy, although it is conceded that experimentation “cannot produce ultimately infallible results” (Lincoln & Guba, 2013, p. 38).
power of a hypothesis test. The ability of a hypothesis test to correctly claim support for the
statement making up the hypothesis.
priority or weighting of components. In a mixed methods study, priority (also referred as weighting) refers to the overall priority given to the components in the study in relation to the overall
logic of inquiry. For example, is the overall study qualitatively or quantitatively driven in terms
of its overriding methodological or philosophical emphasis?
pseudonym. A fictitious name given by researchers to participants or sites in a research study.
The pseudonym is used instead of their participants’ or sites’ real names when reporting and
discussing the research to protect their anonymity.
purposeful (or purposive) sampling. A type of sampling used in qualitative research in which the
people, sites, or texts selected to make up our study sample are chosen because they are
information rich in some way about a situation, event, context, site, or experience we are
interested in knowing something more about—the purpose of the study.
qualitative component (in a mixed methods study). In mixed methods research designs the notations QUAL or qual refer to components in that design that draw on qualitatively derived logics
of inquiry.
qualitative research approach. A research approach designed to enable the emergence of rich
and qualitative interpretations of the perceptions or experiences of people about a specific
aspect(s) of the everyday context(s) in which they exist. This type of approach is often referred
to as naturalistic or interpretive inquiry. It uses nonnumerical data to derive in-depth understandings of people’s perceptions, beliefs, attitudes, and experiences about a situation or
phenomenon.
quantitative component (in a mixed methods study). In mixed methods research designs the notations QUAN or quan refer to components in that design that draw on quantitatively derived
logics of inquiry.
Glossary 295
quantitative research approach. A research approaches that relies on statistical analysis of
numeric data to identify, classify, and quantitatively describe characteristics of, and trends in,
a given (large) group of people (i.e., a study population). The goal of the research is to quantify
the presence of, and links between, objective measures.
quantitative survey. A research method used to collect data from a group of respondents by way
of asking people a set of predefined questions or asking them to respond to a set of predefined
statements. The measurement instrument of a survey can be, but is not always, a questionnaire.
quasi-mixed designs. Research designs claiming to be mixed methods studies in which “two
types of data are collected, but there is little or no integration of findings and inferences from
the study” (Teddlie & Tashakkori, 2011, p. 294.).
random selection process. A probability-based process in which every member of the study
population has an equal probability of being selected as part of the sample in a quantitatively
driven research design.
realism. The ontological view that the world exists and has an external objective reality independent of the perceptions of those living in that world (Schwandt, 2015).
reflexivity/reflexive thinking. The quality or process of reflecting, folding, or bending back
(Finlay & Gough, 2003) on our own thinking to work out why we have come to think about
something in the way that we do. Reflexivity is important for identifying assumptions and
challenging us to work out why we think what we do when designing our research, and
whether there are other possible ways of thinking about that design.
relativism. An ontological position according to which the world is understood as made up of
socially constructed meanings. These everyday social interactions and constructions make
up the way that the world is. The world does not exist independently of those in it.
relevant literature. The body of knowledge built up by the work of others which is relevant to
the various areas that make up your research design. This can be literature related to empirical work in the substantive area of your research, methodological and methods considerations, ethical and theoretical matters. Working with the literature is central to the iterative
process that underpins the development of a research design.
reliability. In quantitative research, the degree of consistency across multiple measures
of the same construct, such as when the same measurement is conducted multiple times
(test-retest reliability), when the same measurement is conducted by different people (interrater reliability), and consistency across individual items designed to measure the same construct in a single measurement instrument (internal consistency reliability).
representative sample. A sample in quantitative research that accurately represents the study
population according to specified criteria, such as the distribution of age, gender, and level of
education in the population, as well as other criteria relevant to the research question(s) the
study is designed to address.
repurposing of data. Data from one study that is reused in a subsequent study or for an additional purpose. The original data collected for one study (or even data collected from several
studies) can be used to undertake a form of secondary analysis to answer questions not part
of the original study.
research area. The broad initial topic(s) or wide area(s) of interest giving rise to the research.
research design. The process by which a research idea is developed into a research project or
plan that can then be carried out by a researcher or research team.
research ethics. The considerations and principles surrounding moral behavior in research
contexts (Wiles, 2013). Ethical issues often identified in relation to the design and conduct
of research include respect for participants’ dignity and privacy, and issues of welfare and
social justice more generally. Thinking through ethics at all points of the research design process is part of responsible research (Kuntz, 2015).
296 Glossary
research problem. The aspect of the research topic or area in which we are interested and on
which our research will focus.
research question. A more specific formulation of the research problem. A research question is
posed in such a way that it can be researched, and when addressed provides information able
to contribute to the body of knowledge relevant to the research problem that the question is
related to.
response rate. The proportion of sampled individuals that are willing to provide information by
responding to the measurement instrument that is part of a specific survey-based quantitative research study.
rigor in qualitative research. The measures and processes put in place to ensure that the
research is conducted appropriately. For example, one measure to improve rigor in qualitative inquiry is to maintain an audit trail, demonstrating that there is sufficient data of the right
type to support any interpretations or conclusions made.
sample size. In qualitative approaches this refers to the number of people interviewed, or
observations made, or texts analyzed. In a qualitative study sample size considerations
include how we know that we have collected and analyzed “enough” data of an appropriate type to be able to reach the conclusions we do at the end of that study. In quantitative
approaches sample size refers to the number of members of the study population who provide the information that comprises the data for that quantitative study. In a quantitative study
sample size considerations include are there enough members of that population to enable
us to conduct the types of statistically and probability-based analyses needed to answer the
questions and/or address our hypotheses.
sample/study sample. A subset, or sample, drawn from the study population. The study sample
comprises the members of the study population who provide the information that comprises
the data of the study.
sampling strategy. A procedure for selecting which members of the study population will
be included in the sample. The logic underpinning this strategy varies depending on the
approach being used in your research design e.g., whether the underlying logic is quantitatively or qualitatively driven.
scientific method. A way of thinking about and doing research derived from the way research is
done in the physical and natural sciences.
semistructured interviews. An interview process used in qualitative research that gives some
structure to what is to be talked about but does not dictate in what order, form, or how it must
be talked about as a closed interview structure does. This enables you to follow up and probe
unanticipated and interesting directions and areas that may arise during the interview.
sensitizing concepts for observations in qualitative inquiry. When making observations in social
settings, sensitizing concepts (Blumer, 1954, p. 7) can play a similar role to that played by
lines of inquiry in qualitative interviews, providing “jumping-off points or lenses” (Tracy, 2020,
p. 29) when collecting data using observations.
social desirability. The tendency to depict oneself as conforming to social norms and common
values, potentially leading to inaccurate responses in surveys or answers in interviews.
standardized observations. Standardized observations are most often used in quantitative
approaches to gain data about the frequency of particular designated activities, events, or
actions that a researcher has decided are of interest. Observations often take the form of
some sort of observational log of the frequency of the event of interest occurring.
statistical validity. Conclusions drawn from a procedure for statistically analyzing the research
data are statistically valid if the analysis procedure is appropriate for drawing that specific
type of conclusion, and if the analysis is performed on a set of data that meets the requirements of that analysis procedure.
Glossary 297
statistically reasonable. A research design is statistically reasonable if it can be justified and
defended in line with the rules of statistics. A statistically justifiable and defensible research
design is necessary for the research to be credible.
structured/closed interviews. A type of interview that uses an interview schedule where each
research participant being interviewed is asked the same predetermined interview questions, using the same wording, in the same order. Usually there is a limited set of response
categories for the person being interviewed to choose from. The aim in using a closed interview is to obtain some form of standardized data that can then be compared across large
numbers of participants. Most often used in quantitative approaches.
study population. The group of people about which you want to be able to say something when
using a quantitative approach to address your research question(s). This study population is
generally a (large) group of people, where every member meets a set of inclusion criteria relevant to the research question(s), and where no one meeting those criteria is excluded from
that group.
substantive area of research. Matters related to the research topic of interest itself (e.g., smok-
ing, or leadership) rather than to theoretical or methodological aspects of the research. For
example, the substantive literature that you might draw on when designing your research is
literature relevant to your problem area to find out what others have and have not done (matters of substance) and how this might affect what you choose to focus on (or not focus on) in
that area.
supplemental component. In a mixed methods study, an incomplete component(s) designed to
supplement the core component in some way.
theory. A set of beliefs or concepts that explain aspects of the world. Ideas or concepts from
specific theories provide orienting ideas that influence all aspects of the research design: the
questions that are asked, the way data is collected, the analysis of that data, the interpretation of that analysis, and consequently the conclusions that can be drawn from the research.
timing or pacing. The order, duration or intervals at which the various components that make
up a mixed methods research design will be undertaken in the study. For example, will the
components be conducted at the same time or will one method or type of collection of data
follow the other?
transcription. The process of converting a sound file into written text. While transcription may
seem to be a simple exercise of converting spoken words to written words, during the process
of transcribing an interview, you revisit what happened and become more familiar with both
what was said and the interactions that led to it being said. In other words, you immerse yourself in that interview data and begin to analyze it.
triangulation. The use of more than one data source, theory, or method to improve the credibil-
ity and therefore trustworthiness of research. “Triangulation means that you take different
perspectives on an issue in your study or in answering your research questions” (Flick, 2020,
p. 187). The idea is that by using multiple sources or methods, fuller, richer, or more nuanced
descriptions, analyses, and therefore interpretations of an issue are possible.
trustworthiness. The degree to which research findings can be accepted and recognized as
significant by the audience of the research. Lincoln and Guba (1985) define trustworthiness
in qualitative inquiry as comprising the elements of rigor, credibility, transferability, dependability, and confirmability.
typology. A system of grouping items into “types” according to their perceived characteristics or relationships. Typologies differ according to the basis of classification. For example,
Greene et al. (1989) developed a typology of five categories of purpose for mixed methods
research design—triangulation, complementarity, development, initiation, and expansion.
unstructured/open interviews. Interviews that have a flexible and nonstandardized structure.
Open-ended questions are often used in qualitative studies and the interview is more a conversation with the researcher picking up on, and probing in more depth, areas that the participant has chosen to talk about when answering those questions.
298 Glossary
validity. Validity is a complex concept, but broadly defined, the validity of research results or
findings refers to the degree to which they represent reality. In quantitative research, we can
say that results or findings are valid when they are credible, well founded, reasonable, justifiable, and defensible. (See also construct validity; content validity)
variable. Anything—a person, object, event, or relationship—that using a quantitively driven
research design the researcher seeks to measure, manipulate, and control in order to produce data for analysis. The variables of a study are identified or determined by the researcher
as the ones that have the potential to enable the researcher to answer a specific research
question related to a study population of interest.
vulnerable populations. A group is considered vulnerable if there is good reason to believe that
individuals in that group may, for some reason, have difficulty providing free and informed
consent to participate in research.
REFERENCES
Agar, M. H. (2008). The professional stranger: An informal introduction to ethnography (2nd ed.).
Emerald Group.
Aksøy, H. (2009). Leading through constant change: What are the issues for nurse managers arising from changes to the home care services? Master’s thesis. Oslo University.
Alabrese, E., Becker, S. O., Fetzer, T., & Novy, D. (2019). Who voted for Brexit? Individual
and regional data combined. European Journal of Political Economy, 59, 132–150. https://doi.
org/10.1016/j.ejpoleco.2018.08.002
Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247–271. https://doi.org/10.5465/amr.2009.0188
Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting
research. SAGE.
American Psychological Association. (2019). Depression assessment instruments. https://
www.apa.org/depression-guideline/assessment
Anderson, E. E., & Corneli, A. (2018). 100 questions (and answers) about research ethics. SAGE.
Attia, M., & Edge, J. (2017). Be(com)ing a reflexive researcher: A developmental approach to
research methodology. Open Review of Educational Research, 4(1), 33–45. https://doi.org/10.10
80/23265507.2017.1300068
Bailey, K. A., Dagenais, M., & Gammage, K. L. (2021). Is a picture worth a thousand words?
Using photo-elicitation to study body image in middle-to-older age women with and
without multiple sclerosis. Qualitative Health Research, 31(8), 1542–1554. https://doi.
org/10.1177/10497323211014830
Baker, S. E., & Edwards, R. (2012). How many qualitative interviews is enough? Expert voices and
early career reflections on sampling and cases in qualitative research (NCRM Methods Review
Papers 19). National Centre for Research Methods. http://eprints.ncrm.ac.uk/2273/
Ballantyne, A. (2019). Adjusting the focus: A public health ethics approach to data research.
Bioethics, 33(3), 357–366. https://doi.org/10.1111/bioe.12551
Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the Beck Depression Inventory-II.
Psychological Corporation.
Beck, C. T. (2019). Secondary qualitative data analysis in the health and social sciences.
Routledge.
Becker, H. S. (1998). Tricks of the trade: How to think about your research while you’re doing it.
The University of Chicago Press.
Bernard, H. R., Wutich, A., & Ryan, G. W. (2017). Analyzing qualitative data: Systematic
approaches (2nd ed.). SAGE.
Bishop, L. (2009). Ethical sharing and reuse of qualitative data. Australian Journal of Social
Issues, 44(3), 255–272. https://doi.org/10.1002/j.1839-4655.2009.tb00145.x
Bjerknes, M. S., & Bjørk, I. T. (2012). Entry into nursing: An ethnographic study of newly qualified nurses taking on the nursing role in a hospital setting. Nursing Research and Practice, 1–7.
https://doi.org/10.1155/2012/690348
Black, T. R. (1999). Doing quantitative research in the social sciences: An integrated approach to
research design, measurement and statistics. SAGE.
299
300 References
Blair, J., Czaja, R. F., & Blair, E. A. (2014). Designing surveys: A guide to decisions and procedures
(3rd ed.). SAGE.
Blee, K. M., & Currier, A. (2011). Ethics beyond the IRB: An introductory essay. Qualitative
Sociology, 34, 401–413. https://doi.org/10.1007/s11133-011-9195-z
Blumer, H. (1954). What is wrong with social theory? American Sociological Review, 19(1), 3–10.
https://doi.org/10.2307/2088165
Boddy, C. R. (2016). Sample size for qualitative research. Qualitative Market Research, 19(4),
426–432. https://doi.org/10.1108/qmr-06-2016-0053
Boellstorff, T., Nardi, B., Pearse, C., & Taylor, T. L. (2012). Ethnography and virtual worlds: A
handbook of method. Princeton University Press.
Bogdan, R. C., & Biklen, S. K. (2011). Qualitative research for education: An introduction to theories and methods (5th ed.). Pearson.
Bolin, A., & Granskog, J. (Eds.). (2003). Athletic intruders: Ethnographic research on women, culture, and exercise. State University of New York Press.
Bors, D. (2018). Data analysis for the social sciences: Integrating theory and practice. SAGE.
Braun, V., & Clarke, V. (2013). Successful qualitative research: A practical guide for beginners.
SAGE.
Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done?
Qualitative Research, 6(1), 97–113. https://doi.org/10.1177/1468794106058877
Bryman, A. (2016). Social research methods (5th ed.). Oxford University Press.
Cadwalladr, C., & Graham-Harrison, E. (2018, March 17). Revealed: 50 million Facebook profiles harvested for Cambrige Analytica in major data breach. The Guardian. https://www.theg
uardian.com/news/2018/mar/17/cambridge-analytica-facebook-influence-us-election
Campbell, D. T., & Russo, M. J. (1999). Social experimentation. SAGE.
Carey, M. A., & Asbury, J. E. (2012). Focus group research. Left Coast Press.
Cashin, A., Newman, C., Eason, M., Thorpe, A., & O’Discoll, C. (2010). An ethnographic study
of forensic nursing culture in an Australian prison hospital. Journal of Psychiatric and Mental
Health Nursing, 17(1), 39–45. https://doi.org/10.1111/j.1365-2850.2009.01476.x
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis.
SAGE.
Charmaz, K. (2014). Constructing grounded theory: A practical guide through qualitative analysis
(2nd ed.). SAGE.
Cheek, J. (2000a). An untold story? Doing funded qualitative research. In N. K. Denzin & Y. S.
Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 401–420). SAGE.
Cheek, J. (2000b). Postmodern and poststructural approaches to nursing research. SAGE.
Cheek, J. (2004). At the margins? Discourse analysis and qualitative research. Qualitative
Health Research, 14(8), 1140–1150. https://doi.org/10.1177/1049732304266820
Cheek, J. (2008). The practice and politics of funded qualitative research. In N. K. Denzin &
Y. S. Lincoln (Eds.), Strategies of qualitative inquiry (3rd ed., pp. 45–74). SAGE.
Cheek, J. (2010). Human rights, social justice, and qualitative research: Questions and hesitations about what we say about what we do. In N. K. Denzin & M. D. Giardina (Eds.), Qualitative
inquiry and human rights (pp. 100–111). Left Coast Press.
Cheek, J. (2017). Qualitative inquiry, research marketplaces, and neoliberalism: Adding
some +s (pluses) to our thinking about the mess in which we find ourselves. In N. K. Denzin &
M. D. Giardina (Eds.), Qualitative inquiry in neoliberal times (pp. 19–36). Routledge.
References 301
Cheek, J. (2018a). The BMJ debate and what it tells us about who says what, when and where,
about our qualitative inquiry. In N. K. Denzin & M. D. Giardina (Eds.), Qualitative inquiry in the
public sphere (pp. 50–65). Routledge.
Cheek, J. (2018b). The marketization of research: Implications for qualitative inquiry. In N. K.
Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (5th ed., pp. 322–340).
SAGE.
Cheek, J. (2021). Maintaining the integrity of qualitatively driven mixed methods: Avoiding the
“this work is part of a larger study” syndrome. Qualitative Health Research, 31(6), 1015–1018.
https://doi.org/10.1177/10497323211003546
Cheek, J. (2022). The impact of funding on ways qualitative research is thought about and
designed. In U. Flick (Ed.), Handbook of qualitative research design (pp. 339–354). SAGE.
Cheek, J., & Ballantyne, A. (2001). Moving them on and in: The process of searching for and
selecting an aged care facility. Qualitative Health Research, 11(2), 221–237. https://doi.org/
10.1177/104973201129119064
Cheek, J., Lipschitz, D. L., Abrams, E. M., Vago, D. R., & Nakamura, Y. (2015). Dynamic reflexivity in action: An armchair walkthrough of a qualitatively driven mixed-method and multiple
methods study of mindfulness training in schoolchildren. Qualitative Health Research, 25(6),
751–762. https://doi.org/10.1177/1049732315582022
Cheek, J., & Morse, J. M. (2022). The power of qualitative research in mixed methods research
designs. In U. Flick (Ed.), Handbook of qualitative research design, (pp. 636–651). SAGE.
Cheek, J., & Øby, E. (2018). Digitalisation, organisations and people: Issues and challenges.
Exploring the digitalisation imperative in higher education institutions in Norway. [Unpublished
funding application]. Østfold University College.
Cheek, J., Onslow, M., & Cream, A. (2004). Beyond the divide: Comparing and contrasting
aspects of qualitative and quantitative research approaches. Advances in Speech-Language
Pathology, 6(3), 147–152. https://doi.org/10.1080/14417040412331282995
Cheek, J., & Rudge, T. (1994). Inquiry into nursing as textually mediated discourse. In P. Chinn
(Ed.), Advances in methods of inquiry for nursing (pp. 59–67). Aspen Publishers.
Chubb, J., & Watermeyer, R. (2017). Artifice or integrity in the marketization of research
impact? Investigating the moral economy of (pathways to) impact statements within research
funding proposals in the UK and Australia. Studies in Higher Education, 42(12), 2360–2372.
https://doi.org/10.1080/03075079.2016.1144182
Coffey, A. (2018). Doing ethnography (2nd ed.). SAGE.
Corbin, J., & Strauss, A. (2015). Basics of qualitative research: Techniques and procedures for
developing grounded theory (4th ed.). SAGE.
Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. SAGE.
Creswell, J. W. (2011). Controversies in mixed methods research. In N. K. Denzin & Y. S.
Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed., pp. 269–284). SAGE.
Creswell, J. W. (2013). Qualitative inquiry & research design: Choosing among five approaches
(3rd ed.). SAGE.
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches
(4th ed.). SAGE.
Creswell, J. W. (2015). A concise introduction to mixed methods research. SAGE.
Creswell, J. W. (2016). 30 essential skills for the qualitative researcher. SAGE.
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative and mixed
methods approaches (5th ed.). SAGE.
Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research
(2nd ed.). SAGE.
302 References
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research
(3rd ed.). SAGE.
Crotty, M. (1998). The foundations of social research: Meaning and perspective in the research
process. Allen & Unwin.
Cutcliffe, J. R., & Harder, H. G. (2012). Methodological precision in qualitative research:
Slavish adherence or “following the yellow brick road?” The Qualitative Report, 17(41), 1–19.
https://doi.org/10.46743/2160-3715/2012.1720
DeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2011). Developing and using a
codebook for the analysis of interview data: An example from a professional development
research project. Field Methods, 23(2), 136–155. https://doi.org/10.1177/1525822X10388468
Demps, E. L. (2013). Excerpts from Elaine Demps: Understanding the faculty experience of
teaching using educational technology. In Y. S. Lincoln & E. G. Guba (Eds.), The constructivist
credo (pp. 83–198). Left Coast Press.
Denzin, N. K. (1971). The logic of naturalistic inquiry. Social Forces, 50(2), 166–182. https://doi.
org/10.1093/SF/50.2.166
Denzin, N. K. (1989a). Interpretive interactionism. SAGE.
Denzin, N. K. (1989b). The research act: A theoretical introduction to sociological methods (3rd
ed.). Prentice-Hall.
Denzin, N. K. (2001). Interpretive interactionism (2nd ed.). SAGE.
Denzin, N. K. (2008). The new paradigm dialogs and qualitative inquiry. International Journal
of Qualitative Studies in Education, 21(4), 315–325. https://doi.org/10.1080/09518390802136995
Denzin, N. K. (2009). Qualitative inquiry under fire: Toward a new paradigm dialog. Left Coast
Press.
Denzin, N. K. (2010). The qualitative manifesto: A call to arms. Left Coast Press.
Denzin, N. K., & Giardina, M. D. (2016). Introduction. In N. K. Denzin & M. D. Giardina (Eds.),
Qualitative inquiry through a critical lens (pp. 1–16). Routledge.
Denzin, N. K., & Lincoln, Y. S. (2000). Preface. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook
of qualitative research (2nd ed., pp. ix–xx). SAGE.
Denzin, N. K., & Lincoln, Y. S. (2005a). Introduction. The discipline and practice of qualitative
research. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd
ed., pp. 1–32). SAGE.
Denzin, N. K., & Lincoln, Y. S. (2005b). Strategies of inquiry. In N. K. Denzin & Y. S. Lincoln
(Eds.), The SAGE handbook of qualitative research (3rd ed., pp. 375–386). SAGE.
Denzin, N. K., & Lincoln, Y. S (Eds.). (2011). The SAGE handbook of qualitative research (4th ed.).
SAGE.
Denzin, N. K., & Lincoln, Y. S (Eds.). (2018a). The SAGE handbook of qualitative research (5th
ed.). SAGE.
Denzin, N. K., & Lincoln, Y. S. (2018b). Introduction. The discipline and practice of qualitative
research. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (5th
ed., pp. 1–26). SAGE.
Denzin, N. K., & Lincoln, Y. S. (2018c). Part II: Paradigms and perspectives in contention. In N.
K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (5th ed., pp. 97–107).
SAGE.
Department of Health and Human Services. (2018). Code of federal regulations Title 45, part 46
– Protection of human subjects, subpart A (‘common rule’). https://www.ecfr.gov/on/2018-07-19
/title-45/subtitle-A/subchapter-A/part-46
References 303
Detrow, S. (2018, March 20). What did Cambridge Analytica do during the 2016 election? NPR.
https://www.npr.org/2018/03/20/595338116/what-did-cambridge-analytica-do-duringthe-2016-election
Duncan, M., & Watson, R. (2010). Taking a stance: Socially responsible ethics and informed
consent. In M. Savin-Baden & C. H. Major (Eds.), New approaches to qualitative research (pp.
49–58). Routledge.
Economic and Social Research Council. (2021). ESRC research data policy. https://www.ukri.
org/publications/esrc-research-data-policy/
Fetters, M. D. (2016). “Haven’t we always been doing mixed methods research?”: Lessons
learned from the development of the horseless carriage. Journal of Mixed Methods Research,
10(1), 3–11. https://doi.org/10.1177/1558689815620883
Finlay, L., & Gough, B. (2003). Prologue. In L. Finlay & B. Gough (Eds.), Reflexivity: A practical
guide for researchers in health and social sciences (pp. ix–xi). Blackwell Science.
Flick, U. (2015a). Introducing research methodology (2nd ed.). SAGE.
Flick, U. (2015b). Qualitative inquiry—2.0 at 20? Developments, trends, and challenges
for the politics of research. Qualitative Inquiry, 21(7), 599–608. https://doi.org/10.1177/
1077800415583296
Flick, U. (2020). Introducing research methodology (3rd ed.). SAGE.
Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and how it can succeed
again. Cambridge University Press.
Fontana, A., & Frey, J. H. (2005). The interview: From neutral stance to political involvement.
In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., pp.
695–727). SAGE.
Fontana, A., & Prokos, A. H. (2007). The interview: From formal to postmodern. Left Coast
Press.
Freeman, M. (2017). Modes of thinking for qualitative data analysis. Routledge.
Frey, J. H., & Fontana, A. (1991). The group interview in social research. Social Science Journal,
28(2), 175–187. https://doi.org/10.1016/0362-3319(91)90003-M
Friels, A. C. (2016). Motivation towards success: A qualitative comparative study illustrating the
differences in motivating factors in achievement between low SES high achieving and low achieving African American high school females. [Doctoral dissertation, University of South Carolina],
https://scholarcommons.sc.edu/etd/3437
Gartner IT. (2018). Digitization. In Gartner IT glossary. https://www.gartner.com/en/
information-technology/glossary/digitization
Geertz, C. (1973). The interpretation of cultures. Basic Books.
Gergen, K. J. (1991). The saturated self: dilemmas of identity in contemporary life (Vol. 166). Basic
Books.
Gernsbacher, M. A. (2018). Writing Empirical Articles: Transparency, Reproducibility, Clarity,
and Memorability. Advances in Methods and Practices in Psychological Science, 1(3), 403–414.
https://doi.org/10.1177/2515245918754485
Gerson, N. (2019). How to protect yourself from predatory publishers and other open access FAQs.
SAGE open access. https://perspectivesblog.sagepub.com/blog/industrynews/oa/howtoprot
ectyourself
Giles, T., King, L., & de Lacey, S. (2013). The timing of the literature review in grounded theory
research. An open mind versus an empty head. Advances in Nursing Science, 36(2), E29–E40.
https://doi.org/10.1097/ANS.0b013e3182902035
Glaser, B. G. (1978). Theoretical sensitivity: Advances in the methodology of grounded theory.
Sociology Press.
304 References
Glaser, B. G. (1998). Doing grounded theory: Issues and discussions. Mill Valley, CA: Sociology
Press.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative
research. Aldine de Gruyter.
Glasson, V. (2017). 6 ways to spot a predatory journal. Rx Communications. https://rxcomms.
com/blog/6-ways-spot-predatory-journal/
Gorard, S. (2003). Quantitative methods in the social sciences: The role of numbers made easy.
Continuum.
Grav, C. M. (2015). Endring før endringen – en studie om organisasjonsendring før en endring er
bestemt. [Master’s thesis, Østfold University College]. http://hdl.handle.net/11250/285200
Greene, J. C. (2002). Mixed-method evaluation: A way of democratically engaging
with difference. Evaluation Journal of Australasia, 2(2), 23–29. https://doi.org/10.1177/
1035719X0200200207
Greene, J. C. (2007). Mixed methods in social inquiry. John Wiley.
Greene, J. C. (2008). Is mixed method social inquiry a distinctive methodology? Journal of
Mixed Method Research, 2(1), 7–22. https://doi.org/10.1177/1558689807309969
Greene, J. C. (2015). Preserving distinctions within the multimethod and mixed methods
research merger. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 606–615). Oxford University Press.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for
mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274.
https://doi.org/10.3102/01623737011003255
Greener, I. (2011). Designing social research. A guide for the bewildered. SAGE.
Greenhalgh, T., Annandale, E., Ashcroft, R., Barlow, J., Black, N., Bleakley, A., Boaden, R.,
Braithwaite, J., Britten, N., Carnevale, F., Checkland, K., Cheek, J., Clark, A., Cohn, S.,
Coulehan, J., Crabtree, B., Cummins, S., Davidoff, F., & Davies, H. . . . & Ziebland, S. (2016). An
open letter to The BMJ editors on qualitative research. BMJ, 352, i563. https://www.bmj.com/c
ontent/352/bmj.i563.short
Greenwood, M. D., & Terry, K. J. (2012). Demystifying mixed methods research: Participation
in a reading group “sign posts” the way. International Journal of Multiple Research Approaches,
6(2), 98–108. https://doi.org/10.5172/mra.2012.6.2.98
Guba, E. G. (Ed.). (1990). The paradigm dialog (2nd ed.). SAGE.
Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In N. K.
Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 105–117). SAGE.
Gubrium, J. F., & Holstein, J. A. (1997). The new language of qualitative method. Oxford
University Press.
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.
org/10.1177/1525822X05279903
Gülay, H., & Önder, A. (2013). A study of social–emotional adjustment levels of preschool children in relation to peer relationships. Education 3-13, 41(5), 514–522. https://doi.org/10.1080/0
3004279.2011.609827
Guyotte, K. W., & Kuntz, A. M. (2018). Becoming openly faithful: Qualitative pedagogy and
paradigmatic slippage. International Review of Qualitative Research, 11(3), 256–270. https://doi.
org/10.1525/irqr.2018.11.3.256
Hall, R. (2013). Mixed methods: In search of a paradigm. In T. Lê & Q. Lê (Eds.), Conducting
research in a changing and challenging world (pp. 71–78). Nova Science Publishers.
Hamilton, M. (1960). A rating scale for depression. Journal of Neurology, Neurosurgery &
Psychiatry, 23, 56–62. https://doi.org/10.1136/jnnp.23.1.56
References 305
Hammond, F. M., Davis, C. S., Hirsch, M. A., Snow, J. M., Kropf, M. E., Schur, L., Kruse, D., &
Ball, A. M. (2021). Qualitative examination of voting empowerment and participation among
people living with traumatic brain injury. Archives of Physical Medicine and Rehabilitation,
102(6), 1091–1101. https://doi.org/10.1016/j.apmr.2020.12.016
Hartup, W. W. (1995). The three faces of friendship. Journal of Social and Personal
Relationships, 12(4), 569–574. https://doi.org/10.1177/0265407595124012
Hartup, W. W. (1996). The company they keep: Friendships and their developmental significance. Child Development, 67(1), 1–13. https://doi.org/10.2307/1131681
Henry, G. T. (1990). Practical sampling (Vol. 21). SAGE.
Henry, G. T. (2009). Practical sampling. In L. Bickman & D. J. Rog (Eds.), The SAGE handbook of
applied research methods (2nd ed., pp. 77–105). SAGE.
Hesse-Biber, S. N. (2010a). Emerging methodologies and methods practices in the
field of mixed methods research. Qualitative Inquiry, 16(6), 415–418. https://doi.org/
10.1177/1077800410364607
Hesse-Biber, S. N. (2010b). Mixed methods research: Merging theory with practice. Guilford
Press.
Hesse-Biber, S. N. (2017). The practice of qualitative research (3rd ed.). SAGE.
Hesse-Biber, S. N., & Leavy, P. (2006). The practice of qualitative research. SAGE.
Hindess, B. (1996). Discourses of power: From Hobbes to Foucault. Blackwell.
Hoza, B. (2007). Peer functioning in children with ADHD. Journal of Pediatric Psychology, 32(6),
655–663. https://doi.org/10.1093/jpepsy/jsm024
Hurst, A. L. (2008). A healing echo: Methodological reflections of a working-class
researcher on class. The Qualitative Report, 13(3), 334–352. https://doi.org/
10.46743/2160-3715/2008.1582
International Council of Nurses. (2021). The ICN code of ethics for nurses. https://www.icn.ch/
system/files/2021-10/ICN_Code-of-Ethics_EN_Web_0.pdf
IoT Agenda. (2016). Definition: Internet of things (IoT). https://internetofthingsagenda.techtarg
et.com/definition/Internet-of-Things-IoT
Jackson, A. Y., & Mazzei, L. A. (2012). Thinking with theory in qualitative research: Viewing data
across multiple perspectives. Routledge.
Janesick, V. J. (2008). Egon Guba: The witty authentic maverick. International Journal of
Qualitative Studies in Education, 21(6), 565–567. https://doi.org/10.1080/09518390802489063
JHL Editorial Team. (2020). References from predatory publishers: Policy statement for
the Journal of Human Lactation. Journal of Human Lactation, 36(2), 219–220. https://doi.
org/10.1177/0890334420912210
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of
mixed method research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.
org/10.1177/1558689806298224
Jones, N. A., Ross, H., Lynam, T., Perez, P., & Leitch, A. (2011). Mental models: An interdisciplinary synthesis of theory and methods. Ecology and Society, 16(1), 46. [online] https://
ecologyandsociety.org/vol16/iss1/art46/
Jorgensen, D. L. (1989). Participant observation: A methodology for human studies (Vol. 15).
SAGE.
Juritzen, T. I., Grimen, H., & Heggen, K. (2011). Protecting vulnerable research participants:
A Foucault-inspired analysis of ethics committees. Nursing ethics, 18(5), 640–650. https://doi.
org/10.1177/0969733011403807
Karsavuran, Z. (2021). Surviving a major crisis: The case of dismissed tourism and hospitality
employees. Journal of Policy Research in Tourism, Leisure and Events, 13(2), 243–265. https://
doi.org/10.1080/19407963.2020.1787421
306 References
Keller, R. (2013). Doing discourse research: An introduction for social scientists (B. Jenner,
Trans.). SAGE.
Kirkegaard, E. O. W., & Bjerrekær, J. D. (2016a). OKCupid Kirkegaard Bjerrekær dataset.
Internet Archive. https://archive.org/download/OKCupid-Kirkegaard-Bjerrekaer-dataset
Kirkegaard, E. O. W., & Bjerrekær, J. D. (2016b). The OKCupid dataset: A very large public dataset of dating site users. Open Differential Psychology. https://doi.org/10.26775/
ODP.2016.11.03
Koro-Ljungberg, M. (2016). Reconceptualizing qualitative research: Methodologies without methodology. SAGE.
Kunnskapsdepartementet. (2017). Digitaliseringsstrategi for universitets- og høyskolesektoren
2017–2021. https://www.regjeringen.no/no/dokumenter/digitaliseringsstrategi-for-universit
ets--og-hoyskolesektoren---/id2571085/
Kuntz, A. M. (2015). The responsible methodologist: Inquiry, truth-telling, and social justice. Left
Coast Press.
Lahman, M. K. E. (2018). Ethics in social science research: Becoming culturally responsive. SAGE.
Lather, P. (2006). Paradigm proliferation as a good thing to think with: Teaching research in
education as a wild profusion. International Journal of Qualitative Studies in Education, 19(1),
35–57. https://doi.org/10.1080/09518390500450144
Leung, D. Y., Kumlien, C., Bish, M., Carlson, E., Chan, P. S., & Chan, E. A. (2021, Article
104851). Using internationalization-at-home activities to enhance the cultural awareness of
health and social science research students: A mixed-method study. Nurse Education Today,
100. https://doi.org/10.1016/j.nedt.2021.104851
Lewins, F. (1992). Social science methodology: A brief but critical introduction. Macmillan
Education Australia.
Liamputtong, P. (2009). Qualitative research methods (3rd ed.). Oxford University Press.
Liamputtong, P. (2011). Focus group methodology: Principles and practice. SAGE.
Liamputtong, P. (2013). Qualitative research methods (4th ed.). Oxford University Press.
Liamputtong, P. (2020). Qualitative research methods, (5th ed.). Oxford University Press.
Lincoln, Y. S. (2015). Critical qualitative research in the 21st century: Challenges of new technologies and the special problem of ethics. In G. S. Cannella, M. S. Pérez, & P. A. Pasque
(Eds.), Critical qualitative inquiry: Foundations and futures (pp. 197–214). Left Coast Press.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE.
Lincoln, Y. S., & Guba, E. G. (2013). The constructivist credo. Left Coast Press.
Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2011). Paradigmatic controversies, contradictions,
and emerging confluences, revisited. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook
of qualitative research (4th ed., pp. 97–128). SAGE.
Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2018). Paradigmatic controversies, contradictions,
and emerging confluences, revisited. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook
of qualitative research (5th ed., pp. 108–150). SAGE.
Locke, L. F., Spirduso, W. W., & Silverman, S. J. (2014). Proposals that work: A guide for planning
dissertations and grant proposals. (3rd ed.). SAGE.
Lofland, J., & Lofland, L. H. (1995). Analyzing social settings: A guide to qualitative observation
and analysis. Wadsworth.
Lumsden, K. (2019). Reflexivity: Theory, method, and practice. Routledge.
Macfarlane, B. (2010). Values and virtues in qualitative research. In M. Savin-Baden & C.
H. Major (Eds.), New approaches to qualitative research: Wisdom and uncertainty (pp. 19–27).
Routledge.
References 307
MacInnes, J. (2019). Little quick fix: See numbers in data. SAGE.
Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). Sample size in qualitative interview
studies: Guided by information power. Qualitative Health Research, 26(13), 1753–1760. https://
doi.org/10.1177/1049732315617444
Markham, A. N. (2018). Ethnography in the digital internet era: From fields to flows, descriptions to interventions. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative
research (5th ed., pp. 650–668). SAGE.
Markham, A. N., Tiidenberg, K., & Herman, A. (2018). Ethics as methods: Doing ethics in
the era of big data research – introduction. Social Media + Society, 4(3), 1–9. https://doi.
org/10.1177/2056305118784502
Martin, A. J. (2001). The student motivation scale: A tool for measuring and enhancing motivation. Journal of Psychologists and Counsellors in Schools, 11, 1–20. https://doi.org/10.1017/
S1037291100004301
Martin, A. J. (2002). Motivation and academic resilience: Developing a model for student enhancement. Australian Journal of Education, 46(1), 34–49. https://doi.org/
10.1177/000494410204600104
Martin, A. J. (2003). The student motivation scale: Further testing of an instrument that measures school students’ motivation. Australian Journal of Education, 47(1), 88–106. https://doi.
org/10.1177/000494410304700107
Maxwell, J. A. (1996). Qualitative research design: An interactive approach. SAGE.
Maxwell, J. A. (2006). Literature reviews of, and for, educational research: A commentary
on Boote and Beile’s “Scholars before researchers.” Educational Researcher, 35(9), 28–31.
https://doi.org/10.3102/0013189X035009028
Maxwell, J. A. (2013). Qualitative Research Design: An Interactive Approach, (3rd ed.). SAGE.
Maxwell, J. A., & Mittapalli, K. (2008). Theory. In L. M. Given (Ed.), The SAGE encyclopedia of
qualitative research methods (pp. 876–880). SAGE.
Maxwell, J., Chmiel, M., & Rogers, S. E. (2015). Designing integration in multimethod and
mixed method research. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of
multimethod and mixed methods research inquiry (pp. 223–239). Oxford University Press.
Mayan, M. J. (2009). Essentials of qualitative inquiry. Left Coast Press.
Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation
(4th ed.). Jossey-Bass.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35(11),
1012–1027. https://doi.org/10.1037/0003-066X.35.11.1012
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices
in Psychological Science, 1(1), 131–144. https://doi.org/10.1177/2515245917747656
Miciak, M., & Daum, C. (2023). Scratching the underbelly of research design: Developing
clear research question(s). In Cheek, J., & Øby, E. (2023). Research Design: Why Thinking About
Design Matters (pp. 59–65). SAGE.
Mikami, A. Y. (2010). The importance of friendship for youth with attention-deficit/hyperactivity disorder. Clinical Child and Family Psychology Review, 13, 181–198. https://doi.org/10.1007/
s10567-010-0067-y
Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). SAGE.
Miles, M. B., Huberman, A. M., & Saldaña, J. (2020). Qualitative data analysis: A methods sourcebook (4th ed.). SAGE.
Mills, K. A. (2018). What are the threats and potentials of big data for qualitative research?
Qualitative Research, 18(6), 591–603. https://doi.org/10.1177/1468794117743465
308 References
Morgan, D. L. (1989). Adjusting to widowhood: Do social networks really make it easier? The
Gerontologist, 29(1), 101–107. https://doi.org/10.1093/geront/29.1.101
Morgan, D. L. (2016). Essentials of dyadic interviewing. Left Coast Press.
Morgan, D. L. (2019). Basic and advanced focus groups. SAGE.
Morse, J. M. (1991). Approaches to qualitative–quantitative methodological triangulation.
Nursing Research, 40(2), 120–123. https://doi.org/10.1097/00006199-199103000-00014
Morse, J. M. (1998). The contracted relationship: Ensuring protection of anonymity and confidentiality. Qualitative Health Research, 8(3), 301–303. https://doi.
org/10.1177/104973239800800301
Morse, J. M. (2008). Deceptive simplicity. Qualitative Health Research, 18(10), 1311. https://doi.
org/10.1177/1049732308322486
Morse, J. M. (2017). Essentials of qualitatively-driven mixed-method designs. Routledge.
Morse, J. M., & Cheek, J. (2014). Making room for qualitatively-driven mixed-method
research. Qualitative Health Research, 24(1), 3–5. https://doi.org/10.1177/1049732313513656
Morse, J. M., & Cheek, J. (2015). Introducing qualitatively-driven mixed-method designs.
Qualitative Health Research, 25(6), 731–733. https://doi.org/10.1177/1049732315583299
Morse, J. M., Cheek, J., & Clark, L. (2018). Data-related issues in qualitatively driven
mixed-method designs: Sampling, pacing, and reflexivity. In U. Flick (Ed.), The SAGE handbook
of qualitative data collection (pp. 564–583). SAGE.
Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Left Coast
Press.
Nagel, D. A., Burns, V. F., Tilley, C., & Aubin, D. (2015). When novice researchers adopt constructivist grounded theory: Navigating less travelled paradigmatic and methodological
paths in PhD dissertation work. International Journal of Doctoral Studies, 10, 365–383. http://
dx.doi.org/10.28945/2300
Nardi, P. M. (2018). Doing survey research: A guide to quantitative methods (4th ed.). Routledge.
National Association of Social Workers. (2021). Read the code of ethics. https://www.socialwor
kers.org/About/Ethics/Code-of-Ethics/Code-of-Ethics-English
National Education Association. (Ed). (2021). Code of ethics of the education profession. In
NEA Handbook 2020/2021 (pp. 429–430).
National Science Foundation. (2014). Grant general conditions (GC-1). https://www.nsf.gov/
pubs/policydocs/gc1/feb14.pdf
Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European
Journal of Social Psychology, 15, 263–280. https://doi.org/10.1002/ejsp.2420150303
Norum, K. E. (2008). Reality and multiple realities. In L. M. Given (Ed.), The SAGE encyclopedia
of qualitative research (Vol. 2, pp. 737–739). SAGE.
O’Leary, Z. (2017). The essential guide to doing your research project (3rd ed.). SAGE.
Oppenheim, A. N. (1992). Questionnaire design, interviewing, and attitude measurement (New
ed.). Pinter Publishers Ltd.
O’Reilly, M., & Kiyimba, N. (2015). Advanced qualitative research: A guide to using theory. SAGE.
Pangle, L. S. (2003). Aristotle and the philosophy of friendship. Cambridge University Press.
Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). SAGE.
Patton, M. Q. (2012). Essentials of utilization-focused evaluation. SAGE.
Patton, M. Q. (2015). Qualitative research & evaluation methods (4th ed.). SAGE.
References 309
Pelto, P. J. (2015). What is so new about mixed methods? Qualitative Health Research, 25(6),
734–745. https://doi.org/10.1177/1049732315573209
Polkinghorne, D. E. (2005). Language and meaning: Data collection in qualitative research.
Journal of Counseling Psychology, 52(2), 137–145. https://doi.org/10.1037/0022-0167.52.2.137
Ponterotto, J. G. (2006). Brief note on the origins, evolution, and meaning of the qualitative research concept thick description. The Qualitative Report, 11(3), 538–549. https://doi.
org/10.46743/2160-3715/2006.1666
Preissle, J., & deMarrais, K. (2011). Teaching qualitative research responsively. In N. K.
Denzin & M. D. Giardina (Eds.), Qualitative inquiry and global crises (pp. 31–39). Left Coast
Press.
Presser, S. (1985). The use of survey data in basic research in the social sciences. In C. E.
Turner & E. Martin (Eds.), Surveying subjective phenomena (Vol. 2, pp. 93–114). Russell Sage
Foundations.
Punch, K. F. (2003). Survey research: The basics. SAGE.
Rager, K. B. (2005). Compassion stress and the qualitative researcher. Qualitative Health
Research, 15(3), 423–430. https://doi.org/10.1177/1049732304272038
Ramcharan, P., & Cutcliffe, J. R. (2001). Judging the ethics of qualitative research:
Considering the “ethics as process” model. Health and Social Care in the Community, 9(6), 358–
366. https://doi.org/10.1046/j.1365-2524.2001.00323.x
Ravitch, S. M., & Riggan, M. (2017). Reason & rigor: How conceptual frameworks guide research
(2nd ed.). SAGE.
Richards, L. (2015). Handling qualitative data: A practical guide (3rd ed.). SAGE.
Richtig, G., Berger, M., Lange-Asschenfeldt, B., Aberer, W., & Richtig, E. (2018). Problems
and challenges of predatory journals. Journal of the European Academy of Dermatology &
Venereology, 32(9), 1441–1449. https://doi.org/10.1111/jdv.15039
Ryen, A. (2011). Ethics and qualitative research. In D. Silverman (Ed.), Qualitative research (3rd
ed., pp. 416–438). SAGE.
Ryle, G. (1971). Collected papers. Volume II collected essays, 1929–1968. Hutchinson.
Saldaña, J. (2021). The coding manual for qualitative researchers (4th ed.). SAGE.
Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health,
18(2), 179–183. https://doi.org/10.1002/nur.4770180211
Saris, W. E., & Gallhofer, I. N. (2014). Design, evaluation, and analysis of questionnaires for survey research (2nd ed.). John Wiley.
Schirmer, J. (2009). Ethical issues in the use of multiple survey reminders. Journal of
Academic Ethics, 7, 125–139. https://doi.org/10.1007/s10805-009-9072-5
Schostak, J. F. (2002). Understanding, designing and conducting qualitative research in education: Framing the project. Open University Press.
Schostak, J., & Schostak, J. (2008). Radical research: Designing, developing and writing
research to make a difference. Routledge.
Schwandt, T. A. (1994). Constructivist, interpretivist approaches to human inquiry. In N. K.
Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 118–137). SAGE.
Schwandt, T. A. (2015). The SAGE dictionary of qualitative inquiry (4th ed.). SAGE.
scrapinghub. (2020). What is web scraping? https://scrapinghub.com/what-is-web-scraping
Silverman, D. (2001). Interpreting qualitative data: Methods for analysing talk, text and interaction
(2nd ed.). SAGE.
310 References
Singer, E., & Bossarte, R. M. (2006). Incentives for survey participation: When are they “coercive”? American Journal of Preventive Medicine, 31(5), 411–418. https://doi.org/10.1016/j.
amepre.2006.07.013
Singer, E., & Ye, C. (2013). The use and effects of incentives in surveys. The ANNALS
of the American Academy of Political and Social Science, 645(1), 112–141. https://doi.
org/10.1177/0002716212458082
Sirkin, R. M. (2006). Statistics for the social sciences (3rd ed.). SAGE.
Skukauskaite, A., Noske, P., & Gonzales, M. (2018). Designing for discomfort: Preparing
scholars for journeys through qualitative research. International Review of Qualitative
Research, 11(3), 334–349. https://doi.org/10.1525/irqr.2018.11.3.334
Small, M. L. (2011). How to conduct a mixed methods study: Recent trends in a rapidly
growing literature. Annual Review of Sociology, 37, 57–86. https://doi.org/10.1146/annurev.
soc.012809.102657
Smith, J. K. (1983). Quantitative versus qualitative research: An attempt to clarify the issue.
Educational Researcher, 12(3), 6–13. https://doi.org/10.3102/0013189X012003006
Sparkes, A. C., & Smith, B. (2014). Qualitative research methods in sport, exercise and health:
From process to product. Routledge.
Spradley, J. P. (1979). The ethnographic interview. Wadsworth/Cengage Learning.
St. Pierre, E. A. (2016). The long reach of logical positivism/logical empiricism. In N. K.
Denzin & M. D. Giardina (Eds.), Qualitative inquiry through a critical lens (pp. 19–29). Routledge.
Stake, R. E. (2010). Qualitative research: Studying how things work. Guilford Press.
Stenbacka, C. (2001). Qualitative research requires quality concepts of its own. Management
Decision, 39(7), 551–555. https://doi.org/10.1108/EUM0000000005801
Stockemer, D. (2019). Quantitative methods for the social sciences: A practical introduction with
examples in SPSS and Stata. Springer Nature.
Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for
developing grounded theory (2nd ed.). SAGE.
Street, A. (1995). Nursing replay: Researching nursing culture together. Churchill Livingstone.
Strong, G. (2019). Understanding quality in research: Avoiding predatory journals. Journal of
Human Lactation, 35(4), 661–664. https://doi.org/10.1177/0890334419869912
Swaminathan, R., & Mulvihill, T. M. (2017). Critical approaches to questions in qualitative
research. Routledge.
Tashakkori, A., & Teddlie, C. (2003). Preface. In A. Tashakkori & C. Teddlie (Eds.), Handbook of
mixed methods in social & behavioral research (pp. ix–xv). SAGE.
Taylor, S. (2013). What is discourse analysis? Bloomsbury Academic.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of
mixed methods in social & behavioral research (pp. 3–50). SAGE.
Teddlie, C., & Tashakkori, A. (2011). Mixed methods research: Contemporary issues in an
emerging field. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research
(4th ed., pp. 285–300). SAGE.
Thorne, S. (2008). Interpretive description. Left Coast Press.
Thorne, S., Oliffe, J., Kimsing, C., Hislop, T.G., Stajduhar, K., Harris, S.R., Armstrong, E.A.,
& Oglov, V. (2010). Helpful communications during the diagnostic period: An interpretive
description of patient preferences. European Journal of Cancer Care, 19, 746–754. https://doi.
org/10.1111/j.1365-2354.2009.01125.x
Tourangeau, R., & Bradburn, N. M. (2010). The psychology of survey response. In P. V.
Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 315–346). Emerald
Group.
References 311
Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative
research. Qualitative Inquiry, 16(10), 837–851. https://doi.org/10.1177/1077800410383121
Tracy, S. J. (2020). Qualitative research methods: Collecting evidence, crafting analysis, communicating impact (2nd ed.). John Wiley .
Tracy, S. J., & Rivera, K. D. (2010). Endorsing equity and applauding stay-at-home moms: How
male voices on work-life reveal aversive sexism and flickers of transformation. Management
Communication Quarterly, 24(1), 3–43. https://doi.org/10.1177/0893318909352248
Tubbs, F. (2018, November 21). The impact of big data on marketing. The Data Administration
Newsletter. https://tdan.com/the-impact-of-big-data-on-marketing/24044
van den Hoonaard, W. C., & van den Hoonaard, D. K. (2013). Essentials of thinking ethically in
qualitative research. Left Coast Press.
Vehovar, V., Toepoel, V., & Steinmetz, S. (2016). Non-probability sampling. In C. Wolf, D. Joye,
T. W. Smith, & Y. Fu (Eds.), The SAGE handbook of survey methodology (pp. 329–345). SAGE.
Wacquant, L. (2002). The curious eclipse of prison ethnography in the age of mass incarceration. Ethnography, 3(4), 371–397. https://doi.org/10.1177/1466138102003004012
Welford, C., Murphy, K., & Casey, D. (2012). Demystifying nursing research terminology: Part
2. Nurse Researcher, 19(2), 29–35. https://doi.org/10.7748/nr2012.01.19.2.29.c8906
Wentzel, A. (2018). A guide to argumentative research writing and thinking: Overcoming challenges. Routledge.
Wiles, R. (2013). What are qualitative research ethics? Bloomsbury Academic.
Wolcott, H. F. (1994). Transforming qualitative data: Description, analysis, and interpretation.
SAGE.
Wolcott, H. F. (2008). Ethnography: A way of seeing (2nd ed.). AltaMira Press.
Woolley, C. M. (2009). Meeting the mixed methods challenge of integration in a sociological study of structure and agency. Journal of Mixed Methods Research, 3(1), 7–25. https://doi.
org/10.1177/1558689808325774
World Medical Association. (2018). WMA Declaration Of Helsinki – ethical principles for medical research involving human subjects. https://www.wma.net/policies-post/wma-declaration
-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/
Yin, R. K. (2006). Mixed methods research: Are the methods genuinely integrated or merely
parallel. Research in the Schools, 13(1), 41–47.
Yin, R. K. (2009). Case study research: Design and methods, (4th ed.). SAGE.
Yin, R. K. (2014). Case study research: Design and methods (5th ed.). SAGE.
Yin, R. K. (2015). Causality, generalizability, and the future of mixed methods research. In
S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods
research inquiry (pp. 652–664). Oxford University Press.
Yin, R. K. (2016). Qualitative research from start to finish (2nd ed.). Guilford Press.
INDEX
anonymity, 27, 29, 32-40, 158, 233
pseudonyms and, 32-34
uses of, 33-39
versus confidentiality, 32-33
audit trail: see rigor in qualitative research
bias: see social desirability bias
CAQDAS: see computer-assisted qualitative
data analysis software
categories: see coding
codebook, use in qualitative analysis, 164-65
coding in qualitative analysis, 125, 152, 154,
155, 161-68, 176, 179
categories, 162-63
choice of strategy, 165-66
codes, 155, 162-65
critiques of, 168
data condensation, 159-61
definition of, 152, 161
in grounded theory, 166
inductive vs deductive approach to,
163-65, 228
when and why to use, 168
computer-assisted qualitative data analysis
software, 161-62
confidentiality of participants, 27, 29, 32-35,
37, 42, 158
description of, 32
versus anonymity, 32
consent: see informed consent
convergent design: see mixed methods
research
core component: see mixed methods
research
correlational approach: see quantitative data
analysis, approaches in
credibility: see qualitative research
data, 12
definition of, 13, 71-73, 206
demographic, 34, 126
digital, 37
internet information as, 37-39
interval/ratio in quantitative research,
201-2
methodological considerations, 72-74
nominal in quantitative research,
199-200, 202
ordinal in quantitative research, 200,
202, 204
photographs, 133
reuse and purposing, 35-37
sharing of, 35-37
data analysis: see qualitative data analysis;
quantitative data analysis
data collection: see qualitative data
collection; quantitative data
collection
data condensation: see coding in qualitative
analysis
‘declaring your hand’, 271-82, 285
deductive reasoning, 55-58, 93-94
description of, 56
inductive reasoning versus, 93
quantitative research use of, 93-94
descriptive approach: see quantitative data
analysis, approaches in
discourse analysis, 107-8
epistemology
definition of, 76
impact on research design, 76-78
see also: inquiry paradigms
ethics, 20-21, 27-43, 231, 283
and incentives for participation, 231
Belmont Report, 39
codes of, 41
definition of, 28
description of, 20-21
issues in, 34-35
professional guidelines, 41
see also: anonymity; confidentiality of
participants; informed consent;
vulnerable populations
ethics committees, 39-41, 173
ethnography: see qualitative research
experimental approach: see quantitative data
analysis, approaches in
external validity: see validity in quantitative
research
feasibility of a research design, 47, 51-53
focus groups, 130-31, 145, 277
moderator in, 130
313
314 Index
generalizability: see quantitative research
grounded theory, 155, 166-67
analytical strategy for, 166
coding in, 166-67
description of, 166
hypothesis
definition of, 56, 197
developing, 94, 196
power of a hypothesis test, 197-98
research question as,
standardized observations and, 128
testing, 94, 103, 197-98
hypothetico-deductive thinking, 94, 97
inductive reasoning
deductive reasoning versus, 93
description of, 57-58, 96-97
inquiry based on, 57, 82. 94-95
qualitative research use of, 94-95, 98, 164
informed consent
considerations for, 30-31
definition of, 29
for internet data, 38-39
layers of, 36-37
in longitudinal study, 31
purpose of, 29
for reusing, repurposing, and sharing of
data, 36
seeking, 30
vulnerable populations and, 31-32
written form, 30
inquiry paradigms, 78-85, 96. 99-101
anti-paradigmatic stance, 260-61
connection to methodology, 78, 82-85
constructivism, 81-82, 99, 100
definition of, 74
description of, 74-75
mixing, 259-61
paradigmatic stance, 74-76, 82-85
positivism, 78-80, 100, 142
post-positivism, 80-81, 100, 142
pragmatism, 260
see also: epistemology; ontology
interpretivism: see inquiry paradigms
interviews in qualitative research
data analysis in, 154-56
individual interviews, 129-31,
interview guide, 132, 134, 137-38
interview probes, 134
lines of inquiry, 132-36, 138
memos in, 155-56
online, 145
questions, designing, 135-36
recording of, 157-58
sample selection, 140-41
sample size, 172-73
structure of, 125-27
transcription, 157-58
trial interviews, 136-37
see also: focus groups
iteration
in qualitative data analysis, 152-61
in research design, 2-4, 18, 58-65,
281-284
schematic example of iterative
process, 283
layers of consent: see informed consent
literature
books and book chapters, 10
categorization of, 8
decision making about, 8-12
gray literature, 11-12
in research question development, 53-55
journal articles, 9-10
open access, 9-10
peer review in, 9
predatory journals, 10-11
relevant, 5-8
working with, 5-8
literature review, 7-8, 179
measurement in quantitative research
consistency, 226
level of, 115, 201-02
reliability of, 226-27
see also: validity in quantitative research
measurement instruments in quantitative
research
definition of, 199, 211
design and development of, 199, 213-29
implementation of, 207, 229-35
presentation of, 213-14, 231
pretesting of, 224-25, 234-35
purpose of,
questionnaires,
surveys: see surveys in quantitative
research
response rates for, 230-31
validity affected by, 229
see also: surveys
measurement items in quantitative research
consistent measurements obtained using,
226-27
definition of, 213
demands on respondent, 224
development of, 22024
form of, 227-29
response types for, 227-28
memos: see qualitative data analysis
methodology
data and, 71-74
definition of, 12, 14, 69, 71
Index 315
mixed methods research
characteristics of, 243-47
as contested field, 246-47
components in, 244-45
definitions of, 242-45
design, types of, 251
designing, 244-57
diagramming in, 262
mixing in, 257-61
paradigmatic-related considerations,
259-61
priority and timing of components, 249-53
purpose in, 248, 253-54
quasi-mixed designs, 261
reasons for using, 247-49, 253-54
strategies for, 262-65
typologies, 253
vs multimethod research, 245
see also: notation in mixed methods
research
mixing: see mixed methods research
moderator: see focus groups
multimethod research: see mixed methods
research
notation in mixed methods research
capitalization in, 249
description of, 249-51
+ sign, 251
→ sign, 251
online information, trustworthiness of, 12
onto-epistemological assumptions, 76-78
ontology
definition of, 76
impact on research design, 76-78
realism, 76
relativism, 76-77
see also: inquiry paradigms
paradigmatic stance: see inquiry paradigms
positivism: see inquiry paradigms
post-positivism: see inquiry paradigms
pseudonyms: see anonymity
qualitative data analysis
coding: see coding in qualitative analysis
interpretation, 168-71
memos, 154-56, 158-59, 162
overview of, 152-54
steps involved in, 153-54, 157-59,
timing of, 153, 157-58, 168-69
iterative and reflexive thinking in, 152-54,
157-61,
strategies for, 157
trustworthiness of, 151, 170-71
see also: coding in qualitative analysis
quantitative data analysis,
approaches to, 102-03
procedures in, 193-94
statistical analysis, 213, 223, 228-29
see also: thick description; thick
interpretation
qualitative data collection,
data saturation in, 171-72
description of, 98–99, 119–122, 145
nonstandardized observations, 129
reflexive thinking in, 127-29
sensitizing concepts, 137-39
strategies for, 157-59
theoretical saturation, 172
timing of, 156-57
quantitative data collection,
description of, 211-12
strategies for, 214-20,
randomized controlled trials, 102-03
standardized observations, 127-28
see also: measurement instruments;
measurement items; surveys in
quantitative research; variables;
validity in quantitative research
qualitative research
basic form of, 105
“big tent” criteria for, 109-11
data produced by, 92, 99
definition of, 92, 96
design considerations, 123
diversity of, 105–106, 110, 112, 121,
123–124
features of, 96-101
inductive reasoning in, 94-95
inquiry strategies, 122-25
ontological position of, 98
purpose of, 92-93
quantitative research and, 92-118
specialized forms of, 106-10
trustworthiness in, 99, 170-71
quantitative research
credibility of, 184-96, 205, 217
deductive reasoning in, 93-94
definition of, 92-93, 96-97
design considerations, 181-85
diversity of, 101-105
features of, 13, 96-101
generalizability of findings, 97, 142,
187-90
methods used in, 103
ontological position of, 97
purpose of, 92-93
qualitative research and, 92-118
quasi-experimental approach, 101-02, 196
statistics in, 182-83
quasi-experimental approach: see
quantitative research
316 Index
quasi-mixed design: see mixed methods
research
questionnaires: see measurement
instruments in quantitative research
randomized controlled trial: see quantitative
data collection
realism: see ontology
reasoning. see deductive reasoning;
inductive reasoning
reflexivity, 1, 18-19, 284
reflexive questioning, 54
reflexive research questions, 58
reflexive thinking, 18–21, 32, 124, 138,
156, 263–264, 266, 280
see also: qualitative data analysis
relativism: see ontology
reliability: see validity in quantitative
research
research
qualitative: see qualitative research
quantitative: see quantitative research;
mixed methods research
substantive area of, 5
theoretical framework of, 14
research area, 21, 50, 182-83
research ethics: see ethics
research design,
cyclic thinking in, 4
definition of, 2
see also: inquiry paradigm; methodology;
qualitative research, design
considerations; quantitative
research, design considerations
research problem
definition of, 50
focusing of, 51
key words in, 52
research questions
development of, 58-65
forms of, 55
as hypothesis, 197
iterative development, 58
literature used in development of, 53-55
response rate: see measurement
instruments in quantitative research
rigor in qualitative research, 168, 170-71
audit trail, 168
see also: qualitative data analysis,
trustworthiness of
sampling in qualitative research
size, 141-42, 173-75
decisions regarding, 140-43
strategies for, 140-45
sampling in quantitative research
considerations in, 185-90
estimation in, 186
size, 185-86
strategies for, 185, 187-191
saturation: see qualitative data collection
scientific method, 79
sensitizing concepts: see qualitative data
collection
social desirability bias, 232
statistical reasonableness: see quantitative
research
study population in quantitative research
analysis procedures for, 193-95
definition of, 185
description of, 185-86
research questions about, 195-96
selection of, 185-87, 192
statistically reasonable claims about, 187
see also: sampling in quantitative
research; variables
surveys in quantitative research, 103, 125,
213-14, 229-34
response rates for, 230-31
uses of, 103
see also: measurement instruments
in quantitative research,
questionnaires
theoretical saturation: see qualitative data
collection
theory
complexity of, 17
definitions of, 15-16
impact on research design, 16-17
thick description, 94, 99, 169
thick interpretation, 57, 99, 169
transcription: see interviews in qualitative
research
triangulation, 84, 171, 248
validity in quantitative research
construct validity, 61, 223
content validity, 61, 220
definition of, 213
internal validity, 90
external validity, 90, 187, 191, 212
of measurement, 226-27
statistical, 183, 212
vs reliability, 226-27
variables
construct of, 216-19
definition of, 56, 192, 199
effect of number of, 189
measurement of, 199-201, 212–222
operational definition of, 216-19, 221
vulnerable populations
definition of, 31
informed consent for, 31-32