Deciding on the type of research design that is best suited to answer your research question can get a bit confusing. This article aims to summarize the basic understanding of the general research design methodologies as well as the type of answers each design can provide.
Ultimately, the question you want to answer will likely dictate the design of your overall project. However, understanding what the research methodology is able to answer is paramount to a well designed project. For example, if your clinical question is simply to determine the prevalence of a disease, then a cross-sectional study is probably what you want to be designing. If it is the incidence you are after, then a cohort study will give you that answer. Having a fundamental understanding of the different design options and the types of questions they can answer will serve you well early in the design process.
One way to look at the different types of clinical trials is those that are observational versus those that are interventional. Observational studies allow researchers to observe the effects of various risk factors, diagnostic tests and treatments without influencing who is or is not exposed to it. Specifically, with observational studies, researchers do not intervene, they just observe study subjects. Conversely, Interventional studies as the name would suggest, allow researchers to introduce an intervention and study the effects of that intervention. The hierarchy of clinical trials is outlined in the figure below. Understanding that each study design has its own purpose will help understand the global picture. Stick with me…
Observational (non-experimental) Studies
Observational studies can be prospective (forward-looking) or retrospective (backward-looking), depending on the specific design. However, you can’t exactly just make up what you want to do. For example, cohort studies can be designed prospective or retrospectively, while case control studies are typically just retrospective. A summary is outlined in the figure below.
Prospective studies: observe a group of people, or compare different groups (cohorts) over time, These studies look for relationships between lifestyle factors or environmental exposures and the development of conditions or diseases.
Retrospective studies: rely on study subjects who have already been diagnosed with a condition or disease, as well as a comparison group of subjects who do not have the condition or disease, called controls. Cases and controls are then asked about past exposures to observe any associations between exposure and disease.
Observational studies by design are typically quicker, cheaper and easier to conduct than interventional studies. Not to mention, significantly easier to get IRB approval. If you have the opportunity to design and implement an interventional study, go for it! However, unless you are working with a team or researchers or residents, this may serve quite difficult to accomplish alone during the course of your residency. In the figure above, I outlined the hierarchy of clinical trials in order of their quality of information.
Don’t get overwhelmed by the information in this article. The overall goal is to generate a baseline understanding of the different research methods, and to serve as a point of reference to point you in the right direction when you begin designing your project.
Let’s take a look at the different designs of observation studies…
Case Reports
Case reports are a description of clinical events of one or several patients in a narrative form. They are a low-hanging fruit for residents looking for a project as they can be written rather quickly. Case reports typically involve a literature search to gain understanding of current knowledge on the topic and this is reviewed in the form of discussion in the published case report and expanded on using the patient case as en example. The key here is the case can’t simply be rare. Case reports aren’t exactly easy to get published, because the market is flooded with them.
Finding cases that provide the reader with a unique clinical case that offers awareness and expanded knowledge will have the highest likelihood of getting published. If you are presented with a unique case that will add to medical literature, I highly recommend writing up a case report. Just don’t spend your entire residency waiting for the opportunity to present itself.
Published examples:
Further readings:
Case Series
There are several definitions of a case series. According to the Dictionary of Epidemiology, a case series is defined as “a collection of patients with common characteristics used to describe some clinical, pathophysiological or operational aspects of a disease, treatment or diagnostic procedures.” This can be retrospective or prospective and may be consecutive or nonconsecutive depending on whether all cases presenting to the reporting authors over a period were included, or only a selection.
While a case report is a description of clinical events of one or several patients, a case series will contain individual patient’s data like demography, diagnosis and management. This data can be presented as a table and pooled together if needed without the need for individual detailed description. There is no agreed upon definition for the minimum or maximum number of patients required to define a case series. Some authors suggest case series should contain no less than four patients, and no more than ten, but there does not appear to be a consensus definition.
Cross-Sectional Studies
A research design in which you collect data from many different individuals at a single point in time. There is no follow-up period, nor an assessment of the subject’s history. In cross-sectional research, you observe variables without influencing them. Such studies provide a snapshot of activity but often cannot determine which came first: the exposure or the disease.
Cross-sectional studies can he descriptive (describes) or analytical (explains). A cross-sectional descriptive survey assesses how frequently, widely, or severely the variable of interest occurs throughout a specific demographic (i.e. to assess the burden of a particular disease). Analytical design is used to investigate the association between a putative risk factor and a health outcome. In this design, investigators survey the risk factors and outcomes simultaneously. Again, you cannot determine which came first with this type of study. In practice, your study design might include descriptive or analytical components, or both.
Outcomes measured:
- Prevalence (fraction of a group of people possessing a clinical condition at a given point in time)
- Odds ratios
Potential Bias:
- Selection bias (sampling, response and non-response bias)
- Information bias (recall and detection bias)
- Confounding
Strengths/Weaknesses:
Cross sectional studies are relatively quick and easy to conduct since all the variables are collected at one point in time. Further, multiple outcomes and exposures can be studies rather rapidly.
These studies are useful at measuring prevalence, but cannot tell whether or not the outcome followed the exposure or in what time it occurred (cannot tell you incidence). Because it is an observational study, it is highly susceptible to bias. Because of this, cross-sectional studies are good for descriptive analysis and for generating hypotheses, but unlikely to give the level of evidence needed to change clinical guidelines.
Published examples:
Protocol examples:
Case Control Study
Case control studies look at groups of people. More specifically, it compares two groups: those with a disease or condition of interest (cases) and those who are similar in age and other demographic characteristics who do not have the condition or disease (controls). Researchers then compare the frequency of exposure prior to the disease diagnosis to one or more risk factors in both groups to help determine whether there’s a link between the past exposure and the development of the disease or condition being studied.
The study starts with an outcome, then traces back to investigate exposures and is therefore, by definition, always retrospective. When the subjects are enrolled in their respective groups, the outcome of each subject is already known by the investigator. This, and not the fact that the investigator usually makes use of previously collected data, is what makes case-control studies ‘retrospective’. Controls should be chosen who are similar to the cases. These factors (i.e., age, sex, time of exposure) are known as the “matching criteria.” This article and this article does a great job discussing the selection process and matching strategies.
Statistical Outcomes:
- Odds ratios (always include a confidence interval when calculating an odds ratio. This is usually done with computer programs such as Epi-info).
- CANNOT tell you anything about incidence or prevalence of a disease
4-steps to conducting a case-control:
- Enroll patients who already have a disease or outcome
- A second control group without the disease or outcome is sampled (should be of similar size and as identical in every other way)
- Subjects are asked about exposures/risk factors
- Calculate an Odds ratio
Potential Bias:
- Recall bias (patients with the outcome are likely to scrutinize the past, remembering negative details more clearly)
- Sampling bias (i.e. enrolling only patients who present to the hospital may not represent the community as a whole)
- Confounding (a variable which is associated with the exposure and is a cause of the outcome, which can be mitigated by matching controls to cases)
Strengths/Weaknesses:
Case-control studies are relatively quick with minimal funding needed. Unlike cross-sectional studies, they can be used for rare diseases (unlike cross-sectional studies) as smaller sample sizes are needed. Further, they can study multiple exposures. However, case-control studies, as with all other observational studies, are subject to bias. Further, case-control studies (like cross-sectional studies) cannot tell you the incidence of a disease.
Published examples:
Protocol examples:
Cohort Study
A cohort is a group of people with defined characteristics who are followed up to determine incidence of, or mortality from, some specific disease, all causes of death, or some other outcome. A cohort study can be retrospective (identifying diseased subjects by interview or written records) or prospective (identifying diseased subjects during a follow-up period).
Prospective cohort studies are designed with specific data collection methods and considered more complete. However, they can cost a lot of money and with a long follow-up period waiting for disease to occur, there is a high risk of losing subjects to follow-up. Loss to follow-up should not exceed 20% of the sample. This article does a great job discussing the selection process and methods to minimize loss to follow-up.
A retrospective (or historical) cohort study is advantageous in that it can be done at a low cost and in a relatively reasonable amount of time. This may be of particular importance to the resident who has 3-4 years to complete a project.
Subject Selection:
Selection of subjects is important, particularly in a cohort study. Subjects who are not at risk of developing an outcome should not be included in the study. The source population is determined by practical considerations, such as sampling. Subjects may be effectively sampled from the hospital, be members of a community, or from a doctor’s individual practice. A subset of these subjects will be eligible for the study. For in depth information about sample selection techniques, check out this article.
General steps of design:
- Identify exposed and unexposed cohort groups
- Prospective study: during follow-up period, identify diseased subjects (incidence of cases)
- Retrospective study: identify diseased subjects by interview or written records
- Analyze the differences (i.e., the incidence or relative risk) among those exposed (cohort 1) and those unexposed (cohort 2)
Outcomes measured:
- Incidence
- Mortality rate
- Case-fatality
- Odds Ratio (OR)
- Hazard Ratio (HR)
- Prevalence
- Relative Risk (RR)
- Risk Difference (RD)
- Standardized incidence ratio (SIR)
- Standardized Mortality Ratio (SMR)
Potential Bias:
- Attrition bias (prospective only)
- Selection bias (retrospective and prospective)
- Recall bias (retrospective only)
- Information bias (retrospective only)
Strengths:
Cohort studies can assess causality as information is gathered regarding a sequence of events. Further, they can examine multiple outcomes for a given exposure and can can determine rates of disease over time (i.e., incidence, relative risk). However, large number of subjects are needed for studying rare exposures. Like other observational trials, cohort studies are subject to bias. Prospective studies may be expensive and require long follow-up and susceptible to loss to follow-up. Retrospective studies are quick and cheaper, but susceptible to recall bias or information bias and there is less control over variables.
Published examples:
Protocol examples:
- Study protocol for a multicentre prospective cohort study to identify predictors of adverse outcome in older medical emergency department patients
- Study protocol for a prospective, longitudinal cohort study investigating the medical and psychosocial outcomes of UK combat casualties from the Afghanistan war: the ADVANCE Study
Meta-analysis
A meta-analysis is a statistical procedure for combining numerical data from multiple separate studies. A meta-analysis cannot be done without also doing a systematic review of the literature. Therefore, this section will contain information that is pertinent to both. Where a systematic review summarizes available literature followed by a critical appraisal, a meta-analysis refers to the statistical analysis of the data from independent primary studies focused on the same question.
Depending on the purpose of a review, reviewers may choose to undertake a rapid or systematic review. While the meta-analytic methodology is similar for systematic and rapid reviews, the scope of literature assessed tends to be significantly narrower for rapid reviews permitting the project to proceed faster.
The stages in conducting a meta-analysis are:
- Formulate a research question: define primary and secondary objectives.
- Identify relevant literature: construct a search strategy (rapid or systematic), screen studies and determine eligibility.
- Extract and consolidate study-level data:collect relevant study-level characteristics and experimental covarities and evaluate the quality of the studies.
- Data appraisal and preparation: compute appropriate outcome measure and evaluate the extent of between-study inconsistency (heterogeneity), perform relevant data transformations and select a meta-analytic model.
- Synthesize study-level data into summary measure: pool data and calculate summary measure and confidence intervals.
- Exploratory analyses: explore potential sources of heterogeneity (biological or experimental) and perform subgroup and meta-regression analyses.
- Knowledge synthesis: interpret findings and provide recommendations for future work.
Transparency and clarity in reporting can an issue for these studies. Reviews from organizations such as Campbell and Cochrane are particularly reliable, as all authors are required to adhere to the same standards of conduct and reporting. Lets briefly discuss these standards…
PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)
In 2009 an international group of experienced authors and methodologists developed PRISMA, which is a 27-item checklist and a four-phase flow diagram (see figures below) that outlines items deemed essential for transparent reporting of a systematic review and meta-analysis. Most reputable journals will require a protocol that includes this formal process for publication.
MECIR (Methodological Expectations of Cochrane Intervention Reviews)
The MECIR is a methodological standard to which all Cochrane Protocols, Review and Updates are expected to adhere to. They provide authors and users of the Cochrane Library with clear and transparent expectations of review conduct and reporting.
The MECIR Standards are divided into four sections:
- Standards for the conduct of new Cochrane Intervention Reviews (C1-C75).
- Standards for reporting of protocols of new Cochrane Intervention Reviews (PR1-PR44).
- Standards for reporting of new Cochrane Intervention Reviews (R1-R109).
- Standards for planning, conducting and reporting of updates of Cochrane Intervention Reviews (U1-U11, UR1-UR7).
Systematic Review
A systematic review is a detailed, systematic and transparent means of gathering, appraising and synthesizing evidence to answer a well-defined questions and summarizes available literature followed by a critical appraisal. It involves comprehensive search strategies that enable reviewers to identify ALL relevant studies of a topic. While it may end with a meta-analysis, it certainly does not have to. The goal of being systematic is to reduce bias by keeping a standardized approach to the review process. Following the appropriate reporting guidelines (PRISMA) and standards (MECIR) is recommended for all Systematic Review and Meta-analyses, as discussed in the previous section.
Comprehensive Review:
To comprehensively search the seemingly never-ending pool of available literature, search criteria must be sensitive enough not to miss any relevant studies. Using Boolean operators AND, OR and NOT should be used to combine keywords and index terms when searching for literature. A true comprehensive review can take up to 2 years to complete and will select a large amount of studies, most not relevant to the topic, yielding a specificity of < 10%. To minimize selection bias, it is recommended to have at least two independent reviewers.
Rapid Review:
The overall goal of a rapid review is to decrease the time needed to gather literature sources. Common shortcuts include narrowing search criteria, imposing date restrictions, using a single reviewer, omitting articles not in English, limiting database searches, etc. It is important to note that rapid reviews have not shown to sacrifice quality compared to comprehensive reviews.
Screening and Selection:
Inclusion and exclusion criteria must be defined before study screening can occur to ensure consistency. The entire screening and selection process should be reported in a PRISMA diagram.
Components of screening and selection include:
- Removing duplicates
- Screening for relevant studies by title and abstract
- Inspecting full texts to ensure they meet infusion and exclusion criteria
Free software tools that can assist with preparing reviews:
Further readings:
- The PRISMA statement for reporting systematic reviews and meta-analyse
- Cochrane Handbook for Systematic Reviews of Interventions
- What are the best methodologies for rapid reviews of the research evidence
- Design of Meta-Analysis Studies
- Meta-Analytic Methodology for Basic Research: A Practical Guide
- A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis
- A step by step guide for conducting a systematic review and meta-analysis with simulation data
- Standards for Systematic Reviews
- Systematic Reviews: A How-To Guide
- Methodology Series Module 6: Systematic analysis
Interventional (experimental) Studies
Interventional studies can be uncontrolled or controlled depending on whether or not they have control groups. Controlled studies can be further subcategorized as randomized (cohort allocation is based on random chance) on non-randomized (patients intentionally placed in different treatment arms).
Clinical trials can also be classified by their purpose:
- Prevention trials: assess ways to prevent diseases using medicine, vaccines, lifestyle changes, etc.
- Screening trials: assess the best way to detect diseases or health conditions
- Diagnostic trials: aim to improve methods to diagnose a disease or condition
- Treatment trials: tests new treatments, novel combinations of treatments or new approaches to surgeries or therapy
- Quality-of-life trials: attempts to find ways to improve the life for people living with diseases or health conditions
Clinical trials can have some extra flair as well:
- Cross-Over: when participants receive both treatment and placebo over time
- Double-blinded: when neither the participant nor the researcher know if the participant is receiving treatment or placebo
- Open label studies: when both the participant and the researcher know the participant is receiving treatment and not placebo
Interventional trial can be conducted incrementally, in phases:
- Phase 1 trials: first-in-human studies conducted in small groups of people to evaluate a new drug, test or procedure. This can include determining the safe dosage of a drug, the most effective way to administer it and identifying any serious side effects. Uncontrolled trials are often conducted in the early phases of interventional trials
- Phase 2 trials: involve slightly larger groups of participants to test efficacy and further test safety.
- Phase 3 trials: involve a large study population and provide a complete picture of safety and efficacy and to determine if the drug, test or procedure is as good as the current standard of care for a disease or condition. This phrase is typically designed as randomized, controlled trials.
- Phase 4 trials: are conducted in large populations after a drug, test or procedure has been introduced on the market to further monitor safety and potential long-term adverse outcomes. Since this phase occurs after an intervention reaches the market, this phase may be done as observational, long-term cohort studies.
Uncontrolled Trials
A control group is made up of a group of patients that do not receive the investigational drug, but instead take the standard treatment, approved for the disease on which the study is being carried out, or takes a placebo, which is a simulation of the investigational drug. An uncontrolled trial is rightfully named because it contains NO control group.
Uncontrolled trials are often used in the early phases of drug research (phases I and II) to determine pharmacokinetic properties or to investigate tolerated dose ranges. They can also be useful to study side effects, biochemical changes in long-term therapies, tolerance, interactions or efficacy of drugs.
As they can generate bias, the results of uncontrolled trials are considered less valid than those of controlled trials. The results obtained are usually compared with those obtained in previous studies or that have been published by other researchers.
Non-randomized Controlled Trial
Non-randomized trials use controls, but the study participants are not randomly assigned into each study arm (treatment group vs control group).
Controls commonly used:
- Placebo control
- “No treatment” control
- Active treatment control (i.e. comparing a new drug to a standard drug or to standard care)
- Control with dose comparison
- Historical control (information from controls are from patients treated at an earlier time or in a different setting)
Randomized Controlled Trial
A Randomized Controlled Trial (RCT) is generally used in experiments testing the effectiveness and/or safety of one or more interventions. Subjects are randomly assigned participants into an experimental group or a control group. As the study is conducted, the only expected difference between the control and experimental groups is the outcome variable being studied.
RCTs are suitable both for preclinical and clinical research. There should be sufficient uncertainty about the utility of an intervention. This is referred to as equipoise. RCTs are not suitable for investigating etiology or natural history of disease.
Observational studies are prone to bias. Bias is the tendency of any factors associated with the design, implementation, evaluation and interpretation of the results of a study to make the estimate of the effect of a treatment or intervention deviate from its true value. If two or more groups are being compared in an observational study, there are often systematic differences between the groups, so much so that the outcome of the groups may be different because of these differences rather than actual exposure or intervention. This is known as confounding. Randomization is the best way to minimize confounding by increasing the probability that the observed outcome is likely to be due to the intervention rather than other factors.
Outcomes measured:
Potential Bias:
- Volunteer bias (the population that participated may not be representative of the whole)
- Susceptibility
- Performance
- Detection follow-up
Strengths/Weaknesses:
Good randomization will wash out any population bias. RCT’s are easier to blind/mask than observational studies. Results can be analyzed with well known statistical tools and populations of participating individuals are clearly identified. However, RCT’s can be expensive to conduct, can take a very long time to conduct and therefore patient loss to follow-up is a concern.
Example Protocols:
The details of designing each individual methodology here exceeds the scope of what I am hoping to introduce here. Hopefully this article will give resident physicians and other junior researches a structured overview of the methodologies used in clinical research trials. Stay tuned for future articles in our introduction to clinical research series.
- Monoclonal Antibodies for COVID-19 – An evidence based update - January 18, 2022
- COVID-19 Vaccines in Pregnancy - January 6, 2022
- COVID-19 Boosters….are they necessary? - October 8, 2021