Uniform Framework for Evaluating Research
The following excerpt from Life Journey Through Autism: A Parent’s Guide To Research is provided with the kind permission of The Organization for Autism Research.
There is a standard format used for reporting research in most professional journals that publish research studies. This section will help you read and analyze research articles by providing you with a framework for evaluating them. Research articles are typically organized into six sections in the following order: (1) Abstract, (2) Background or Introduction, (3) Methods or Methodology, (4) Results, (5) Discussion, and (6) References. This is the most common format for reporting research. You will probably find that some journals use a different format; nevertheless, you can still use this standard format as a guide for evaluating a research article.
The abstract of an article presents a summary of the research study. Its purpose is to provide the reader with a brief overview or synopsis of the article. As a general rule, the abstract contains a sentence or two from each of the five sections of the article and usually includes the following key points:
- The purpose of the study, or why the authors think it was important to conduct the study
- Information about the participants in the study; their ages, ethnic background, etc.
- The procedure used in the study
- The major results or findings of the study
- A summary of the findings of the study and why they are important
Often, as a first step in conducting research, the only section from an article that is needed is the abstract. The abstract can be printed from the Internet and usually provides enough information for you to decide whether the article is relevant to your child.
Background or Introduction
Research articles usually begin with some background information about autism and the specific research areas related to the study. Key points to look for in this section include:
What has previous research found about the topic?
How will this study add to the previous research?
A research study is not done in isolation; it is connected or related to other research. Therefore, research articles begin by summarizing previous research on the topic of interest. The purpose of summarizing previous research is to provide a framework or context for the present study. When referring to other research, authors cite the author and date of publication of each study, either in the text or at the end of a sentence, as illustrated in the box below. The full reference is provided in the references section at the end of the article. If during your reading, you come across a previous study that sounds interesting, you can get the full reference to the article in the reference section at the end of the article.
An Example of Background Information
In a previous study, Jones, Meyers, and Klein (2000) found that children with autism take longer to learn new tasks than other children. This finding has been confirmed in numerous other studies (Brown and Rogers 1989; Roberts 1999; Williamson, et al. 1995).
After summarizing previous research, authors will state how the study conducted in the present article adds to what has already been done.
What is the research question and hypothesis guiding the study?
Often (but not always), authors will state their research question(s) in the introduction and make a hypothesis about what they expect the study to find. A research question states the topic broadly. A hypothesis makes a specific prediction about what the researcher expects to find. Authors design a research study to answer a research question and/or test a hypothesis. A study can answer one or more than one research question. The boxes below provide examples of a research question and a hypothesis.
An Example of a Research Question
Does early intervention make a difference in the need for special education services once children with autism are in elementary school?
An Example of a Hypothesis
Children with autism who receive early intervention will be less likely to need special education services in elementary school.
Methods or Methodology
The purpose of the methods section is to provide readers with a blueprint of how the study was designed and conducted. This section is usually divided into subsections of: (1) research design, (2) subjects, (3) procedure, and (4) measures and instruments.
This section tells the reader about the design of the study. Things to look for in this section include
What type of study was used in the research?
Sometimes, the authors will tell the reader what type of study they used. Often, however, the type of study will not be explicitly stated, but clues will be evident in the study design. See Appendix C for specific examples of different types of studies.
Where was the research conducted?
Was it conducted in a laboratory, clinic, or “real-world” setting?
This section tells the reader about the people who participated in the study. Important points to look for in this section include:
How many subjects were in the study?
With the exception of case studies that are frequently based on only one subject or one case, it is important to know how many subjects were used in the study. As a general rule, the more subjects used, the more reliable the results of the study. However, it is also important to bear in mind that studies that conduct research with people with disorders such as autism will generally have fewer subjects than studies that use participants from the general population. This is because there are simply fewer people with autism overall.
What are the characteristics of the subjects in the study?
It is important to know something about the characteristics of the subjects in the study, such as their age and gender, to determine if they are representative of the larger population under investigation. For example, we know that among children with autism, four boys are affected to every one girl. Therefore, we would expect a good study on children with autism to show a similar ratio.
It is also desirable to have a racially and ethnically diverse group of subjects. With special populations, however, this can be difficult. Many studies are conducted in laboratories or special schools where participants volunteer to participate in the study. In this case, the subjects are self-selected and may not be representative of the general population.
What are the inclusion and exclusion criteria for subject selection?
As a general rule, research studies should include all eligible participants. However, there are circumstances when certain people might need to be excluded. For example, a study might want to look at characteristics of boys and not girls. Therefore, if the study excluded some possible participants, it should clearly state the reasons why they were excluded.
How were the participants assigned to groups?
If there is more than one group in the study, the methods section should explain how the participants were assigned to each group. Good research studies “randomly assign” participants to groups. Random assignment is a lot like “flipping a coin.” It means that each participant has an equally likely chance of being in any one group. The purpose of random assignment is to guard against bias being introduced into a study; that is, to ensure that there are no differences in the basic characteristics of participants in any one group as compared with any other group.
The procedure section explains how the study was conducted. Important things to look for regarding how the research was conducted include:
What steps did the researchers take to set up and complete the study?
This section should give a fairly detailed explanation of the procedure used in the study, including the method of collecting data. The purpose of this section is to enable another person to recreate the study exactly, given they had the equipment or skills to do so.
What kinds of data were collected?
The procedure section should also give specifics about what kinds of data were collected from the participants in the study, such as measures of knowledge or behavior. A good study collects sufficient data to answer all of the research questions posed at the beginning of the study.
Measures and Instruments
This section should provide specific information about the instruments and measures used to collect the data. Specific points to look for in this section include:
Do the instruments measure what they are supposed to measure?
Researchers should discuss whether the instruments measure what they are supposed to measure. In research terminology, this is referred to as validity. Authors will talk about validity in numerical terms, expressed as a decimal (e.g., .80). A perfectly valid instrument will have a validity of 1.0. Generally, instruments with percentage points above .75 are considered to have good validity. However, a lower score may be considered acceptable depending on whether or not other studies have used the instrument successfully.
Have other researchers used the instruments?
Generally, instruments that have been used by other researchers have more credibility. Authors should refer to other studies that have used the same instrument by referencing the author and the date of publication of the study in the text. If it is a new measure, not used in other studies or created specifically for this study, the authors should give a lot of detail about the measure, as well as an explanation about why it was chosen for use in the current study.
What are the findings of the study?
Research studies use sophisticated statistical methods to evaluate a study. The average layperson (and even some researchers!) will be unable to evaluate the statistical methods used in conducting research. Therefore, the best strategy for evaluating the findings of the study may be for you to get a general idea by looking at the tables and figures provided.
The tables present the average or mean scores for each group on the measures used in the study. By looking at these scores, the reader can get a general idea if one group scored much higher or lower than another. An example of a table presenting mean scores is provided below. The table tells us that the maximum score on the measures is 100. On the verbal skills test, the experimental group scored higher than the control group (82 vs. 71). There is an asterisk after this score to show that the difference is statistically significant. On the spatial skills test, the experimental group still scored higher than the control group (70 vs. 68), but this difference isn’t as large and therefore isn’t statistically significant. (See the next section for more about statistical significance).
The figures present the same information as the tables in a graphic format (often figures are presented as bar graphs, as illustrated in Figure 1 or line graphs as illustrated in Figure 2). By looking at the exhibits, the reader can see if there is a general pattern in which one group differs from another.
Not all studies present the findings in both tabular and/or graphic formats. Therefore, sometimes the reader will have to interpret the findings as best they can by reading the text. When reading the text, look for mean differences between groups.
What do the findings mean?
Differences in group scores do not necessarily indicate that the study is meaningful. In research, the findings are considered meaningful only if they are “statistically significant.” The findings are considered to be “statistically significant,” if the probability that the difference in scores could be due to chance is less than 5%. In research studies, statistical significance is expressed in text as “p < .05 or p < .01.”
The discussion section will explain the findings of the study in terms that are more easily understandable.
In the discussion section, researchers summarize the findings of the study, give their interpretation of the findings, and present their conclusions about the study. Key points to look for in this section include:
How do the authors summarize the findings?
The discussion section should begin with a summary of the findings of the study. Most authors summarize the findings in the first paragraph of this section.
How do the authors interpret the findings?
The discussion section gives authors the most “flexibility.” This is because the findings of a study are always open to interpretation. Authors usually present a very good argument for their conclusions about the findings of the study.
It is important to bear in mind that the discussion section of an article reflects the author's own interpretation of the findings of the study.
In addition to presenting their conclusions about the findings, authors should also discuss the limitations of their study. For example, was it limited by a small number of participants, or to a specific setting?
Are the findings applicable to different participants and other settings?
In the discussion section, authors should also discuss whether or not the findings are applicable to different participants and other settings. This will depend on the characteristics of the participants used in the study and whether or not the setting is applicable to the real world or limited to a laboratory setting. The researchers may end this section by talking about what research still needs to be done to answer the question more fully.
As stated previously, research does not occur in isolation. Authors conduct a literature review of relevant studies before conducting a study. In the references section, the authors list all research studies that they cited in the text. You can find full references to these articles in this section. If you want to look up a reference on one of the online databases, you can enter the author’s name in the author field or the title in the title field.
Beware of a Conflict of Interest
As an educated consumer of information, it is important for you to be aware that researchers may sometimes be influenced by a “conflict of interest.” For example, drug companies may offer funding to researchers to conduct studies testing their products. And, because their research is dependent on the company for funding, researchers can be under increased pressure to find evidence that supports the use of these products. Therefore, it is always wise to investigate the source of funding behind the research. You can find this information in the research article. In most professional research articles, authors acknowledge their source of funding by referencing the funding agency and grant number where applicable, either on the first page or right before the reference section at the end of an article.
In general, be skeptical when evaluating the research. Don’t be afraid to assess the merit of a study based on your own common sense. Always bear in mind that you are your child’s most important advocate, and there are many knowledgeable parents and professionals in the autism community with whom you can discuss misgivings and questions about the research.
In the final analysis, it is up to you to decide if a particular study makes sense, or has what researchers refer to as face validity. A parent is the leading expert when it comes to his or her own child. If something doesn’t “ring true” to you, trust your own instincts and make your own assessment.
The results section explains the statistical analyses of the data used in the study and presents the findings of the analyses in three formats: (1) narrative (written in text), (2) graphic (depicted in graphs), and (3) tabular (presented in tables). Key points to look for in this section include: