Qualitative Research Studies

Results Overview

The purpose of this research is to determine if there is a link between supplemental media from Khan Academy and student academic performance. Furthermore, the research also seeks to determine the attitudes of teachers and students towards toward Khan Academy, particularly during the intervention period. The results are parsed and analyzed with the intention of providing clear conclusions as well as reasonable answers to research questions. First, the general findings of the data will be presented. These will be general conclusions derived from detailed analysis of the quantitative and qualitative data. The detailed analysis of the quantitative data is presented in terms of increasing complexity beginning with averages

derived from the data and ending with conclusions derived by hypothesis testing. The qualitative data is presented beginning with results from likert survey data for the teachers, and followed by student likert survey data. The last section of the data analysis will examine conclusions derived from teacher after-action interviews.
Data-Driven Findings
The findings in the data are interesting but statistically non-conclusive. Both the quantitative and qualitative research lacked sufficient data points to rule out chance being the primary factor in the results. Quantitatively, while there seemed to be a relatively consistent improvement of both mean and median average scores in the intervention group, when tested, the differences in the scores were well within non-rejection values, meaning that there was not a large enough difference or there were not enough data points to imply causation. This does not mean that there was no difference between the two groups, rather that the observed difference was not statistically robust enough to make confident assumptions about and causative relationship.
Qualitatively, teacher and student attitudes toward Khan Academy were positive and generally increased over time. The teachers that were part of the intervention group began with a generally positive view of Khan Academy, and this generally positive view seemed to increase during and immediately after the intervention. These attitudes were further confirmed in the after-action interviews of the teachers. The general sentiment of the after-action interviews was that Khan Academy videos reduced the total workload for the teachers in terms of both grading and lesson planning. These teachers generally concluded that the intervention was more efficient, and seemed to allow more time with individual students.
The students participating in the intervention group seemed to have a more initially positive view of Khan Academy. On average, the students seemed to find the website and mobile application easy to use. Many of the students personally expressed sentiments that claimed that their positive scores were primarily related to the ability to work on homework on their mobile phones when they had free time. Regardless of exact reasons, this positive sentiment of the students increased over time. While it is impossible to directly compare students and teacher attitudes, the general overlap of the question allows the conclusion that, by the end of the study, the students viewed the intervention even more favorably than the teachers.
Data Analysis
The data is broken into sections examining quantitative data from student assessment tests, qualitative likert data for teachers, qualitative likert data for students, and after-action interviews.

Quantitative Student Assessment Data. Student quantitative assessment data is broken into seven sections to most clearly present the data. Beginning with the most general conclusions and ending with specific hypothesis testing.

Mean average. Both the control group and intervention group of students had initial baseline assessments in the failing range, though this was not surprising considering the intervention specifically measured students that had not yet begun algebra instruction and likely only had pre-algebra instruction to guide them in completing the assessments. The largest improvement in test scores for both groups was between the baseline assessment test and the first test after the initial block of algebra instruction. Between the baseline and first test the control group seemed to improve more, though this improvement could be explained by regression to the mean considering the control group had lower initial scores. After the initial improvement, the intervention group improved more and improved more consistently than the control group. At the last data point, two weeks after the intervention, the intervention group had an average score approximately three and a half points higher than the control group (see Figure

1).

Figure 1. The change in the control and intervention means over time.
Median average. The median average of the two groups has similar results as the mean average, in that the intervention group improved slightly more than the control group, but the median average shows a larger difference between the two. The intervention group begins with a median average one point higher and ends with a median average almost five points higher than the control group. At the first test, both groups showed similar improvements over baseline, but after the first test, the scores for the intervention group improved significantly more than the control group.

Figure 2. Change in control and intervention medians over time.
Distribution and skew of Scores.

The standard deviation of scores decreased across both groups during the intervention, but spiked at the post-test. Given that the post-test was given two weeks after the end of the intervention, it seems plausible that increased algebra ability and increased familiarity with the assessment test caused the difference between the best and worse scores to decrease, but that this effect reversed after two weeks passed with no algebra instruction. It was also discovered through discussion with participating students that many of the highest scoring students continued using Khan Academy of their own volition outside of

school.

Figure 3. Measure of control and intervention standard deviations over time.
The skew of the test seems to indicate that a small group of high scoring of students in both the control and intervention group were responsible for the increase in scores at the post-test. Where the control group had significantly positive skews on the two tests after baseline, the intervention group seemed to be more symmetrically distributed. Both the control and intervention groups had significantly negative skews on the post-test implying that many moderately low scores were balanced by a few high scores in both groups.

Figure 4. Measure of control and intervention skew over time.