KUMON: HYPE OR HIGH YIELD, A META-ANALYSIS

I wanted to look at the efficacy of Kumon math programs. Theoretically, Kumon makes a lot of sense to me. Students have an individualized curriculum, based on their individual learning needs. Students progress at their own rate, and are primarily taught through ‘skill and drill.’ This is very similar to how I teach math, moreover, there is significant evidence that an individualized curriculum can improve student achievement outcomes. A 2016 meta-analysis by Steenbergen-Hu et al, showed that individualized curriculum for students had a mean effect size of 2.35, which is so high, it meets the literal definition of a “super effect size”.

That being said, when I look to establish the efficacy of an intervention, I normally rely on meta-analysis. However, to the best of my ability, I could not find a meta-analysis on the topic of Kumon, so I decided to conduct my own meta-analysis. Upon searching Education Source, Jstor, and Google Scholar, I was able to find 5 relevant studies. Three studies showed a large effect size in favor of Kumon, one study showed a moderate effect size in favor of Kumon, and one study showed a moderate negative effect size against Kumon. I re-calculated the effect sizes of each study using a Hedge’s g effect size formula.

 
Results:

Kumon 1.png
Kumon 2.png
Kumon 3.png

Summary of Individual Studies:

I searched through the Jstor, and Education Source catalogue for this analysis. However, I was only able to locate 1 relevant study from these databases. I was able to find 6 additional studies on Google Scholar, where they conducted an experiment focusing on Kumon interventions. I excluded one study because it included insufficient data to come to an effect size. I excluded another study because the sample size was 5.

Effectiveness of Kumon Teaching Method for Academic Achievement of Children in Mathematics:

This paper was written by, Jamila Begum, Muhammad Maqsood, Alam Bukhari, and Aisha Akbar, in 2018, for the Pakistan Journal of Education. Admittedly, I think this paper/experiment was by far the best-done experiment of the 7 mentioned studies. This study included 215 grade 5 students. The study was 12 weeks long with a well-managed control group. Control groups and experimental groups were made based on pre-test data so that both groups had the same mean-pre-test results. The experiment group received no additional training after the initial training. Both groups were divided so teachers had the same amount of training and experience.  

 

Developing Latent Mathematics Abilities in Economically Disadvantaged Students:

This paper was written by Michele A. McKenna Patricia L. Hollingsworth Laura L. B. Barnes in 2005 for Roeper Review. The study included 421 students from grades 2-4, from a disadvantaged school. The control group came from the same school. This study was very interesting, as it broke down its results in a variety of ways. Interestingly the Kumon group outperformed the control group in every category. Aswell, they also re-tested their students a year after the initial experiment. Not only did the Kumon students outperform the control students a year later, but they also did so by a very large effect size. For conceptual math, the Kumon group outperformed by an effect size of .85 and for computation, the Kumon group outperformed by an effect size of 1.04. This was especially interesting as a common critique of Kumon instruction is that it is too focused on procedural math. However, this study clearly indicated a strong benefit for conceptual knowledge. I will also say that this study had the second-best study design. 

 

 Study of the Effects of the Kumon Method Upon the Mathematical Development of a Group of Inner-City Junior High School Students.

 

This study was written by Suzanne Medina in 1989. This study conducted an experiment on 103 students from grades 6 to 8. They did not have a control group and used a pre-test post-analysis. This type of analysis is not the best experimental type and often produces inflated results. However, the results were moderate to high. Their study was 7 months long, 5 days a week, for 50 minutes per day. This intervention was quite intensive and I was surprised that the results were so moderate. They used both a CAT test and a Kumon test. However, the CAT test data did not include enough information to calculate a Hedge’s g effect size, so I was forced to use their Kumon test data. This study showed an effect size of .69. This is a larger than average effect size, but still within the moderate range. Their study showed a substantially higher benefit to procedural knowledge over conceptual knowledge. 



The effect of Kumon learning model on mathematics learning outcomes in cognitive style view:

This paper was written by Sulasteri et al, in 2020 for the Journal of Physics. This study was specifically looking to see if Kumon benefited different styles of learners. While the study did have a control group, the experimental group was only 32 grade 8 students. Moreover, this was the only study with a negative result for the Kumon group, which made me think its data was an outlier. Moreover, the study did not include very much information in regards to how they designed the experiment, nor did they include the duration of the experiment. 

 

The Effect of Application Kumon Learning Method in Learning Mathematics of Ability Troubleshooting Mathematics of Students: 

This paper was written by Usmadi et al, in 2020 for the Journal of Physics. This study did not use a control group and used a pre-test post comparison, which usually inflates effect size results. The study also did not include the age of the students or the duration of the experiment. Moreover, they did not include their own standard deviation, or enough of their raw data to properly calculate their standard deviation. I attempted to calculate it based on the data they did include, however, the resulting effect size was extremely large. I therefore think the effect size was an outlier, and is meaningless for comparison. 
 

Discussion: 

With only 5 studies included and only 2 well-designed studies, these results are far from conclusive. However, they are very promising. In general, the effect sizes found in this literature suggest that Kumon interventions are a high yield intervention. That being said, Kumon is a complex program with many components. It is impossible to say for sure, which parts of Kumon lead to its success, for example, individualization has been shown to have a very large effect size in itself and Kumon uses individualization. Indeed the effect size for Individualization is more than double the mean effect size for Kumon. Perhaps most interesting, was how the results differed for conceptual vs procedural knowledge. As stated previously, a common criticism of Kumon is the fact that it appears weighted for Procedural knowledge. That being said, while the data did show a greater benefit for procedural knowledge, the data also showed that Kumon benefitted students’ conceptual knowledge by a large degree.

References:

McKenna, M. A., Hollingsworth, P. L., & Barnes, L. L. B. (2005). Developing Latent Mathematics Abilities in Economically Disadvantaged Students. Roeper Review, 27(4), 222–227. https://doi-org.ezproxy.lakeheadu.ca/10.1080/02783190509554322

Begum, et al. (2020). Effectiveness of Kumon Teaching Method for Academic Achievement of Children in Mathematics. Pakistan Journal of Education. Retrieved from <https://pdfs.semanticscholar.org/3ba0/d3b4a989aa3e30dedcb6dc2d5bf94121fa5f.pdf>. 

S, Medina. (1989). Study of the Effects of the Kumon MethodUpon the Mathematical Development of a Group of Inner-City Junior High School Students. Retrieved from <https://files.eric.ed.gov/fulltext/ED331700.pdf>.

Usmadi, et al. (2020). The Effect of Application Kumon Learning Method in Learning Mathematics of Ability Troubleshooting Mathematics of Students. Journal of Physics. Retrieved from <https://iopscience.iop.org/article/10.1088/1742-6596/1429/1/012005/pdf>. 

Sulasteri, et al. (2020). The effect of Kumon learning model on mathematics learning outcomes

in cognitive style view. Journal of Physics. 

NELSON, G.; MCMASTER, K. L. The Effects of Early Numeracy Interventions for Students in Preschool and Early Elementary: A Meta-Analysis. Journal of Educational Psychology, [s. l.], v. 111, n. 6, p. 1001–1022, 2019. DOI 10.1037/edu0000334. Disponível em: http://search.ebscohost.com.ezproxy.lakeheadu.ca/login.aspx?direct=true&db=eue&AN=137730895&site=ehost-live. Acesso em: 25 jul. 2020.

Methe, S., Kilgus, S., Neiman, C., & Riley-Tillman, T. (2012). Meta-Analysis of Interventions for Basic Mathematics Computation in Single-case Research. Journal of Behavioral Education, 21(3), 230-253. Retrieved July 25, 2020, from www.jstor.org/stable/43551228Copy

R, Getsen, Et al. (2009). A Meta-analysis of Mathematics Instructional Interventions for Students with Learning Disabilities: Technical Report. Instructional Research Group. Retrieved from <https://www.researchgate.net/publication/266864172_A_Meta-analysis_of_Mathematics_Instructional_Interventions_for_Students_with_Learning_Disabilities>