Recently, the English First English Proficiency Index (EPI) – Third Edition was released. People I talked with about this index have significant reservations about this index. However, before getting to these, let’s first get to the methodology.
As EF’s website states, “The EF English Proficiency Index calculates a country’s average adult English skill level using data from two different EF English tests completed by hundreds of thousands of adults every year. One test is open to any internet user for free. The second is an online placement test used by EF during the enrollment process before students start an English course. Both include grammar, vocabulary, reading, and listening sections.”
One of the tests is adaptive, while the other is not. There is also no testing of spoken English skills, which typically significantly lags the skills that were tested. The EF website does not indicate what the relative weighting of the two tests or the weighting of the skills tested. Finally, we do not know the sample sizes for each country.
Given the lack of detailed data, it is hard to assess how valid the results of the English First EPI are. Being based in Asia and travelling extensively throughout Asia, I have a hard time believing many of the rankings. At Qooco, we are focused on helping students with their spoken English proficiency using speech interactive mobile technology, so these results are important to us.
A useful starting point would be to compare the Programme for International Student Assessment (PISA) Scores. A cursory look will show that there is little correlation between the EPI and PISA in Asia. If we then look at a couple of countries, Singapore and Malaysia, nobody I know in these countries believe that Malaysia ranks higher than Singapore in English. Both businesses and educators in Malaysia have lamented the poor level of English in Malaysia for decades. In contrast, in Singapore, most people I know think that Singapore has quite a good level of English proficiency. Now, in fairness to the EF EPI, the samples in these two countries could be quite biased and as a result not be at all representative of the reality that much larger samples would show.
However, the limitations are not just restricted to these two countries. Given that Korea spends among the highest amount per capita on English tuition and its education system ranks among the highest in the world, it is hard to believe it ranks just above Indonesia, which has significant challenges with its educational system. These examples make it very difficult to take the EPI seriously in Asia. To address these shortcomings, we believe EF would be well-served in releasing this data for researchers to analyse properly and come to conclusions that can be justified. The data, as presented, do not help people in Asia, either from a policy perspective or a learning perspective. Perhaps the data are more useful for other regions, but we have no way of knowing.
What we would like to see is some assessment of spoken English, as this is by far the largest challenge in Asia. Furthermore, there has been no consistent correlation shown among many English tests with spoken English fluency. For example, the scores in the widely-taken College English Test in China have no demonstrable correlation with spoken English fluency. As a result, for most employers these scores have little meaning. It would be great to have the English First EPI include a spoken English Fluency component with enough data so that robust conclusions could be made by both external researchers and those at EF.
In December 2013, the 2012 PISA scores will be released, so we can analyse the correlation between the individual country EPI and PISA. It will be interesting to see these results.