Sentiment and Sentence Similarity as Predictors of Integrated and Independent L2 Writing Performance
Keywords:
EFL Writing Performance, Independent Writing, Integrated Writing, Sentiment Analysis, Sentence Similarity, Task TypeAbstract
This study aimed to utilize sentiment and sentence similarity analyses, two Natural Language Processing techniques, to see if and how well they could predict L2 Writing Performance in integrated and independent task conditions. The data sources were an integrated L2 writing corpus of 185 literary analysis essays and an independent L2 writing corpus of 500 argumentative essays, both of which were compiled in higher education contexts. Both essay groups were scored between 0 and 100. Two Python libraries, TextBlob and SpaCy, were used to generate sentiment and sentence similarity data. Using sentiment (polarity and subjectivity) and sentence similarity variables, regression models were built and 95% prediction intervals were compared for integrated and independent corpora. The results showed that integrated L2 writing performance could be predicted by subjectivity and sentence similarity. However, only subjectivity predicted independent L2 writing performance. The prediction interval of subjectivity for independent writing model was found to be narrower than the same interval for integrated writing. The results show that the sentiment and sentence similarity analysis algorithms can be used to generate complementary data to improve more complex multivariate L2 writing performance prediction models.
Downloads
References
Araque, O., Zhu, G., & Iglesias, C. A. (2019). A semantic similarity-based perspective of affect lexicons for sentiment analysis. Knowledge-Based Systems, 165, 346-359.
Azamnouri, N., Pishghadam, R., & Meidani, E. N. (2020). The role of emotioncy in cognitive load and sentence comprehension of language learners. Issues in Language Teaching, 9(1), 29-55. https://doi.org/10.22054/ilt.2020.51543.485
Bailey, S. (2011). Academic writing: A handbook for international students (3rd ed.). Abingdon/New York, NY: Routledge.
Baştürkmen, H., & von Randow, J. (2014). Guiding the reader (or not) to re-create coherence: Observations on postgraduate student writing in an academic argumentative writing task. Journal of English for Academic Purposes, 16, 14-22.
Biber, D., Gray, B., & Staples, S. (2016). Predicting patterns of grammatical complexity across language exam task types and proficiency levels. Applied Linguistics, 37(5), 639-668.
Bird, S., Loper, E., & Klein, E. (2009). Natural language processing with Python. Sebastopol: O'Reilly Media Inc.
Carlson, S. (1988). Cultural differences in writing and reasoning skills. In A. C. Purver (Ed), Writing across languages and cultures: Issues in contrastive rhetoric (pp. 109-137). Newbury Park, CA: Sage.
Casal, J. E., & Lee, J. J. (2019). Syntactic complexity and writing quality in assessed first-year L2 writing. Journal of Second Language Writing, 44, 51-62.
Chen, I., Chang, C. (2009). Cognitive Load Theory: An Empirical Study of Anxiety and Task Performance in Language Learning. Electronic Journal of Research in Educational Psychology, 7(18), 729-746. http://dx.doi.org/10.25115/ejrep.v7i18.1369
Cheng, Y. S. (2004). A measure of second language writing anxiety: Scale development and preliminary validation. Journal of Second Language Writing, 13(4), 313-335. https://doi.org/10.1016/j.jslw.2004.07.001
Connor, U. (1996). Contrastive rhetoric: Cross-cultural aspects of second language writing. Cambridge, England: CUP.
Cooper, G. (1998, December). Research into cognitive load theory and instructional design at UNSW. Sydney, Australia: University of New South Wales. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.470.3428&rep=rep1&type=pdf
Crossley, S. & McNamara, D. (2011). Text coherence and judgments of essay quality: models of quality and coherence. In L. Carlson, C. Hoelscher, & T. F. Shipley (Eds.), Proceedings of the 29th annual conference of the cognitive science society (pp. 1236-1241). Austin, TX: Cognitive Science Society.
Crossley, S. A. (2013). Advancing research in second language writing through computational tools and machine learning techniques: A research agenda. Language Teaching, 46(2), 256-271.
Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35(2), 115-135.
Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 32, 1–16. https://doi.org/10.1016/j.jslw.2016.01.003
Crossley, S., Paquette, L., Dascalu, M., McNamara, D. S., & Baker, R. S. (2016). Combining click-stream data with NLP tools to better understand MOOC completion. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 6-14). ACM. https://doi.org/10.1145/2883851.2883931
Crystal, D. (2008) Dictionary of linguistics and phonetics (6th ed.). Oxford: Blackwell.
Cumming, A., Kantor, R., Baba, K., Eouanzoui, K., Erdosy, U., & Jamse, M. (2005). Analysis of discourse features and verification of scoring levels for independent and integrated prototype written tasks for the new TOEFL®. ETS Research Report Series, 2005(1), i-77.
DeCoursey, C. A., & Hamad, A. N. (2019). Emotions across the essay: What second-language writers feel across four weeks’ writing a research essay. English Studies at NBU, 5(1), 114-134. https://doi.org/10.33919/esnbu.19.1.6
Dewaele, J. M., & Alfawzan, M. (2018). Does the effect of enjoyment outweigh that of anxiety in foreign language performance? Studies in Second Language Learning and Teaching, 8(1). https://doi.org/10.14746/ssllt.2018.8.1.2
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
Fulwiler, T. (2002). College writing: A personal approach to academic writing (3rd ed.). Portsmouth, NH: Heinemann Boynton/Cook
Ghosal, T., Das, S. K., & Bhattacharjee, S. (2015). Sentiment analysis on (Bengali horoscope) corpus. In 2015 Annual IEEE India Conference (INDICON) (pp. 1-6), New Delhi, India. https://doi.org/10.1109/INDICON.2015.7443551.
Grabe, W., & Kaplan, R. B. (2014). Theory and Practice of Writing. Abingdon/New York, NY: Routledge.
Graham, S., Berninger, V., & Abbott, R. (2012). Are attitudes toward writing and reading separable constructs? A study with primary grade children. Reading & Writing Quarterly, 28(1), 51–69.
Graham, S., Harris, K. R., Kiuhara, S. A., & Fishman, E. J. (2017). The relationship among strategic writing behavior, writing motivation, and writing performance with young, developing writers. The Elementary School Journal, 118(1), 82-104.
Guo, J. D. (2018). Effect of EFL writing self-concept and self-efficacy on writing performance: Mediating role of writing anxiety. Foreign Language Research, 2, 69-74.
Guo, J., Zou, T., & Peng, D. (2018). Dynamic influence of emotional states on novel word learning. Frontiers in Psychology, 9, 1-12. https://10.3389/fpsyg.2018.00537
Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238. https://10.1016/j.asw.2013.05.002
Hall, C., & Sheyholislami, J. (2013). Using appraisal theory to understand rater values: An examination of rater comments on ESL test essays. Journal of Writing Assessment, 6(1), 1-17.
Halliday, M. A., & Matthiessen, C. M. (2014). Halliday's introduction to functional grammar. Oxford: Routledge.
Han, J., & Hiver, P. (2018). Genre-based L2 writing instruction and writing-specific psychological factors: The dynamics of change. Journal of Second Language Writing, 40(1), 44-59. https://doi.org/10.1016/j.jslw.2018.03.001
Harispe S., Ranwez S., Janaqi S., & Montmain J. (2015). Semantic similarity from natural language and ontology analysis. Synthesis Lectures on Human Language Technologies, 8, 1–254. https://doi.org/10.2200/S00639ED1V01Y201504HLT027
Hinkel, E. (1999). Objectivity and credibility in L1 and L2 academic writing. Culture in second language teaching and learning. Cambridge: Cambridge University Press.
Honnibal, M., & Johnson, M. (2015). An improved non-monotonic transition system for dependency parsing. In L. Màrquez, C. Callison-Burch, & J. Su (eds.), Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1373-1378). Lisbon, Portugal: Association for Computational Linguistics.
Honnibal, M., & Montani, I. (2017). spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. https://spacy.io/
Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI
Hwang, S. & Lee, M. (2008). Syntactic and referential markers ensuring objectivity in EFL essay writing. English Teaching, 63(4), 29-47.
Ishikawa, S. (2018). The ICNALE edited essays; A dataset for analysis of L2 English learner essays based on a new integrative viewpoint. English Corpus Studies, 25, 117-130.
Jacobs, H.L., Zinkgraf, S.A., Wormouth, D.R., Hartfiel, V.F., & Hughey, J.B. (1981). Testing ESL composition: A practical approach. Rowely, MA: Newbury House.
JASP Team (2020). JASP (Version 0.12.2)[Computer software]. Retrieved from https://jasp-stats.org/
Jianqiang, Z., & Xiaolin, G. (2017). Comparison Research on Text Pre-processing Methods on Twitter Sentiment Analysis. IEEE Access, 5, 2870-2879. https://doi.org/10.1109/ACCESS.2017.2672677.
Jung, Y., Crossley, S., & McNamara, D. (2019). Predicting Second Language Writing Proficiency in Learner Texts Using Computational Tools. The Journal of Asia TEFL, 16(1), 37-52. https://dx.doi.org/10.18823/asiatefl.2019.16.1.3.37
Kluyver, T., Ragan-Kelley, B., Pérez, F., Granger, B. E., Bussonnier, M., Frederic, J., ... & Ivanov, P. (2016). Jupyter Notebooks-a publishing format for reproducible computational workflows. In F. Loizides & B. Schmidt, Positioning and power in academic publishing: Players, agents and agendas (pp. 87-90). Amsterdam: IOS Press.
Kormos, J. (2012). The role of individual differences in L2 writing. Journal of Second Language Writing, 21, 390-403. https://doi.org/10.1016/j.jslw.2012.09.003.
Kulkarni, A., & Shivananda, A. (2019). Natural Language Processing Recipes. Unlocking Text Data with Machine Learning and Deep Learning using Python. Berkeley, CA: Apress. https://doi.org/10.1007/978-1-4842-4267-4.
Kumar, A., & Sebastian, T. M. (2012). Sentiment analysis: A perspective on its past, present and future. International Journal of Intelligent Systems and Applications, 4(10), 1-14.
Kyle, K. (2020). The relationship between features of source text use and integrated writing quality. Assessing Writing, 45, 100467. https://doi.org/10.1016/j.asw.2020.100467
Kyle, K., & Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34, 12-24.
Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge
Lee, C., Wong, K. C., Cheung, W. K., & Lee, F. S. (2009). Web-based essay critiquing system and EFL students' writing: A quantitative and qualitative investigation. Computer Assisted Language Learning, 22(1), 57-72.
Liu, B. (2010). Sentiment analysis and subjectivity. In N. Indurkhya & F. J. Damerau (eds.), Handbook of Natural Language Processing (2nd ed.) (pp. 627–666). Boca Raton, FL: Chapman & Hall/CRC.
Lo, J., & Hyland, F. (2007). Enhancing students’ engagement and motivation in writing: The case of primary students in Hong Kong. Journal of Second Language Writing, 16, 219-237. https://doi.org/10.1016/j.jslw.2007.06.002
Loria, S. (2020). TextBlob Documentation (Release 0.16.0). Retrieved from https://buildmedia.readthedocs.org/media/pdf/textblob/latest/textblob.pdf
MacIntyre, P. D., & Gregersen, T. (2012a). Emotions that facilitate language learning: The positive-broadening power of the imagination. Studies in Second Language Learning and Teaching, 2, 193-213. doi: 10.14746/ssllt.2012.2.2.4
McArthur, C.A., Jennings, A. & Philippakos, Z.A. (2019) Which linguistic features predict quality of argumentative writing for college basic writers, and how do those features change with instruction? Reading and Writing, 32, 1553–1574. https://doi.org/10.1007/s11145-018-9853-6
Miller, Z. F., Fox, J., Moser, J. S., & Godfroid, A. (2018). Playing with fire: Effects of negative mood induction and working memory on vocabulary acquisition. Cognition and Emotion, 32, 1105–1113. https://doi.org/10.1080/02699931.2017.1362374.
Munezero, M., Montero, C. S., Sutinen, E., & Pajunen, J. (2014). Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text. IEEE Transactions of Affective Computing, 5(2), 101-111. https://doi.org/10.1109/TAFFC.2014.2317187.
Nawal, A. F. (2018). Cognitive load theory in the context of second language academic writing. Higher Education Pedagogies, 3(1), 385-402. https://doi.org/10.1080/23752696.2018.1513812
Oliphant, T. E. (2006). A guide to NumPy. USA: Trelgol Publishing.
Parra G., L., & Calero S., X. (2019). Automated writing evaluation tools in the improvement of the writing skill. International Journal of Instruction, 12(2), 209-226. https://doi.org/10.29333/iji.2019.12214a
Peñafiel, M., Vásquez, S., Vásquez, D., Zaldumbide J., & Luján-Mora, S. (2018). Data Mining and Opinion Mining: A Tool in Educational Context. In Proceedings of the 2018 International Conference on Mathematics and Statistics (ICoMS 2018) (pp. 74-78). New York, NY: Association for Computing Machinery https://doi.org/10.1145/3274250.3274263
Plakans, L., & Gebril, A. (2017). Exploring the relationship of organization and connection with scores in integrated writing assessment. Assessing Writing, 31, 98–112. doi:10.1016/j.asw.2016.08.005
Provoost, S., Ruwaard, J., van Breda, W., Riper, H., & Bosse, T. (2019). Validating automated sentiment analysis of online cognitive behavioral therapy patient texts: An exploratory study [Provisional PDF]. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01065
Ranjan, S., & Sood, S. (2019). Investor community sentiment analysis for predicting stock price trends. International Journal of Management, Technology and Engineering, 9(5), 6012-6020.
Richards, J. C., & Miller, S. K. (2005). Doing academic writing in education: Connecting the personal and the professional. Mahwah, NJ: Erlbaum
Roscoe, R. D., Crossley, S. A., Snow, E. L., Varner, L. K., & McNamara, D. S. (2014). Writing quality, knowledge, and comprehension correlates of human and automated essay scoring. In W. Eberle & C. Boonthum-Denecke (eds.), Proceedings of the 27th International Florida Artificial Intelligence Research Society (FLAIRS) Conference (pp. 393-398). Palo Alto, CA: AAAI Press.
Sanders, T., & Maat, H. P. (2006). Cohesion and coherence: Linguistic approaches. Reading, 99, 440–466.
Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37, 141-188.
Wang, Y. (2020). Emotion and syntactic complexity in L2 writing: A corpus-based study on Chinese college-level students’ English writing. The Asian Journal of Applied Linguistics, 7(1), 1-17.
Weigle, C. S., & Parker, K. (2012). Source text borrowing in an integrated reading/writing assessment. Journal of Second Language Writing, 21(2), 118-133. https://doi.org/10.1016/j.jslw.2012.03.004.
Wiebe, J., Wilson, T., Bruce, R., Bell, M., & Martin, M. (2004). Learning subjective language. Computational linguistics, 30(3), 277-308.
Yang, W., & Sun, Y. (2012). The use of cohesive devices in argumentative writing by Chinese EFL learners at different proficiency levels. Linguistics and education, 23(1), 31-48.
Yoon, H., & Hirvela, A. (2004). ESL student attitudes toward corpus use in L2 writing. Journal of Second Language Writing, 13, 257–283. https://doi.org/10.1016/j.jslw.2004.06.002
