Document Type : Articles
Author
Institute for Humanities and Cultural Studies
Abstract
This paper provides a comparative analysis of cross-lingual word embedding by studying the impact of different variables on the quality of the embedding models within the distributional semantics framework. Distributional semantics is a method for the semantic representation of words, phrases, sentences, and documents. This method aims at capturing as much information as possible from the contextual information in a vector space. The early study in this domain focused on monolingual word embedding. Further progress used cross-lingual data to capture the contextual semantic information across different languages. The main contribution of this research is to make a comparative study to find out the superior impact of the learning methods, supervised and unsupervised in training and post-training approaches in different embedding algorithms, to capture semantic properties of the words in cross-lingual embedding models to be applicable in tasks that deal with multi-languages, such as question retrieval. To this end, we study the cross-lingual embedding models created by BilBOWA, VecMap, and MUSE embedding algorithms along with the variables that impact the embedding models' quality, namely the size of the training data and the window size of the local context. In our study, we use the unsupervised monolingual Word2Vec embedding model as the baseline and evaluate the quality of embeddings on three data sets: Google analogy, mono- and cross-lingual words similar lists. We further investigated the impact of the embedding models in the question retrieval task.
Keywords
AleAhmad, A., Amiri, H., Darrudi, E., Rahgozar, M. & Oroumchian, F. (2009). Hamshahri: A standard Persian text collection. Knowledge-Based Systems, 22(5), 382–387.
Ansari, E., Sadreddini, M., Tabebordbar, A. & Wallace, R. (2014). Extracting Persian–English parallel sentences from document level aligned comparable Corpus using bi-directional translation. Advances in Computer Science: An International Journal, 3(5), 59–65.
Artetxe, M., Labaka, G. & Agirre, E. (2018a). A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics,1, (pp: 789–798), Melbourne, Australia. Association for Computational Linguistics.
Artetxe, M., Labaka, G. &Agirre, E. (2017). Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,1, (pp: 451–462), Vancouver, Canada. Association for Computational Linguistics.
Artetxe, M., Labaka, G., Agirre, E. & Cho, K. (2018b). Unsupervised neural machine translation.
Assi, S. (1997). Farsi linguistic database (FLDB). International Journal of Lexicography, 10(3), 5.
Bijankhan, M. (2004). naqše peykarehāye zabāni dar neveštane dasture zabān: mo'arrefiye yek narmafzāre rāyāneyi ["The role of corpora in writing a grammar: Introducing a software"]. Journal of Linguistics, 19(2), 48–67.
Bijankhan, M., Sheykhzadegan, J., Bahrani, M. & Ghayoomi, M. (2011). Lessons from building a Persian written corpus: Peykare. Language Resources and Evaluation, 45(2), 143–164.
Blei, D. M., Ng, A. & Jordan, M. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
Camacho-Collados, J., Pilehvar, M., Collier, N. & Navigli, R. (2017). SemEval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), (pp. 15–26), Vancouver, Canada. Association for Computational Linguistics.
Chandar, S., Lauly, S., Larochelle, H., Khapra, M. M., Ravindran, B., Raykar, V. & Saha, A. (2014). An auto-encoder approach to learning bilingual word representations. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, , (pp. 1853–1861).
Chaudhary, A., Zhou, C., Levin, L., Neubig, G., Mortensen, D. R. & Carbonell, J. G. (2018) Adapting word embeddings to new languages with morphological and phonological subword representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, (pp. 3285–3295).
Conneau, A., Lample, G., Ranzato, M., Denoyer, L. & Jégou, H. (2018). Word translation without parallel data. In sixth International Conference on Learning Representations.
Coulmance, J., Marty, J. M., Wenzek, G. & Benhalloum, A. (2015). Transgram, fast cross-lingual word-embeddings. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, (pp. 1109–1113).
Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. (2019). BERT: Pretraining of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (pp. 4171–4186), Minneapolis, Minnesota. Association for Computational Linguistics.
Doval, Y., Camacho-Collados, J., Espinosa Anke, L. & Schockaert, S. (2020) On the robustness of unsupervised and semi-supervised cross-lingual word embedding learning, In Proceedings of the 12th Language Resources and Evaluation Conference, (pp. 4013–4023), Marseille, France, European Language Resources Association.
Doval, Y., Camacho-Collados, J., Espinosa-Anke, L. & Schockaert, S. (2018). Improving cross-lingual word embeddings by meeting in the middle, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, (pp. 294–304), Brussels, Belgium. Association for Computational Linguistics.
Duong, L., Kanayama, H., Ma, T., Bird, S. & Cohn, T. (2017). Multilingual training of cross-lingual word embeddings, In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, (pp. 894–904), Valencia, Spain. Association for Computational Linguistics.
Duong, L., Kanayama, H., Ma, T., Bird, S., & Cohn, T. (2016). Learning cross-lingual word embeddings without bilingual corpora, In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, USA, Novem
Espinosa-Anke L., Palmer G., Corcoran P., Filimonov M., Spasić I. & Knight D. (2021). English–Welsh cross-lingual embeddings, Applied Sciences, 11(14), 6541.
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G. & Ruppin, E. (2002). Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20, 116–131.
Firth, J. R. (1957). A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis. (Special Volume of the Philological Society), 1–32.
Ghayoomi, M. (2019). Finding the meaning of Persian words automatically using word embedding. Iranian Journal of Information Processing & Management, 35(1), 25–50.
Gouws, S. & Søgaard, A. (2015). Simple task-specific bilingual word embeddings. In HLT-NAACL (pp. 1386-1390).
Gouws, S., Bengio, Y. & Corrado, G. (2015). Bilbowa: Fast bilingual distributed representations without word alignments. In International Conference on Machine Learning (pp. 748-756). PMLR.
Gu, J., Hassan, H., Devlin, J. & Li, V. O. K. (2018a). Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (pp. 344–354).
Gu, J., Wang, Y., Chen, Y., Cho, K. & Li, V. O. K. (2018b). Meta-learning for low-resource neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, (pp. 3622–3631).
Guo, J., Che, W., Yarowsky, D., Wang, H. & Liu, T. (2015). Cross-lingual dependency parsing based on distributed representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1234-1244).
Ha, T. L., Niehues, J., Sperber, M., Pham, N. Q. & Waibel, A. (2018). KIT-Multi: A translation-oriented multilingual embedding corpus. In Proceedings of the 11th International Conference on Language Resources and Evaluation, (pp. 3904–3907), Miyazaki, Japan. European Language Resources Association (ELRA).
Hadifar, A. & Momtazi, S. (2018). The impact of corpus domain on word representation: a study on persian word embeddings. Language Resources and Evaluation, 52(4), 997–1019.
HajiAminShirazi, S. & Momtazi, S. (2020). Cross-lingual embedding for cross-lingual question retrieval in low-resource community question answering. Machine Translation, 34(4).
Harris, Z. S. (1954). Distributional structure. Word, 23(10), 146–162.
Harris, Z. S. (1991). A Theory of Language and Information: A Mathematical Approach. Oxford University Press, Oxford, England.
Hermann, K. M. & Blunsom, P. (2013). The role of syntax in vector space models of compositional semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 1, (pp. 894–904), Sofia, Bulgaria.
Jabbari, F., Bakshaei, S., Ziabary, S. M. M. & Khadivi, S. (2012). Developing an open-domain English-Farsi translation system using AFEC: Amirkabir bilingual Farsi-English corpus. In Fourth Workshop on Computational Approaches to Arabic-Script-based Languages (pp. 17-23).
Jurafsky, D. & Martin, J. H. (2000). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall, Upper Saddle River, New Jersey.
Landauer, T. K. & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240.
Lauly, S., Boulanger, A. & Larochelle, H. (2014). Learning multilingual word representations using a bag-of-words autoencoder. arXiv preprint arXiv:1401.1803.
Levy, O. & Goldberg, Y. (2014). Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 302-308).
Lin, D. (1998). Automatic retrieval and clustering of similar words. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2 (pp. 768-774).
Luong, M. T., Pham, H. & Manning, C. D. (2015). Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st workshop on vector space modeling for natural language processing (pp. 151-159).
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Miller, G. A. & Charles, W. G. (1991). Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1), 1–28.
Ney, H., Och, F. J. & Vogel, S. (2000). Statistical translation of spoken dialogues in the Verbmobil system. In Workshop on Multilingual Speech Communication, (pp. 69–74), Kyoto, Japan.
Padó, S. & Lapata, M. (2007). Dependency-based construction of semantic space models. Computational Linguistics, 33(2), 161–199.
Pan, X., Gowda, T., Ji, H., May, J. & Miller, S. (2019). Cross-lingual joint entity and word embedding to improve entity linking and parallel sentence mining. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019) (pp. 56-66).
Peirsman, Y. & Geeraerts, D. (2009) Predicting strong associations on the basis of corpus data. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009) (pp. 648-656).
Pennington, J., Socher, R. & Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).
Ruder, S., Vulic, I. & Søgaard, A. (2019). A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65(1), 569–631.
Sahlgren, M. (2006). The Word-Space Model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces (Doctoral dissertation, Institutionen för lingvistik).
Salton, G. M., Wong, A. & Yang, C.-S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620.
Schuster, S., Gupta, S., Shah, R. & Lewis, M. (2019). Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (pp. 3795–3805), Minneapolis, MN, USA.
Shaoul, C. & Westbury, C. (2010). The Westbury Lab Wikipedia Corpus. Retrieved from http://www.psych.ualberta.ca/~westburylab/downloads/westburylab.wikicorp.download.html.
Shi, T., Liu, Z., Liu, Y. & Sun, M. (2015). Learning cross-lingual word embeddings via matrix co-factorization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, 2, (pp.567–572).
Song, L., Wang, Z., Mi, H. & Gildea, D. (2016). Sense embedding learning for word sense induction. arXiv preprint arXiv:1606.05409.
Vyas, Y. & Carpuat, M. (2016). Sparse bilingual word representations for cross-lingual lexical entailment. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies (pp. 1187-1197).
Wittgenstein, L. (1953). Philosophical Investigations. Blackwell Publishing Ltd, Oxford, UK.
Xu, L., Ouyang, W., Ren, X., Wang, Y. & Jiang, L. (2018). Enhancing Semantic Representations of Bilingual Word Embeddings with Syntactic Dependencies. In IJCAI (pp. 4517-4524).
Zahedi, M. S., Bokaei, M. H., Shoeleh, F., Yadollahi, M. M., Doostmohammadi, E. & Farhoodi, M. (2018). "Persian word embedding evaluation benchmarks," In Proceedings of IEEE Iranian Conference of Electrical Engineering, (pp:1583-1588).
', './data/ijism/coversheet/661672549534.jpg'))