Representation Learning for Natural Language Processing
by Zhiyuan Liu, Associate Professor of Department of Computer Science and Technology at Tsinghua University, China
Representation learning aims to learn informative representations of objects from raw data automatically. The learned representations can be further fed as input to machine learning systems for prediction or classification. In this way, machine learning algorithms will be more flexible and desirable while handling large-scale and noisy unstructured data, such as speech, images, videos, time series, and texts.
We focus on the distributed representation for natural language processing (NLP). NLP aims to build linguistic-specific programs for machines to understand languages. Natural language texts are typical unstructured data, with multiple granularities, multiple tasks, and multiple domains, which make NLP challenging to achieve satisfactory performance. By distributed representation learning, all objects that we are interested in are projected into a unified low-dimensional semantic space. The geometric distance between two objects in the semantic space indicates their semantic relatedness, and the semantic meaning of an object is related to which objects are close to it. With the aid of representation learning, knowledge could be transferred across multiple language entries, multiple NLP tasks, and multiple application domains, and therefore the effectiveness and robustness of NLP systems gain significant improvement.
First, we give a thorough introduction to word representation to show the basic ideas for representation learning for NLP. Words are usually considered as the smallest meaningful units of speech or writing in human languages. For human beings, to understand a language, it is crucial to understand the meanings of words. It is essential to accurately represent words, which could help models better understand, categorize, or generate text in NLP tasks. Moreover, high-level structures in a language are further composed of words. Compositionality enables natural languages to construct complex semantic meanings from the combinations of simpler semantic elements. This property is often captured with the following principle: the semantic meaning of a whole is a function of its several parts’ semantic meanings. Therefore, the semantic meanings of complex structures will depend on how their semantic elements combine. Based on this, we further talk about how to compositionally acquire the representation for higher-level language components, from phrases, sentences to documents.
Second, we introduce two forms of knowledge representation that are closely related to NLP. On the one hand, sememe representation tries to encode linguistic and commonsense knowledge in natural languages. Sememe is defined as the minimum indivisible unit of semantic meaning. With the help of sememe representation learning, we can get more interpretable and more robust NLP models. On the other hand, world knowledge representation studies how to encode world facts into continuous semantic space. It can not only help with knowledge graph tasks but also benefit knowledge-guided NLP applications.
Third, we discuss the representation learning of the network, which is also a natural way to represent objects and their relationships. In this part, we study how to embed vertices and edges in a network and how these elements interact with each other. Through the applications, we further show how network representations can help NLP tasks. Besides, we introduce another interesting topic related to NLP, the cross-modal representation, which studies how to model unified semantic representations across different modalities (e.g., text, audios, images, videos, etc.).
Finally, we look into the future directions of representation learning techniques for NLP. To be more specific, we consider the following directions including using more unsupervised data, utilizing few labeled data, employing deeper neural architectures, improving model interpretability and fusing the advantages of other areas.
In summary, we review and present the recent advances of distributed representation learning for NLP, including why representation learning can improve NLP, how representation learning takes part in various important topics of NLP, and what challenges are still not well addressed by distributed representation.
Representation Learning for Natural Language Processing Open Access
Authors: Liu, Zhiyuan, Lin, Yankai, Sun, Maosong 
- Provides a comprehensive overview of the representation learning techniques for natural language processing.
- Presents a systematic and thorough introduction to the theory, algorithms and applications of representation learning.
- Shares insights into the future research directions for each topic as well as for the overall field of representation learning for natural language processing.
The book
About the author
Zhiyuan Liu is an Associate Professor at the Department of Computer Science and Technology at Tsinghua University, China. His research interests include representation learning, knowledge graphs and social computation, and he has published more than 80 papers in at leading conferences and in respected journals. He has received several awards/honors, including Excellent Doctoral Dissertation awards from Tsinghua University and the Chinese Association for Artificial Intelligence, and was named as one of MIT Technology Review Innovators Under 35 China (MIT TR-35 China). He has served as area chair for various conferences, including ACL, EMNLP, COLING.
点击观看直播回放
在线直播：知识指导的自然语言处理
Prof. Zhiyuan Liu
Line