Overview
- Offers an overview of pre-trained language models such as BERT, GPT, and sequence-to-sequence Transformer
- Explains the key techniques to improve the performance of pre-trained models
- Presents advanced pre-trained models for a broad range of NLP tasks
- This book is open access, which means that you have free and unlimited access
Part of the book series: Artificial Intelligence: Foundations, Theory, and Algorithms (AIFTA)
Buy print copy
Tax calculation will be finalised at checkout
Table of contents (8 chapters)
-
Front Matter
-
Back Matter
Keywords
About this book
This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts.
Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models.
After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI.Authors and Affiliations
About the authors
Sven Giesselbach is the leader of the Natural Language Understanding (NLU) team at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), where he has specialized in Artificial Intelligence and Natural Language Processing. He and his team develop solutions in the areas of medical, legal and general document understanding, which in their core build upon Foundation Models. Sven Giesselbach is also part of the Competence Center for Machine Learning Rhine-Ruhr (ML2R), where he works as a research scientist and investigates Informed Machine Learning, a paradigm in which knowledge is injected into machine learning models, in conjunction with language modeling. He has published more than 10 papers on Natural Language Processing and Understanding, which focus on the creation of application-ready NLU systems and the integration of expert knowledge in various stages of the solution design. He led the development of the Natural Language Understanding Showroom, a platform for showcasing state-of-the-art Natural Language Understanding models. He regularly gives talks about NLU at summer schools, conferences and AI-Meetups.
Bibliographic Information
Book Title: Foundation Models for Natural Language Processing
Book Subtitle: Pre-trained Language Models Integrating Media
Authors: Gerhard Paaß, Sven Giesselbach
Series Title: Artificial Intelligence: Foundations, Theory, and Algorithms
DOI: https://doi.org/10.1007/978-3-031-23190-2
Publisher: Springer Cham
eBook Packages: Computer Science, Computer Science (R0)
Copyright Information: The Editor(s) (if applicable) and The Author(s) 2023
Hardcover ISBN: 978-3-031-23189-6Published: 24 May 2023
Softcover ISBN: 978-3-031-23192-6Published: 24 May 2023
eBook ISBN: 978-3-031-23190-2Published: 23 May 2023
Series ISSN: 2365-3051
Series E-ISSN: 2365-306X
Edition Number: 1
Number of Pages: XVIII, 436
Number of Illustrations: 13 b/w illustrations, 112 illustrations in colour
Topics: Natural Language Processing (NLP), Computational Linguistics, Artificial Intelligence, Knowledge based Systems, Machine Learning