The Elements of Statistical Learning

During the past decade there has been an explosion in computation and information technology. With it have come vast amounts of data in a variety of fields such as medicine, biology, finance, and marketing. The challenge of understanding these data has led to the development of new tools in the field of statistics, and spawned new areas such as data mining, machine learning, and bioinformatics. Many of these tools have common underpinnings but are often expressed with different terminology. This book describes the important ideas in these areas in a common conceptual framework. While the approach is statistical, the emphasis is on concepts rather than mathematics. Many examples are given, with a liberal use of color graphics. It is a valuable resource for statisticians and anyone interested in data mining in science or industry. The book's coverage is broad, from supervised learning (prediction) to unsupervised learning. The many topics include neural networks, support vector machines, classification trees and boosting---the first comprehensive treatment of this topic in any book.

This major new edition features many topics not covered in the original, including graphical models, random forests, ensemble methods, least angle regression & path algorithms for the lasso, non-negative matrix factorization, and spectral clustering. There is also a chapter on methods for ``wide'' data (p bigger than n), including multiple testing and false discovery rates.

• Boosting
• Random Forest
• Support Vector Machine
• algorithms
• bioinformatics
• classification
• clustering
• data mining
• ensemble method
• learning
• machine learning
• neural networks
• statistics
• supervised learning
• unsupervised learning

From the reviews:

SIAM REVIEW

"The book is very well written and color is used throughout. Color adds a dimension that can be used to help the reader visualize high-dimensional data, and it is also very useful to help the eye see patterns and clusters more easily. This makes color effective in the book and not just a pleasing gimmick. This is the first book of its kind to treat data mining from a statistical perspective that is comprehensive and up-to-date on the statistical methods…I found the book to be both innovative and fresh. It provides an important contribution to data mining and statistical pattern recognition. It should become a classic…It is especially good for statisticians interested in high-dimensional and high-volume data such as can be found in telephone records, satellite images, and genetic microarrays. It can be used for an advanced special topics course in statistics for graduate students."

TECHNOMETRICS

"[This] is a vast and complex book. Generally, it concentrates on explaining why and how the methods work, rather than how to use them. Examples and especially the visualizations are principle features…As a source for the methods of statistical learning…it will probably be a long time before there is a competitor to this book."

SHORT BOOK REVIEWS

"This book describes modern tools for data analysis. With the exception of the last chapter, it is concerned with "supervised" methods - those methods in which a sample of cases is available, including values of an outcome variable, and on which one can build a model allowing one to predict the value of the outcome variable for new cases. The authors are amongst the leaders in this area, having developed many of the modern tools. Such methods have seen extraordinary development in recent decades, primarily because of progress in computer technology, but also because of the huge range of applications. Furthermore, the practical development of these modeling and inferential tools has resulted in a deeper theoretical understanding of the modeling process... The book includes many special cases and examples, which give insights into the ideas and methods. It explains very clearly the relationships between the methods, and covers both standard statistical staples, such as linear and logistic regression, as well as modern tools. It is not overburdened with unnecessary mathematics but uses only what is necessary for the practical application of the methods...The book has been beautifully produced. It was a pleasure to read. I strongly recommend it."

MATHEMATICAL REVIEWS

"The book provides a comprehensive and up-to-date introduction to the field of statistical pattern recognition, now commonly referred to as statistical learning…Browsing through the book, one is immediately attracted to the skillful use of color plots to stress the different behaviors of algorithms on real-world datasets. This tells a lot about the books style: intuition about a learning technique is built by looking at the behavior on the data, then the statistical analysis follows. However, even in its most technical parts, the presentation flows very smoothly, avoiding the definition-theorem-proof writing style…this is a very complete and up-to-date work covering all the most important learning techniques, which are presented in a rigorous but accessible statistical framework."

JOURNAL OF CLASSIFICATION, JUNE 2004

"This is a great book. All three authors have track records for clear exposition and are famously gifted for finding intuitive explanations that illuminate technical results…In particular, we admire the book for its:

-outstanding use of real data examples to motivate problems and methods;

-unified treatment of flexible inferential procedures in terms of maximization of an objective function subject to a complexity penalty;

-lucid explanation of the amazing performance of the AdaBoost algorithm in improving classification accuracy for almost any rule;

-clear account of support vector machines in terms of traditional statistical paradigms;

-regular introduction of some new insight, such as describing self-organizing maps as constrained k-means clustering.

…No modern statistician or computer scientist should be without this book."

JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, JUNE 2004

"In the words of the authors, the goal of this book was to ‘bring together many of the important new ideas in learning, and explain them in a statistical framework.’ The authors have been quite successful in achieving this objective, and their work is a welcome addition to the statistics and learning literatures…A strength of the book is the attempt to organize a plethora of methods into a coherent whole. The relationships among the methods are emphasized. I know of no other book that covers so much ground."

"The book byHastie et al. covers a wider number of topics such as supervised learning based on linear models, nearest neighbor methods, decision theory, function approximations, roughness, and kernels … . The charts and graphs are done in color to distinguish different patterns. The book has both challenging and easier exercise problems in each chapter. The book is suitable for a graduate level data mining course. I learned a lot from this well written book and recommend it highly." (Ramalingam Shanmugam, Journal of Statistical Computation & Simulation, Vol. 74 (4), 2004)

"One of the great features of the book is that it really contains more or less all modern methods for statistical learning, so it gives the reader a very good overview of this important field. … The author worked very hard on presentation of the material, in particular they illustrated the material by many colored graphics. … I think this book is valuable for anyone interested in statistical learning and its application, and I am happy to have it on my desk." (Michael Kohler, Metrika, February, 2003)

"For anyone who … wants to learn the new terminology and to understand what the ‘competition’ is doing, this is the book to buy. … the thinking is still very much statistical. This makes the book very easy to digest and pleasant to read for people with a statistical background. The many superb graphs add to this pleasure. … The book is important because it shows that interaction between statistics and machine learning can be profitable for both fields." (Hans C. van Houwelingen, Statistics in Medicine, Vol. 23, 2004)

"This is a great book. … We have taught a large graduate course (for statisticians and computer scientists) in data mining from this book. In developing this course we spoke to many other faculty members at a range of institutions, and we found no one who did not enjoy reading and teaching from this text. … there is no other book worth considering for such a course. … Thebook has beautiful graphics … . No modern statistician or computer scientist should be without this book." (David Banks and Feng Liang, Journal of Classification, Vol. 21 (1), 2004)

"The book provides a long-sought link between Statistics and Data Mining. It provides an excellent reference for researchers in information sciences … . Written by well-known specialists in applied statistics, the book provides a good practical orientation, with related theoretical issues coming out quite clearly. … this is the first book to address different aspects of data mining, inference and prediction in a coherently interdisciplinary context. … this book will always be remembered for laying the foundation of that scientific pyramid." (Kassim Said Mwitondi, Journal of Applied Statistics, Vol. 30 (1), 2003)

"The emphasis is on concepts rather than mathematics, and several examples are given as illustration. … This book is designed for researchers and students in a broad variety of fields such as statistics, artificial intelligence, engineering and finance. It should be a valuable resource for those who are interested in data mining in science or industry. I believe that it will be a very useful addition to any scholarly library." (Theofanis Sapatinas, Journal of the Royal Statistical Society Series A: Statistics in Society, Vol. 157 (1), 2004)

"A mere glance at the table of contents gives an idea of the breadth and depth of coverage of this remarkable book. … The style of this beautifully presented book is friendly and intuitive, and at the same time clear and rigorous. All the techniques dealt with are presented and compared through examples with real and simulated data. … The book can be used as a basis for courses of different levels, from the purely practical to the thoroughly theoretical. … a wonderful book!" (Ricardo Maronna, Statistical Papers, Vol. 44 (3), 2003)

"The book covers two topics: 12 chapters discuss statistical methods of supervisedlearning, the final chapter is on unsupervised learning. … The getup of the book is outstanding … . The book is an excellent and comprehensive treatment of the topics for which the authors are well known … . The book may well serve as a textbook for an advanced course; the illustrating examples and the discussion of computational aspects make the book useful for those who want to apply the methods." (Peter Hackl, Statistical Papers, Vol. 43 (3), 2002)

"This book describes modern tools for data analysis. … The book includes many special cases and examples, which give insights into the ideas and methods. It explains very clearly the relationship between the methods, and covers both standard statistical staples, such as linear and logistic regression, as well as modern tools. It is not overburdened with unnecessary mathematics but uses only what is necessary for the practical application … . The book has been beautifully produced. It was a pleasure to read. I strongly recommend it." (D. J. Hand, Short Book Reviews, Vol. 22 (1), 2002)

"This book is designed for researchers and students in the fields of statistics, artificial, intelligence, engineering, finance, and others. Some of the most important learning methods with the underlying concepts are described. … Bibliographic notes giving background references for the material, as well as computational considerations and exercises are provided at the end of each chapter. … The website for this book is located at http://www.stat.standford.edu/ElemStatLearn, which includes many of the datasets used." (Roland Fahrion, Zentralblatt MATH, Vol. 973, 2001)

• Department of Statistics, and Department of Health Research & Policy, Sequoia Hall, Stanford University, Stanford, USA

  Trevor Hastie

• Department of Statistics, Sequoia Hall, Stanford University, Stanford, USA

  Jerome Friedman

• Department of Health Research & Policy, and Department of Statistics, HRP Redwood Building, Stanford University, Stanford, USA

  Robert Tibshirani

Trevor Hastie, Robert Tibshirani, and Jerome Friedman are professors of statistics at Stanford University. They are prominent researchers in this area: Hastie and Tibshirani developed generalized additive models and wrote a popular book of that title. Hastie co-developed much of the statistical modeling software and environment in R/S-PLUS and invented principal curves and surfaces. Tibshirani proposed the lasso and is co-author of the very successful An Introduction to the Bootstrap. Friedman is the co-inventor of many data-mining tools including CART, MARS, projection pursuit and gradient boosting.

Policies and ethics