Skip to main content
SpringerLink
Log in
Book cover

Neural Networks: Tricks of the Trade

  • Book
  • © 1998

Overview

Editors:
  1. Genevieve B. Orr

    1. Department of Computer Science, Willamette University, Salem, USA

    View editor publications

    You can also search for this editor in PubMed   Google Scholar

  2. Klaus-Robert Müller

    1. GMD First (Forschungszentrum Informationstechnik), Berlin, Germany

    View editor publications

    You can also search for this editor in PubMed   Google Scholar

  • This book makes a unique assessment and evaluation of the tricks for efficiently exploiting neural network techniques
  • Includes supplementary material: sn.pub/extras

Part of the book series: Lecture Notes in Computer Science (LNCS, volume 1524)

This is a preview of subscription content, log in via an institution to check access.

Access this book

Log in via an institution
eBook USD 74.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Other ways to access

Licence this eBook for your library

Institutional subscriptions

Table of contents (18 chapters)

  1. Front Matter

    Pages I-VI
    Download chapter PDF

  2. Introduction

    1. Introduction

      Pages 1-5

  3. Speeding Learning

    1. Front Matter

      Pages 7-8
      Download chapter PDF

    2. Efficient BackProp

      • Yann LeCun, Leon Bottou, Genevieve B. Orr, Klaus -Robert Müller
      Pages 9-50

  4. Regularization Techniques to Improve Generalization

    1. Front Matter

      Pages 51-53
      Download chapter PDF

    2. Early Stopping - But When?

      • Lutz Prechelt
      Pages 55-69

    3. A Simple Trick for Estimating the Weight Decay Parameter

      • Thorsteinn S. Rognvaldsson
      Pages 71-92

    4. Controlling the hyperparameter search in MacKay’s Bayesian neural network framework

      • Tony Plate
      Pages 93-112

    5. Adaptive Regularization in Neural Network Modeling

      • Jan Larsen, Claus Svarer, Lars Nonboe Andersen, Lars Kai Hansen
      Pages 113-132

    6. Large Ensemble Averaging

      • David Horn, Ury Naftaly, Nathan Intrator
      Pages 133-139

  5. Improving Network Models and Algorithmic Tricks

    1. Front Matter

      Pages 141-143
      Download chapter PDF

    2. Square Unit Augmented Radially Extended Multilayer Perceptrons

      • Gary William Flake
      Pages 145-163

    3. A Dozen Tricks with Multitask Learning

      • Rich Caruana
      Pages 165-191

    4. Solving the Ill-Conditioning in Neural Network Learning

      • Patrick van der Smagt, Gerd Hirzinger
      Pages 193-206

    5. Centering Neural Network Gradient Factors

      • Nicol N. Schraudolph
      Pages 207-226

    6. Avoiding roundoff error in backpropagating derivatives

      • Tony Plate
      Pages 227-233

  6. Representing and Incorporating Prior Knowledge in Neural Network Training

    1. Front Matter

      Pages 235-237
      Download chapter PDF

    2. Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation

      • Patrice Y. Simard, Yann A. LeCun, John S. Denker, Bernard Victorri
      Pages 239-274

    3. Combining Neural Networks and Context-Driven Search for Online, Printed Handwriting Recognition in the Newton

      • Yaeger Larry S., Webb Brandyn J., Lyon Richard F.
      Pages 275-298

    4. Neural Network Classification and Prior Class Probabilities

      • Steve Lawrence, Ian Burns, Andrew Back, Ah Chung Tsoi, C. Lee Giles
      Pages 299-313
Back to top

Keywords

About this book

It is our belief that researchers and practitioners acquire, through experience and word-of-mouth, techniques and heuristics that help them successfully apply neural networks to di cult real world problems. Often these \tricks" are theo- tically well motivated. Sometimes they are the result of trial and error. However, their most common link is that they are usually hidden in people’s heads or in the back pages of space-constrained conference papers. As a result newcomers to the eld waste much time wondering why their networks train so slowly and perform so poorly. This book is an outgrowth of a 1996 NIPS workshop called Tricks of the Trade whose goal was to begin the process of gathering and documenting these tricks. The interest that the workshop generated motivated us to expand our collection and compile it into this book. Although we have no doubt that there are many tricks we have missed, we hope that what we have included will prove to be useful, particularly to those who are relatively new to the eld. Each chapter contains one or more tricks presented by a given author (or authors). We have attempted to group related chapters into sections, though we recognize that the di erent sections are far from disjoint. Some of the chapters (e.g., 1, 13, 17) contain entire systems of tricks that are far more general than the category they have been placed in.

Editors and Affiliations

  • Department of Computer Science, Willamette University, Salem, USA

    Genevieve B. Orr

  • GMD First (Forschungszentrum Informationstechnik), Berlin, Germany

    Klaus-Robert Müller

Bibliographic Information

Publish with us

Policies and ethics

Back to top

Search

Navigation