Neural Networks for Conditional Probability Estimation

1. Front Matter

   Pages i-xxiii

   PDF

2. Introduction

   • Dirk Husmeier

   Pages 1-19

3. A Universal Approximator Network for Predicting Conditional Probability Densities

   • Dirk Husmeier

   Pages 21-37

4. A Maximum Likelihood Training Scheme

   • Dirk Husmeier

   Pages 39-55

5. Benchmark Problems

   • Dirk Husmeier

   Pages 57-67

6. Demonstration of the Model Performance on the Benchmark Problems

   • Dirk Husmeier

   Pages 69-85

7. Random Vector Functional Link (RVFL) Networks

   • Dirk Husmeier

   Pages 87-97

8. Improved Training Scheme Combining the Expectation Maximisation (EM) Algorithm with the RVFL Approach

   • Dirk Husmeier

   Pages 99-119

9. Empirical Demonstration: Combining EM and RVFL

   • Dirk Husmeier

   Pages 121-135

10. A simple Bayesian regularisation scheme

    • Dirk Husmeier

    Pages 137-145

11. The Bayesian Evidence Scheme for Regularisation

    • Dirk Husmeier

    Pages 147-163

12. The Bayesian Evidence Scheme for Model Selection

    • Dirk Husmeier

    Pages 165-177

13. Demonstration of the Bayesian Evidence Scheme for Regularisation

    • Dirk Husmeier

    Pages 179-191

14. Network Committees and Weighting Schemes

    • Dirk Husmeier

    Pages 193-201

15. Demonstration: Committees of Networks Trained with Different Regularisation Schemes

    • Dirk Husmeier

    Pages 203-219

16. Automatic Relevance Determination (ARD)

    • Dirk Husmeier

    Pages 221-227

17. A Real-World Application: The Boston Housing Data

    • Dirk Husmeier

    Pages 229-250

18. Summary

    • Dirk Husmeier

    Pages 251-253

19. Appendix: Derivation of the Hessian for the Bayesian Evidence Scheme

    • Dirk Husmeier

    Pages 255-265

20. Back Matter

    Pages 267-275

    PDF

Conventional applications of neural networks usually predict a single value as a function of given inputs. In forecasting, for example, a standard objective is to predict the future value of some entity of interest on the basis of a time series of past measurements or observations. Typical training schemes aim to minimise the sum of squared deviations between predicted and actual values (the 'targets'), by which, ideally, the network learns the conditional mean of the target given the input. If the underlying conditional distribution is Gaus­ sian or at least unimodal, this may be a satisfactory approach. However, for a multimodal distribution, the conditional mean does not capture the relevant features of the system, and the prediction performance will, in general, be very poor. This calls for a more powerful and sophisticated model, which can learn the whole conditional probability distribution. Chapter 1 demonstrates that even for a deterministic system and 'be­ nign' Gaussian observational noise, the conditional distribution of a future observation, conditional on a set of past observations, can become strongly skewed and multimodal. In Chapter 2, a general neural network structure for modelling conditional probability densities is derived, and it is shown that a universal approximator for this extended task requires at least two hidden layers. A training scheme is developed from a maximum likelihood approach in Chapter 3, and the performance ofthis method is demonstrated on three stochastic time series in chapters 4 and 5.

• algorithms
• dynamical systems
• neural network
• neural networks
• noise
• pattern
• pattern recognition
• training

• Neural Systems Group, Department of Electrical & Electronic Engineering, Imperial College, London, UK

  Dirk Husmeier

Policies and ethics