Skip to main content
SpringerLink
Log in
Book cover

Handbook of Markov Decision Processes

Methods and Applications

  • Book
  • © 2002

Overview

Editors:
  1. Eugene A. Feinberg

    1. State University of New York at Stony Brook, USA

    View editor publications

    You can also search for this editor in PubMed   Google Scholar

  2. Adam Shwartz

    1. Technion—Israel Institute of Technology, Israel

    View editor publications

    You can also search for this editor in PubMed   Google Scholar

Part of the book series: International Series in Operations Research & Management Science (ISOR, volume 40)

This is a preview of subscription content, log in via an institution to check access.

Access this book

Log in via an institution
eBook USD 299.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book USD 379.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

Licence this eBook for your library

Institutional subscriptions

Table of contents (17 chapters)

  1. Front Matter

    Pages i-viii
    Download chapter PDF

  2. Introduction

    1. Introduction

      • Eugene A. Feinberg, Adam Shwartz
      Pages 1-17

  3. Finite State and Action Models

    1. Front Matter

      Pages 19-19
      Download chapter PDF

    2. Finite State and Action MDPS

      • Lodewijk Kallenberg
      Pages 21-87

    3. Bias Optimality

      • Mark E. Lewis, Martin L. Puterman
      Pages 89-111

    4. Singular Perturbations of Markov Chains and Decision Processes

      • Konstantin E. Avrachenkov, Jerzy Filar, Moshe Haviv
      Pages 113-150

  4. Infinite State Models

    1. Front Matter

      Pages 151-151
      Download chapter PDF

    2. Average Reward Optimization Theory for Denumerable State Spaces

      • Linn I. Sennott
      Pages 153-171

    3. Total Reward Criteria

      • Eugene A. Feinberg
      Pages 173-207

    4. Mixed Criteria

      • Eugene A. Feinberg, Adam Shwartz
      Pages 209-229

    5. Blackwell Optimality

      • Arie Hordijk, Alexander A. Yushkevich
      Pages 231-267

    6. The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations

      • Armand M. Makowski, Adam Shwartz
      Pages 269-303

    7. Stability, Performance Evaluation, and Optimization

      • Sean P. Meyn
      Pages 305-346

    8. Convex Analytic Methods in Markov Decision Processes

      • Vivek S. Borkar
      Pages 347-375

    9. The Linear Programming Approach

      • OnĂ©simo Hernández-Lerma, Jean B. Lasserre
      Pages 377-407

    10. Invariant Gambling Problems and Markov Decision Processes

      • Lester E. Dubins, Ashok P. Maitra, William D. Sudderth
      Pages 409-428

  5. Applications

    1. Front Matter

      Pages 429-429
      Download chapter PDF

    2. Neuro-Dynamic Programming: Overview and Recent Trends

      • Benjamin Van Roy
      Pages 431-459

    3. Markov Decision Processes in Finance and Dynamic Options

      • Manfred Schäl
      Pages 461-487

    4. Applications of Markov Decision Processes in Communication Networks

      • Eitan Altman
      Pages 489-536
Back to top

Keywords

About this book

Eugene A. Feinberg Adam Shwartz This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the re­ spective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts ofSection 1.2. Most chap­ ters should be accessible by graduate or advanced undergraduate students in fields of operations research, electrical engineering, and computer science. 1.1 AN OVERVIEW OF MARKOV DECISION PROCESSES The theory of Markov Decision Processes-also known under several other names including sequential stochastic optimization, discrete-time stochastic control, and stochastic dynamic programming-studiessequential optimization ofdiscrete time stochastic systems. The basic object is a discrete-time stochas­ tic system whose transition mechanism can be controlled over time. Each control policy defines the stochastic process and values of objective functions associated with this process. The goal is to select a "good" control policy. In real life, decisions that humans and computers make on all levels usually have two types ofimpacts: (i) they cost orsavetime, money, or other resources, or they bring revenues, as well as (ii) they have an impact on the future, by influencing the dynamics. In many situations, decisions with the largest immediate profit may not be good in view offuture events. MDPs model this paradigm and provide results on the structure and existence of good policies and on methods for their calculation.

Editors and Affiliations

  • State University of New York at Stony Brook, USA

    Eugene A. Feinberg

  • Technion—Israel Institute of Technology, Israel

    Adam Shwartz

Bibliographic Information

Publish with us

Policies and ethics

Back to top

Search

Navigation