Skip to main content
SpringerLink
Log in
Book cover

Markov Decision Processes in Practice

  • Book
  • © 2017

Overview

Editors:
  1. Richard J. Boucherie

    1. Stochastic Operations Research, University of Twente, Enschede, The Netherlands

    View editor publications

    You can also search for this editor in PubMed   Google Scholar

  2. Nico M. van Dijk

    1. Stochastic Operations Research, University of Twente, Enschede, The Netherlands

    View editor publications

    You can also search for this editor in PubMed   Google Scholar

  • Promotes, illustrates and disseminates the present day possibilities and value of classical MDP for real life practical application and optimization
  • Provides “smart” starting rules for approximate procedures like ADP
  • Contains a collection of current applications that applies the latest numerical capacities and expansions?

Part of the book series: International Series in Operations Research & Management Science (ISOR, volume 248)

This is a preview of subscription content, log in via an institution to check access.

Access this book

Log in via an institution
eBook USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book USD 299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

Licence this eBook for your library

Institutional subscriptions

Table of contents (21 chapters)

  1. Front Matter

    Pages i-xxxii
    Download chapter PDF

  2. General Theory

    1. Front Matter

      Pages 1-1
      Download chapter PDF

    2. One-Step Improvement Ideas and Computational Aspects

      • Henk Tijms
      Pages 3-32

    3. Value Function Approximation in Complex Queueing Systems

      • Sandjai Bhulai
      Pages 33-62

    4. Approximate Dynamic Programming by Practical Examples

      • Martijn R. K. Mes, Arturo Pérez Rivera
      Pages 63-101

    5. Server Optimization of Infinite Queueing Systems

      • András Mészáros, Miklós Telek
      Pages 103-129

    6. Structures of Optimal Policies in MDPs with Unbounded Jumps: The State of Our Art

      • H. Blok, F. M. Spieksma
      Pages 131-186

  3. Healthcare

    1. Front Matter

      Pages 187-187
      Download chapter PDF

    2. Markov Decision Processes for Screening and Treatment of Chronic Diseases

      • Lauren N. Steimle, Brian T. Denton
      Pages 189-222

    3. Stratified Breast Cancer Follow-Up Using a Partially Observable MDP

      • J. W. M. Otten, A. Witteveen, I. M. H. Vliegen, S. Siesling, J. B. Timmer, M. J. IJzerman
      Pages 223-244

    4. Advance Patient Appointment Scheduling

      • Antoine Sauré, Martin L. Puterman
      Pages 245-268

    5. Optimal Ambulance Dispatching

      • C. J. Jagtenberg, S. Bhulai, R. D. van der Mei
      Pages 269-291

    6. Blood Platelet Inventory Management

      • Rene Haijema, Nico M. van Dijk, Jan van der Wal
      Pages 293-317

  4. Transportation

    1. Front Matter

      Pages 319-319
      Download chapter PDF

    2. Stochastic Dynamic Programming for Noise Load Management

      • T. R. Meerburg, Richard J. Boucherie, M. J. A. L. van Kraaij
      Pages 321-335

    3. Allocation in a Vertical Rotary Car Park

      • M. Fackrell, P. Taylor
      Pages 337-370

    4. Dynamic Control of Traffic Lights

      • Rene Haijema, Eligius M. T. Hendrix, Jan van der Wal
      Pages 371-386

    5. Smart Charging of Electric Vehicles

      • Pia L. Kempker, Nico M. van Dijk, Werner Scheinhardt, Hans van den Berg, Johann Hurink
      Pages 387-404

  5. Production

    1. Front Matter

      Pages 405-405
      Download chapter PDF

    2. Analysis of a Stochastic Lot Scheduling Problem with Strict Due-Dates

      • Nicky D. van Foreest, Jacob Wijngaard
      Pages 407-423
Back to top

Keywords

About this book

This book presents classical Markov Decision Processes (MDP) for real-life applications and optimization. MDP allows users to develop and formally support approximate and simple decision rules, and this book showcases state-of-the-art applications in which MDP was key to the solution approach.  The book is divided into six parts. Part 1 is devoted to the state-of-the-art theoretical foundation of MDP, including approximate methods such as policy improvement, successive approximation and infinite state spaces as well as an instructive chapter on Approximate Dynamic Programming. It then continues with five parts of specific and non-exhaustive application areas. Part 2 covers MDP healthcare applications, which includes different screening procedures, appointment scheduling, ambulance scheduling and blood management. Part 3 explores MDP modeling within transportation.  This ranges from public to private transportation, from airports and traffic lights to car parking or charging your electric car . Part 4 contains three chapters that illustrates the structure of approximate policies for  production or manufacturing structures. In Part 5, communications is highlighted as an important application area for MDP. It includes Gittins indices, down-to-earth call centers and wireless sensor networks. Finally Part 6 is dedicated to financial modeling, offering an instructive review to account for financial portfolios and derivatives under proportional transactional costs. The MDP applications in this book illustrate a variety of both standard and non-standard aspects of MDP modeling and its practical use. This book should appeal to readers for practitioning, academic research and educational purposes, with a background in, among others, operations research, mathematics, computer science, and industrial engineering.

Editors and Affiliations

  • Stochastic Operations Research, University of Twente, Enschede, The Netherlands

    Richard J. Boucherie, Nico M. van Dijk

About the editors

Richard Boucherie received M.Sc. degrees in 1988 in applied mathematics and theoretical physics from the Universiteit Leiden, and received the Ph.D. degree in econometrics in 1992 from the Vrije Universiteit, Amsterdam. Since 2000 he is with the department of Applied Mathematics of the University of Twente, where he was appointed in 2003 as full professor of Stochastic Operations Research.

His research interests are in queueing theory, Petri nets and random walks with application areas including wireless and sensor networks, healthcare, road traffic, and network intrusion detection and prevention. Richard is co-founder of the University of Twente Center for Healthcare Operations Improvement and Research (CHOIR) in the area of healthcare logistics, and chair of the Postdoctorate programme in healthcare logistics. In 2014 he co-founded the spin-off company Rhythm, that carries out actual implementations of healthcare logistics solutions in healthcare organisations.  <

Nico M. van Dijk  has been active in the area of Stochastic Operations Research for over 30 years. He has always been stimulated by real-life stochastics , as reflected by his Ph.D. on Controlled Markov Processes: Time-discretization (1983), by his Wiley book on Queueing Networks and Product Forms : A systems approach (1993) and by practical application papers such as on communications, call centers, railways and ICU (Intensive care unit) systems. For a decade he works in close cooperation with the Dutch Blood banks. For joint work on blood supply, the OR team that he guided, became finalist for the EURO 2009 Excellence in OR Practice Award and received the INFORMS 2011 First Prize Interactive Poster Award. He is affiliated with the Stochastic Operations Research group at the University of Twente.

Bibliographic Information

Publish with us

Policies and ethics

Back to top

Search

Navigation