MDP Design

We consider a problem in which we seek to optimally design a Markov decision process (MDP). That is, subject to resource constraints we first design the action sets that will be available in each state when we later optimally control the process. The control policy is subject to additional constraints governing state-action pair frequencies, and we allow randomized policies. When the design decision is made, we are uncertain of some of the parameters governing the MDP, but we assume a distribution for these stochastic parameters is known. We focus on transient MDPs with a finite number of states and actions. We formulate, analyze and solve a two-stage stochastic integer program that yields an optimal design. A simple example threads its way through the paper to illustrate the development. The paper concludes with a larger application involving optimal design of malaria intervention strategies in Nigeria.

You can also view the related talks on budgeted disease interdiction and the malaria intervention strategies.


Nedialko B. Dimitrov and David P. Morton. Combinatorial Design of a Stochastic Markov Decision Process. Book chapter in "Operations Research and Cyber-Infrastructure", December, 2008

Comments and Questions

Add a comment

Add A Comment

Visual Captcha
Code in the picture:
Your Name(*):

© Copyright 2004-2019 - Ned Dimitrov