Markov Decision Processes: A Survey

1. Markov Decision Processes: A Survey Adviser：Yeong-Sung Lin Graduate Student：Cheng-Ta Lee Network Optimization Research Group March 22, 2004 Markov Decision Processes: A Survey

2. Outline • Introduction • Markov Theory • Markov Decision Processes • Conclusion • Future Work Markov Decision Processes: A Survey

3. Introduction Decision Theory • Probability Theory + • Utility Theory = • Decision Theory Describes what an agent should believe based on evidence. Describes what an agent wants. Describes what an agent should do. Markov Decision Processes: A Survey

4. Introduction • Markov decision processes (MDPs) theory has developed substantially in the last three decades and become an established topic within many operational research. • Modeling of (infinite) sequence of recurring decision problems (general behavioral strategies) • MDPs defined • Objective functions • Policies Markov Decision Processes: A Survey

5. Markov Theory • Markov process • A mathematical model that us useful in the study of complex systems. • The basic concepts of the Markov process are those “state” of a system and state “transition”. • A graphic example of a Markov process is presented by a frog in a lily pond. • State transition system • Discrete-time process • Continuous-time process Markov Decision Processes: A Survey

6. Markov Theory • To study the discrete-time process • Suppose that there are N states in the system numbered from 1 to N. If the system is a simple Markov process, then the probability of a transition to state j during the next time interval, given that the system now occupies state i, is a function only of i and j and not of any history of the system before its arrival in i. • In other words, we may specify a set of conditional probabilitypij. • where Markov Decision Processes: A Survey

7. The Toymaker Example • First state: the toy is great favor. • Second state: the toy is out of favor. • Matrix form • Transition diagram Markov Decision Processes: A Survey

8. The Toymaker Example • , the probability that the system will occupy state i after n transitions. • If its state at n=0 is known. It follow that Markov Decision Processes: A Survey

9. The Toymaker Example • If the toymaker starts with a successful toy, then and , so that Markov Decision Processes: A Survey

10. The Toymaker Example • Table 1.1 Successive State Probabilities of Toymaker Starting with a Successful Toy • Table 1.2 Successive State Probabilities of Toymaker Starting without a Successful Toy Markov Decision Processes: A Survey

11. The Toymaker Example • The row vector with components is thus the limit as n approaches infinity of Markov Decision Processes: A Survey

12. For the study of transient behavior and for theoretical convenience, it is useful to study the Markov process from the point of view of the generating function or, as we shall call it, the z-transform. Consider a time function f(n) that takes on arbitrary values f(0), f(1), f(2), and so on, at nonnegative, discrete, integrally spaced points of time and that is zero for negative time. Such a time function is shown in Fig. 2.4 Fig. 2.4 An Arbitrary discrete-time function z-Transformation Markov Decision Processes: A Survey

13. z-transform F(z) such that Table 1.3. z-Transform Pairs z-Transformation Markov Decision Processes: A Survey

14. z-Transformation • Consider first the step functionthe z-transform is orFor the geometric sequence f(n)=αn,n≧0, or Markov Decision Processes: A Survey

15. z-Transformation • We shall now use the z-transform to analyze Markov processes.In this expression I is the identity matrix. Markov Decision Processes: A Survey

16. z-Transformation • Let us investigate the toymaker’s problem by z-transformation. Let the matrix H(n) be the inverse transform of (I-zP)-1 on an element-by-element basis Markov Decision Processes: A Survey

17. z-Transformation • If the toymaker starts in the successful state 1, then π(0)=[1 0] and or , • If the toymaker starts in the unsuccessful state 2, then π(0)=[0 1] and or , • We have now obtained analytic forms for the data in Table 1.1 and 1.2. Markov Decision Processes: A Survey

18. We shall extend our previous work to the case in which the process may make transitions at random time intervals. The Laplace transform of a time function f(t) which is zero for t<0 is defined by Table 2.4. Laplace Transform Pairs Laplace Transformation Markov Decision Processes: A Survey

19. Laplace Transformation Markov Decision Processes: A Survey

20. Laplace Transformation • We shall now use the Laplace transform to analyze Markov processes. For discrete processes, or Markov Decision Processes: A Survey

21. Laplace Transformation • Recall the toymaker’s initial policy, for which the transition-probability matrix was Markov Decision Processes: A Survey

22. Laplace Transformation • Let the matrix H(t) be the inverse transform (sI-A)-1 • Then becomes by means of inverse transformation Markov Decision Processes: A Survey

23. Laplace Transformation • If the toymaker starts in the successful state 1, then π(0)=[1 0] and or , • If the toymaker starts in the unsuccessful state 2, then π(0)=[0 1] and or , • We have now obtained analytic forms for the data in Table 1.1 and 1.2. Markov Decision Processes: A Survey

24. Markov Decision Processes • MDPs applies dynamic programming to the solution of a stochastic decision with a finite number of states. • The transition probabilities between the states are described by a Markov chain. • The reward structure of the process is described by a matrix that represents the revenue (or cost) associated with movement from one state to another. • Both the transition and revenue matrices depend on the decision alternatives available to the decision maker. • The objective of the problem is to determine the optimal policy that maximizes the expected revenue over a finite or infinite number of stages. Markov Decision Processes: A Survey

25. Markov Process with Rewards • Suppose that an N-state Markov process earns rijdollars when it makes a transition from state i to j. • We call rij the “reward” associated with the transition from i to j. • The rewards need not be in dollars, they could be voltage levels, unit of production, or any other physical quantity relevant to the problem. • Let us define vi(n) as the expected total earnings in the next n transitions if the system is now in state i. Markov Decision Processes: A Survey

26. Markov Process with Rewards • Recurrence relation v(n)=q+Pv(n-1) Markov Decision Processes: A Survey

27. The Toymaker Example • Table 3.1. Total Expected Reward for Toymaker as a Function of State and Number of Weeks Remaining Markov Decision Processes: A Survey

28. Toymaker’s problem: total expected reward in each state as a function of week remaining Markov Decision Processes: A Survey

29. z-Transform Analysis of the Markov Process with Rewards • The z-Transform of the total-value vector v(n) will be called where v(0)=0 Markov Decision Processes: A Survey

30. z-Transform Analysis of the Markov Process with Rewards The total-value vector v(n) is then F(n)q by inverse transformation of , and, since ， Let the matrix F(n) be the inverse transform of We see that, as n becomes very large. Both v1(n) and v2(n) have slope 1 and v1(n)-v2(n)=10. Markov Decision Processes: A Survey

31. Optimization Techniques in General Markov Decision Processes • Value Iteration • Exhaustive Enumeration • Policy Iteration • Linear Programming • Lagrangian Relaxation Markov Decision Processes: A Survey

32. Value Iteration • Original • Advertising? No Yes • Research? No Yes Markov Decision Processes: A Survey

33. Diagram of States and Alternatives Markov Decision Processes: A Survey

34. The Toymaker’s Problem Solved by Value Iteration • The quantity is the expected reward from a single transition from state i under alternative k. Thus, • The alternatives for the toymaker are presented in Table 3.1. Markov Decision Processes: A Survey

35. The Toymaker’s Problem Solved by Value Iteration • We call di(n) the “decision” in state i at the nth stage. When di(n) has been specified for all i and all n, a “policy” has been determined. The optimal policy is the one that maximizes total expected return for each i and n. • To analyze this problem. Let us redefine as the total expected return in n stages starting from state i if an optimal policy is followed. It follows that for any n • “Principle of optimality” of dynamic programming: in an optimal sequence of decisions or choices, each subsequence must also be optimal. Markov Decision Processes: A Survey

36. The Toymaker’s Problem Solved by Value Iteration Table 3.6 Toymaker’s Problem Solved by Value Iteration Markov Decision Processes: A Survey

37. The Toymaker’s Problem Solved by Value Iteration • Note that for n=2, 3, and 4, the second alternative in each state is to be preferred. This means that the toymaker is better advised to advertise and to carry on research in spite of the costs of these activities. • For this problem the convergence seems to have taken place at n=2, and the second alternative in each state has been chosen. However, in many problems it is difficult to tell when convergence has been obtained. Markov Decision Processes: A Survey

38. Evaluation of the Value-Iteration Approach • Even though the value-iteration method is not particularly suited to long-duration processes. Markov Decision Processes: A Survey

39. Exhaustive Enumeration • The methods for solving the infinite-stage problem. • The method calls for evaluating all possible stationary policies of the decision problem. • This is equivalent to an exhaustive enumeration process and can be used only if the number of stationary policies is reasonably small. Markov Decision Processes: A Survey

40. Exhaustive Enumeration • Suppose that the decision problem has S stationary policies, and assume that Ps and Rs are the (one-step) transition and revenue matrices associated with the policy, s=1, 2, …, S. Markov Decision Processes: A Survey

41. Exhaustive Enumeration • The steps of the exhaustive enumeration method are as follows. • Step 1. Compute vsi, the expected one-step (one-period) revenue of policy s given state i, i=1, 2, …, m. • Step 2. Compute πsi, the long-run stationary probabilities of the transition matrix Ps associated with policy s. These probabilities, when they exist, are computed from the equationsπsPs =πsπs1 +πs2 +…+πsm =1where πs =(πs1 , πs2 , …, πsm ). • Step 3. Determine Es, the expected revenue of policy s per transition step (period), by using the formula • Step 4. The optimal policy s* id determined such that Markov Decision Processes: A Survey

42. Exhaustive Enumeration • We illustrate the method by solving the gardener problem for an infinite-period planning horizon. • The gardener problem has a total of eight stationary policies, as the following table shows: Markov Decision Processes: A Survey

43. Exhaustive Enumeration • The matrices Ps and Rs for policies 3 through 8 are derived from those of policies 1 and 2 and are given as Markov Decision Processes: A Survey

44. Exhaustive Enumeration • Step1: • The values of vsi can thus be computed as given in the following table. Markov Decision Processes: A Survey

45. Exhaustive Enumeration • Step 2: • The computations of the stationary probabilities are achieved by using the equationsπsPs =πsπs1 +πs2 +…+πsm =1 • As an illustration, consider s=2. The associated equations areThe solution yieldsIn this case, the expected yearly revenue is Markov Decision Processes: A Survey

46. Exhaustive Enumeration • Step 3&4: • The following table summarizes πs and Es for all the stationary policies. • Policy 2 yields the largest expected yearly revenue. The optimum long-range policy calls for applying fertilizer regardless of the system. 2.256= Markov Decision Processes: A Survey

47. Policy Iteration • The system is completely ergodic, the limiting state probabilities πi are independent of the starting state, and the gain g of the system iswhere qi is the expected immediate return in state i defined by Markov Decision Processes: A Survey

48. Policy Iteration • A possible five-state problem. • The alternative thus selected is called the “decision” for that state; it is no longer a function of n. The set of X’s or the set of decisions for all states is called a “policy”. Markov Decision Processes: A Survey

49. Policy Iteration • It is possible to describe the policy by a decision vector d whose elements represent the number of the alternative selected in each state. In this case • An optimal policy is defined as a policy that maximizes the gain, or average return per transition. Markov Decision Processes: A Survey

50. Policy Iteration • In five-state problem diagrammed, there are different policies. • However feasible this may be for 120 policies, it becomes unfeasible for very large problem. • For example, a problem with 50 states and 50 alternatives in each state contains 5050(≒1085) policies. • The policy-iteration method that will be described will find the optimal policy in a small number of iterations. • It is composed of two parts, the value-determination operation and the policy-improvement routine. Markov Decision Processes: A Survey