1 / 10

Introduction to Continuous-Time Markov Chains (CTMCs)

This lecture on Continuous-Time Markov Chains (CTMCs) covers the formal definition and key concepts such as transient solutions, steady-state solutions, and practical applications, particularly focusing on the M/M/1 queue and machine repair models. The session delves into the stochastic processes that define CTMCs, providing insight into state transition rates and equilibrium distributions. Examples and graphical representations are utilized to clarify complex concepts. This lecture is ideal for those looking to understand the foundational elements of CTMCs in probability theory and applications.

karik
Télécharger la présentation

Introduction to Continuous-Time Markov Chains (CTMCs)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 547: Lecture 25 Continuous-time Markov Chains Mary K. Vernon Fall 2003

  2. Today’s Outline • Formal definition of CTMCs • Transient solution: P[X(t)=k] • Steady state solution • Applications • M/M/1 queue • Machine repair model Reference: AA 4.3, 5.0-5.2; LK1 2.4, 3.1

  3. Continuous Time Markov Chains Definition: a stochastic process {X(t), tT} is a Markov process if t1,t2,…,tn+1, t1 t2 … tn+1 and  x1,x1,…, xn+1, P[X(tn+1)xn+1 | X(t1)x1,X(t2)x2, …, X(tn)xn]  P[X(tn+1)xn+1 | X(tn)xn] Continuous-timeMarkov chain (CTMC): state space is discrete, T is continuous e.g., Poisson counting process, or Q(t) in M/M/1 queue Goal: derive Pk(t)  P[X(t)  j] and/or j P[X(t) = j] Notation: pi,,j (t1,t2) = P[X(t2)j| X(t1)i]

  4.    0 1 2 3     CTMC: Graphical Representation state transition rates: , i  j time-homogeneous: qi,j qi,j(t) pi,,j(t,t+h)  qi,jh + o(h), i  j pi,i(t,t+h)  1  ( qi,j)h + o(h) e.g.,X(t) is the queue length of an M/M/1 queue at time t qi,i+1  qi,i-1  State transitions are labelled with the state transition rate

  5. CTMC: Pk(t) pi,j(t1,t2) P[X(t2)j| X(t1)  i] P(t)  P[X(t)  j] theorem of total probability where Qi,i   qi,j , i.e., pi,,i(t,t+h)  1 + qi,ih + o(h) Qi,j qi,j , ij

  6. CTMC: Pk(t) pi,j(t1,t2) P[X(t2)j| X(t1)  i] P(t)  P[X(t)  j] st or thus, and completely define a time-homogeneous CTMC

  7.    0 1 2 3     CTMC Example: Pk(t) X(t) = queue length of M/M/1 queue at time t , k  1 solution for Pk(t): LK1 pp. 74-77 (very complex)

  8. 0 1 2  2 CTMC Example: Pk(t) X(t) = number of machines that are operational at time t Solution for P0(t): AA Example 4.3.3, pp. 217-219

  9. CTMC: P[X(t) = j] This “equilibrium distribution” or “stationary distribution” exists and is independent of the initial state if the DPMC is irreducible is non-trivial if the CTMC is irreducible & recurrent non-null in this case, and

  10.    0 1 2 3     k k U/(1  U) CTMC Example: k X(t) = queue length of M/M/1 queue at time t irreducible 0  - flux out of state k + flux into state k 0  -0 + 1 ; 0  -( + )1 + 0 + 2 ; 1 (/) 0 ; 2 (/)1  (/)2 0; … k (/)k 0 : 0[ 1 + (/)k ]  0 Uk  1 0 [ 1  U]1  1,    or U  1 0  1  U, k (1  U)Uk E[Q] =

More Related