1 / 36

390 likes | 929 Vues

TCOM 501: Networking Theory & Fundamentals. Lecture 3 & 4 January 23 & 30, 2002 Prof. Yannis A. Korilis. Topics. Markov Chains Discrete-Time Markov Chains Calculating Stationary Distribution Global Balance Equations Detailed Balance Equations Birth-Death Process

Télécharger la présentation
## TCOM 501: Networking Theory & Fundamentals

**An Image/Link below is provided (as is) to download presentation**
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.
Content is provided to you AS IS for your information and personal use only.
Download presentation by click this link.
While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

**TCOM 501: Networking Theory & Fundamentals**Lecture 3 & 4 January 23 & 30, 2002 Prof. Yannis A. Korilis**Topics**• Markov Chains • Discrete-Time Markov Chains • Calculating Stationary Distribution • Global Balance Equations • Detailed Balance Equations • Birth-Death Process • Generalized Markov Chains • Continuous-Time Markov Chains**Markov Chain**• Stochastic process that takes values in a countable set • Example: {0,1,2,…,m}, or {0,1,2,…} • Elements represent possible “states” • Chain “jumps” from state to state • Memoryless (Markov) Property: Given the present state, future jumps of the chain are independent of past history • Markov Chains: discrete- or continuous- time**Discrete-Time Markov Chain**• Discrete-time stochastic process {Xn: n = 0,1,2,…} • Takes values in {0,1,2,…} • Memoryless property: • Transition probabilities Pij • Transition probability matrix P=[Pij]**Chapman-Kolmogorov Equations**• n step transition probabilities • Chapman-Kolmogorov equations • is element (i, j) in matrix Pn • Recursive computation of state probabilities**State Probabilities – Stationary Distribution**• State probabilities (time-dependent) • In matrix form: • If time-dependent distribution converges to a limit • p is called the stationary distribution • Existence depends on the structure of Markov chain**Aperiodic:**State i is periodic: Aperiodic Markov chain: none of the states is periodic Irreducible: States i and j communicate: Irreducible Markov chain: all states communicate 0 0 1 1 2 2 4 4 3 3 Classification of Markov Chains**Limit Theorems**Theorem 1: Irreducible aperiodic Markov chain • For every state j, the following limitexists and is independent of initial state i • Nj(k): number of visits to state j up to time k • j: frequency the process visits state j**Existence of Stationary Distribution**Theorem 2: Irreducible aperiodic Markov chain. There are two possibilities for scalars: • j= 0, for all states j No stationary distribution • j> 0, for all states j is the unique stationary distribution Remark: If the number of states is finite, case 2 is the only possibility**Ergodic Markov Chains**• Markov chain with a stationary distribution • States are positive recurrent: The process returns to state j “infinitely often” • A positive recurrent and aperiodic Markov chain is called ergodic • Ergodic chains have a unique stationary distribution • Ergodicity ⇒ Time Averages = Stochastic Averages**A. Finite number of states**Solve explicitly the system of equations Numerically from Pn which converges to a matrix with rows equal to Suitable for a small number of states B. Infinite number of states Cannot apply previous methods to problem of infinite dimension Guess a solution to recurrence: Calculation of Stationary Distribution**Markov chain formulation**i is the number of umbrellas available at her current location Transition matrix 0 2 1 Example: Finite Markov Chain Absent-minded professor uses two umbrellas when commuting between home and office. If it rains and an umbrella is available at her location, she takes it. If it does not rain, she always forgets to take an umbrella. Let p be the probability of rain each time she commutes. What is the probability that she gets wet on any given day?**Example: Finite Markov Chain**• Taking p = 0.1: • Numerically determine limit of Pn • Effectiveness depends on structure of P**Global Balance Equations**• Markov chain with infinite number of states • Global Balance Equations (GBE) • is the frequency of transitions from j to i • Intuition: j visited infinitely often; for each transition out of j there must be a subsequent transition into j with probability 1**Global Balance Equations**• Alternative Form of GBE • If a probability distribution satisfies the GBE, then it is the unique stationary distribution of the Markov chain • Finding the stationary distribution: • Guess distribution from properties of the system • Verify that it satisfies the GBE • Special structure of the Markov chain simplifies task**S**Sc 1 2 n n+1 0 Birth-Death Process • One-dimensional Markov chain with transitions only between neighboring states: Pij=0, if |i-j|>1 • Detailed Balance Equations (DBE) • Proof: GBE with S ={0,1,…,n} give:**1**0 2 n n+1 Example: Discrete-Time Queue • In a time-slot, one arrival with probability p or zero arrivals with probability 1-p • In a time-slot, the customer in service departs with probability q or stays with probability 1-q • Independent arrivals and service times • State: number of customers in system**1**2 n n+1 0 Example: Discrete-Time Queue**Example: Discrete-Time Queue**• Have determined the distribution as a function of 0 • How do we calculate the normalization constant 0? • Probability conservation law: • Noting that**Detailed Balance Equations**• General case: • Imply the GBE • Need not hold for a given Markov chain • Greatly simplify the calculation of stationary distribution Methodology: • Assume DBE hold – have to guess their form • Solve the system defined by DBE and ii= 1 • If system is inconsistent, then DBE do not hold • If system has a solution {i:i=0,1,…}, then this is the unique stationary distribution**Generalized Markov Chains**• Markov chain on a set of states {0,1,…}, that whenever enters state i • The next state that will be entered is j with probability Pij • Given that the next state entered will be j, the time it spends at state i until the transition occurs is a RV with distribution Fij • {Z(t):t ≥ 0} describing the state the chain is in at time t: Generalized Markov chain, or Semi-Markov process • It does not have the Markov property: future depends on • The present state, and • The length of time the process has spent in this state**Generalized Markov Chains**• Ti: time process spends at state i, before making a transition – holding time • Probability distribution function of Ti • Tii: time between successive transitions to i • Xnis the nth state visited. {Xn: n=0,1,…} • Is a Markov chain: embedded Markov chain • Has transition probabilities Pij • Semi-Markov process irreducible: if its embedded Markov chain is irreducible**Limit Theorems**Theorem 3: Irreducible semi-Markov process, E[Tii] < ∞ • For any state j, the following limit exists and is independent of the initial state. • Tj(t): time spent at state j up to time t • pj is equal to the proportion of time spent at state j**Occupancy Distribution**Theorem 4: Irreducible semi-Markov process; E[Tii] < ∞. Embedded Markov chain ergodic; stationary distribution π • Occupancy distribution of the semi-Markov process • πj proportion of transitions into state j • E[Tj] mean time spent at j • Probability of being at j is proportional to πjE[Tj]**Continuous-Time Markov Chains**Continuous-time process {X(t): t ≥0} taking values in {0,1,2,…}. Whenever it enters state i • Time it spends at state i is exponentially distributed with parameter νi • When it leaves state i, it enters state j with probability Pij, where j Pij= 1 • Continuous-time Markov chain is a semi-Markov process with • Exponential holding times: a continuous-time Markov chain has the Markov property**Continuous-Time Markov Chains**• When at state i, the process makes transitions to state j≠i with rate: • Total rate of transitions out of state i • Average time spent at state i before making a transition:**Occupancy Probability**• Irreducible and regular continuous-time Markov chain • Embedded Markov chain is irreducible • Number of transitions in a finite time interval is finite with probability 1 • From Theorem 3: for any state j, the limit exists and is independent of the initial state • pj is the steady-state occupancy probability of state j • pj is equal to the proportion of time spent at state j[Why?]**Global Balance Equations**• Two possibilities for the occupancy probabilities: • pj= 0, for all j • pj> 0, for all j, and j pj= 1 • Global Balance Equations • Rate of transitions out of j = rate of transitions into j • If a distribution {pj: j = 0,1,…} satisfies GBE, then it is the unique occupancy distribution of the Markov chain • Alternative form of GBE:**Detailed Balance Equations**• Detailed Balance Equations • Simplify the calculation of the stationary distribution • Need not hold for any given Markov chain • Examples: birth-death processes, and reversible Markov chains**S**Sc 1 2 n n+1 0 Birth-Death Process • Transitions only between neighboring states • Detailed Balance Equations • Proof: GBE with S ={0,1,…,n} give:**Birth-Death Process**• Use DBE to determine state probabilities as a function of p0 • Use the probability conservation law to find p0 • Using DBE in problems: • Prove that DBE hold, or • Justify validity (e.g. reversible process), or • Assume they hold – have to guess their form – and solve system**1**0 2 n n+1 M/M/1 Queue • Arrival process: Poisson with rate λ • Service times: iid, exponential with parameter µ • Service times and interarrival times: independent • Single server • Infinite waiting room • N(t): Number of customers in system at time t (state)**1**0 2 n n+1 M/M/1 Queue • Birth-death process → DBE • Normalization constant • Stationary distribution**The M/M/1 Queue**• Average number of customers • Applying Little’s Theorem, we have • Similarly, the average waiting time and number of customers in the queue is given by

More Related