IERG5300 Tutorial 3 Stopping Time & Martingale

IERG5300 Tutorial 3Stopping Time & Martingale Peter Chen Peng Adapted from Qiwen Wang’s Tutorial Materials

Outline • Stopping Time and Wald’s Equation • Definitions and Theorems • Exmaples • Martingale and Martingale Stopping Theorem • Definitions and Theorems • Examples • Summary

Wald’s Equation For i.i.d random variables X1, X2 … Xn …, and a discrete random variable N ≥ 0, • If N is independent of all Xn , then • The above equation also holds in a looser condition: Wald’s equation If N is a Stopping Time for the sequence of i.i.d Xk , and E[N]<∞

Stopping Time • Definition: For a sequence X1, X2 … Xn … of random variables, a nonnegative r.v. N is called a stopping time if • For every n, (the characteristic variable of) the event {N n} is independent of the random variable Xn . • How to understand a stopping time? • Xn represents the outcome of the nth experiment. We may stop the experiment after round n if some condition is satisfied. • If this condition is a stopping time, whether we will continue the experiment after round n-1 is independent of the outcome in round n. i.e. after round n-1, we have enough information to see whether we can stop. • e.g. Stake $1 until a net earning $100 or merely $10 in our pocket.

Wald’s Equation ― Example • 1-D symmetric random walk is not positive recurrent. • Let Xn be the winning on the n-th game in a fair gamble (either 1 or -1) and X1, X2, X3, ... are i.i.ds. • Let T01 be the waiting time for ∑n=1Xn = 1. Clearly T01 is a stopping time for the Markov chain. • 1 = ET01 EX1 = ET01 0 = 0 \\ • Thus ET01 must be . Similarly, ET10 =  = ET1,0. Conditioning on the first move, we have ET00 = 1 + ET10 / 2 + ET1,0 / 2 = 

Wald’s Equation ― Understandings • Differences with Markov chain: we (secretly) introduced a value assignment for each node, which we originally don’t have in Markov chain • When each node is assigned a value, all nodes are in the real line. Thus any distribution will have a expectation value • Example. The naive way to assign value to random walk • And we can see the trend of expectation in a random process, with the help of Wald’s Equation • Example. Symmetric random walk: Expectation doesn’t move at all. It always stay where it is • Asymmetric random walk: Expectation moves steadily by p-q in each transition to the right (or q-p to the left)

Martingale and Martingale Stopping Theorem • A process X1, X2, ... , Xn, ... is called a Martingaleif, for all k, E|Xk| <  and E[Xk+1 | Xk = xk, … , X1 = x1] = xk • e.g. Xk = net winning after k games in a fair gamble. (which is the key meaning of Martingale) Xk = the state after k steps in a symmetric random walk. • (Martingale Stopping Theorem) Let X1, X2, ... , Xn, ... be a Martingale and N a stopping time. If E|XN| <  and lim inf (N>k) |Xk| dP = 0, then EXN = EX1. Think of the little constrains when you seems to get a contradiction in your logic.

Martingale and Martingale Stopping Theorem―Example on Gambler’s ruin problem • In a Gambler’s ruin problem, we have the result Pi = P{will reach N | start with i} Instant calculation by Martingale: (Why should we assign the values like that?) ri=EXT= (1 pi)1 + pi rn

Martingale and Martingale Stopping Theorem―Example on Gambler’s ruin problem • Why not using Wald’s Equation? • It involves expected time, which we don’t want • The value assignment when using Wald’s euqation: (Why should we assign the values like that?) • It is used to calculate expected time later: • E[pi n + (1pi )  0 + (qp)W] = i

Martingale and Martingale Stopping Theorem―Example on Strategy Pertaining • Start with 10000+0.5*(-20000) =0 • Fair Gamble -> Don’t hope to earn money, it’s impossible • A more practical goal: Collect 20000 to pay boss no matter what happens • At point (0,1), we have possibility p to lose 20000, p is a constant and can be calculated • Do we have enough cash to cancel that part? • No -> has possibility that fail to pay 20000 to boss • Yes, and even more cash -> Since it’s fair gamble, some other points will have not enough cash -> some other points will has possibility to fail • Yes, and just enough -> Bingo! Keep that all the time (i.e. for all points) • In the end, p is either 0 or 1, and we are done (0, 0) (0, 1) (1, 0) (2, 0) (0, 2) (1, 1) (3, 0) (2, 1) (0, 3) (1, 2) (3, 1) (1, 3) (2, 2) Lose 20000 Lose 0 (2, 3) (3, 2) So, calculate p for each point, then decide how much to bet (3, 3)

Martingale and Martingale Stopping Theorem ―Pattern Race Roll a special die repeatedly, where P{a} = 1/2 P{b} = 1/3 P{c} = 1/6 The average waiting time for the pattern aba= ? Solution: Team Gamble

Martingale and Martingale Stopping Theorem ―Pattern Race When The team wins, we always have the 1st and 3rd man left Total (Team) receipt = 12 + 0 + 2 regardless of the outcome of die rolling. Net gain of the team upon stopping = 12 + 0 + 2 – N, where the r.v. N represents the waiting time for the pattern aba. 3rd last gambler has $12 in the end Because this is fair gamble, 0 = E[Net gain upon stopping] = 14 – EN 1 EN= 14 0+ 1/2 Last gambler has $2 in the end 1+ 1/3 = 14 1/2

Martingale and Martingale Stopping Theorem ―Pattern Race • Key concept: Profit = Cost (Since Martingale) • Two things to verify: Where to start and what is the goal(stopping time) • The above example: start with nothing ($0 profit earned), end with aba ($14 profit earned by calculating aba*aba) • If we start with acab, we start with $6 earned by calculating acab*aba • X*Y means: The final goal is Y, the current situation is X now, how much money do I(The team) have? • What is X*X interpreted then? X*Y 0 ------> X ------> Y

Martingale and Martingale Stopping Theorem ―Pattern Race • Two Patterns (A & B) race • EN = pAB+ (1p) BB • EN = (1p) BA + pAA • How to interpret these two? • p : (1p) = (BBBA) : (AAAB) • How to interpret? 0 ------> A ------> B 0 ------> B

Martingale and Martingale Stopping Theorem ―Pattern Race How to interpret each line? How to interpret each line?

Summary • Stopping time • Wald’s Equation • Martingale & Martingale Stopping Theorem • Problems revisited with Martingale approach • Gambler’s ruin problem • Runs & Patterns solved by artificial casino Questions?

IERG5300 Tutorial 3 Stopping Time & Martingale