Fun and Games (and Graphs) with EBSDEs
Samuel N. Cohen
Mathematical Institute
University of Oxford
Research Supported by:
The Oxford–Man Institute for Quantitative Finance
The Oxford–Nie Financial Big Data Laboratory
Based on joint work with
Andrew Allan, Robert Elliott, Victor Fedyashov, Ying Hu
Control and BSDEs
Fun, Games and Graphs: Control and BSDEs
2
A basic control problem
I
To begin, let’s consider a fairly classical control problem.
I
We have a (for simplicity, finite dimensional) semimartingale X , where we
can control the drift and the rate of jumps.
I
We wish to implement a control, which minimizes the expected total of a
terminal cost, interim costs, and costs of our control.
I
We can formulate our problem in a ‘weak’ sense, where our control affects
the measure directly.
Fun, Games and Graphs: Control and BSDEs
3
A basic control problem
I
Combining martingale representations and measure changes naturally leads
us to BSDEs.
I
For a P-Brownian
motion W with the PRP, consider the measure P̃ under
R
which Wt − [0,t] γs ds is a martingale, for γ a sufficiently integrable scalar
predictable process. (We need E(γ • W ) to be a martingale.)
I
Applying the MRT under this measure, we see that we can write any
FT -measurable P̃-integrable random variable ξ as
Z
ξ = E P̃ [ξ|Ft ] +
Zs> (dWs − γs ds)
]t,T ]
Z
Zs> γs ds
= Yt −
]t,T ]
I
Z
+
Zs> dWs
]t,T ]
We see that a solution (Y , Z ) to this equation embeds the measure
change, as Yt = E P̃ [ξ|Ft ]
Fun, Games and Graphs: Control and BSDEs
4
A basic control problem
More generally, if W is a cylindrical BM and µ̃ is a compensated random
measure, we consider the situation
I
with density d P̃/dP = E(γ • W + γ 0 ∗ µ̃)T ,
I
assuming (W , µ̃) has the PRP and deterministic and absolutely continuous
predictable-quadratic-variation,
I
for any P̃ integrable FT -measurable ξ,
I
we apply the PRP under P̃ to see that Yt = E P̃ [ξ|Ft ] solves (for some
Z , Θ)
Z
dYt = −f (ω, t, Zt , Θt )dt + Zt> dWt +
Θt (ζ)µ̃(dζ, dt)
Z
with terminal condition YT = ξ
I
where the ‘driver’ of the BSDE is the function
Z
f (ω, t, z, θ) = γt> z +
θ(ζ)γt0 (ζ)ν(dζ).
Z
Fun, Games and Graphs: Control and BSDEs
5
A basic control problem
I
To really exploit these equations for control, we need to go beyond these
linear cases.
I
Considering this equation directly, it is fairly straightforward to prove:
Theorem (Pardoux–Peng and subsequent extensions)
Suppose f : (ω, t, y , z, θ) → R is LipschitzRcontinuous, in the appropriate
Hilbert-type norms for y , z, θ, and that E [ [0,T ] |f (ω, t, 0, 0, 0)|2 dt] < ∞. Then
for any ξ ∈ L2 (FT , P), the BSDE
dYt = −f (ω, t, Zt , Θt )dt + Zt> dWt +
Z
Θt (ζ)µ̃(dζ, dt)
Z
with terminal condition YT = ξ has a unique solution in
S 2 × L2 (`2 × [0, T ]) × L2 (L2 (ν) × [0, T ])
I
The pair (f , ξ) are the ‘data’ of the BSDE.
I
Proof: Stability estimate + contraction in some equivalent norm
Fun, Games and Graphs: Control and BSDEs
6
A basic control problem
I
Given a solution (Y , Z , Θ) to a BSDE, we can linearize the driver.
φt = f (ω, t, 0, 0, 0),
ρt = f (ω, t, Yt , 0, 0) − f (ω, t, 0, 0, 0) /Yt
γt =
Z
f (ω, t, Yt , Zt , 0) − f (ω, t, Yt , 0, 0)
Zt
kZt k2
γ 0 (ζ)Θ(ζ)ν(dζ) = f (ω, t, Yt , Zt , Θt ) − f (ω, t, Yt , Zt , 0)
Z
f (ω, t, Yt , Zt , Θt ) = φt + ρt Yt + γt> Zt +
γ 0 (ζ)Θ(ζ)ν(dζ)
Z
Z
0
I
Provided γ > −1, the BSDE solution corresponds to a measure change.
I
Novikov/Lépingle–Mémin criteria from the Lipschitz continuity of f .
I
The measure is ‘dynamically’ chosen by considering Y , Z , Θ.
I
ρ corresponds to a discount factor.
Fun, Games and Graphs: Control and BSDEs
7
A basic control problem
I
Extending this, we say that f is balanced if for any y , z, z 0 , θ, θ0 , we can
find a sufficiently integrable γ and γ 0 > −1 such that
γt =
Z
f (ω, t, y , z, θ0 ) − f (ω, t, y , z 0 , θ0 )
(z − z 0 )
kz − z 0 k2
γ 0 (ζ) θ(ζ) − θ0 (ζ) ν(dζ) = f (ω, t, y , z, θ) − f (ω, t, y , z, θ0 )
Z
I
This is a strictly stronger condition than is needed for a measure change.
I
When we have no jumps, Y is 1D and f is Lipschitz, then f is always
balanced.
Fun, Games and Graphs: Control and BSDEs
8
A basic control problem
Applying this logic to the difference of two solutions, as we know that the
conditional expectation (under any measure) is monotone, we observe the
following result
Theorem (Comparison theorem)
Let (f , ξ), (f 0 , ξ 0 ) be data for two BSDEs, with
ξ ≥ ξ0
and
f (ω, t, 0, 0, 0) ≥ f 0 (ω, t, 0, 0, 0).
Suppose f is balanced. Then the solutions satisfy Y ≥ Y 0 .
Furthermore, if there exists s ∈ [0, T ] and A ∈ Fs such that Yt = Yt0 on A for
some t ≥ s, then Yt = Yt0 on A for all t ≥ s. (It follows that Zt = Zt0 and
Θt = Θ0t also.)
Fun, Games and Graphs: Control and BSDEs
9
A basic control problem
I
I
How to use this for control?
RT
For any control u, if the total cost is given by 0 L(ω, t, u)dt + ξ, then the
remaining cost is
i
hZ T
Yt = inf Eu
L(ω, s, u)ds + ξ Ft .
u
I
t
Writing f as
Z
n
o
f (ω, t, z, θ) = inf L(ω, t, u) + γt (u)> z +
γt0 (ζ; u)θ(ζ)ν(dζ)
u
Z
we know f is balanced!
I
By comparison, Y is the minimal cost and solves the BSDE with driver f ,
terminal value YT = ξ
Fun, Games and Graphs: Control and BSDEs
10
Ergodic BSDEs
I
When dealing with ergodic control, we use a cost of the form
Z
i
1 h T
λ = lim supT Eu
L(Xs , us )ds
T
0
I
X is a (weakly controlled) Markov process (with some dissipativity in its
generator).
I
The key step in obtaining a BSDE approach is to show that, for every
control u (at least among Markov controls), there exists a bound on the
ergodic behaviour of X , ie
kPtu δx0 − π u kTV ≤ R(1 + kx0 k2 )e −ρt
where Ptu is the transition operator associated with X under control u, π u
its stationary distribution, and R, ρ do not depend on u.
I
Splitting techniques are usually used, under a Lipschitz assumption on f
(defined as before)
Fun, Games and Graphs: Control and BSDEs
11
Ergodic BSDEs
I
Given this, we consider an ‘ergodic BSDE’
dYt = − f (Xs , us ) − λ ds + Zs> dWs
I
Jumps, infinite dimensional W and cyclic time dependence can be
included.
I
Existence/Uniqueness Proof: Discounted BSDE to infinite horizon, send
discount to zero, bound using ergodicity. Then compactness,
subsequences, and uniqueness by comparison.
I
We interpret Yt = v (Xt ) and λ as
Z
i Z
∗
1 h T
L(Xs , us )ds =
L(x, u ∗ )dπ u
λ = lim supT Eu
T
0
X
Z Z
∗
Y0 = v (x0 ) =
L(x, u ∗ )(Pt δx0 (dx) − π u (dx))dt
R+
I
X
Given a balanced driver and bounded cost, λ is unique bounded, Y is
unique among bounded Markov solutions.
Fun, Games and Graphs: Control and BSDEs
12
Ergodic Games
Fun, Games and Graphs: Ergodic Games
13
Setup of the problem
We consider a non-zero sum game between two players with the following
properties:
I
Each player can control the drift of a forward process.
I
Both optimise some average functional in infinite horizon.
We aim to find a Nash equilibrium for such a game.
The forward process satisfies the (multidimensional) SDE
dXt = AXt + F (Xt ) dt + σ(Xt )dWt ; X0 = x0 ∈ Rd ,
where A enforces weak dissipativity (hence ergodicity), F is bounded and σ is
not degenerate. We assume W has the predictable representation property.
Fun, Games and Graphs: Ergodic Games
14
Mathematical model
I
Take a weak formulation of the game – in other words, the agents affect
the drift by changing measure.
I
For admissible controls (u, v ) ∈ U1 × U2 , the players introduce a bounded
drift R(u, v ) in the Brownian motion W , over any horizon [0, T ].
I
For each agent (i = 1, 2) ergodic payoffs are
Z
J i (x0 , u, v ) = lim supT →∞ T −1 Eu,v ,T
T
Li (Xt , ut , vt )dt .
0
I
∗
∗
The goal is to find an admissible control (u , v ), such that
J 1 (x0 , u ∗ , v ∗ ) ≤ J 1 (x0 , u, v ∗ ) and J 2 (x0 , u ∗ , v ∗ ) ≤ J 2 (x0 , u ∗ , v )
holds for all admissible (u, v ).
Fun, Games and Graphs: Ergodic Games
15
Generalised Isaac’s condition
Define the Hamiltonian functions {Hi }i=1,2 , as
Hi (x, zi , u, v ) = zi R(u, v ) + Li (x, u, v )
Assumption
(i) There exist measurable maps u1∗ , u2∗ , defined on R3N , with values in U1
and U2 respectively, such that
H1 x, z1 , u1∗ (x, z1 , z2 ), u2∗ (x, z1 , z2 ) ≤ H1 (x, z1 , u, u2∗ (x, z1 , z2 ))
and
H2 x, z1 , u1∗ (x, z1 , z2 ), u2∗ (x, z1 , z2 ) ≤ H2 (x, z1 , u1∗ (x, z1 , z2 ), v )
holds for all (x, z1 , z2 , u, v ) ∈ R3N × U1 × U2 .
(ii) The mapping (z1 , z2 ) ∈ R2N → (H1∗ , H2∗ )(x, z1 , z2 ) is continuous for any
fixed x.
Fun, Games and Graphs: Ergodic Games
16
An example:
For the sake of simplicity restrict ourselves to one dimensional setting:
I
Let U1 = [0, 1], U2 = [−1, 1] be the space of controls.
I
Define the cost functions by
L1 (x, u, v ) = u 2 + v 2 ,
I
L2 (x, u, v ) = 2u 2 +
1 2
v .
2
The additional drift is given by
R(u, v ) = u + v .
I
The ‘optimal’ Hamiltonian for the first player is given by
H1 (x, z, u ∗ (z, y ), v ∗ (z, y )) = z(z̄ + ȳ ) + z̄ 2 + ȳ 2 ,
where z̄ = −1 ∨ (− z2 ) ∧ 1, ȳ = 0 ∨ (−y ) ∧ 1. By considering when z is
large, we see that H1 is not Lipschitz wrt y .
Fun, Games and Graphs: Ergodic Games
17
Connection to BSDEs
As in classical ergodic control, we look at the system of ergodic BSDEs
corresponding to the players’ values:
Z T
Z T
[Hi∗ (Xs , Zs1 , Zs2 ) − λi ]ds −
(Zsi )> dWs ,
Yti = YTi +
t
(1)
t
In order to establish the existence of a Nash equilibrium, we need to answer two
questions:
I
Assuming there exists a solution to (1), do we have a saddle point for the
game?
I
Can we prove that the system (1) indeed admits a solution?
Fun, Games and Graphs: Ergodic Games
18
Ideas behind the proof:
I
By standard change of measure technique, we show that the answer to the
first question is yes.
I
The second question is tricky: not only is (1) a multidimensional ergodic
BSDE, but also the driver is not Lipschitz.
I
However, we notice that for fixed u, v the Hamiltonian has a diagonal
structure.
I
This suggests an iterative approach similar to Picard iteration.
I
Player 1 chooses a strategy u n+1 which is optimal given the value function
induced by (u n , v n ), and vice versa.
I
To get a limit, we need some compactness (but we don’t have
comparisons)
Fun, Games and Graphs: Ergodic Games
19
Ideas behind the proof:
On each step n ≥ 0, for i = 1, 2, obtain a weak comparison principle for the
ergodic component λi,n . In other words, we make use of the following result:
Lemma
Let f i , i = 1, 2 be two drivers, such that for any (x, z1 , z2 )
|f i (x, z1 , z2 )| ≤ C kzi k + C̄
for some constants C , C̄ > 0. Suppose that there exist Markov solutions
(Y i , Z i , λi ), i = 1, 2 to the corresponding EBSDEs. Let λ be the ergodic part
of the bounded Markovian solution (Y , Z , λ) to the EBSDE (for scalar Y )
−dYt = [(C kZt k + C̄ ) − λ]ds − Zt> dWt .
Then λi ≤ λ for i = 1, 2.
Proof: Measure change using only the scalar BSDE
Fun, Games and Graphs: Ergodic Games
20
Ideas behind the proof:
I
We conclude that λ’s lie in a compact rectangle.
I
Prove that a number of useful estimates from the standard theory of
ergodic BSDEs still hold uniformly in n, for each Picard iterate.
I
Extract a convergent subsequence on a dense subset for Markovian
solutions {y i,n (·)}i=1,2 . Then extend by continuity.
I
Use the generalised Isaac’s condition to show that all other parts of the
solution converge as well.
Bonus result:
(Markovian) simultaneous best response strategies converge to a Nash
equilibrium (at least on a subsequence).
Fun, Games and Graphs: Ergodic Games
21
EBSDEs with continuous coefficients
This also gives us a new approach to scalar EBSDEs:
Lemma
Suppose we are given a function f : Rd → R, that is continuous and of linear
growth, that is, there exists a constant κ > 0, such that |f (x)| ≤ κ(1 + kxk)
for all x ∈ RN . Then there exist two uniformly bounded functions φ(·), ψ(·),
such that
f (x) = φ(x)x + ψ(x),
holds for all x ∈ Rd .
Remark
Notice that we have no continuity assumption on the individual components
φ, ψ. One can clearly see that this decomposition is inspired by the structure of
the Hamiltonian in the games framework.
Fun, Games and Graphs: Ergodic Games
22
EBSDEs with continuous coefficients (cont.)
We are interested in the following equation:
Z T
Z
Yt = YT +
[f (Xs , Zs ) − λ]ds −
t
T
Zs> dWs ,
t
where one can find functions φ(·), ψ(·), such that f (x, z) = φ(x, z)z + ψ(x, z),
and there exist constants c, C > 0, such that
kφ(x, z)k ≤ c,
kψ(x, z)k ≤ C .
Theorem
Suppose the driver f : RN × RN → R is a measurable map, such that f (x, ·) is
continuous for all x ∈ RN and there exist a constant κ > 0, such that
f (x, z) ≤ κ(1 + kzk).
Then there exists a solution (Y , Z , λ) to the EBSDE above.
Fun, Games and Graphs: Ergodic Games
23
Generalisation to the case of n > 2 players
I
Our results extend in a straightforward manner to the case of finitely many
agents.
I
With countably many players, the situation is a bit more involved. The
reason is that we assume continuity, but in infinite dimensions that
involves specifying the topology over the player-space as well.
I
For any finite measure µ over the players (parameterized by N), the
infinite dimensional z in the BSDEs converge in L2 (µ), dt × dP-a.e.
I
Therefore, if Hi is L2 -continuous with respect to z for every i
(dt × dP-a.e., not uniformly in i), we obtain a Nash equilibrium.
Fun, Games and Graphs: Ergodic Games
24
Asymmetric players
I
A related question is to understand whether there is a qualitative
difference between an ergodic player (as above), and one who discounts
the future at a small discount rate.
I
We define the payoff structure as follows:
Z T
J 1 (x0 , u, v ) = lim supT →∞ T −1 Eu,v ,T
L1 (Xt , ut , vt )dt ,
0
Z ∞
2
u,v ,T
−αt
J (x0 , u, v ) = E
e
L2 (Xt , ut , vt )dt ,
0
where α is some positive constant.
I
The same techniques give a Nash equilibrium in this case.
I
We also establish convergence to the ergodic system as α ↓ 0.
Fun, Games and Graphs: Ergodic Games
25
Graphs and Markov Chains
Fun, Games and Graphs: Graphs and Markov Chains
26
Markov chains
I
Consider a time homogenous finite/countable-state continuous-time
Markov chain X .
I
Discrete time works just as well.
I
X is a process in (Ω, F, {Ft }t≥0 , P), where {Ft } is the natural filtration of
X . The value of X at time t is Xt (ω) or (by abuse of notation) Xt .
I
X takes values in X , the standard basis of RN where
N = number of states ≤ ∞.
I
X jumps from state Xt− to state ei ∈ X at rate ei> AXt− , so
Z
Xt = X0 +
AXu− du + Mt
]0,t]
for some RN -valued martingale M.
I
A is a RN×N matrix, with [A]ij ≥ 0 for i 6= j, and 1∗ Aei =
Fun, Games and Graphs: Graphs and Markov Chains
P
j [A]ij
≡ 0.
27
Markov chains
Example
For N = 4, we can consider the Markov chain with matrix


−3
1
0
0
 1
−3
2
2 

A=
 2
2
−3
1 
0
0
1
−3
Which can be drawn as a graph
1
1
2
1
2
2
3
2
2
1
4
1
Fun, Games and Graphs: Graphs and Markov Chains
28
Martingale Representation
I
We have the representation
Z
Xt = X0 +
AXu− du + Mt
]0,t]
I
M is a RN martingale with predictable quadratic variation
I
>
> >
dhMi = φt dt = (diag(AXt− ) + AXt− Xt−
+ Xt− Xt−
A )dt
R >
2
R >
R
We have E
Z dM ] = E
Z φt Zdt] =: E
kZ k2Mt dt
I
We can give a martingale representation theorem with respect to M,
namely for any scalar martingale L
Z
Lt = L0 +
Zt> dMt
]0,t]
and Z is unique up equality in
kzk2M
= z > φt z. Note k1k2Mt ≡ 0.
Fun, Games and Graphs: Graphs and Markov Chains
29
Changing the probabilities
Now supposePwe were in the world where X had rates λ (with the convention
λ> Xt− = − ei 6=Xt− ei> λ). This corresponds to a different choice of probability
measure P̃. Then
Z
Xt = X0 +
λu− du + M̃t ,
]0,t]
where M̃ is a new martingale under the new probabilities. Rearranging, we see
that
Z
M̃t =
(AXu− − λu− )du + Mt .
]0,t]
Therefore, we can write
E P̃ [ξ|Ft ] = E P̃ [ξ] +
Z
]0,t]
Z
Zu> d M̃u = ξ −
Z
Zu> d M̃u
]t,T ]
Zu> (λu− − AXu− )du −
=ξ+
Z
]t,T ]
Fun, Games and Graphs: Graphs and Markov Chains
Zu> dMu
]t,T ]
30
BSDEs
A BSDE is an now equation of the form
Z
Z
Yt = ξ +
f (ω, s, Ys− , Zs )ds −
]t,T ]
Zs> dMs
]t,T ]
or equivalently,
dYt = −f (ω, t, Yt− , Zt )dt + Zt> dMt ,
I
YT = ξ.
Classically, they are closely linked to semilinear PDEs.
Simple examples:
I
f ≡ 0 (Martingale representation)
I
f (Zs ) = Z > (λs − AXs− ) (Measure change)
I
These are the same as considered earlier (with jumps)
Fun, Games and Graphs: Graphs and Markov Chains
31
BSDE Existence and Uniqueness
Theorem
Let f be a predictable function with
|f (ω, t, y , z) − f (ω, t, y 0 , z 0 )|2 ≤ c(|y − y 0 |2 + kz − z 0 k2Mt )
R
for some c > 0, and E [ ]0,T ] |f (ω, t, 0, 0)|2 dt] < ∞. Then for any ξ ∈ L2 (FT )
the BSDE
Z
Z
Yt = ξ +
f (ω, t, Yu− , Zu )du −
Zu> dMu
]t,T ]
]t,T ]
has a unique adapted solution (Y , Z ) with appropriate integrability.
Fun, Games and Graphs: Graphs and Markov Chains
32
γ-balanced drivers
To get a comparison theorem, we need the following definition.
Definition
f is γ-balanced if there exists a random field λ, with λ(·, ·, z, z 0 ) predictable
and λ(ω, t, ·, ·) Borel measurable, such that
I
f (ω, t, y , z) − f (ω, t, y , z 0 ) = (z − z 0 )> (λ(ω, t, z, z 0 ) − AXt− ),
I
for each ei ∈ X ,
ei> λ(ω, t, z, z 0 )
∈ [γ, γ −1 ]
ei> AXt−
for some γ > 0, where 0/0 := 1,
I
1> λ(ω, t, z, z 0 ) ≡ 0 and
I
λ(ω, t, z + α1, z 0 ) = λ(ω, t, z, z 0 ) for all α ∈ R.
Fun, Games and Graphs: Graphs and Markov Chains
33
Comparison theorem
The purpose of this definition is to obtain:
Theorem (Comparison theorem)
Let f be γ-balanced for some γ > 0. Then if
ξ ≥ ξ 0 and f (ω, t, y , z) ≥ f 0 (ω, t, y , z),
the associated BSDE solutions satisfy Yt ≥ Yt0 for all t, and Yt = Yt0 on
A ∈ Ft iff Ys = Ys0 on A for all s ≥ t.
Fun, Games and Graphs: Graphs and Markov Chains
34
Comparison theorem
As mentioned before, if B is another rate matrix, we write E B for the
expectation under the corresponding measure.
Lemma
If B ∼γ A (to be defined) then f (ω, t, y , z) = z > (B − A)Xt− is γ-balanced
with λ(· · · ) = BXt− , and Yt = E B [ξ|Ft ].
Lemma
If f (u; · · · ) is γ-balanced for each u, then g (· · · ) := ess infu {f (u; · · · )} is
γ-balanced (given it is always finite).
Fun, Games and Graphs: Graphs and Markov Chains
35
Definition
For γ > 0, we say ‘A γ-controls B’ (and write A γ B) if B − γA is also a rate
matrix.
If A γ B and B γ A we write A ∼γ B.
Example

−3
 1
A=
 2
0
1
−3
2
0
0
2
−3
1

0
2 

1 
−3

−4
 2
B=
 2
0
1
−3
2
0
0
2
−3
1

0
2 

1 
−3
Then for γ ≤ 1/2, A ∼γ B.
Fun, Games and Graphs: Graphs and Markov Chains
36
Ergodicity
I
As in the continuous case, we can show ergodicity in these settings.
I
An irreducible finite-state Markov chain is always ergodic
Theorem
For a fixed rate matrix A, there exists R, ρ > 0 depending only on A and γ
such that for every B ∼γ A we have
ke Bt δx0 − π B k`1 ≤ Re −ρt .
Fun, Games and Graphs: Graphs and Markov Chains
37
Markov chain Ergodic BSDEs
Fun, Games and Graphs: Markov chain Ergodic BSDEs
38
EBSDE Existence/Uniqueness
Theorem
The ‘ergodic’ BSDE
dYt = −(f (Xt , Zt ) − λ)dt + Zt> dMt
admits a bounded solution (Y , Z , λ) with Yt = u(Xt ) and u(x̂) = 0, whenever
f is γ-balanced with |f (x, 0)| < C .
Any other bounded solution has the same λ, any other bounded Markovian
solution has Yt0 = Yt + c.
A Feynman–Kac type result gives, with Y = u(Xt ) = Xt> u, Z = u,
f(u) − λ = −A> u
which is readily calculable for small dimensions.
Fun, Games and Graphs: Markov chain Ergodic BSDEs
39
Applications: Ergodic distributions of Graphs
I
Consider the driver f (x, z) = I{x∈Ξ} + g (x, z). Then if g ≡ 0, the EBSDE
solutions will be λ = π(Ξ).
I
If g is concave, we will have a ‘convex ergodic probability’,
I
If g (x, z) = inf B∈B {z > (B − A)x}, when B has ‘column-exchangeability’,
then we can show that
λΞ = inf π B (Ξ)
I
The global balance equation Aπ ≡ 0 becomes the Isaac’s condition
nZ
o
inf
sup{v > B 0 x}dπ B (x) ≡ 0
for all v ∈ RN .
B
B
X B0
Fun, Games and Graphs: Markov chain Ergodic BSDEs
40
Nonlinear Graph Centrality
I
A common problem when we study graphs is to find the most ‘central’
nodes.
I
One method (connected to ‘random walk centrality’) is to look at the
ergodic distribution of a Markov chain on the graph.
I
What happens if we instead dynamically take the minimum over a range of
rate matrices?
Fun, Games and Graphs: Markov chain Ergodic BSDEs
41
Nonlinear Graph Centrality
Consider this undirected graph.
2
1
7
4
5
6
3
8
Our Markov chain moves between states randomly, at a constant (exit) rate.
Fun, Games and Graphs: Markov chain Ergodic BSDEs
42
Nonlinear Graph Centrality
The ergodic probabilities under this basic model can be simply calculated.
11.1%
11.1%
5.6%
16.7%
11.1%
16.7%
16.7%
11.1%
We have three states which are equally highest ranked, even though they have
different positions in the graph.
Fun, Games and Graphs: Markov chain Ergodic BSDEs
43
Nonlinear Graph Centrality
Now suppose a bias can be introduced dynamically, so that the relative rate of
jumping to the right/left can be doubled. Then the EBSDE can be solved with
drivers
f (x, z) = I{x∈Ξ} + min{z > (B − A)x}
B
for each Ξ ⊆ X . The corresponding values λΞ (for Ξ singletons) are
5.5%
4.3%
2.0%
10.3%
4.8%
7.8%
11.4%
6.8%
Fun, Games and Graphs: Markov chain Ergodic BSDEs
44
Nonlinear Graph Centrality
2
1
7
4
5
6
3
I
We can also assess collections of states, so if
I
I
I
I
I
I
I
8
Ξ = {1, 3, 4}, then λΞ = 0.3067
Ξ = {2, 3, 4}, then λΞ = 0.3164
Ξ = {3, 4, 5}, then λΞ = 0.3531
Ξ = {4, 5, 6}, then λΞ = 0.2899.
Note all these collections have ergodic measure 0.444 under the basic model.
Peculiarly, λ{2} = 0.055 > 0.048 = λ{5} , but λ{2,3,4} < λ{3,4,5} .
In this way we can assess the ‘minimal importance’ of a collection of states.
Fun, Games and Graphs: Markov chain Ergodic BSDEs
45
Nonlinear Graph Centrality
I
To assess centrality (as opposed to importance), it is more interesting to
look at the change in ergodic probability when the system is controlled.
I
The ‘controllability centrality’
controlled probability CC = log
uncontrolled probability
gives interesting results.
I
Nodes near the edge change more (so have CC ↓ −∞)
Fun, Games and Graphs: Markov chain Ergodic BSDEs
46
Bank networks
I
Consider an interbank liability network.
I
Taking liabilities as links, we obtain a weighted directed graph.
I
We can consider a continuous time Markov chain with these values as
rates of change.
I
Data from Austrian banking system, Dec 2008. (Thanks to Martin
Summer of the Österreichische Nationalbank)
In the following:
I
Size related to total liability
I
Colour based on log(Ergodic probability).
Thanks to Lucas Jeub for visualization help.
Fun, Games and Graphs: Markov chain Ergodic BSDEs
47
Ergodic distribution
15
10
5
0
−5
−10
−15
−20
−20
−15
−10
−5
0
5
Fun, Games and Graphs: Markov chain Ergodic BSDEs
10
15
48
Controlled ergodic distribution
15
10
5
0
−5
−10
−15
−20
−20
−15
−10
−5
0
5
10
15
Determined using a 10% change in any/all connection strength, corresponding
to BSDE driver
X
f (x, z) = −0.1
Aij |(ej − x)> z|.
ej 6=x
Fun, Games and Graphs: Markov chain Ergodic BSDEs
49
Controlled ergodic distribution
0
10
−1
10
−2
10
−3
10
−4
Minimal Probability
10
−5
10
−6
10
−7
10
−8
10
−9
10
−10
10
−10
10
−9
10
−8
10
−7
10
−6
10
−5
10
Ergodic Probability
−4
10
−3
10
−2
10
−1
10
0
10
Figure: Ergodic probabilities vs Controlled Ergodic probabilities (log scales)
Fun, Games and Graphs: Markov chain Ergodic BSDEs
50
Controllability Centrality
−0.45
−0.5
−0.55
Controllability Centrality
−0.6
−0.65
−0.7
−0.75
−0.8
−0.85
−9
10
−8
10
−7
10
−6
10
−5
10
Ergodic Probability
−4
10
−3
10
−2
10
−1
10
Figure: Ergodic probabilities (log scale) vs CC
Fun, Games and Graphs: Markov chain Ergodic BSDEs
51
Controllability Centrality
15
10
5
0
−5
−10
−15
−20
−20
−15
−10
−5
0
5
Fun, Games and Graphs: Markov chain Ergodic BSDEs
10
15
52
Bank networks
I
This gives a natural notion of ‘robustness’ of the network.
I
If a node has a high CC, then only a small change in the network is needed
to noticeably change its significance.
I
Conversely, low CC means large structural changes would be needed to
alter the node’s importance.
Fun, Games and Graphs: Markov chain Ergodic BSDEs
53
BSDEs to stopping times
Fun, Games and Graphs: BSDEs to stopping times
54
BSDEs to stopping times
I
Another interesting class of BSDEs/Control problems is where the
terminal time is a stopping time.
I
Here we want
Z
Z
Yt = ξ +
Zu> dMu
f (ω, u, Yu− , Zu )du +
]t,τ ]
]t,τ ]
when τ is a stopping time, ξ is Fτ -measurable.
I
If
f (ω, t, y , z) − f (ω, t, y 0 , z)
≤ −α
y − y0
for some α > 0,
then we can use the discounted methods considered earlier.
I
This does not cover the interesting case f (x, z) = z ∗ (B − A)x.
Fun, Games and Graphs: BSDEs to stopping times
55
BSDEs to stopping times
I
We will assume sufficient bounds on ξ and τ that our BSDE admits a
unique solution with polynomial growth.
I
This is similar to the backwards heat equation admitting only one
polynomial growth solution.
I
Ergodicity plays a role again, due to the connection between expected
hitting times and ergodic convergence.
I
For simplicity, we consider the Markov chain case, and write Qγ for the
class of measures with generators B ∼γ A for some A.
Fun, Games and Graphs: BSDEs to stopping times
56
BSDEs to stopping times
Assumption
For some nondecreasing K , K̃ : R+ → [1, ∞[, constants β, β̃ > 0,
E Q [|ξ||Ft ] ≤ K (t),
E Q [(1 + τ )1+β |Ft ] ≤ K (t),
E Q [K (τ )1+β̃ |Ft ] ≤ K̃ (t),
all P-a.s. for all Q ∈ Qγ and all t.
Theorem
If τ is the first hitting time of a set Ξ ⊆ X , and ξ = g (τ, Xτ ) for some
g (t, x) ≤ c(1 + t β̂ ), then these assumptions are satisfied.
Fun, Games and Graphs: BSDEs to stopping times
57
BSDEs to stopping times
Theorem
Let ξ, τ satisfy the previous bounds, and suppose f is γ-balanced, and such
that for any y , y 0 , z,
|f (ω, t, 0, 0)| ≤ c(1 + t β̂ )
f (ω, t, y , z) − f (ω, t, y 0 , z)
∈ [−c, 0]
y − y0
for some c ∈ R, some β̂ ∈ [0, β[. Then the BSDE
Z
Z
Yt = ξ +
f (ω, u, Yu− , Zu )du +
]t,τ ]
Zu> dMu
]t,τ ]
admits a unique adapted solution satisfying the bound
|Yt | ≤ (1 + c)K (t).
Proof: Measure change and estimates
Fun, Games and Graphs: BSDEs to stopping times
58
BSDEs to stopping times
I
If we don’t assume f is γ-balanced, then this becomes more difficult.
I
Suppose f is Markov, continuous and satisfies, for every z,
f (x, z) = z > (B − A)x
where B ∼γ A may depend on z, x
I
In this case we can write Y as a conditional expectation (but not the
differences of Y s) and is therefore bounded (uniformly in the Picard
iterations)
Fun, Games and Graphs: BSDEs to stopping times
59
BSDEs to stopping times
Theorem
Let ξ satisfy the previous bounds, τ be a hitting time and suppose f is as on
the previous slide and
|f (ω, t, 0)| ≤ c(1 + t β )
for some c ∈ R, some β < ∞. Then the BSDE
Z
Z
Yt = ξ +
f (ω, u, Yu− , Zu )du +
]t,τ ]
Zu> dMu
]t,τ ]
admits an adapted solution satisfying the bound
|Yt | ≤ (1 + c)K (t).
Proof: Compactness, subsequences
Fun, Games and Graphs: BSDEs to stopping times
60
Toy application: Nonlinear circuits
Fun, Games and Graphs: Toy application: Nonlinear circuits
61
Non-Ohmic Circuits
I
Markov chains can be used to model electronic circuits of resistors.
I
If wij = 1/Rij is the resistance of a connection, then the voltage potential
v satisfies
(
v (i) = φ(i),
i ∈Ξ
P
P
v (i) j wij = j wij v (j), i ∈
/Ξ
for Ξ a source set.
I
This depends on Ohm’s Law and Kirtchoff’s current laws.
I
For finite circuits, the voltage potential is the expected value at the first
hitting time of Ξ, for a Markov chain with Aij = wji for i 6= j.
Fun, Games and Graphs: Toy application: Nonlinear circuits
62
Non-Ohmic Circuits
I
Diodes, Transistors, etc. do not satisfy Ohm’s law.
I
For a diode, the effective resistance is
Rijv =
v (j) − v (i)
>0
ci (exp([v (j) − v (i)]/cv ) − 1)
I
Write B v for the rate matrix with entries Bijv = 1/Rjiv .
I
We can write the voltage potential as solving the BSDE to the first hitting
time of Ξ with driver
f (x, z) = z > (B z − A)x
where A is generated from a reference resistor circuit.
I
One can show this gives a balanced driver, so for any finite source
potential we have unique solutions.
Fun, Games and Graphs: Toy application: Nonlinear circuits
63
Non-Ohmic Circuits
I
A transistor (or op-amp) has effective resistance depending on the
potential between different nodes
I
The corresponding driver is not balanced
I
This can be seen as we can encode a NOT logical gate
I
This allows us to encode an approximate-probabilistic liar paradox
I
Non-uniqueness is perhaps unsurprising
Fun, Games and Graphs: Toy application: Nonlinear circuits
64