Example 2.2.1. Long-term Behavior of Markov Processes.
Recall that a Markov process describes the evolution of one state \(\vb{x}_{n}\) into a future state \(\vb{x}_{n+1}\) using the matrix equation \(A\vb{x}_n = \vb{x}_{n+1}\text{.}\) In such a process, the matrix \(A\) is a square matrix with non-negative entries whose columns sum to \(1\text{.}\) Starting from an initial state \(\vb{x}_0\text{,}\) we are often interested in whether the long-term evolution approaches a specific state vector \(\vb{x}\text{.}\) In symbols, we want to determine if \(A^n\vb{x}_0 \to \vb{x}\) as \(n\to\infty\text{.}\)
For such a vector, we should have \(A\vb{x} = \lim_{n\to\infty}A(A^n\vb{x}) = \vb{x}\text{.}\) In other words, \(\vb{x}\) is an eigenvector of \(A\) with eigenvalue \(1\). We call this vector a steady-state vector of the Markov process.
Now let's suppose we model the weather with a Markov process with transition matrix
\begin{equation*}
A = \mqty[0.33 & 0.25 & 0.40 \\ 0.52 & 0.42 & 0.40 \\ 0.15 & 0.33 & 0.20],
\end{equation*}
corresponding states \(S, C\) and \(R\) (i.e., "sunny", "cloudy" and "rainy") and we use an initial state vector of \(\vb{x}_0 = \smqty[0 & 1 & 0]^T\text{.}\) To figure out the long-term probability that it will be a cloudy day, we can try computing \(A^n\vb{x}_0\) for large values of \(n\text{.}\) See the Octave cell below. If we do so, it appears that the long-term probability of a cloudy day settles in around \(44.6\%\text{.}\)
We can make this analysis more precise by looking for the steady state vector using
eig. If we take this approach, then we see that \(A\) has \(1\) as an eigenvalue and corresponding eigenvector \(\smqty[0.52 & 0.74 & 0.41]^{T}\text{.}\) This is not a state vector since the values do not add to \(1\) and therefore can't be probabilities. However, we can convert this into a state vector by dividing each entry by the sum \(0.52+0.74+0.41 = 1.678\) which in turn gives the eigenvector
\begin{equation*}
\vb{x} = \mqty[0.3113 \\ 0.4463 \\ 0.2425]
\end{equation*}
confirming our earlier guess. We can also see the long-term probabilities of sunny and rainy days from this steady-state vector as well.
