Markov chains with finite state space
Basic concepts: transition probabilities, state distributions, properties of states
Analysis of transitions and duration of stay
Costs with finitely many time steps and costs in the long run (asymptotics)
Markov Chain Monte Carlo as a simulation method based on Markov chains
Point processes
Poisson processes
Renewal processes
Cumulative processes
Time-continuous Markov processes with finite state space
Basic concepts: transition, rate and generator matrices, state distributions
Analysis of transitions, duration of stay
Kulkarni, V.G. (2011). Introduction to Modeling and Analysis of Stochastic Systems, Second Edition, Springer.
Waldmann, K.H., Stocker, U.M. (2004). Stochastische Modelle. Eine anwendungsorientierte Einführung, Springer.