Lec. 8

• Markov models, also known as Markov processes or Markov chains, are mathematical models used to describe systems that exhibit a type of memoryless property called the Markov property.
• In a Markov model, the future state of a system depends only on its current state and is independent of its previous states.
• Markov models are useful when a decision problem involves risk that is continuous over time, when the timing of events is important, and when important events may happen more than once.
• In health sciences, Markov models are widely used as analytical tools to assess diseases from an economic point of view.
• A patient may be assessed in a finite number of discrete states of health, in which the important clinical events are modeled as transitions from one state to another.
• A Markov model describes a system in terms of a set of states, which represent the possible conditions or configurations of the system at any given time.
• In a weather prediction model, states could be "sunny," "cloudy," and "rainy."
• Markov models define the probabilities of transitioning from one state to another.
• The transition probabilities in a Markov model are often represented in a transition matrix, where each entry describes the probability of moving from one state to another in one time step.
• The central assumption in Markov models is that the future state of the system depends only on its current state, not on how it arrived at that state.
• This property is often referred to as the "memorylessness" of the system.
• The evolution of the Markov process in the future depends only on the present state and does not depend on history.
• Markov models often assume that the transition probabilities do not change over time, which is known as time homogeneity.
• In some applications, non-homogeneous Markov models are used where transition probabilities may change over time.
• Finance: To model stock prices, interest rates, and other financial variables.
• Natural Language Processing: For text and speech processing, including hidden Markov models for speech recognition.
• Genetics: To model DNA sequences and protein structure.
• Operations Research: To model queuing systems, inventory management, and more.
• Epidemiology: For modeling the spread of diseases and healthcare planning.
• Physics: To describe particle interactions, quantum systems, and more.
• Discrete-Time Markov Chains: The state of the system changes at discrete time intervals, such as days, hours, or steps.
• Continuous-Time Markov Chains: The state of the system changes continuously over time, often described using differential equations.
• Hidden Markov Models (HMMs): HMMs are a type of Markov model where the true state of the system is hidden and can only be inferred based on observed data.