```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE)
library(kableExtra)
set.seed(123)
```

## Topics

Deterministic vs stochastic elements

Regression with autocorrelated errors

Regression with temporal random effects


## A very simple model

Consider this simple model, consisting of a mean $\mu$ plus error

$$
y_i = \mu + e_i ~ \text{with} ~ e_i \sim \text{N}(0,\sigma^2)
$$


## A very simple model

The right-hand side of the equation is composed of _deterministic_ and _stochastic_ pieces

$$
y_i = \underbrace{\mu}_{\text{deterministic}} + \underbrace{e_i}_{\text{stochastic}}
$$


## A very simple model

Sometime these pieces are referred to as _fixed_ and _random_

$$
y_i = \underbrace{\mu}_{\text{fixed}} + \underbrace{e_i}_{\text{random}}
$$


## A very simple model

This can also be seen by rewriting the model 

$$
y_i = \mu + e_i ~ \text{with} ~ e_i \sim \text{N}(0,\sigma^2)
$$

as

$$
y_i \sim \text{N}(\mu,\sigma^2)
$$


## Simple linear regression

We can expand the deterministic part of the model, as with linear regression

$$
y_i = \underbrace{\alpha + \beta x_i}_{\text{mean}} + e_i ~ \text{with} ~ e_i \sim \text{N}(0,\sigma^2)
$$

so

$$
y_i \sim \text{N}(\alpha + \beta x_i,\sigma^2)
$$


## A simple time series model

Consider a simple model with a mean $\mu$ plus white noise

$$
y_t = \mu + e_t ~ \text{with} ~ e_t \sim \text{N}(0,\sigma^2)
$$


## Time series model with covariates

We can expand the deterministic part of the model, as before with linear regression

$$
y_t = \underbrace{\alpha + \beta x_t}_{\text{mean}} + e_t ~ \text{with} ~ e_t \sim \text{N}(0,\sigma^2)
$$

so

$$
y_t \sim \text{N}(\alpha + \beta x_t,\sigma^2)
$$


## Example of linear model

```{r ts_with_AR_errors}
set.seed(123)
tt <- 30
ee <- rnorm(tt)
ww <- rnorm(tt)
phi <- 0.7
for(t in 2:tt) {
  ee[t] <- phi * ee[t-1] + ww[t]
}

aa <- 5
bb <- 1
xx <- rnorm(tt) + 2

yy <- aa + bb * xx + ee

par(mai = c(0.9,0.9,0.3,0.1), omi = c(0,0,0,0))

plot.ts(yy, ylim = c(min(xx,yy), max(xx, yy)), las = 1,
        col = "dodgerblue", lwd = 2,
        ylab = expression(italic(x[t])~~or~~italic(y[t])))
lines(xx, col = "blue", lwd = 2)
text(xx[1], expression(italic(x[t])), pos = 3, col = "blue")
text(yy[1], expression(italic(y[t])), pos = 3, col = "dodgerblue")
mtext(side = 3, expression(italic(y[t]) == 1~+~5~italic(x[t])~+~italic(e[t])),
      line = 0.4, adj = 0)
```


## Model residuals

```{r model_residuals}

par(mai = c(0.9,0.9,0.3,0.1), omi = c(0,0,0,0))

plot.ts(ee, col = "darkred", lwd = 2,
        ylab = expression(italic(e[t])))
```

These do _not_ look like white noise!


## ACF of model residuals

```{r ACF_model_residuals}

par(mai = c(0.9,0.9,0.3,0.1), omi = c(0,0,0,0))

acf(ee)
```

There is significant autocorrelation at lag = 1


## Model with autocorrelated errors

We can expand the stochastic part of the model to have autocorrelated errors

$$
y_t = \alpha + \beta x_t + e_t \\
e_t = \phi e_{t-1} + w_t
$$

with $w_t \sim \text{N}(0,\sigma^2)$


## Model with autocorrelated errors

We can expand the stochastic part of the model to have autocorrelated errors

$$
y_t = \alpha + \beta x_t + e_t \\
e_t = \phi e_{t-1} + w_t
$$

with $w_t \sim \text{N}(0,\sigma^2)$

We can write this model as our standard state-space model


## State-space model | Observation equation

$$
\begin{align}
  y_t &= \alpha + \beta x_t + e_t \\
      &= e_t + \alpha + \beta x_t\\
      &\Downarrow \\
  y_t &= x_t + a + D d_t + v_t\\
\end{align}
$$

with

$x_t = e_t$, $a = \alpha$, $D = \beta$, $d_t = x_t$, $v_t = 0$


## State-space model | State equation


$$
\begin{align}
  e_t &= \phi e_{t-1} + w_t \\
      &\Downarrow \\
  x_t &= B x_t + w_t\\
\end{align}
$$

with

$x_t = e_t$ and $B = \phi$


## State-space model | Full form

$$
y_t = \alpha + \beta x_t + e_t \\
e_t = \phi e_{t-1} + w_t \\
\Downarrow \\
y_t = a + D d_t + x_t\\
x_t = B x_t + w_t
$$


## State-space model | Observation model in `MARSS()`

$$
y_t = a + D d_t + x_t \\
\Downarrow \\
y_t = Z x_t + a + D d_t + v_t
$$

```{r, echo=TRUE, eval=FALSE}
y = data         ## [1 x T] matrix of data
a = matrix("a")  ## intercept
D = matrix("D")  ## slope
d = covariate    ## [1 x T] matrix of measured covariate
Z = matrix(1)    ## no multiplier on x 
R = matrix(0)    ## v_t ~ N(0,R); want y_t = 0 for all t
```


## State-space model | State model in `MARSS()`

$$
x_t = B x_t + w_t \\
\Downarrow \\
x_t = B x_t + u + C c_t + w_t
$$

```{r, echo=TRUE, eval=FALSE}
B = matrix("b")  ## AR(1) coefficient for model errors
Q = matrix("q")  ## w_t ~ N(0,Q); var for model errors
u = matrix(0)    ## u = 0
C = matrix(0)    ## C = 0
c = matrix(0)    ## c_t = 0 for all t
```


## Expanding the random effect

Recall our simple model

$$
y_t = \underbrace{\mu}_{\text{fixed}} + \underbrace{e_t}_{\text{random}}
$$


## Expanding the random effect

We can expand the random portion

$$
y_t = \underbrace{\mu}_{\text{fixed}} + ~ \underbrace{f_t + e_t}_{\text{random}}
$$

$$
e_t \sim \text{N}(0, \sigma) \\
f_t \sim \text{N}(f_{t-1}, \gamma)
$$


## Expanding the random effect

We can expand the random portion

$$
y_t = \underbrace{\mu}_{\text{fixed}} + ~ \underbrace{f_t + e_t}_{\text{random}}
$$

$$
e_t \sim \text{N}(0, \sigma) \\
f_t \sim \text{N}(f_{t-1}, \gamma)
$$

This is simply a random walk observed with error


## Random walk observed with error

$$
y_t = \mu + f_t + e_t ~ \text{with} ~ e_t \sim \text{N}(0, \sigma) \\
f_t = f_{t-1} + w_t ~ \text{with} ~ w_t \sim \text{N}(0, \gamma) \\
\Downarrow \\
y_t = a + x_t + v_t ~ \text{with} ~ v_t \sim \text{N}(0, R) \\
x_t = x_{t-1} + w_t ~ \text{with} ~ w_t \sim \text{N}(0, Q)
$$


## Expanding fixed & random effects

We can expand the fixed portion

$$
y_t = \underbrace{\alpha + \beta x_t}_{\text{fixed}} + ~ \underbrace{f_t + e_t}_{\text{random}}
$$

$$
e_t \sim \text{N}(0, \sigma) \\
f_t \sim \text{N}(f_{t-1}, \gamma)
$$


## Fixed & random effects | In familiar state-space form

$$
y_t = \alpha + \beta x_t + f_t + e_t ~ \text{with} ~ e_t \sim \text{N}(0, \sigma) \\
f_t = f_{t-1} + w_t ~ \text{with} ~ w_t \sim \text{N}(0, \gamma) \\
\Downarrow \\
y_t = a + D d_t + x_t + v_t ~ \text{with} ~ v_t \sim \text{N}(0, R) \\
x_t = x_{t-1} + w_t ~ \text{with} ~ w_t \sim \text{N}(0, Q)
$$

