Getting started

Introduction

This vignette briefly outlines the functionality of EpiSoon. To get started load the required packages.

Load the package (bsts for models, ggplot2 for plotting, and cowplot for theming)

library(EpiSoon)
library(bsts)
library(fable)
library(cowplot)
library(dplyr)

Forecast Rts, score and plot

We use an example dataframe built into the package but this could be replaced with your own data.

EpiSoon::example_obs_rts
#>          rt       date
#> 1  2.490547 2020-03-01
#> 2  2.442588 2020-03-02
#> 3  2.402473 2020-03-03
#> 4  2.335572 2020-03-04
#> 5  2.266551 2020-03-05
#> 6  2.192293 2020-03-06
#> 7  2.146429 2020-03-07
#> 8  2.104371 2020-03-08
#> 9  2.059281 2020-03-09
#> 10 2.027134 2020-03-10
#> 11 2.014678 2020-03-11
#> 12 1.998946 2020-03-12
#> 13 1.968350 2020-03-13
#> 14 1.947376 2020-03-14
#> 15 1.906984 2020-03-15
#> 16 1.812842 2020-03-16
#> 17 1.718532 2020-03-17
#> 18 1.665646 2020-03-18
#> 19 1.639927 2020-03-19
#> 20 1.633795 2020-03-20
#> 21 1.682025 2020-03-21
#> 22 1.561653 2020-03-22

Fit a bsts model and produce a Rt forecast. Any appropriately wrapped model can be used (see bsts_model and fable_model for an examples).

rt_forecast <- forecast_rt(EpiSoon::example_obs_rts[1:10, ],
  model = function(...) {
    EpiSoon::bsts_model(model = function(ss, y) {
      bsts::AddAutoAr(ss, y = y, lags = 10)
    }, ...)
  },
  horizon = 21, samples = 10
)

rt_forecast
#> # A tibble: 210 × 4
#>    sample date          rt horizon
#>     <int> <date>     <dbl>   <int>
#>  1      1 2020-03-11  2.00       1
#>  2      2 2020-03-11  1.96       1
#>  3      3 2020-03-11  2.01       1
#>  4      4 2020-03-11  2.02       1
#>  5      5 2020-03-11  1.91       1
#>  6      6 2020-03-11  1.89       1
#>  7      7 2020-03-11  1.83       1
#>  8      8 2020-03-11  2.07       1
#>  9      9 2020-03-11  1.90       1
#> 10     10 2020-03-11  2.01       1
#> # ℹ 200 more rows

Score the forecast

rt_scores <- score_forecast(rt_forecast, EpiSoon::example_obs_rts)

rt_scores
#>           date horizon  bias       dss       crps overprediction
#>         <Date>   <int> <num>     <num>      <num>          <num>
#>  1: 2020-03-11       1  -0.4 -4.691287 0.02729385    0.000000000
#>  2: 2020-03-12       2  -0.4 -3.908083 0.04200669    0.000000000
#>  3: 2020-03-13       3  -0.6 -3.591938 0.04325256    0.000000000
#>  4: 2020-03-14       4  -0.6 -3.696009 0.05094181    0.000000000
#>  5: 2020-03-15       5  -0.8 -3.329451 0.04230191    0.000000000
#>  6: 2020-03-16       6   0.2 -3.670295 0.02571250    0.000465794
#>  7: 2020-03-17       7   0.2 -3.309348 0.04734411    0.007724726
#>  8: 2020-03-18       8   0.2 -2.912233 0.05015259    0.002787545
#>  9: 2020-03-19       9   0.0 -2.906609 0.04617460    0.000000000
#> 10: 2020-03-20      10  -0.2 -3.106047 0.04853196    0.000000000
#> 11: 2020-03-21      11  -0.6 -2.631192 0.09326165    0.000000000
#> 12: 2020-03-22      12  -0.4 -2.665554 0.05926225    0.000000000
#>     underprediction dispersion  log_score        mad   ae_median      se_mean
#>               <num>      <num>      <num>      <num>       <num>        <num>
#>  1:     0.004027416 0.02326643 -1.4136279 0.08539850 0.038630658 2.992559e-03
#>  2:     0.012533805 0.02947288 -1.0663732 0.09710996 0.060879176 5.639814e-03
#>  3:     0.014437996 0.02881456 -1.0963544 0.08642888 0.064686907 6.960322e-03
#>  4:     0.026254458 0.02468736 -0.9796007 0.08410731 0.082653933 8.384510e-03
#>  5:     0.016954170 0.02534774 -1.1152168 0.07898306 0.055312628 1.021943e-02
#>  6:     0.000000000 0.02524670 -1.2436471 0.08543024 0.004731511 1.248033e-03
#>  7:     0.000000000 0.03961938 -0.7326964 0.13541446 0.057754708 9.560754e-04
#>  8:     0.000000000 0.04736505 -0.7160134 0.15442834 0.066255456 1.086381e-03
#>  9:     0.000000000 0.04617460 -0.5536151 0.20024155 0.005356842 5.227182e-06
#> 10:     0.007356902 0.04117506 -0.7056845 0.15753011 0.052630441 5.228729e-04
#> 11:     0.043738648 0.04952300 -0.2885369 0.15218223 0.140996106 1.744156e-02
#> 12:     0.007342486 0.05191977 -0.4615967 0.22193245 0.060378040 4.515357e-03

Summarise the forecast scores

summarise_scores(rt_scores)
#> # A tibble: 10 × 8
#>    score             bottom    lower   median     mean    upper      top      sd
#>    <chr>              <dbl>    <dbl>    <dbl>    <dbl>    <dbl>    <dbl>   <dbl>
#>  1 ae_median        4.90e-3  0.0491   0.0591   5.75e-2  6.51e-2  0.125   0.0352 
#>  2 bias            -7.45e-1 -0.6     -0.4     -2.83e-1  5   e-2  0.2     0.356  
#>  3 crps             2.61e-2  0.0422   0.0468   4.80e-2  5.03e-2  0.0839  0.0171 
#>  4 dispersion       2.37e-2  0.0253   0.0345   3.61e-2  4.65e-2  0.0513  0.0110 
#>  5 dss             -4.48e+0 -3.68    -3.32    -3.37e+0 -2.91e+0 -2.64    0.589  
#>  6 log_score       -1.37e+0 -1.10    -0.856   -8.64e-1 -6.68e-1 -0.336   0.340  
#>  7 mad              8.04e-2  0.0854   0.116    1.28e-1  1.55e-1  0.216   0.0494 
#>  8 overprediction   0        0        0        9.15e-4  1.16e-4  0.00637 0.00229
#>  9 se_mean          1.48e-4  0.00105  0.00375  5.00e-3  7.32e-3  0.0155  0.00516
#> 10 underprediction  0        0        0.00735  1.11e-2  1.51e-2  0.0389  0.0132

Summarise the forecast

summarised_rt_forecast <- summarise_forecast(rt_forecast)

summarised_rt_forecast
#> # A tibble: 21 × 9
#>    date       horizon median  mean     sd bottom lower upper   top
#>    <date>       <int>  <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-11       1   1.98  1.96 0.0753   1.83  1.96  2.07  2.07
#>  2 2020-03-12       2   1.94  1.92 0.120    1.73  1.87  2.01  2.12
#>  3 2020-03-13       3   1.90  1.88 0.146    1.64  1.85  1.96  2.13
#>  4 2020-03-14       4   1.86  1.86 0.120    1.62  1.82  1.93  2.03
#>  5 2020-03-15       5   1.85  1.81 0.160    1.46  1.85  1.91  2.01
#>  6 2020-03-16       6   1.82  1.78 0.164    1.38  1.75  1.87  2.01
#>  7 2020-03-17       7   1.78  1.75 0.199    1.31  1.69  1.87  2.04
#>  8 2020-03-18       8   1.73  1.70 0.243    1.16  1.61  1.79  2.08
#>  9 2020-03-19       9   1.63  1.64 0.246    1.14  1.60  1.84  2.05
#> 10 2020-03-20      10   1.58  1.61 0.222    1.16  1.54  1.74  2.00
#> # ℹ 11 more rows

Plot the forecast against observed data

plot_forecast(summarised_rt_forecast, EpiSoon::example_obs_rts)

Forecast cases, score and plot

Forecasting cases requires the observed cases on which the observed Rt estimates were based

EpiSoon::example_obs_cases
#> # A tibble: 63 × 2
#>    cases date      
#>    <dbl> <date>    
#>  1     1 2020-01-20
#>  2     0 2020-01-21
#>  3     1 2020-01-22
#>  4     0 2020-01-23
#>  5     0 2020-01-24
#>  6     0 2020-01-25
#>  7     1 2020-01-26
#>  8     0 2020-01-27
#>  9     0 2020-01-28
#> 10     0 2020-01-29
#> # ℹ 53 more rows

It also requires an assumption to be made about the serial interval (defined using probability distribution).

EpiSoon::example_serial_interval
#>         1    2    3    4    5    6    7    8    9   10   11   12   14 
#> 0.00 0.03 0.25 0.17 0.09 0.15 0.13 0.05 0.05 0.03 0.02 0.01 0.01 0.01

Forecast cases (using the case data on which the observed Rt estimates were based)

case_forecast <- forecast_cases(EpiSoon::example_obs_cases, rt_forecast,
  serial_interval = EpiSoon::example_serial_interval
)

case_forecast
#>      sample       date cases horizon
#>       <num>     <Date> <int>   <int>
#>   1:      1 2020-03-11   178       1
#>   2:      1 2020-03-12   186       2
#>   3:      1 2020-03-13   224       3
#>   4:      1 2020-03-14   253       4
#>   5:      1 2020-03-15   309       5
#>  ---                                
#> 206:     10 2020-03-27  1375      17
#> 207:     10 2020-03-28  1554      18
#> 208:     10 2020-03-29  1735      19
#> 209:     10 2020-03-30  1927      20
#> 210:     10 2020-03-31  2083      21

Score the cases forecast

case_scores <- score_case_forecast(case_forecast, EpiSoon::example_obs_cases)
#> Warning: Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.

case_scores
#>           date horizon  bias       dss   crps overprediction underprediction
#>         <Date>   <int> <num>     <num>  <num>          <num>           <num>
#>  1: 2020-03-11       1   0.2  4.862912   4.49   2.000000e+00             0.0
#>  2: 2020-03-12       2   0.1  5.206750   3.79   8.881784e-16             0.0
#>  3: 2020-03-13       3   0.4  6.700315  11.63   4.000000e+00             0.0
#>  4: 2020-03-14       4   0.2  7.656474  13.96   4.000000e-01             0.0
#>  5: 2020-03-15       5   0.4  7.756487  16.27   6.400000e+00             0.0
#>  6: 2020-03-16       6   0.8 10.134662  58.61   4.380000e+01             0.0
#>  7: 2020-03-17       7   0.8 11.328143  83.73   6.620000e+01             0.0
#>  8: 2020-03-18       8   0.8 10.496891  78.34   4.640000e+01             0.0
#>  9: 2020-03-19       9   0.6 10.358460  57.35   1.990000e+01             0.0
#> 10: 2020-03-20      10   0.6 10.531157  54.81   1.660000e+01             0.0
#> 11: 2020-03-21      11  -0.2 10.494836  43.05   0.000000e+00             4.1
#> 12: 2020-03-22      12   0.8 12.409442 183.37   1.244000e+02             0.0
#>     dispersion log_score      mad ae_median  se_mean
#>          <num>     <num>    <num>     <num>    <num>
#>  1:       2.49  3.761955   9.6369      10.0    22.09
#>  2:       3.79  3.867026  17.0499       0.5     0.01
#>  3:       7.63  4.741225  20.7564      21.0   118.81
#>  4:      13.56  4.880791  48.9258      16.5   538.24
#>  5:       9.87  4.994165  38.5476      29.5   552.25
#>  6:      14.81  6.003877  57.8214     101.5  6577.21
#>  7:      17.53  6.443136  77.8365     119.0 13479.21
#>  8:      31.94  6.297812 120.0906     120.5 13225.00
#>  9:      37.45  6.207223 156.4143      66.5  9447.84
#> 10:      38.21  6.226776 156.4143      54.5  9467.29
#> 11:      38.95  6.178953 154.1904      23.5     3.24
#> 12:      58.97  7.048915 243.8877     222.0 85497.76

Summarise the cases scores

summarise_scores(case_scores)
#> # A tibble: 10 × 8
#>    score              bottom lower  median      mean    upper      top        sd
#>    <chr>               <dbl> <dbl>   <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
#>  1 ae_median        3.11e+ 0 19.9    42       65.4     106.     194.      65.0  
#>  2 bias            -1.18e- 1  0.2     0.5      0.458     0.8      0.8      0.332
#>  3 crps             3.98e+ 0 13.4    48.9     50.8      63.5    156.      50.5  
#>  4 dispersion       2.85e+ 0  9.31   16.2     22.9      37.6     53.5     17.7  
#>  5 dss              4.96e+ 0  7.42   10.2      8.99     10.5     12.1      2.47 
#>  6 log_score        3.79e+ 0  4.85    6.09     5.55      6.24     6.88     1.07 
#>  7 mad              1.17e+ 1 34.1    67.8     91.8     155.     220.      73.6  
#>  8 overprediction   2.44e-16  1.60   11.5     27.5      44.4    108.      37.6  
#>  9 se_mean          8.98e- 1 94.6  3565.   11577.    10407.   65693.   23895.   
#> 10 underprediction  0         0       0        0.342     0        2.97     1.18

Summarise the cases forecast

summarised_case_forecast <- summarise_case_forecast(case_forecast)

summarised_case_forecast
#> # A tibble: 21 × 9
#>    date       horizon median  mean    sd bottom lower upper   top
#>    <date>       <int>  <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-11       1   177   172.  10.8    152   177   186   186
#>  2 2020-03-12       2   194.  194.  14.2    173   194   214   214
#>  3 2020-03-13       3   229   219.  27.5    176   212   240   254
#>  4 2020-03-14       4   268.  274.  40.3    223   223   282   331
#>  5 2020-03-15       5   302.  296.  43.2    219   282   334   370
#>  6 2020-03-16       6   368.  347.  59.4    225   361   408   408
#>  7 2020-03-17       7   415   412.  72.2    267   357   453   507
#>  8 2020-03-18       8   464.  458  114.     227   441   586   586
#>  9 2020-03-19       9   466   497. 147.     227   382   593   707
#> 10 2020-03-20      10   508.  551. 170.     238   436   647   824
#> # ℹ 11 more rows

Plot the forecast against observed case data

plot_forecast(summarised_case_forecast, EpiSoon::example_obs_cases)

Use iterative fitting to explore a forecast

To explore the quality of a models forecast it can help to iteratively forecast from each available data point. This is supported in EpiSoon using the following:

it_rt_forecast <- iterative_rt_forecast(EpiSoon::example_obs_rts,
  model = function(...) {
    EpiSoon::bsts_model(model = function(ss, y) {
      bsts::AddAutoAr(ss, y = y, lags = 10)
    }, ...)
  },
  horizon = 7, samples = 10, min_points = 4
)

it_rt_forecast
#> # A tibble: 1,260 × 5
#>    forecast_date sample date          rt horizon
#>    <chr>          <int> <date>     <dbl>   <int>
#>  1 2020-03-05         1 2020-03-06  2.17       1
#>  2 2020-03-05         2 2020-03-06  2.31       1
#>  3 2020-03-05         3 2020-03-06  2.23       1
#>  4 2020-03-05         4 2020-03-06  2.15       1
#>  5 2020-03-05         5 2020-03-06  2.27       1
#>  6 2020-03-05         6 2020-03-06  1.64       1
#>  7 2020-03-05         7 2020-03-06  2.37       1
#>  8 2020-03-05         8 2020-03-06  2.21       1
#>  9 2020-03-05         9 2020-03-06  2.38       1
#> 10 2020-03-05        10 2020-03-06  2.21       1
#> # ℹ 1,250 more rows

We can then iteratively forecast cases using the following:

it_cases_forecast <- iterative_case_forecast(
  it_fit_samples = it_rt_forecast,
  cases = EpiSoon::example_obs_cases,
  serial_interval = EpiSoon::example_serial_interval
)

it_cases_forecast
#>       forecast_date sample       date cases horizon
#>              <char>  <num>     <Date> <int>   <int>
#>    1:    2020-03-05      1 2020-03-06    86       1
#>    2:    2020-03-05      1 2020-03-07    79       2
#>    3:    2020-03-05      1 2020-03-08   104       3
#>    4:    2020-03-05      1 2020-03-09   134       4
#>    5:    2020-03-05      1 2020-03-10   138       5
#>   ---                                              
#> 1256:    2020-03-22     10 2020-03-25   706       3
#> 1257:    2020-03-22     10 2020-03-26   726       4
#> 1258:    2020-03-22     10 2020-03-27   726       5
#> 1259:    2020-03-22     10 2020-03-28   839       6
#> 1260:    2020-03-22     10 2020-03-29   889       7

All functionality shown above is also supported for iterative forecasting.

Evaluate a model

In real world use we are likely to want to evaluate a model by iteratively forecasting Rts and cases, summarising these forecasts, scoring them and then returning them in a sensible format. These steps are all contained in the evaluate_model function.

model_eval <- evaluate_model(EpiSoon::example_obs_rts,
  EpiSoon::example_obs_cases,
  model = function(...) {
    EpiSoon::bsts_model(model = function(ss, y) {
      bsts::AddAutoAr(ss, y = y, lags = 10)
    }, ...)
  },
  horizon = 21, samples = 10,
  serial_interval = EpiSoon::example_serial_interval
)
#> Warning: Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.

model_eval
#> $forecast_rts
#> # A tibble: 399 × 10
#>    forecast_date date       horizon median  mean     sd bottom lower upper   top
#>    <chr>         <date>       <int>  <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-04    2020-03-05       1   2.28  2.29 0.0418   2.23  2.27  2.30  2.38
#>  2 2020-03-04    2020-03-06       2   2.24  2.27 0.0796   2.20  2.20  2.24  2.44
#>  3 2020-03-04    2020-03-07       3   2.17  2.20 0.0972   2.08  2.12  2.20  2.42
#>  4 2020-03-04    2020-03-08       4   2.13  2.17 0.129    2.06  2.06  2.14  2.46
#>  5 2020-03-04    2020-03-09       5   2.05  2.12 0.160    1.96  1.96  2.07  2.47
#>  6 2020-03-04    2020-03-10       6   2.00  2.08 0.169    1.89  1.89  2.01  2.37
#>  7 2020-03-04    2020-03-11       7   1.94  2.03 0.179    1.85  1.85  1.94  2.41
#>  8 2020-03-04    2020-03-12       8   1.88  1.98 0.209    1.75  1.84  1.97  2.44
#>  9 2020-03-04    2020-03-13       9   1.82  1.90 0.208    1.71  1.71  1.84  2.37
#> 10 2020-03-04    2020-03-14      10   1.80  1.86 0.229    1.61  1.69  1.82  2.38
#> # ℹ 389 more rows
#> 
#> $rt_scores
#>      forecast_date       date horizon  bias       dss       crps overprediction
#>             <char>     <Date>   <int> <num>     <num>      <num>          <num>
#>   1:    2020-03-04 2020-03-05       1   0.6 -6.199857 0.01067544    0.002713175
#>   2:    2020-03-04 2020-03-06       2   1.0 -4.092500 0.03913039    0.023854950
#>   3:    2020-03-04 2020-03-07       3   0.6 -4.405747 0.02560759    0.007773878
#>   4:    2020-03-04 2020-03-08       4   0.2 -3.907413 0.03109359    0.005804566
#>   5:    2020-03-04 2020-03-09       5   0.0 -3.621503 0.03411773    0.000000000
#>  ---                                                                           
#> 167:    2020-03-19 2020-03-21       2  -1.0 -2.391009 0.11229863    0.000000000
#> 168:    2020-03-19 2020-03-22       3  -0.4 -3.304818 0.03443609    0.000000000
#> 169:    2020-03-20 2020-03-21       1  -0.8 -4.109734 0.05829671    0.000000000
#> 170:    2020-03-20 2020-03-22       2   0.2 -3.935531 0.02550214    0.001908332
#> 171:    2020-03-21 2020-03-22       1   0.8 -3.364868 0.05479628    0.029404531
#>      underprediction dispersion  log_score        mad  ae_median      se_mean
#>                <num>      <num>      <num>      <num>      <num>        <num>
#>   1:     0.000000000 0.00796226 -2.3116337 0.03271227 0.01289396 0.0004011467
#>   2:     0.000000000 0.01527544 -1.4381730 0.05075759 0.04852127 0.0061268006
#>   3:     0.000000000 0.01783372 -1.5726870 0.06115264 0.02435990 0.0030693302
#>   4:     0.000000000 0.02528902 -1.1538320 0.09866985 0.03019366 0.0043856667
#>   5:     0.000000000 0.03411773 -0.9779854 0.09813755 0.01085950 0.0034041445
#>  ---                                                                         
#> 167:     0.091555152 0.02074347  0.2977267 0.06137922 0.12306800 0.0309605269
#> 168:     0.005927061 0.02850903 -1.1771941 0.10443211 0.03977151 0.0066907550
#> 169:     0.041212439 0.01708428 -0.6675415 0.06528006 0.09281170 0.0060060091
#> 170:     0.000000000 0.02359381 -1.3401668 0.07677724 0.02794667 0.0021779371
#> 171:     0.000000000 0.02539175 -1.1093058 0.08649769 0.08376115 0.0124168608
#> 
#> $forecast_cases
#> # A tibble: 171 × 10
#>    forecast_date date       horizon median  mean    sd bottom lower upper   top
#>    <chr>         <date>       <int>  <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-04    2020-03-05       1   67.5  71.4  10.5     59    59    70    88
#>  2 2020-03-04    2020-03-06       2   83    84    12.6     63    72    87   102
#>  3 2020-03-04    2020-03-07       3  103   102.   12.8     80   102   116   116
#>  4 2020-03-04    2020-03-08       4  128.  129.   15.5    112   112   129   163
#>  5 2020-03-04    2020-03-09       5  148.  146.   14.3    119   139   153   164
#>  6 2020-03-04    2020-03-10       6  166.  171.   32.9    114   160   204   220
#>  7 2020-03-04    2020-03-11       7  212.  207.   40.3    135   203   252   262
#>  8 2020-03-04    2020-03-12       8  240.  242.   52.8    163   191   255   336
#>  9 2020-03-04    2020-03-13       9  269   283.   78.1    183   211   291   451
#> 10 2020-03-04    2020-03-14      10  297   322.  101.     214   234   310   536
#> # ℹ 161 more rows
#> 
#> $case_scores
#>      sample forecast_date       date horizon  bias       dss   crps
#>      <char>        <char>     <Date>   <int> <num>     <num>  <num>
#>   1:      1    2020-03-04 2020-03-05       1   0.4  5.310300   4.02
#>   2:      1    2020-03-04 2020-03-06       2   0.6  5.809870   6.40
#>   3:      1    2020-03-04 2020-03-07       3   0.6  6.229501   8.87
#>   4:      1    2020-03-04 2020-03-08       4   1.0  8.778775  19.19
#>   5:      1    2020-03-04 2020-03-09       5   1.0 10.033621  22.29
#>  ---                                                               
#> 167:      1    2020-03-19 2020-03-21       2  -1.0 14.969724 121.30
#> 168:      1    2020-03-19 2020-03-22       3   0.8 12.827055 109.28
#> 169:      1    2020-03-20 2020-03-21       1  -1.0 24.405494 101.01
#> 170:      1    2020-03-20 2020-03-22       2   1.0 18.134923 159.51
#> 171:      1    2020-03-21 2020-03-22       1   1.0 28.606997 196.00
#>      overprediction underprediction dispersion log_score     mad ae_median
#>               <num>           <num>      <num>     <num>   <num>     <num>
#>   1:            0.8             0.0       3.22  3.460153 10.3782       4.5
#>   2:            3.2             0.0       3.20  3.830918 13.3434      10.0
#>   3:            5.6             0.0       3.27  4.119981 17.7912      15.0
#>   4:           16.0             0.0       3.19  5.082311 14.8260      26.5
#>   5:           19.4             0.0       2.89  4.888529 10.3782      32.5
#>  ---                                                                      
#> 167:            0.0           110.7      10.60 15.444145 34.0998     136.5
#> 168:           91.4             0.0      17.88  6.845924 63.0105     166.5
#> 169:            0.0            94.1       6.91 13.038058 27.4281     112.5
#> 170:          153.3             0.0       6.21 46.246028 25.2042     172.0
#> 171:          182.9             0.0      13.10 18.290501 54.8562     212.0
#>       se_mean
#>         <num>
#>   1:    70.56
#>   2:   121.00
#>   3:   182.25
#>   4:   734.41
#>   5:   882.09
#>  ---         
#> 167: 21933.61
#> 168: 20534.89
#> 169: 13595.56
#> 170: 34077.16
#> 171: 49684.41

All functionality outlined above can be applied to this output but a special plotting function (plot_forecast_evaluation) is also provided. First evaluate the Rt forecast against observed values.

plot_forecast_evaluation(model_eval$forecast_rts,
  EpiSoon::example_obs_rts,
  horizon_to_plot = 7
)

Then evaluate forecast cases against observed values.

plot_forecast_evaluation(model_eval$forecast_cases,
  EpiSoon::example_obs_cases,
  horizon_to_plot = 7
)

Wrapper functions

EpiSoon provides several wrapper functions (compare_models and compare_timeseries). These both wrap evaluate_model and can be used to rapidly explore several forecasting models (compare_models) against multiple time series (compare_timeseries). All lower level summary and plotting functions can be then used with the output of these wrappers to explore the results. See the function documentation for further details.

Supporting generic modelling packages

EpiSoon supports the use of generic forecasting models if they are used in a wrapper that accepts a standardised set of inputs and outputs its forecast in the form the package expects. Examples of model wrappers are those for the bsts and fable packages (bsts_model and fable_model). See the examples and documentation for fable_model for further details.