Getting started

Introduction

This vignette briefly outlines the functionality of EpiSoon. To get started load the required packages.

Load the package (bsts for models, ggplot2 for plotting, and cowplot for theming)

library(EpiSoon)
library(bsts)
library(fable)
library(cowplot)
library(dplyr)

Forecast Rts, score and plot

We use an example dataframe built into the package but this could be replaced with your own data.

EpiSoon::example_obs_rts
#>          rt       date
#> 1  2.490547 2020-03-01
#> 2  2.442588 2020-03-02
#> 3  2.402473 2020-03-03
#> 4  2.335572 2020-03-04
#> 5  2.266551 2020-03-05
#> 6  2.192293 2020-03-06
#> 7  2.146429 2020-03-07
#> 8  2.104371 2020-03-08
#> 9  2.059281 2020-03-09
#> 10 2.027134 2020-03-10
#> 11 2.014678 2020-03-11
#> 12 1.998946 2020-03-12
#> 13 1.968350 2020-03-13
#> 14 1.947376 2020-03-14
#> 15 1.906984 2020-03-15
#> 16 1.812842 2020-03-16
#> 17 1.718532 2020-03-17
#> 18 1.665646 2020-03-18
#> 19 1.639927 2020-03-19
#> 20 1.633795 2020-03-20
#> 21 1.682025 2020-03-21
#> 22 1.561653 2020-03-22

Fit a bsts model and produce a Rt forecast. Any appropriately wrapped model can be used (see bsts_model and fable_model for an examples).

rt_forecast <- forecast_rt(EpiSoon::example_obs_rts[1:10, ],
  model = function(...) {
    EpiSoon::bsts_model(model = function(ss, y) {
      bsts::AddAutoAr(ss, y = y, lags = 10)
    }, ...)
  },
  horizon = 21, samples = 10
)

rt_forecast
#> # A tibble: 210 × 4
#>    sample date          rt horizon
#>     <int> <date>     <dbl>   <int>
#>  1      1 2020-03-11  2.02       1
#>  2      2 2020-03-11  2.05       1
#>  3      3 2020-03-11  1.97       1
#>  4      4 2020-03-11  2.05       1
#>  5      5 2020-03-11  1.99       1
#>  6      6 2020-03-11  2.09       1
#>  7      7 2020-03-11  1.98       1
#>  8      8 2020-03-11  1.94       1
#>  9      9 2020-03-11  1.99       1
#> 10     10 2020-03-11  1.93       1
#> # ℹ 200 more rows

Score the forecast

rt_scores <- score_forecast(rt_forecast, EpiSoon::example_obs_rts)

rt_scores
#>           date horizon  bias       dss       crps overprediction
#>         <Date>   <int> <num>     <num>      <num>          <num>
#>  1: 2020-03-11       1  -0.2 -6.002330 0.01706757    0.000000000
#>  2: 2020-03-12       2  -0.2 -4.598654 0.02157014    0.000000000
#>  3: 2020-03-13       3   0.0 -4.456673 0.03177013    0.000000000
#>  4: 2020-03-14       4  -0.2 -4.787475 0.02731943    0.000000000
#>  5: 2020-03-15       5   0.0 -4.526320 0.03332107    0.000000000
#>  6: 2020-03-16       6   0.2 -4.341815 0.04517277    0.016204925
#>  7: 2020-03-17       7   0.6 -3.696656 0.06170477    0.028250979
#>  8: 2020-03-18       8   0.6 -3.484140 0.07530746    0.051509580
#>  9: 2020-03-19       9   0.8 -3.634539 0.07376509    0.055246760
#> 10: 2020-03-20      10   0.2 -3.701116 0.03464709    0.004081392
#> 11: 2020-03-21      11   0.0 -3.681546 0.05257577    0.000000000
#> 12: 2020-03-22      12   0.4 -3.493082 0.05712293    0.008904683
#>     underprediction dispersion   log_score        mad   ae_median      se_mean
#>               <num>      <num>       <num>      <num>       <num>        <num>
#>  1:     0.005030529 0.01203704 -1.73104979 0.05805938 0.025704350 0.0001651079
#>  2:     0.001258836 0.02031131 -1.50902884 0.04379888 0.009144337 0.0028050395
#>  3:     0.000000000 0.03177013 -1.15527951 0.11449140 0.032291868 0.0022842837
#>  4:     0.007071851 0.02024758 -1.27457259 0.09139324 0.039333511 0.0011851609
#>  5:     0.000000000 0.03332107 -1.02455061 0.12858929 0.020190690 0.0005879743
#>  6:     0.000000000 0.02896784 -0.80007481 0.09673911 0.090404142 0.0014317075
#>  7:     0.000000000 0.03345379 -0.72963747 0.11338981 0.095507513 0.0068247201
#>  8:     0.000000000 0.02379788 -0.43856507 0.09256732 0.133454484 0.0086675675
#>  9:     0.000000000 0.01851833 -0.09738934 0.06432833 0.115830072 0.0076754828
#> 10:     0.000000000 0.03056570 -0.98531675 0.12388296 0.025881705 0.0014546529
#> 11:     0.000000000 0.05257577 -0.61140448 0.18424730 0.044822014 0.0013558676
#> 12:     0.000000000 0.04821824 -0.48876039 0.21354367 0.070051172 0.0024514095

Summarise the forecast scores

summarise_scores(rt_scores)
#> # A tibble: 10 × 8
#>    score             bottom    lower   median     mean    upper      top      sd
#>    <chr>              <dbl>    <dbl>    <dbl>    <dbl>    <dbl>    <dbl>   <dbl>
#>  1 ae_median        1.22e-2  0.0258   0.0421   0.0586   9.17e-2  0.129   0.0412 
#>  2 bias            -2   e-1 -0.05     0.1      0.183    4.5 e-1  0.745   0.346  
#>  3 crps             1.83e-2  0.0307   0.0399   0.0443   5.83e-2  0.0749  0.0197 
#>  4 dispersion       1.38e-2  0.0203   0.0298   0.0295   3.34e-2  0.0514  0.0119 
#>  5 dss             -5.67e+0 -4.54    -4.02    -4.20    -3.67e+0 -3.49    0.741  
#>  6 log_score       -1.67e+0 -1.19    -0.893   -0.904   -5.81e-1 -0.191   0.471  
#>  7 mad              4.77e-2  0.0846   0.105    0.110    1.25e-1  0.205   0.0494 
#>  8 overprediction   0        0        0.00204  0.0137   1.92e-2  0.0542  0.0205 
#>  9 se_mean          2.81e-4  0.00131  0.00187  0.00307  3.81e-3  0.00839 0.00292
#> 10 underprediction  0        0        0        0.00111  3.15e-4  0.00651 0.00237

Summarise the forecast

summarised_rt_forecast <- summarise_forecast(rt_forecast)

summarised_rt_forecast
#> # A tibble: 21 × 9
#>    date       horizon median  mean     sd bottom lower upper   top
#>    <date>       <int>  <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-11       1   1.99  2.00 0.0506   1.93  1.93  1.99  2.09
#>  2 2020-03-12       2   1.99  1.95 0.0854   1.80  1.99  2.02  2.02
#>  3 2020-03-13       3   1.94  1.92 0.100    1.76  1.89  2.02  2.04
#>  4 2020-03-14       4   1.91  1.91 0.0885   1.75  1.85  1.95  2.03
#>  5 2020-03-15       5   1.89  1.88 0.107    1.71  1.79  1.93  2.05
#>  6 2020-03-16       6   1.90  1.85 0.113    1.65  1.89  1.97  1.97
#>  7 2020-03-17       7   1.81  1.80 0.135    1.51  1.76  1.91  1.97
#>  8 2020-03-18       8   1.80  1.76 0.148    1.40  1.77  1.88  1.92
#>  9 2020-03-19       9   1.76  1.73 0.136    1.39  1.71  1.77  1.91
#> 10 2020-03-20      10   1.66  1.67 0.161    1.30  1.63  1.79  1.87
#> # ℹ 11 more rows

Plot the forecast against observed data

plot_forecast(summarised_rt_forecast, EpiSoon::example_obs_rts)

Forecast cases, score and plot

Forecasting cases requires the observed cases on which the observed Rt estimates were based

EpiSoon::example_obs_cases
#> # A tibble: 63 × 2
#>    cases date      
#>    <dbl> <date>    
#>  1     1 2020-01-20
#>  2     0 2020-01-21
#>  3     1 2020-01-22
#>  4     0 2020-01-23
#>  5     0 2020-01-24
#>  6     0 2020-01-25
#>  7     1 2020-01-26
#>  8     0 2020-01-27
#>  9     0 2020-01-28
#> 10     0 2020-01-29
#> # ℹ 53 more rows

It also requires an assumption to be made about the serial interval (defined using probability distribution).

EpiSoon::example_serial_interval
#>         1    2    3    4    5    6    7    8    9   10   11   12   14 
#> 0.00 0.03 0.25 0.17 0.09 0.15 0.13 0.05 0.05 0.03 0.02 0.01 0.01 0.01

Forecast cases (using the case data on which the observed Rt estimates were based)

case_forecast <- forecast_cases(EpiSoon::example_obs_cases, rt_forecast,
  serial_interval = EpiSoon::example_serial_interval
)

case_forecast
#>      sample       date cases horizon
#>       <num>     <Date> <int>   <int>
#>   1:      1 2020-03-11   144       1
#>   2:      1 2020-03-12   207       2
#>   3:      1 2020-03-13   189       3
#>   4:      1 2020-03-14   258       4
#>   5:      1 2020-03-15   274       5
#>  ---                                
#> 206:     10 2020-03-27  1282      17
#> 207:     10 2020-03-28  1390      18
#> 208:     10 2020-03-29  1468      19
#> 209:     10 2020-03-30  1642      20
#> 210:     10 2020-03-31  1810      21

Score the cases forecast

case_scores <- score_case_forecast(case_forecast, EpiSoon::example_obs_cases)
#> Warning: Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.

case_scores
#>           date horizon  bias       dss   crps overprediction underprediction
#>         <Date>   <int> <num>     <num>  <num>          <num>           <num>
#>  1: 2020-03-11       1   0.0  5.773623   6.37            0.0               0
#>  2: 2020-03-12       2   0.2  5.453304   2.93            0.2               0
#>  3: 2020-03-13       3   0.6  6.760452  11.63            8.2               0
#>  4: 2020-03-14       4   0.4  6.833202   9.07            2.8               0
#>  5: 2020-03-15       5   0.8  8.011295  17.67            8.6               0
#>  6: 2020-03-16       6   1.0 12.044913  61.16           49.8               0
#>  7: 2020-03-17       7   1.0 11.950872  72.87           62.8               0
#>  8: 2020-03-18       8   0.8 11.383748  85.57           73.8               0
#>  9: 2020-03-19       9   0.8 11.329568  86.00           69.8               0
#> 10: 2020-03-20      10   0.8 10.672793  92.76           70.2               0
#> 11: 2020-03-21      11   0.4  9.626493  48.68           24.6               0
#> 12: 2020-03-22      12   1.0 15.576734 268.73          236.7               0
#>     dispersion log_score      mad ae_median   se_mean
#>          <num>     <num>    <num>     <num>     <num>
#>  1:       6.37  4.229843  23.7216       3.5      0.81
#>  2:       2.73  3.557565  13.3434       1.0      3.61
#>  3:       3.43  4.809004  13.3434      18.5    306.25
#>  4:       6.27  4.276023  25.9455      12.0    272.25
#>  5:       9.07  4.672724  25.9455      28.5   1108.89
#>  6:      11.36  6.028916  34.0998      92.5   6724.00
#>  7:      10.07  6.595815  34.0998      99.5   9702.25
#>  8:      11.77  7.292956  42.2541     126.5  11946.49
#>  9:      16.20  6.432625  59.3040     137.5  14089.69
#> 10:      22.56  6.788190  91.1799     158.5  14592.64
#> 11:      24.08  6.155408  63.7518      80.5   1391.29
#> 12:      32.03  7.577648 131.9514     377.5 121661.44

Summarise the cases scores

summarise_scores(case_scores)
#> # A tibble: 10 × 8
#>    score            bottom  lower  median     mean    upper      top        sd
#>    <chr>             <dbl>  <dbl>   <dbl>    <dbl>    <dbl>    <dbl>     <dbl>
#>  1 ae_median        1.69    16.9    86.5     94.7    129.     317.     105.   
#>  2 bias             0.055    0.4     0.8      0.65     0.85     1        0.332
#>  3 crps             3.88    11.0    54.9     63.6     85.7    220.      73.2  
#>  4 dispersion       2.92     6.35   10.7     13.0     17.8     29.8      9.05 
#>  5 dss              5.54     6.82   10.1      9.62    11.5     14.6      3.08 
#>  6 log_score        3.74     4.57    6.09     5.70     6.64     7.50     1.33 
#>  7 mad             13.3     25.4    34.1     46.6     60.4    121.      35.2  
#>  8 overprediction   0.0550   6.85   37.2     50.6     69.9    192.      65.8  
#>  9 se_mean          1.58   298.   4058.   15150.   12482.   92218.   34038.   
#> 10 underprediction  0        0       0        0        0        0        0

Summarise the cases forecast

summarised_case_forecast <- summarise_case_forecast(case_forecast)

summarised_case_forecast
#> # A tibble: 21 × 9
#>    date       horizon median  mean    sd bottom lower upper   top
#>    <date>       <int>  <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-11       1   170.  168.  18.9    143   152   181   196
#>  2 2020-03-12       2   195   192.  16.0    161   193   207   212
#>  3 2020-03-13       3   226.  226.  21.2    189   216   234   267
#>  4 2020-03-14       4   263   268.  25.4    235   243   272   310
#>  5 2020-03-15       5   302.  306.  35.4    258   284   319   370
#>  6 2020-03-16       6   358.  348   39.5    280   350   393   393
#>  7 2020-03-17       7   396.  394.  50.6    316   375   419   504
#>  8 2020-03-18       8   470.  452.  65.0    301   444   499   532
#>  9 2020-03-19       9   537   518.  74.6    368   502   582   623
#> 10 2020-03-20      10   612.  575. 105.     334   605   675   675
#> # ℹ 11 more rows

Plot the forecast against observed case data

plot_forecast(summarised_case_forecast, EpiSoon::example_obs_cases)

Use iterative fitting to explore a forecast

To explore the quality of a models forecast it can help to iteratively forecast from each available data point. This is supported in EpiSoon using the following:

it_rt_forecast <- iterative_rt_forecast(EpiSoon::example_obs_rts,
  model = function(...) {
    EpiSoon::bsts_model(model = function(ss, y) {
      bsts::AddAutoAr(ss, y = y, lags = 10)
    }, ...)
  },
  horizon = 7, samples = 10, min_points = 4
)

it_rt_forecast
#> # A tibble: 1,260 × 5
#>    forecast_date sample date          rt horizon
#>    <chr>          <int> <date>     <dbl>   <int>
#>  1 2020-03-05         1 2020-03-06  2.19       1
#>  2 2020-03-05         2 2020-03-06  2.20       1
#>  3 2020-03-05         3 2020-03-06  2.18       1
#>  4 2020-03-05         4 2020-03-06  2.21       1
#>  5 2020-03-05         5 2020-03-06  2.28       1
#>  6 2020-03-05         6 2020-03-06  2.13       1
#>  7 2020-03-05         7 2020-03-06  2.18       1
#>  8 2020-03-05         8 2020-03-06  2.27       1
#>  9 2020-03-05         9 2020-03-06  2.18       1
#> 10 2020-03-05        10 2020-03-06  2.18       1
#> # ℹ 1,250 more rows

We can then iteratively forecast cases using the following:

it_cases_forecast <- iterative_case_forecast(
  it_fit_samples = it_rt_forecast,
  cases = EpiSoon::example_obs_cases,
  serial_interval = EpiSoon::example_serial_interval
)

it_cases_forecast
#>       forecast_date sample       date cases horizon
#>              <char>  <num>     <Date> <int>   <int>
#>    1:    2020-03-05      1 2020-03-06    75       1
#>    2:    2020-03-05      1 2020-03-07    94       2
#>    3:    2020-03-05      1 2020-03-08   108       3
#>    4:    2020-03-05      1 2020-03-09   137       4
#>    5:    2020-03-05      1 2020-03-10   159       5
#>   ---                                              
#> 1256:    2020-03-22     10 2020-03-25   645       3
#> 1257:    2020-03-22     10 2020-03-26   771       4
#> 1258:    2020-03-22     10 2020-03-27   719       5
#> 1259:    2020-03-22     10 2020-03-28   780       6
#> 1260:    2020-03-22     10 2020-03-29   850       7

All functionality shown above is also supported for iterative forecasting.

Evaluate a model

In real world use we are likely to want to evaluate a model by iteratively forecasting Rts and cases, summarising these forecasts, scoring them and then returning them in a sensible format. These steps are all contained in the evaluate_model function.

model_eval <- evaluate_model(EpiSoon::example_obs_rts,
  EpiSoon::example_obs_cases,
  model = function(...) {
    EpiSoon::bsts_model(model = function(ss, y) {
      bsts::AddAutoAr(ss, y = y, lags = 10)
    }, ...)
  },
  horizon = 21, samples = 10,
  serial_interval = EpiSoon::example_serial_interval
)
#> Warning: Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.
#> Predictions appear to be integer-valued.
#> ! The log score uses kernel density estimation, which may not be appropriate
#>   for integer-valued forecasts.
#> ℹ See the scoringRules package for alternatives for discrete probability
#>   distributions.

model_eval
#> $forecast_rts
#> # A tibble: 399 × 10
#>    forecast_date date       horizon median  mean    sd bottom lower upper   top
#>    <chr>         <date>       <int>  <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-04    2020-03-05       1   2.27  2.21 0.181  1.90   2.19  2.35  2.48
#>  2 2020-03-04    2020-03-06       2   2.25  2.20 0.275  1.70   2.23  2.47  2.60
#>  3 2020-03-04    2020-03-07       3   2.13  2.08 0.351  1.42   2.10  2.44  2.59
#>  4 2020-03-04    2020-03-08       4   2.10  2.02 0.443  1.12   2.07  2.48  2.69
#>  5 2020-03-04    2020-03-09       5   2.03  1.94 0.533  0.972  2.01  2.46  2.76
#>  6 2020-03-04    2020-03-10       6   1.94  1.91 0.595  0.892  1.83  2.50  2.86
#>  7 2020-03-04    2020-03-11       7   1.94  1.81 0.670  0.655  1.79  2.43  2.86
#>  8 2020-03-04    2020-03-12       8   1.85  1.74 0.663  0.651  1.81  2.35  2.87
#>  9 2020-03-04    2020-03-13       9   1.76  1.60 0.738  0.478  1.63  2.23  2.91
#> 10 2020-03-04    2020-03-14      10   1.62  1.55 0.762  0.469  1.51  2.31  2.96
#> # ℹ 389 more rows
#> 
#> $rt_scores
#>      forecast_date       date horizon  bias       dss       crps overprediction
#>             <char>     <Date>   <int> <num>     <num>      <num>          <num>
#>   1:    2020-03-04 2020-03-05       1   0.0 -3.411202 0.03991464  -6.938894e-18
#>   2:    2020-03-04 2020-03-06       2   0.4 -2.687349 0.06250200   1.597906e-02
#>   3:    2020-03-04 2020-03-07       3  -0.2 -2.151413 0.06993905   0.000000e+00
#>   4:    2020-03-04 2020-03-08       4   0.0 -1.695038 0.06744857   0.000000e+00
#>   5:    2020-03-04 2020-03-09       5  -0.2 -1.310320 0.09529686   0.000000e+00
#>  ---                                                                           
#> 167:    2020-03-19 2020-03-21       2  -1.0 -2.464737 0.06797658   0.000000e+00
#> 168:    2020-03-19 2020-03-22       3  -0.2 -5.257135 0.01749182   0.000000e+00
#> 169:    2020-03-20 2020-03-21       1  -0.8 -3.773278 0.05008078   0.000000e+00
#> 170:    2020-03-20 2020-03-22       2   0.0 -4.080440 0.02636530   0.000000e+00
#> 171:    2020-03-21 2020-03-22       1   0.8 -4.267754 0.05982192   4.591034e-02
#>      underprediction dispersion   log_score        mad   ae_median      se_mean
#>                <num>      <num>       <num>      <num>       <num>        <num>
#>   1:    0.0000000000 0.03991464 -0.70104246 0.11642891 0.007408170 0.0034433769
#>   2:    0.0000000000 0.04652294 -0.53254948 0.19005238 0.056131145 0.0000196604
#>   3:    0.0022939819 0.06764507 -0.08960756 0.33277556 0.019683761 0.0050583672
#>   4:    0.0000000000 0.06744857 -0.17985186 0.32271588 0.001162727 0.0067057059
#>   5:    0.0043578499 0.09093901  0.18427562 0.46164197 0.025815706 0.0135188376
#>  ---                                                                           
#> 167:    0.0542756419 0.01370094 -0.69981893 0.06050234 0.074064917 0.0093890876
#> 168:    0.0003442201 0.01714760 -1.49624273 0.06189629 0.002166230 0.0002676288
#> 169:    0.0320568253 0.01802395 -1.26407900 0.05711470 0.068668199 0.0083780021
#> 170:    0.0000000000 0.02636530 -1.21665256 0.10732353 0.009071721 0.0003736223
#> 171:    0.0000000000 0.01391157 -0.59730261 0.01792708 0.113976702 0.0051327722
#> 
#> $forecast_cases
#> # A tibble: 171 × 10
#>    forecast_date date       horizon median  mean     sd bottom lower upper   top
#>    <chr>         <date>       <int>  <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl>
#>  1 2020-03-04    2020-03-05       1   73.5  71.9   9.42     50    69    75    85
#>  2 2020-03-04    2020-03-06       2   84.5  82.4  17.6      51    81    98   107
#>  3 2020-03-04    2020-03-07       3  101    99.1  22.2      68    95   126   126
#>  4 2020-03-04    2020-03-08       4  118.  109.   30.6      48   110   133   146
#>  5 2020-03-04    2020-03-09       5  144.  134.   48.2      56   133   184   207
#>  6 2020-03-04    2020-03-10       6  150.  157.   74.6      39   134   221   287
#>  7 2020-03-04    2020-03-11       7  175   181.  105.       30   152   259   397
#>  8 2020-03-04    2020-03-12       8  225   225.  147.       31   201   319   547
#>  9 2020-03-04    2020-03-13       9  218   241.  197.       24   176   365   701
#> 10 2020-03-04    2020-03-14      10  260.  295   267.       25    85   315   943
#> # ℹ 161 more rows
#> 
#> $case_scores
#>      sample forecast_date       date horizon  bias       dss   crps
#>      <char>        <char>     <Date>   <int> <num>     <num>  <num>
#>   1:      1    2020-03-04 2020-03-05       1   0.8  5.372139   6.93
#>   2:      1    2020-03-04 2020-03-06       2   0.4  5.944579   7.48
#>   3:      1    2020-03-04 2020-03-07       3   0.4  6.372494   9.19
#>   4:      1    2020-03-04 2020-03-08       4   0.4  6.790483  10.82
#>   5:      1    2020-03-04 2020-03-09       5   0.4  7.792983  18.30
#>  ---                                                               
#> 167:      1    2020-03-19 2020-03-21       2  -1.0 34.915059 102.72
#> 168:      1    2020-03-19 2020-03-22       3   1.0 24.692744 147.69
#> 169:      1    2020-03-20 2020-03-21       1  -1.0 13.213070  85.52
#> 170:      1    2020-03-20 2020-03-22       2   1.0 18.790561 136.03
#> 171:      1    2020-03-21 2020-03-22       1   1.0 27.034101 179.02
#>      overprediction underprediction dispersion log_score     mad ae_median
#>               <num>           <num>      <num>     <num>   <num>     <num>
#>   1:            5.6             0.0       1.33  4.382427  4.4478      10.5
#>   2:            3.6             0.0       3.88  4.245305 18.5325      11.5
#>   3:            2.2             0.0       6.99  4.453154 30.3933      13.0
#>   4:            4.6             0.0       6.22  4.666473 20.7564      16.5
#>   5:            8.6             0.0       9.70  5.163744 44.4780      27.5
#>  ---                                                                      
#> 167:            0.0            96.7       6.02 22.090362 25.9455     110.5
#> 168:          140.1             0.0       7.59 11.448417 39.2889     171.5
#> 169:            0.0            72.1      13.42  6.563312 40.0302     125.5
#> 170:          126.1             0.0       9.93  8.109663 31.8759     166.0
#> 171:          170.1             0.0       8.92 15.695009 20.0151     215.5
#>       se_mean
#>         <num>
#>   1:    79.21
#>   2:    88.36
#>   3:   123.21
#>   4:    43.56
#>   5:   309.76
#>  ---         
#> 167: 13156.09
#> 168: 28425.96
#> 169: 12656.25
#> 170: 25856.64
#> 171: 41575.21

All functionality outlined above can be applied to this output but a special plotting function (plot_forecast_evaluation) is also provided. First evaluate the Rt forecast against observed values.

plot_forecast_evaluation(model_eval$forecast_rts,
  EpiSoon::example_obs_rts,
  horizon_to_plot = 7
)

Then evaluate forecast cases against observed values.

plot_forecast_evaluation(model_eval$forecast_cases,
  EpiSoon::example_obs_cases,
  horizon_to_plot = 7
)

Wrapper functions

EpiSoon provides several wrapper functions (compare_models and compare_timeseries). These both wrap evaluate_model and can be used to rapidly explore several forecasting models (compare_models) against multiple time series (compare_timeseries). All lower level summary and plotting functions can be then used with the output of these wrappers to explore the results. See the function documentation for further details.

Supporting generic modelling packages

EpiSoon supports the use of generic forecasting models if they are used in a wrapper that accepts a standardised set of inputs and outputs its forecast in the form the package expects. Examples of model wrappers are those for the bsts and fable packages (bsts_model and fable_model). See the examples and documentation for fable_model for further details.