NEWS


scoringutils 1.2.2.9000

This update represents a major rewrite of the package and introduces breaking changes. If you want to keep using the older version, you can download it using remotes::install_github("epiforecasts/[email protected]").

The update aims to make the package more modular and customisable and overall cleaner and easier to work with. In particular, we aimed to make the suggested workflows for evaluating forecasts more explicit and easier to follow (see visualisation below). To do that, we clarified input formats and made them consistent across all functions. We refactord many functions to S3-methods and introduced forecast objects with separate classes for different types of forecasts. A new function, as_forecast() was introduced to validate the data and convert inputs into a forecast object (a data.table with a forecast class and an additional class corresponding to the forecast type (see below)). Another major update is the possibility for users to pass in their own scoring functions into score(). We updated and improved all function documentation and added new vignettes to guide users through the package. Internally, we refactored the code, improved input checks, updated notifications (which now use the cli package) and increased test coverage.

The most comprehensive documentation for the new package after the rewrite is the revised version of our original scoringutils paper.

Package updates

score()

Creating a forecast object

Updated workflows

scores <- score(forecast_quantile)

- Overall, we updated the suggested workflows for how users should work with the package. The following gives an overview (see the  [updated paper](https://drive.google.com/file/d/1URaMsXmHJ1twpLpMl1sl2HW4lPuUycoj/view?usp=drive_link) for more details). 
  ![package workflows](./man/figures/workflow.png)
  
### Input formats
- We standardised input formats both for `score()` as well as for the scoring rules exported by `scoreingutils`. The following plot gives a overview of the expected input formats for the different forecast types in `score()`. 
![input formats](./man/figures/required-inputs.png)

- Support for the interval format was mostly dropped (see PR #525 by @nikosbosse and reviewed by @seabbs). The co-existence of the quantile and interval format let to a confusing user experience with many duplicated functions providing the same functionality. We decided to simplify the interface by focusing exclusively on the quantile format. 
    - The function `bias_range()` was removed (users should now use `bias_quantile()` instead)
    - The function `interval_score()` was made an internal function rather than being exported to users. We recommend using `wis()` instead. 

### (Re-)Validating forecast objects
- To create and validate a new `forecast` object, users can use `as_forecast()`. To revalidate an existing `forecast` object users can call `assert_forecast()` (which validates the input and returns `invisible(NULL)`. `assert_forecast()` is a generic with methods for the different forecast types. Alternatively, `validate_forecast()` can be used (which calls `assert_forecast()`), which returns the input and is useful in a pipe. Lastly, users can simply print the object to obtain additional information. 
- Users can test whether an object is of class `forecast_*()` using the function `is_forecast()`. Users can also test for a specific `forecast_*` class using the appropriate `is_forecast.forecast_*` method. For example, to check whether an object is of class `forecast_quantile`, you would use you would use `scoringutils:::is_forecast.forecast_quantile()`.

### Pairwise comparisons and relative skill
- The functionality for computing pairwise comparisons was now split from `summarise_scores()`. Instead of doing pairwise comparisons as part of summarising scores, a new function, `add_relative_skill()`, was introduced that takes summarised scores as an input and adds columns with relative skill scores and scaled relative skill scores.
- The function `pairwise_comparison()` was renamed to `get_pairwise_comparisons()`, in line with other `get_`-functions. Analogously, `plot_pairwise_comparison()` was renamed to `plot_pairwise_comparisons()`.
- Output columns for pairwise comparisons have been renamed to contain the name of the metric used for comparing.
- Replaced warnings with errors in `get_pairwise_comparison` to avoid returning `NULL`

### Computing coverage values
- `add_coverage()` was replaced by a new function, `get_coverage()`. This function comes with an updated workflow where coverage values are computed directly based on the original data and can then be visualised using `plot_interval_coverage()` or `plot_quantile_coverage()`. An example workflow would be `example_quantile |> as_forecast() |> get_coverage(by = "model") |> plot_interval_coverage()`.

### Obtaining and plotting forecast counts
- The function `avail_forecasts()` was renamed to `get_forecast_counts()`. This represents a change in the naming convention where we aim to name functions that provide the user with additional useful information about the data with a prefix "get_". Sees Issue #403 and #521 and PR #511 by @nikosbosse and reviewed by @seabbs for details. 
  - For clarity, the output column in `get_forecast_counts()` was renamed from "Number forecasts" to "count".
  - `get_forecast_counts()` now also displays combinations where there are 0 forecasts, instead of silently dropping corresponding rows.
  - `plot_avail_forecasts()` was renamed `plot_forecast_counts()` in line with the change in the function name. The `x` argument no longer has a default value, as the value will depend on the data provided by the user.
  
### Renamed functions
- The function `find_duplicates()` was renamed to `get_duplicate_forecasts()`. 
- Renamed `interval_coverage_quantile()` and `interval_coverage_dev_quantile()` to `interval_coverage()` and `interval_coverage_deviation()`, respectively. 
- "range" was consistently renamed to "interval_range" in the code. The "range"-format (which was mostly used internally) was renamed to "interval"-format
- Renamed `correlation()` to `get_correlations()` and `plot_correlation()` to `plot_correlations()`
- `pit()` was renamed to `get_pit()`. 

### Deleted functions
- Removed abs_error and squared_error from the package in favour of `Metrics::ae` and `Metrics::se`.`get_duplicate_forecasts()` now sorts outputs according to the forecast unit, making it easier to spot duplicates. In addition, there is a `counts` option that allows the user to display the number of duplicates for each forecast unit, rather than the raw duplicated rows.
- Deleted the function `plot_ranges()`. If you want to continue using the functionality, you can find the function code [here](https://github.com/epiforecasts/scoringutils/issues/462) or in the Deprecated-visualisations Vignette.
- Removed the function `plot_predictions()`, as well as its helper function `make_NA()`, in favour of a dedicated Vignette that shows different ways of visualising predictions. For future reference, the function code can be found [here](https://github.com/epiforecasts/scoringutils/issues/659) (Issue #659) or in the Deprecated-visualisations Vignette.
- Removed the function `plot_score_table()`. You can find the code in the Deprecated-visualisations Vignette. 
- Removed the function `merge_pred_and_obs()` that was used to merge two separate data frames with forecasts and observations. We moved its contents to a new "Deprecated functions"-vignette.
- Removed `interval_coverage_sample()` as users are now expected to convert to a quantile format first before scoring.

### Function changes
- `bias_quantile()` changed the way it handles forecasts where the median is missing: The median is now imputed by linear interpolation between the innermost quantiles. Previously, we imputed the median by simply taking the mean of the innermost quantiles.
- In contrast to the previous `correlation` function, `get_correlations` doesn't round correlations by default. Instead, `plot_correlations` now has a `digits` argument that allows users to round correlations before plotting them. Alternatively, using `dplyr`, you could call something like `mutate(correlations, across(where(is.numeric), \(x) signif(x, digits = 2)))` on the output of `get_correlations`. 

### Internal package updates
- The deprecated `..density..` was replaced with `after_stat(density)` in ggplot calls.
- Files ending in ".Rda" were renamed to ".rds" where appropriate when used together with `saveRDS()` or `readRDS()`.
- Added a subsetting `[` operator for scores, so that the score name attribute gets preserved when subsetting.
- Switched to using `cli` for condition handling and signalling, and added tests for all the `check_*()` and `test_*()` functions. See #583 by @jamesmbaazam and reviewed by @nikosbosse and @seabbs.
- `scoringutils` now requires R >= 4.0


### Documentation and testing
- Updates documentation for most functions and made sure all functions have documented return statements
- Documentation pkgdown pages are now created both for the stable and dev versions.
- Added unit tests for many functions

# scoringutils 1.2.2

## Package updates
- `scoringutils` now depends on R 3.6. The change was made since packages `testthat` and `lifecycle`, which are used in `scoringutils` now require R 3.6. We also updated the Github action CI check to work with R 3.6 now. 
- Added a new PR template with a checklist of things to be included in PRs to facilitate the development and review process

## Bug fixes
- Fixes a bug with `set_forecast_unit()` where the function only worked with a data.table, but not a data.frame as an input. 
- The metrics table in the vignette [Details on the metrics implemented in `scoringutils`](https://epiforecasts.io/scoringutils/articles/metric-details.html) had duplicated entries. This was fixed by removing the duplicated rows.

# scoringutils 1.2.1

## Package updates
- This minor update fixes a few issues related to gh actions and the vignettes displayed at epiforecasts.io/scoringutils. It
  - Gets rid of the preferably package in _pkgdown.yml. The theme had a toggle between light and dark theme that didn't work properly
  - Updates the gh pages deploy action to v4 and also cleans up files when triggered
  - Introduces a gh action to automatically render the Readme from Readme.Rmd
  - Removes links to vignettes that have been renamed

# scoringutils 1.2.0

This major release contains a range of new features and bug fixes that have been introduced in minor releases since `1.1.0`. The most important changes are:

- Documentation updated to reflect changes since version 1.1.0, including new transform and workflow functions.
- New `set_forecast_unit()` function allows manual setting of forecast unit.
- `summarise_scores()` gains new `across` argument for summarizing across variables.
- New `transform_forecasts()` and `log_shift()` functions allow forecast transformations. See the documentation for `transform_forecasts()` for more details and an example use case.
- Input checks and test coverage improved for bias functions.
- Bug fix in `get_prediction_type()` for integer matrix input.
- Links to scoringutils paper and citation updates.
- Warning added in `interval_score()` for small interval ranges.
- Linting updates and improvements.

Thanks to @nikosbosse, @seabbs, and @sbfnk for code and review contributions. Thanks to @bisaloo for the suggestion to use a linting GitHub Action that only triggers on changes, and @adrian-lison for the suggestion to add a warning to `interval_score()` if the interval range is between 0 and 1.

## Package updates

- The documentation was updated to reflect the recent changes since `scoringutils 1.1.0`. In particular, usage of the functions `set_forecast_unit()`, `check_forecasts()` and `transform_forecasts()` are now documented in the Vignettes. The introduction of these functions enhances the overall workflow and help to make the code more readable. All functions are designed to be used together with the pipe operator. For example, one can now use something like the following:

```r
example_quantile |> 
  set_forecast_unit(c("model", "location", "forecast_date", "horizon", "target_type")) |> 
  check_forecasts() |> 
  score()

Documentation for the transform_forecasts() has also been extended. This functions allows the user to easily add transformations of forecasts, as suggested in the paper "Scoring epidemiological forecasts on transformed scales". In an epidemiological context, for example, it may make sense to apply the natural logarithm first before scoring forecasts, in order to obtain scores that reflect how well models are able to predict exponential growth rates, rather than absolute values. Users can now do something like the following to score a transformed version of the data in addition to the original one:

data <- example_quantile[true_value > 0, ]
data |>
  transform_forecasts(fun = log_shift, offset = 1) |> 
  score() |> 
  summarise_scores(by = c("model", "scale"))

Here we use the log_shift() function to apply a logarithmic transformation to the forecasts. This function was introduced in scoringutils 1.1.2 as a helper function that acts just like log(), but has an additional argument offset that can add a number to every prediction and observed value before applying the log transformation.

Feature updates

scoringutils 1.1.7

Release by @seabbs in #305. Reviewed by @nikosbosse and @sbfnk.

Breaking changes

Package updates

Bug fixes

scoringutils 1.1.6

Feature updates

scoringutils 1.1.5

Feature updates

scoringutils 1.1.4

Package updates

scoringutils 1.1.3

Package updates

Package updates

scoringutils 1.1.2

Feature updates

scoringutils 1.1.1

scoringutils 1.1.0 (2023-01-30)

A minor update to the package with some bug fixes and minor changes.

Feature updates

Package updates

Bug fixes

scoringutils 1.0.0 (2022-05-13)

Major update to the package and most package functions with lots of breaking changes.

Feature updates

New functions and function changes

Bug fixes

Package data updated

Other breaking changes

scoringutils 0.1.8

Feature updates

scoringutils 0.1.7.2 (2021-07-21)

Package updates

scoringutils 0.1.7 (2021-07-14)

Feature updates

Package updates

scoringutils 0.1.5

Package updates

scoringutils 0.1.4 (2020-11-17)

Feature updates

package updates

scoringutils 0.1.3

(Potentially) Breaking changes

Feature updates

scoringutils 0.1.2

(Potentially) Breaking changes

Feature updates

Package updates

scoringutils 0.1.1

Feature updates

Package updates

scoringutils 0.1

Feature updates

Package updates