Package: scoringutils 2.2.0

Nikos Bosse

scoringutils: Utilities for Scoring and Assessing Predictions

Facilitate the evaluation of forecasts in a convenient framework based on data.table. It allows user to to check their forecasts and diagnose issues, to visualise forecasts and missing data, to transform data before scoring, to handle missing forecasts, to aggregate scores, and to visualise the results of the evaluation. The package mostly focuses on the evaluation of probabilistic forecasts and allows evaluating several different forecast types and input formats. Find more information about the package in the Vignettes as well as in the accompanying paper, <doi:10.48550/arXiv.2205.07090>.

Authors:Nikos Bosse [aut, cre], Sam Abbott [aut], Hugo Gruson [aut], Johannes Bracher [ctb], Toshiaki Asakura [ctb], James Mba Azam [ctb], Sebastian Funk [aut], Michael Chirico [ctb]

scoringutils_2.2.0.tar.gz
scoringutils_2.2.0.zip(r-4.7)scoringutils_2.2.0.zip(r-4.6)scoringutils_2.2.0.zip(r-4.5)
scoringutils_2.2.0.tgz(r-4.6-any)scoringutils_2.2.0.tgz(r-4.5-any)
scoringutils_2.2.0.tar.gz(r-4.7-any)scoringutils_2.2.0.tar.gz(r-4.6-any)
scoringutils_2.2.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
card.svg |card.png
scoringutils/json (API)
NEWS

# Install 'scoringutils' in R:
install.packages('scoringutils', repos = c('https://epiforecasts.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/epiforecasts/scoringutils/issues

Pkgdown/docs site:https://epiforecasts.io

Datasets:

On CRAN:

Conda:

forecast-evaluationforecasting

11.50 score 61 stars 7 packages 384 scripts 1.5k downloads 69 exports 31 dependencies

Last updated from:e8853873e9 (on v2.2.0). Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-x86_64OK181
source / vignettesOK216
linux-release-x86_64OK197
macos-release-arm64OK220
macos-oldrel-arm64OK163
windows-develOK393
windows-releaseOK172
windows-oldrelOK188
wasm-releaseOK120

Exports:add_relative_skillae_median_quantileae_median_sampleas_forecast_binaryas_forecast_multivariate_pointas_forecast_multivariate_sampleas_forecast_nominalas_forecast_ordinalas_forecast_pointas_forecast_quantileas_forecast_sampleassert_forecastbias_quantilebias_samplebrier_scorecrps_sampledispersion_quantiledispersion_sampledss_sampleenergy_score_multivariateget_correlationsget_coverageget_duplicate_forecastsget_forecast_countsget_forecast_unitget_groupingget_metricsget_pairwise_comparisonsget_pit_histograminterval_coverageis_forecastis_forecast_binaryis_forecast_multivariate_pointis_forecast_multivariate_sampleis_forecast_nominalis_forecast_ordinalis_forecast_pointis_forecast_quantileis_forecast_samplelog_shiftlogs_binarylogs_categoricallogs_samplemad_samplenew_forecastoverprediction_quantileoverprediction_samplepit_histogram_sampleplot_correlationsplot_forecast_countsplot_heatmapplot_interval_coverageplot_pairwise_comparisonsplot_quantile_coverageplot_wisquantile_scorerps_ordinalscorese_mean_sampleselect_metricssummarise_scoressummarize_scorestheme_scoringutilstransform_forecastsunderprediction_quantileunderprediction_samplevariogram_score_multivariatevariogram_score_multivariate_pointwis

Dependencies:backportscheckmateclicpp11data.tableevaluatefarverggplot2gluegtablehighrisobandknitrlabelinglifecyclemagrittrMASSpurrrR6RColorBrewerRcppRcppArmadillorlangS7scalesscoringRulesvctrsviridisLitewithrxfunyaml

Deprecated functions

Rendered fromDeprecated-functions.Rmdusingknitr::rmarkdownon May 05 2026.

Last update: 2024-10-30
Started: 2024-04-04

Deprecated Visualisations

Rendered fromDeprecated-visualisations.Rmdusingknitr::rmarkdownon May 05 2026.

Last update: 2026-02-11
Started: 2024-03-18

Scoring multivariate forecasts

Rendered fromscoring-multivariate-forecasts.Rmdusingknitr::rmarkdownon May 05 2026.

Last update: 2026-03-08
Started: 2025-09-23

Scoring rules in scoringutils

Nikos Bosse

Rendered fromscoring-rules.Rmdusingknitr::rmarkdownon May 05 2026.

Last update: 2026-03-30
Started: 2024-04-08

Readme and manuals

Help Manual

Help pageTopics
Add relative skill scores based on pairwise comparisonsadd_relative_skill
Absolute error of the median (quantile-based version)ae_median_quantile
Absolute error of the median (sample-based version)ae_median_sample
Create a 'forecast' object for binary forecastsas_forecast_binary as_forecast_binary.default
General information on creating a 'forecast' objectas_forecast_doc_template
Common functionality for as_forecast_<type> functionsas_forecast_generic
Create a 'forecast' object for multivariate point forecastsas_forecast_multivariate_point as_forecast_multivariate_point.default
Create a 'forecast' object for sample-based multivariate forecastsas_forecast_multivariate_sample as_forecast_multivariate_sample.default
Create a 'forecast' object for nominal forecastsas_forecast_nominal as_forecast_nominal.default
Create a 'forecast' object for ordinal forecastsas_forecast_ordinal as_forecast_ordinal.default
Create a 'forecast' object for point forecastsas_forecast_point as_forecast_point.default as_forecast_point.forecast_quantile
Create a 'forecast' object for quantile-based forecastsas_forecast_quantile as_forecast_quantile.default as_forecast_quantile.forecast_sample
Create a 'forecast' object for sample-based forecastsas_forecast_sample as_forecast_sample.default
Assert Inputs Have Matching Dimensionsassert_dims_ok_scalar
Validation common to all forecast typesassert_forecast_generic
Assert that forecast type is as expectedassert_forecast_type
Assert that input is a forecast object and passes validationsassert_forecast assert_forecast.default assert_forecast.forecast_binary assert_forecast.forecast_multivariate_point assert_forecast.forecast_multivariate_sample assert_forecast.forecast_point assert_forecast.forecast_quantile assert_forecast.forecast_sample
Assert that inputs are correct for binary forecastassert_input_binary
Assert that inputs are correct for categorical forecastsassert_input_categorical
Assert that inputs are correct for interval-based forecastassert_input_interval
Assert that inputs are correct for sample-based forecastassert_input_multivariate_sample
Assert that inputs are correct for nominal forecastsassert_input_nominal
Assert that inputs are correct for ordinal forecastsassert_input_ordinal
Assert that inputs are correct for point forecastassert_input_point
Assert that inputs are correct for quantile-based forecastassert_input_quantile
Assert that inputs are correct for sample-based forecastassert_input_sample
Determines bias of quantile forecastsbias_quantile
Determine bias of forecastsbias_sample
Check column names are present in a data.framecheck_columns_present
Check Inputs Have Matching Dimensionscheck_dims_ok_scalar
Check that there are no duplicate forecastscheck_duplicates
Check that inputs are correct for binary forecastcheck_input_binary
Check that inputs are correct for interval-based forecastcheck_input_interval
Check that inputs are correct for point forecastcheck_input_point
Check that inputs are correct for quantile-based forecastcheck_input_quantile
Check that inputs are correct for sample-based forecastcheck_input_sample
Check that all forecasts have the same number of rowscheck_number_per_forecast
Check whether an input is an atomic vector of mode 'numeric'check_numeric_vector
Helper function to convert assert statements into checkscheck_try
(Continuous) ranked probability scorecrps_sample dispersion_sample overprediction_sample underprediction_sample
Dawid-Sebastiani scoredss_sample
Energy score for multivariate forecastsenergy_score_multivariate
Binary forecast example dataexample_binary
Multivariate forecast example dataexample_multivariate_sample
Nominal example dataexample_nominal
Ordinal example dataexample_ordinal
Point forecast example dataexample_point
Quantile example dataexample_quantile
Continuous forecast example dataexample_sample_continuous
Discrete forecast example dataexample_sample_discrete
Calculate correlation between metricsget_correlations
Get quantile and interval coverage values for quantile-based forecastsget_coverage
Find duplicate forecastsget_duplicate_forecasts
Count number of available forecastsget_forecast_counts
Get forecast type from forecast objectget_forecast_type
Get unit of a single forecastget_forecast_unit
Get grouping for a multivariate forecastget_grouping
Get metricsget_metrics
Get default metrics for binary forecastsget_metrics.forecast_binary
Get default metrics for multivariate point forecastsget_metrics.forecast_multivariate_point
Get default metrics for sample-based forecastsget_metrics.forecast_multivariate_sample
Get default metrics for nominal forecastsget_metrics.forecast_nominal
Get default metrics for nominal forecastsget_metrics.forecast_ordinal
Get default metrics for point forecastsget_metrics.forecast_point
Get default metrics for quantile-based forecastsget_metrics.forecast_quantile
Get default metrics for sample-based forecastsget_metrics.forecast_sample
Get names of the metrics that were used for scoringget_metrics.scores
Obtain pairwise comparisons between modelsget_pairwise_comparisons
Probability integral transformation histogramget_pit_histogram get_pit_histogram.default get_pit_histogram.forecast_quantile get_pit_histogram.forecast_sample
Get type of a vector or matrix of observed values or predictionsget_type
Interval coverage (for quantile-based forecasts)interval_coverage
Interval scoreinterval_score
Test whether an object is a forecast objectis_forecast is_forecast_binary is_forecast_multivariate_point is_forecast_multivariate_sample is_forecast_nominal is_forecast_ordinal is_forecast_point is_forecast_quantile is_forecast_sample
Log transformation with an additive shiftlog_shift
Log score for categorical outcomeslogs_categorical
Logarithmic score (sample-based version)logs_sample
Determine dispersion of a probabilistic forecastmad_sample
Probability integral transformation for countspit_histogram_sample
Plot correlation between metricsplot_correlations
Visualise the number of available forecastsplot_forecast_counts
Create a heatmap of a scoring metricplot_heatmap
Plot interval coverageplot_interval_coverage
Plot heatmap of pairwise comparisonsplot_pairwise_comparisons
Plot quantile coverageplot_quantile_coverage
Plot contributions to the weighted interval scoreplot_wis
Print information about a forecast objectprint.forecast
Quantile scorequantile_score
Ranked Probability Score for ordinal outcomesrps_ordinal
Evaluate forecastsscore score.forecast_binary score.forecast_multivariate_point score.forecast_multivariate_sample score.forecast_nominal score.forecast_ordinal score.forecast_point score.forecast_quantile score.forecast_sample
Metrics for binary outcomesbrier_score logs_binary scoring-functions-binary
Squared error of the mean (sample-based version)se_mean_sample
Select metrics from a list of functionsselect_metrics
Set unit of a single forecast manuallyset_forecast_unit
Summarise scores as produced by 'score()'summarise_scores summarize_scores
Test whether all column names are present in a data.frametest_columns_present
Scoringutils ggplot2 themetheme_scoringutils
Transform forecasts and observed valuestransform_forecasts transform_forecasts.default transform_forecasts.forecast transform_forecasts.forecast_multivariate_point transform_forecasts.forecast_multivariate_sample
Validate metricsvalidate_metrics
Variogram score for multivariate forecastsvariogram_score_multivariate
Variogram score for multivariate point forecastsvariogram_score_multivariate_point
Weighted interval score (WIS)dispersion_quantile overprediction_quantile underprediction_quantile wis