Package: scoringutils 2.0.0

Nikos Bosse

scoringutils: Utilities for Scoring and Assessing Predictions

Facilitate the evaluation of forecasts in a convenient framework based on data.table. It allows user to to check their forecasts and diagnose issues, to visualise forecasts and missing data, to transform data before scoring, to handle missing forecasts, to aggregate scores, and to visualise the results of the evaluation. The package mostly focuses on the evaluation of probabilistic forecasts and allows evaluating several different forecast types and input formats. Find more information about the package in the Vignettes as well as in the accompanying paper, <doi:10.48550/arXiv.2205.07090>.

Authors:Nikos Bosse [aut, cre], Sam Abbott [aut], Hugo Gruson [aut], Johannes Bracher [ctb], Toshiaki Asakura [ctb], James Mba Azam [ctb], Sebastian Funk [aut], Michael Chirico [ctb]

scoringutils_2.0.0.tar.gz
scoringutils_2.0.0.zip(r-4.5)scoringutils_2.0.0.zip(r-4.4)scoringutils_2.0.0.zip(r-4.3)
scoringutils_2.0.0.tgz(r-4.4-any)scoringutils_2.0.0.tgz(r-4.3-any)
scoringutils_2.0.0.tar.gz(r-4.5-noble)scoringutils_2.0.0.tar.gz(r-4.4-noble)
scoringutils_2.0.0.tgz(r-4.4-emscripten)scoringutils_2.0.0.tgz(r-4.3-emscripten)
scoringutils.pdf |scoringutils.html✨
scoringutils/json (API)
NEWS

# Install 'scoringutils' in R:
install.packages('scoringutils', repos = c('https://epiforecasts.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/epiforecasts/scoringutils/issues

Datasets:

On CRAN:

forecast-evaluationforecasting

11.11 score 48 stars 5 packages 272 scripts 1.0k downloads 58 exports 41 dependencies

Last updated 22 days agofrom:efb41afd99. Checks:OK: 7. Indexed: yes.

TargetResultDate
Doc / VignettesOKOct 31 2024
R-4.5-winOKOct 31 2024
R-4.5-linuxOKOct 31 2024
R-4.4-winOKOct 31 2024
R-4.4-macOKOct 31 2024
R-4.3-winOKOct 31 2024
R-4.3-macOKOct 31 2024

Exports:add_relative_skillae_median_quantileae_median_sampleas_forecast_binaryas_forecast_nominalas_forecast_pointas_forecast_quantileas_forecast_sampleassert_forecastbias_quantilebias_samplebrier_scorecrps_sampledispersion_quantiledispersion_sampledss_sampleget_correlationsget_coverageget_duplicate_forecastsget_forecast_countsget_forecast_unitget_metricsget_pairwise_comparisonsget_pit_histograminterval_coverageis_forecastis_forecast_binaryis_forecast_nominalis_forecast_pointis_forecast_quantileis_forecast_samplelog_shiftlogs_binarylogs_nominallogs_samplemad_samplenew_forecastoverprediction_quantileoverprediction_samplepit_histogram_sampleplot_correlationsplot_forecast_countsplot_heatmapplot_interval_coverageplot_pairwise_comparisonsplot_quantile_coverageplot_wisquantile_scorescorese_mean_sampleselect_metricssummarise_scoressummarize_scorestheme_scoringutilstransform_forecastsunderprediction_quantileunderprediction_samplewis

Dependencies:backportscheckmateclicolorspacedata.tableevaluatefansifarverggplot2gluegtablehighrisobandknitrlabelinglatticelifecyclemagrittrMASSMatrixMetricsmgcvmunsellnlmepillarpkgconfigpurrrR6RColorBrewerRcppRcppArmadillorlangscalesscoringRulestibbleutf8vctrsviridisLitewithrxfunyaml

Deprecated functions

Rendered fromDeprecated-functions.Rmdusingknitr::rmarkdownon Oct 31 2024.

Last update: 2024-10-30
Started: 2024-04-04

Deprecated Visualisations

Rendered fromDeprecated-visualisations.Rmdusingknitr::rmarkdownon Oct 31 2024.

Last update: 2024-10-30
Started: 2024-03-18

Scoring rules in scoringutils

Nikos Bosse

Rendered fromscoring-rules.Rmdusingknitr::rmarkdownon Oct 31 2024.

Last update: 2024-09-16
Started: 2024-04-08

Readme and manuals

Help Manual

Help pageTopics
Add relative skill scores based on pairwise comparisonsadd_relative_skill
Absolute error of the median (quantile-based version)ae_median_quantile
Absolute error of the median (sample-based version)ae_median_sample
Create a 'forecast' object for binary forecastsas_forecast_binary
General information on creating a 'forecast' objectas_forecast_doc_template
Common functionality for as_forecast_<type> functionsas_forecast_generic
Create a 'forecast' object for nominal forecastsas_forecast_nominal
Create a 'forecast' object for point forecastsas_forecast_point as_forecast_point.default as_forecast_point.forecast_quantile
Create a 'forecast' object for quantile-based forecastsas_forecast_quantile as_forecast_quantile.default as_forecast_quantile.forecast_sample
Create a 'forecast' object for sample-based forecastsas_forecast_sample
Assert Inputs Have Matching Dimensionsassert_dims_ok_point
Validation common to all forecast typesassert_forecast_generic
Assert that forecast type is as expectedassert_forecast_type
Assert that input is a forecast object and passes validationsassert_forecast assert_forecast.default assert_forecast.forecast_binary assert_forecast.forecast_point assert_forecast.forecast_quantile assert_forecast.forecast_sample
Assert that inputs are correct for binary forecastassert_input_binary
Assert that inputs are correct for interval-based forecastassert_input_interval
Assert that inputs are correct for nominal forecastsassert_input_nominal
Assert that inputs are correct for point forecastassert_input_point
Assert that inputs are correct for quantile-based forecastassert_input_quantile
Assert that inputs are correct for sample-based forecastassert_input_sample
Determines bias of quantile forecastsbias_quantile
Determine bias of forecastsbias_sample
Check column names are present in a data.framecheck_columns_present
Check Inputs Have Matching Dimensionscheck_dims_ok_point
Check that there are no duplicate forecastscheck_duplicates
Check that inputs are correct for binary forecastcheck_input_binary
Check that inputs are correct for interval-based forecastcheck_input_interval
Check that inputs are correct for point forecastcheck_input_point
Check that inputs are correct for quantile-based forecastcheck_input_quantile
Check that inputs are correct for sample-based forecastcheck_input_sample
Check that all forecasts have the same number of rowscheck_number_per_forecast
Check whether an input is an atomic vector of mode 'numeric'check_numeric_vector
Helper function to convert assert statements into checkscheck_try
(Continuous) ranked probability scorecrps_sample dispersion_sample overprediction_sample underprediction_sample
Dawid-Sebastiani scoredss_sample
Binary forecast example dataexample_binary
Nominal example dataexample_nominal
Point forecast example dataexample_point
Quantile example dataexample_quantile
Continuous forecast example dataexample_sample_continuous
Discrete forecast example dataexample_sample_discrete
Calculate correlation between metricsget_correlations
Get quantile and interval coverage values for quantile-based forecastsget_coverage
Find duplicate forecastsget_duplicate_forecasts
Count number of available forecastsget_forecast_counts
Get forecast type from forecast objectget_forecast_type
Get unit of a single forecastget_forecast_unit
Get metricsget_metrics
Get default metrics for binary forecastsget_metrics.forecast_binary
Get default metrics for nominal forecastsget_metrics.forecast_nominal
Get default metrics for point forecastsget_metrics.forecast_point
Get default metrics for quantile-based forecastsget_metrics.forecast_quantile
Get default metrics for sample-based forecastsget_metrics.forecast_sample
Get names of the metrics that were used for scoringget_metrics.scores
Obtain pairwise comparisons between modelsget_pairwise_comparisons
Probability integral transformation histogramget_pit_histogram get_pit_histogram.default get_pit_histogram.forecast_quantile get_pit_histogram.forecast_sample
Get type of a vector or matrix of observed values or predictionsget_type
Interval coverage (for quantile-based forecasts)interval_coverage
Interval scoreinterval_score
Test whether an object is a forecast objectis_forecast is_forecast_binary is_forecast_nominal is_forecast_point is_forecast_quantile is_forecast_sample
Log transformation with an additive shiftlog_shift
Log score for nominal outcomeslogs_nominal
Logarithmic score (sample-based version)logs_sample
Determine dispersion of a probabilistic forecastmad_sample
Probability integral transformation for countspit_histogram_sample
Plot correlation between metricsplot_correlations
Visualise the number of available forecastsplot_forecast_counts
Create a heatmap of a scoring metricplot_heatmap
Plot interval coverageplot_interval_coverage
Plot heatmap of pairwise comparisonsplot_pairwise_comparisons
Plot quantile coverageplot_quantile_coverage
Plot contributions to the weighted interval scoreplot_wis
Print information about a forecast objectprint.forecast
Quantile scorequantile_score
Evaluate forecastsscore score.forecast_binary score.forecast_nominal score.forecast_point score.forecast_quantile score.forecast_sample
Metrics for binary outcomesbrier_score logs_binary scoring-functions-binary
Squared error of the mean (sample-based version)se_mean_sample
Select metrics from a list of functionsselect_metrics
Set unit of a single forecast manuallyset_forecast_unit
Summarise scores as produced by 'score()'summarise_scores summarize_scores
Test whether column names are NOT present in a data.frametest_columns_not_present
Test whether all column names are present in a data.frametest_columns_present
Scoringutils ggplot2 themetheme_scoringutils
Transform forecasts and observed valuestransform_forecasts
Validate metricsvalidate_metrics
Weighted interval score (WIS)dispersion_quantile overprediction_quantile underprediction_quantile wis