Skip to contents

A decision is a misclassification when it disagrees with the candidate's standing against the intended standard, `theta >= theta_standard`: the measure-scale cut for measure-based rules, or for raw rules the ability at which an average-severity panel's expected total equals the raw cut. For each candidate the probability of misclassification is computed under three panels, and the expected number of wrong decisions is split into:

measurement

Error expected with an average-severity panel. It comes from having a finite number of ratings and no rater can remove it.

assignment

Observed-panel error minus average-panel error: what the raters actually assigned added (or, if negative, removed). Reported net and as gross harm and gross benefit, because lenient raters can rescue a truly passing borderline candidate while wrongly passing another.

design

Random-panel error minus average-panel error: what the random assignment design costs in expectation, before anyone is scored. `df_design()` will try to reduce this.

Usage

df_attribute(counterfactual, components = "severity")

Arguments

counterfactual

A `df_counterfactual`.

components

Rater effects to separate. Only `"severity"` is estimable from the current model (no halo or drift terms yet).

Value

A `df_attribution` data frame, one row per candidate: `person`, `panel`, `pass_observed`, `p_true_pass` (probability the candidate truly meets the standard), `err_observed`, `err_average`, `err_random`, `assignment` (= err_observed - err_average), `false_pass` and `false_fail` (the two parts of err_observed), and `realized_error` when the counterfactual was computed from known truth. `summary()` gives the aggregate decomposition.

Details

With estimated parameters, error probabilities integrate over each candidate's ability posterior. Net components are well recovered in simulation, but the gross harm/benefit split and the false-pass / false-fail split are attenuated by posterior shrinkage; report them from estimates with that caveat.

Examples

sim <- df_simulate(n_persons = 200, n_items = 3, n_raters = 6, seed = 1)
cf <- df_counterfactual(sim, df_cut(12, "raw_total"))
df_attribute(cf)
#> <df_attribution> rule = raw_total | cut = 12 | standard at theta = 0.001 | n = 200 
#> 
#>  component                                        expected_count per_1000
#>  expected misclassifications (observed panels)    40.7           203.3   
#>    measurement error (average panel)              29.6           147.8   
#>    rater assignment, net (observed - average)     11.1            55.5   
#>      gross harm (panels that added error)         17.6            88.1   
#>      gross benefit (panels that removed error)    -6.5           -32.6   
#>    of which false passes                          28.1           140.6   
#>    of which false fails                           12.5            62.7   
#>  random-assignment design cost (random - average) 10.5            52.4   
#>  realized misclassifications (known truth)        40.0           200.0