vinference

The goal of vinference is to assist with the design and analysis of visual inference experiments.

For more information on visual inference, check out the nullabor R package or Wickham et al. (2010).

Installation

vinference is not yet on CRAN; you can install the development version from GitHub with:

# install.packages("pak")
pak::pak("heike/vinference")

Example

In visual inference, we can run experiments under 3 scenarios:

  1. Scenario 1: in each of K evaluations a different data set and a different set of (m-1) null plots is shown.
  2. Scenario 2: in each of K evaluations the same data set but a different set of (m-1) null plots is shown.
  3. Scenario 3: the same lineup, i.e. same data and same set of null plots, is shown to K different observers.

To calculate a p-value for a set of 20-panel single-target lineups generated under Scenario 1 where the data plot was selected 10 times out of 30 observations,

pVis(10, K = 30, m = 20, scenario = 1, lower.tail = F)
#> [1] 1.161521e-06

Under Scenario 3, however, we would use the same 20-panel single-target lineup and have 30 different participants evaluate it. Because we are using the same panels for each participant evaluation, there is dependence between the evaluations of different participants and we must account for this using a parameter α\alpha which provides information about the level of visual interest of the null panels. See XXX Stat paper citation for more information.

Suppose, for now, we know α≈0.1\alpha \approx 0.1. Then our p-value for the lineup would be

pVis(10, K = 30, m = 20, scenario = 3, alpha = 0.1, lower.tail = F)
#> [1] 0.2232121

Probability (dVis) and quantile (qVis) functions are also available.

As discussed in XXX Cite stat paper XXX, we need to estimate the hyperparameter α\alpha, which describes the number of visually interesting null plots generated by the null model.

This approach is appropriate when you have several single or dual target lineup evaluations with similar (non-zero) number of null panel selections. Eliminate target panel selections and work with the remaining KK total null panel selections and a m0m_0 equal to the number of null panels. Determine how many selections cc are required for a panel to be considered interesting (by default, we recommend K/m0K/m_0). Using your values for cc, KK and m0m_0, the estimate_alpha_visual function will create a plot with the expected number of null panels selected at least cc times; simulated experiments will be plotted around the line to provide some measure of variability.

estimate_alpha_visual(m0 = 19, K = 30, c = 1)
#> Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
#> ℹ Please use `linewidth` instead.
#> ℹ The deprecated feature was likely used in the vinference package.
#>   Please report the issue to the authors.
#> This warning is displayed once per session.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.

Scatterplot of the average number of panels with at least 1 selection (on x) by the value of alpha on a log scale. With increasing alpha, the average number of selected panels follows a logistic S curve (shown in blue). The observed values from a simulation of ten points in each value of alpha. The observed values largely follow the S curve. The largest deviations can be seen for values of alpha between 0.1 and 0.25. About half of those simulations have around 9 interesting panels. If you have experimental data available, there is an additional function which may assist you with visual estimation: observed_band. The observed band function leverages an average number of panels or a range of panels to add additional cues to the plot. Suppose that we have several lineups which have K≈30K\approx30 evaluations; between 4 and 7 panels were selected in each lineup, with an average of 4.75 panels selected.

estimate_alpha_visual(m0 = 19, K = 30, c = 1) + observed_band(5.5, c(4, 7))
#> Warning in scale_x_log10(name = expression(alpha), breaks = bks, minor_breaks = mb, : log-10 transformation introduced infinite values.
#> log-10 transformation introduced infinite values.

The same scatterplot as before is now overlaid with two gray bands demonstrating how to use this chart to estimate alpha visually. Starting from a number of 4.75 observed panels in a study, we mark that value on the y-axis and draw a horizontal line to the S curve, to find the corresponding value on the x-axis, i.e. we visually find the inverse of the S curve. An additional grey band is drawn at the interval from the minimum number of selected panels (4) to the maximum (7) in our hypothetical study. This covers a range of alpha values around 0.1 that are also plausible candidates. Using this cue, values of α\alpha between 0.075 and 0.12 might be most appropriate, with the best numerical estimate around α=0.1\alpha=0.1.

We can then examine whether our p-value of 9.670497^{-4} corresponding to α=0.1\alpha=0.1 results in a similar conclusion for all plausible α\alpha values in our observed range. At α=0.0175\alpha=0.0175, the corresponding p-value is 0.0015002. At α=0.012\alpha=0.012, the corresponding p-value is 6.6426736^{-4}. In both cases, our lineup plot is significant, and we come to the same conclusion: the target plot is a significant visual statistic.